How to Integrate Sikuli With Selenium And To How Work With It

Introduction of Sikuli:

Today we are going to discuss "Sikuli". When we are performing image processing automation, in that case, Sikuli will help. Sikuli is an open source and developed by MTI. It's a simple tool and some utilities are available and it's totally based on image processing. It will just read the images and on the basis of that it will perform the action or click on that particular image. Let's see especially for flash objects because in selenium web driver we don't have any flash API's. Lets suppose you want to pause and play the video in that case you have to use "Sikuli". It is not performed by selenium web driver.

How Sikuli Works:

First You capture the images and on the basis of that, you have to perform actions as .click(). So start testing on the images or flash object you have to add Sikuli jar files. Earlier it was known by Sikuli but now it is known by SikuliX API. Here I am creating a Maven project so I directly add the dependency of SikuliX API.

Open the IDE tool and create a maven project. And Add The Below Dependencies:

1) Sikuli   https://mvnrepository.com/artifact/com.sikulix/sikulixapi/1.1.0

2. WebDriver  https://docs.seleniumhq.org/download/maven.jsp

Add these two dependencies and save that. It takes time to add all the jar files. After complete, the process checks in the Maven Dependencies it shows all the Jar and API's add over. like that:

Here is code for click on the Play and Pause Button or we can say image which we take from screenshots.

import java.util.concurrent.TimeUnit;

import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.sikuli.script.FindFailed;
import org.sikuli.script.Pattern;
import org.sikuli.script.Screen;

public class YouTubeVedioTest
{
   public static WebDriver driver;   
   
   public static void main( String[] args ) throws FindFailed
   {
      System.setProperty("webdriver.chrome.driver", "/home/nargis/Downloads/chromedriver");
      driver = new ChromeDriver();
      driver.get("https://www.youtube.com/watch?v=Zrv3WbdfkDQ&t=20s");

      driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
      driver.manage().window().maximize();

      //create the object of screen class-Sikuli
      Screen s = new Screen();
      
      //Click on settingimage
      Pattern setting=new Pattern("YT_Settingimage.png");
      s.wait( setting,2000 );
      
      s.click();
      
      
      //Click on AutoImage
      Pattern auto=new Pattern("YT_Auto.png");
      s.wait( auto,2000 );
      
      s.click();
      
      
      //Click on Pause
      Pattern pasueimage=new Pattern("YT_Pause.png");
      s.wait( pasueimage,2000 );
      
      s.click();
      s.click();
      
      //Click on Play
      Pattern playimage=new Pattern("YT_Play.png");
      s.wait( playimage,2000 );
      
      s.click();
      s.click();

      
      
   }

}
        

Sometimes click on play and pause images is very fast so I have clicked two times in my code so I can easily see watch clicking actions. It's not happened everytime it might be possible that it will perform fine on one click.

Steps to click play and pause buttons and images:

1. Take screenshots of that play, pause, setting, mute and etc. like that

2. Rename those images with suitable names.

3. Copy those images then Go to eclipse and paste in Project folder. It will show in the Project.

4. Create a class writes the above code. Here are steps to run the script:

i) Launch the browser.

ii) Enter the Url which you will copy from the running video.

iii) The first Step to click on the Setting icon which you can easily in the youtube videos.

iv) then Click on the Pause button and then play that.

Result in the console:

Conclusion: Sikuli is the very useful tool. We can perform easily those clicks which we are able to click with XPath. It will help videos screaming and adds where adds comes for short times.

Hope You Like It