35.基于图像匹配的自动化测试

图像匹配的自动化测试

介绍技术

价值与用途

• 解决特殊场景的自动化测试问题
  • 不可识别控件，自定义控件、图片控件、滑块
  • 安全性较高控件，比如安全键盘
  • 特殊渲染引擎，比如游戏 app
• 作为常规自动化测试的补充
  • 图像识别速度较慢
  • 游戏控件通过自定义可以实现控件识别

基于图像匹配的自动化测试框架

• airtest，适合纯游戏自动化测试场景
• appium，适合常规与特殊控件的结合场景
• sikuliX，老牌工具

appium 图像匹配能力支持

• appium 1.8 版本新增
• 基于 opencv + 匹配算法
• 图像匹配相关的 API
• 自动化相关的 API

图像匹配 API

• 子图像查找 findImageOccurrence
• 匹配算法 matchImagesFeatures
• 相似度计算 getImagesSimilarity

环境准备

• 安装 opencv4nodejs
• 注意 appium 的报错
  • Cannot find module /usr/local/lib/node_modules/opencv4nodejs/build/Release/opencv4nodejs
  • Encountered internal error running command: Error: ‘opencv4nodejs’ module is required to use OpenCV features. Please install it first (‘npm i -g opencv4nodejs’) and restart Appium. Read https://github.com/justadudewhohacks/opencv4nodejs#how-to-install for more details on this topic.

安装 opencv4nodejs

npm install -g opencv4nodejs

# mac安装
export OPENCV4NODEJS_DISABLE_AUTOBUILD=1
npm install -g opencv4nodejs

cd $(npm root -g)/opencv4nodejs/
export OPENCV4NODEJS_AUTOBUILD_FLAGS=-DBUILD_LIST=core,imgproc,imgcodecs,videoio,highgui,video,calib3d,features2d,objdetect,dnn,ml,flann,photo,stitching,gapi
npm i --save opencv-build
node install/install.js

capability 与 settings 设置

        desiredCapabilities.setCapability("settings[fixImageFindScreenshotDims]", false);
        desiredCapabilities.setCapability("settings[fixImageTemplateSize]", true);
        desiredCapabilities.setCapability("settings[getMatchedImageResult]", true)
        desiredCapabilities.setCapability("settings[imageMatchThreshold]", 0.4)

基于图像识别的自动化测试 java 版

public class ImageTest {

    private static final ObjectMapper mapper = new ObjectMapper(new YAMLFactory());
    private static AndroidDriver driver;

    @BeforeAll
    public static void beforeAll() throws MalformedURLException {
        DesiredCapabilities desiredCapabilities = new DesiredCapabilities();
        desiredCapabilities.setCapability("platformName", "android");
        desiredCapabilities.setCapability("noReset", true);
        desiredCapabilities.setCapability("appPackage", "io.appium.android.apis");
        desiredCapabilities.setCapability("appActivity", "io.appium.android.apis.ApiDemos");
//        desiredCapabilities.setCapability("settings[fixImageFindScreenshotDims]", false);
//        desiredCapabilities.setCapability("settings[fixImageTemplateSize]", true);
//        desiredCapabilities.setCapability("settings[getMatchedImageResult]", true);

        URL url = new URL("http://127.0.0.1:4723/wd/hub");
        driver = new AndroidDriver(url, desiredCapabilities);
        driver.manage().timeouts().implicitlyWait(Duration.ofSeconds(10));

    }

    /**
     * 测试appium-opencv图像识别测试
     */
    @Test
    public void testOpenCV() throws InterruptedException, IOException {
        driver.findElement(AppiumBy.accessibilityId("Graphics")).click();
        driver.findElement(AppiumBy.accessibilityId("BitmapMesh")).click();


        WebElement noseElement = driver.findElement(AppiumBy.image(getImageAsBase64String("src/test/java/com/ceshiren/hogwarts/appium/nose.png")));
        System.out.println(mapper.writeValueAsString(noseElement.getRect()));
        noseElement.click();

        driver.setSetting(Setting.IMAGE_MATCH_THRESHOLD, 0.5);
        WebElement flowerElement = driver.findElement(AppiumBy.image(getImageAsBase64String("src/test/java/com/ceshiren/hogwarts/appium/flower.png")));
        System.out.println(mapper.writeValueAsString(flowerElement.getRect()));
        flowerElement.click();
    }

    private String getImageAsBase64String(String imagePath) throws IOException {
        return Base64.getEncoder().encodeToString(Files.readAllBytes(Path.of(imagePath)));
    }
}

基于图像识别的自动化测试 python 版

class TestImage:
    def setup_class(self):
        capabilities = {
            'platformName': 'android',
            'appPackage': 'io.appium.android.apis',
            'appActivity': 'io.appium.android.apis.ApiDemos'
        }
        self.driver = webdriver.Remote('http://localhost:4723/wd/hub', capabilities)
        self.driver.implicitly_wait(10)

    def test_image(self):
        self.driver.find_element(AppiumBy.ACCESSIBILITY_ID, 'Graphics').click()
        self.driver.find_element(AppiumBy.ACCESSIBILITY_ID, 'BitmapMesh').click()
        self.driver.find_element_by_image("nose.png").click()
        self.driver.find_element_by_image("flower.png").click()