Overview of surface automation

Surface automation is automation based on recognition of images and texts visible on a computer's screen. Use surface automation to work around limitations to map application controls or extract texts from the computer's screen when you can't do it by getting their attributes or properties. Some cases where you need to use surface automation involve virtual desktop automation, and automation of web or desktop applications without access to their controls.

Recognizing and comparing images

IBM RPA uses a proprietary Vision's driver to recognize and compare images. It recognizes images by pixel-by-pixel comparison between an anchor image and the computer's screen. Recognition of texts in images is done with optical character recognition (OCR) technology.

IBM RPA can recognize image or text references based on the degree of similarity between an anchor image or text and the image that is displayed on the computer's screen. This comparison ensures that application controls are identified even if anchor images have minimal differences with the computer's screen as in different screen resolutions.

The following video shows how IBM RPA recognize images by pixel-by-pixel comparison according to the degree of similarity.