Click by OCR
Clicks on a user interface control based on the text entered, using the OCR features.
Command availability: IBM RPA SaaS and IBM RPA on premises
Description
Clicks on a user interface control based on the Text entered, using the OCR features. Controls are, on Windows, elements that allow the user to interact with an application. Controls can be buttons, data entry fields, menus and various other elements.
You can use this command to leverage surface automation capabilities to your script. For more information, see Actions for surface automation.
Script syntax
IBM RPA's proprietary scripting language has a syntax similar to other programming languages. The script syntax defines the command's syntax in the script file. You can work with this syntax in IBM RPA Studio's Script mode.
ocrClick [--handleError(Boolean)] [--safesearch(Boolean)] --ocrprovider(Nullable<OpticalCharacterRecognitionProvider>) [--googlevisionclientsecret(String)] --comparison(OcrStringComparison) --fuzzyalgorithm(Nullable<FuzzyStringComparisonAlgorithms>) --tolerance(Nullable<FuzzyStringComparisonTolerance>) --manualTolerance(Numeric) --segmentation(StringSegmentation) --text(String) [--occurrencetype(FindOccurrenceTypes)] --occurrence(Numeric) [--doubleclick(Boolean)] [--clickonposition(Boolean)] [--corner(Nullable<ControlCorner>)] --position(Point) --language(String) [--region(Rectangle)] [--invertcolors(Boolean)] [--forcerefresh(Boolean)] --selector(ControlSelectors) --id(String) --name(String) --innertext(String) --tagname(String) --xpath(String) --classname(String) --elementvalue(String) --controltype(Nullable<ControlTypes>) --index(Numeric) --control(Control) [--usetable(Boolean)] [--searchbycolumn(Boolean)] --searchcolumn(String) --searchvalue(String) --returncolumn(String) --row(Numeric) [--timeout(TimeSpan)] (Boolean)=success (Image)=image (String)=text (Rectangle)=bounds
Input parameter
The following table displays the list of input parameters available in this command. In the table, you can see the parameter name when working in IBM RPA Studio's Script mode and its Designer mode equivalent label.
| Designer mode label | Script mode name | Required | Accepted variable types | Description |
|---|---|---|---|---|
| Handle Error | handleError |
Optional |
Boolean |
When enabled, it allows the script to not halt if an error occurs during that run, in addition to the error information being returned in the IBM RPA Studio console. Otherwise, it will stop running when an error occurs. |
| Selector Type | selector |
Required |
ControlSelectors |
Type of selector used to identify user interface controls. See the selector parameter options.You can use the IBM RPA Studio's recorder to map the controls and obtain the information needed for this parameter. |
| Id | id |
Only whenSelector is Id, IdAndName |
Text |
Id of the UI control. |
| Safe Search | safesearch |
Optional |
Boolean |
When enabled, allows the window to be focused and use superior image resolution for comparison. |
| OCR Provider | ocrprovider |
Required |
OpticalCharacterRecognitionProvider |
OCR service provider used for character recognition. See the ocrprovider parameter options |
| XPath | xpath |
Required when the Selector type parameter is XPath |
Text |
XPath expression that contains the desired control. |
| API Parameters | googlevisionclientsecret |
Optional |
Text |
The absolute path to the JSON file containing the API parameters. Refer to the Google Cloud Vision™ documentation for details about the JSON format. |
| Comparison | comparison |
Required |
OcrStringComparison |
Comparison mode that should be used between the text of the desired region and the text obtained from the specified interface control. See the comparison parameter options |
| Fuzzy Algorithm | fuzzyalgorithm |
Only whenComparison is ApproximatelyEquals |
FuzzyStringComparisonAlgorithms |
Type of fuzzy algorithm used as a basis for comparison. See the fuzzyalgorithm parameter options |
| Tolerance | tolerance |
Only whenComparison is ApproximatelyEquals |
FuzzyStringComparisonTolerance |
Degree of similarity of the text of the region sought for comparison with the specified fuzzy algorithm. See The Tolerance parameter options |
| Tolerance Value | manualTolerance |
Only whenTolerance is Manual |
Number |
Defines the minimum percentage from 0 to 100 to be applied as a tolerance to the selected Fuzzy Algorithm. |
| Segmentation | segmentation |
Required |
StringSegmentation |
Type of text to search for. -Phrase; -Word |
| Text | text |
Required |
Text |
Internal text of the user interface control. |
| Ocurrence Type | occurrencetype |
Optional |
FindOccurrenceTypes |
Type of occurrence used to find Text: See the ocurrencetype parameter options |
| Index | occurrence |
Only whenOcurrence Type is N |
Number |
Index of the reported occurrence used to locate the Text. |
| Double Click | doubleclick |
Optional |
Boolean |
When enabled, double clicks on the region specified for the OCR click. |
| Click on Position | clickonposition |
Optional |
Boolean |
When enabled, it allows the OCR click to be performed in a specific corner of the screen. |
| Corner | corner |
Optional |
ControlCorner |
Delimits in which corner of the screen the OCR click will be performed. See The corner parameters option |
| X offset (Obsolete) | deltax |
Optional |
Number |
The horizontal offset in pixels. This parameter is obsolete, use the Region parameter instead. |
| Y offset (Obsolete) | deltay |
Optional |
Number |
The vertical offset in pixels. This parameter is obsolete, use the Region parameter instead. |
| Position | position |
Only whenClick on Position is True |
Point |
Coordinates, in pixels, for the click on one of the corners to be executed. The specified coordinates can be changed according to the screen resolution of the machine on which the script is being executed. |
| Language | language |
Required |
Text, Culture |
Language of the text that should be retrieved. The user can override the default supported languages with a valid culture code that is supported by the OCR provider. For supported languages see Supported languages. |
| Region | region |
Optional |
Rectangle |
Region used to get the text. |
| Invert Colors | invertcolors |
Optional |
Boolean |
When enabled, allows inverting the control colors. |
| Update screen cache | forcerefresh |
Optional |
Boolean |
When enabled, reads the screen again before running the command to identify new elements and update the screen cache. This is needed if the screen changes frequently. |
| Name | name |
Only whenSelector is Name, NameAndValue, IdAndName, TypeAndName |
Text |
Name of the UI control. |
| Text | innertext |
Only whenSelector is InnerTextAndTag |
Text |
Inner text of the UI control. |
| Element Type | tagname |
Only whenSelector is InnerTextAndTag |
Text |
Type of the element or tag. |
| Name | classname |
Only whenSelector is ClassAndValue, ClassName |
Text |
Class of the UI control. |
| Element Value | elementvalue |
Only whenSelector is ClassAndValue, NameAndValue |
Text |
Current value of the UI control. |
| Type | controltype |
Only whenSelector is TypeAndIndex, TypeAndName |
ControlTypes |
UI control type. See the controltype parameter options |
| Index | index |
Only whenSelector is TypeAndIndex |
Number |
Index of the UI control. |
| Control | control |
Only whenSelector is Instance |
Control |
Control instance. This variable is obtained using the Search Controlcommand. |
| Element in Table | usetable |
Optional |
Boolean |
When enabled, allows fetching an element within a table. |
| Search by Column | searchbycolumn |
Optional |
Boolean |
When enabled, fetches the element by column, not by row. |
| Column | searchcolumn |
Only whenElement in Table is True |
Text |
Name or number of the column from which the element is obtained. |
| Value | searchvalue |
Only whenSearch by Column is True |
Text |
Value used to find the element in the table. |
| Return Column | returncolumn |
Only whenSearch by Column is True |
Text |
Column that contains the control. |
| Row | row |
Only whenSearch by Column is False |
Number |
Table row from which the element is obtained. |
| Timeout | timeout |
Optional |
Time Span, Number, Text |
Maximum waiting time for running the command. When blank, the default timeout is 5 seconds. |
selector parameter options
The following table displays the options available for the selector input parameter. The table shows the options available when working in Script mode and the equivalent label in the Designer mode.
| Designer mode label | Script mode name | Description |
|---|---|---|
| Id | Id |
It matches the control's unique identifier. |
| Name | Name |
It matches the control name. |
| Name and value | NameAndValue |
It matches the control name and the value. |
| XPath | XPath |
It matches the path through the control tree to the target control. |
| Id and name | IdAndName |
It matches the unique identifier and the control name. |
| Class and value | ClassandValue |
It matches the control class name and the value. |
| Class Name | ClassName |
It matches the class name that instantiates the control. |
| Inner text and control type | InnerTextAndTag |
It matches the control inner text and its type attributes. |
| Instance | instance |
It matches the control instance. |
| Type and index | typeandindex |
It matches the control type and the index of its position among controls of equal type. |
| Type and name | typeandname |
It matches the control type and the name. |
ocrprovider parameter options
The following table displays the options available for the ocrprovider input parameter. The table shows the options available when working in Script mode and the equivalent label in the Designer mode.
| Designer mode label | Script mode name | Description |
|---|---|---|
Google |
Google Tesseract™ | |
| Google Cloud Vision | GoogleVision |
Google Cloud Vision™ |
| Abbyy | Abbyy |
Abbyy™ |
corner parameter options
The following table displays the options available for the corner input parameter. The table shows the options available when working in Script mode and the equivalent label in the Designer mode.
| Designer mode label | Script mode name | Description |
|---|---|---|
| Bottom left | BottomLeft |
It matches the bottom left corner relative to the screen position. |
| Bottom right | Bottomright |
It matches the bottom right corner relative to the screen position. |
| Top left | TopLeft |
It matches the top left corner relative to the screen position. |
| Top right | TopRight |
It matches the top right corner relative to the screen position. |
Controltype parameter options
The following table displays the options available for the Controltype input parameter. The table shows the options available when working in Script mode and the equivalent label in the Designer mode.
| Designer mode label | Script mode name | Description |
|---|---|---|
| Button | Button |
It matches the Button control in the application. |
| Check box | CheckBox |
It matches the Check box control in the application. |
| Child window | ChildWindow |
It matches the Child window control in the application. |
| Combo box | ComboBox |
It matches the Combo box control in the application. |
| Generic | Generic |
It matches the Generic control in the application. |
| Image | Image |
It matches the Image control in the application. |
| Label | Label |
It matches the Label control in the application. |
| List box | ListBox |
It matches the List box control in the application. |
| Menu bar | MenuBar |
It matches the Menu bar control in the application. |
| Menu item | MenuItem |
It matches the Menu item control in the application. |
| Progress bar | ProgressBar |
It matches the Progress bar control in the application. |
| Radio button | RadioButton |
It matches the Radio button control in the application. |
| Scroll bar | ScrollBar |
It matches the Scroll bar control in the application. |
| Slider | Slider |
It matches the Slider control in the application. |
| Spinner | Spinner |
It matches the Spinner control in the application. |
| Status bar | StatusBar |
It matches the Status bar control in the application. |
| Tab page | TabPage |
It matches the Tab page control in the application. |
| Tab panel | TabPanel |
It matches the Tab panel control in the application. |
| Table | Table |
It matches the Table control in the application. |
| Text box | TextBox |
It matches the Text box control in the application. |
| Toggle button | ToggleButton |
It matches the Toggle button control in the application. |
| Tree view | TreeView |
It matches the Tree view control in the application. |
| TreeTable | TreeTable |
It matches the TreeTable control in the application. |
| Window | Window |
It matches the Window control in the application. |
fuzzyalgorithm parameter options
The following table displays the options available for the fuzzyalgorithm input parameter. The table shows the options available when working in Script mode and the equivalent label in the Designer mode. Fuzzy algorithms are string
comparison algorithms
Fuzzy algorithms are string comparison algorithms.
| Designer mode label | Script mode name |
|---|---|
| Dice Coefficient | DiceCoefficient |
| Hamming Distance | HammingDistance |
| Jaccard Distance | JaccardDistance |
| Jaro Distance | JaroDistance |
| Jaro Winkler Distance | JaroWinklerDistance |
| Levenshtein Distance | LevenshteinDistance |
| Longest Common Subsequence | LongestCommonSubsequence |
| Longest Common Substring | LongestCommonSubstring |
| Overlap Coefficient | OverlapCoefficient |
| Ratcliff Obershelp Similarity | RatcliffObershelpSimilarity |
| Sorensen Dice Distance | SorensenDiceDistance |
| Tanimoto Coefficient | TanimotoCoefficient |
occurrencetype parameter options
The following table displays the options available for the occurrencetype input parameter. The table shows the options available when working in Script mode and the equivalent label in the Designer mode.
| Designer mode label | Script mode name | Description |
|---|---|---|
| First occurrence | First |
First ocurrence to find text. |
| Last occurrence | Last |
Last ocurrence to find text. |
| Nth occurrence | N |
Nth ocurrence to find text. |
comparison parameter options
The following table displays the options available for the comparison input parameter. The table shows the options available when working in Script mode and the equivalent label in the Designer mode.
| Designer mode label | Script mode name |
|---|---|
| Approximately Equals | ApproximatelyEquals |
| Begins with | Begins_With |
| Ends with | Ends_With |
| Equal to | Equal_To |
| Matches | Matches |
The Approximately Equals option uses a degree of tolerance as a method of comparison along with the fuzzy algorithm.
tolerance parameter options
The following table displays the options available for the tolerance input parameter. The table shows the options available when working in Script mode and the equivalent label in the Designer mode.
| Designer mode label | Script mode name | Description |
|---|---|---|
| Manual | Manual |
Manual Degree of similarity of the text of the region sought for comparison. |
| Normal | Normal |
Normal Degree of similarity of the text of the region sought for comparison. |
| Strong | Strong |
Strong Degree of similarity of the text of the region sought for comparison. |
| Weak | Weak |
Weak Degree of similarity of the text of the region sought for comparison. |
Output parameter
| Designer mode label | Script mode name | Accepted variable types | Description |
|---|---|---|---|
| Success | success |
Boolean |
Returns true if the text was successfully obtained, or false otherwise. |
| Image | image |
Image |
Image that has been processed. |
| Text | text |
Text |
Returns the text that has been recognized. |
| Region | bounds |
Rectangle |
Returns the region where the text was found in the image. |
Example
A Paint executable is opened, using the Click by OCR command to recognize the text indicated by the Region parameter, with "Google Cloud Vision", in order to search and click on a specific field previously mapped in the IBM RPA Studio's recorder.
defVar --name paintWindow --type Window
defVar --name paintImage --type Image
defVar --name clickedText --type String
defVar --name success --type Boolean
defVar --name clickRegion --type Rectangle
// Opens the Paint executable.
launchWindow --executablepath "C:\\Windows\\System32\\mspaint.exe" paintWindow=value
// Extracts the text contained in the informed control, using the Google Cloud Vision OCR provider.
ocrClick --ocrprovider "GoogleVision" --comparison "Contains" --segmentation "Word" --text View --occurrencetype "First" --language "eng" --region "5,3,42,20" --selector "Name" --name View success=success paintImage=image clickedText=text clickRegion=bounds
logMessage --message "Success: ${success}\r\nClicked Text: ${clickedText}\r\nClick Region: ${clickRegion}" --type "Info"
closeWindow --window ${paintWindow}
// The return varies according to the current execution context. In the case of this example:
// Success: True
// Clicked Text: View
// Click Region: 8,5,20,9