UI Automation is a critical part of any RPA software, because it enables bots to mimic user behavior with mouse and keyboard on computer system.

Before bots can do actions like click, set text on the target applications, it needs to identify the UI Elements that it will act on.

There are three ways that a bot can recognize the UI Elements on computer’s screen.

  1. UI Element Properties Based
    The underlying technologies that are used to implement the applications like .NET, Java, Web have exposed all the properties of any UI Element.
    A client application like RPA software can access and use these properties via an API to uniquely identity the UI Element.
  2. Region, Position Based
    You can define a specific coordinates inside the screen or the region and do a click action at this coordinates.
  3. Image Based
    In environments like Citrix where RPA software cannot access directly to the target applications, the only thing that it has is the image of the full screen.
    To identify location of an UI Element, RPA software will capture the image of that UI Element and then use template matching techniques to identity it’s location on the screen.
  4. Text Based
    The same with Image Based but use text instead with the helping from OCR techniques.