Chill and Relax When Running Robotic Process Automation

Simple and easy tasks can turn to be annoying tasks when they are in large numbers and happen to be repetitive. Our time can be wasted doing the same things over and over again when we can spend the time doing something else. Even when we already know how to do those tasks, there is a possibility that we lose focus, make mistakes, and mess things up.

Photo by Andy Kelly on Unsplash

However, we can say goodbye to those annoying tasks and say hello to our savior, which is known as Robotic Process Automation (RPA). RPA utilizes a graphical user interface to capture data and manipulate applications so it can emulate human interaction within digital systems. In short, we have a virtual robot that will copy our behavior to do our jobs. We can leave all those tasks to the robot and do something else, or maybe give ourselves some time to rest. All of these sound impressive, right?

Well, it depends. From the point of view of the users, RPA is the best thing ever. It can cut man-hours, reduce costs, reduce human errors, and have more benefits. The users just need to watch the automation running. Meanwhile, the hardships of RPA go to the developers, especially the first-timers like me who did not know a single thing about RPA and how to make automation back then.

I have not explored RPA tools much, but I find SikuliX to suit my current condition to do RPA. It uses a programming language that is easy enough to understand even by someone who does not have an IT background. Like other RPA tools, SikuliX uses image recognition to identify graphical user interface (GUI) components. Surely I was fascinated when I used SikuliX for the first time. The mouse cursor was moving on its own and empty text fields were filled by self-typing words when I ran my automation script. I felt happy until the error messages appeared uninvited. Well, they are more of exception messages rather than error messages, but they are red-colored so I called them error messages.

After looking for some explanation of those errors, I learned several tricks that can help us in creating an automation script in SikuliX. Maybe these tricks will also help us when using other RPA tools.

1. Wait

No, I do not mean to wait and stop reading. I mean we should wait for a certain time after doing something until something else happens. Sometimes, our computer has so many background apps running that it starts to take a long time just to open an app. For example, if we want to type something in a cell in an excel document, after double-clicking the document, we only have to click the cell that we want to edit and start typing.

The simple script of editing a cell in an excel document.

But after double-clicking the document, the spreadsheet does not always open immediately, while SikuliX will continue the process onto the next line, which is clicking the cell. This can raise a FindFailed Exception because the cell image has not appeared on the screen yet. So in this case, we should wait until the cells appear before clicking those cells. In SikuliX, we can use the wait() command.

Adding wait() command to wait for the document to be open.

There is another way to wait for an image to appear. We can use the if exists() method for better error handling. Both wait() and exists() commands can put a number as the second parameter. This number specifies how many seconds SikuliX will wait for the image to appear, instead of the default three seconds.

Using if exists() method for error handling. If the image does not appear within 5 seconds, we jump to the else part.

Both of these commands will also be useful when opening web pages which sometimes takes a long time to finish loading. Waiting for an application or a web page to be fully loaded is one of the important parts of automation.

2. Utilize commands rather than using image recognition

Well, this trick is inspired by the mistake I made. There was one incident where I waited for an excel document to be open. Back then, SikuliX always raised a FindFailed exception. After investigating, I found that my LibreOffice Calc zoom factor was changed to 60% rather than the normal set of 100%. So the cell size was also getting smaller and SikuliX failed to recognize the image that I wanted to click.

Even though this problem could be solved by lowering the matching threshold of the image, I do not think it is a good idea because then SikuliX can unintentionally click a similar image which we do not intend to click.

Decreasing the matching threshold can lead to a fatal mistake. Imagine intending to click B3 cell but the system clicks D3 cell.

Then I thought to use the keyboard to navigate to the wanted cell. Using the type() command gives me access to use the keyboard, hence I can navigate to the wanted cell using type(Key.DOWN) and type(Key.RIGHT). Both using image recognition and using keyboard buttons have the same result but the latter is not affected when the zoom factor changed.

Another reason for using commands instead of image recognition is we will have broader access to the involved software. We can use hotkeys or keyboard shortcuts to do something in the software so we can have a faster automation process. For example, when saving an excel document, we prefer to click Ctrl+S rather than clicking File and then Save in the taskbar.

Do not be afraid when we do not know the correct syntax of the commands. SikuliX — and surely other RPA tools — have its documentation web page and we can also browse the internet for the syntax.

3. Use fixed GUI components

Based on the same problem from trick #2, I was forced to change the expected image to be put as a parameter in wait() command because the image I wanted to click was not recognized. Well, the answer was easy this time. I used the LibreOffice Calc taskbar as the image in wait()/exists() command because the taskbar is not affected by the zoom factor. I also used the alphabet of the column header. While the size of the column may change, the font size remains the same.

Using the taskbar as exists() args, and also using right and down key from keyboard instead of clicking to navigate to the wanted cell.

If we use other software in our automation, I suggest that we identify the taskbar or any other kind of unchanging GUI first.

Those were the tricks that helped me to minimize errors when I was automating tasks. Hopefully, those can help you too. Saying that I am not an expert in RPA, so I am sure there are other ways that can be better than mine to help us in developing robotic process automation.

Reference:

[1] https://www.uipath.com/rpa/robotic-process-automation

[2] http://sikulix.com