Robust UI Automation Using Deep Learning and Optical Character Recognition (OCR)

In the complete software development life cycle (SDLC), testing and maintenance stand slowest, and the reason being entire process is manual or manually maintained automation. Over time, companies have realized that it is very costly and time consuming. The biggest reason for manual maintenance of automation is its dependency on dynamic properties of the UI elements (xpath, class, ID) which directly depend on document object model (DOM). Cloud releases are very frequent, and changes in the UI properties are expected. In this paper, we are presenting a UI technology agnostic approach, which does not depend on the DOM and UI properties of the application rather behaves like a human and visually parses the screen. We use artificial intelligence to detect the UI elements, and even, just a mock-up of a screen can be parsed this way. This also enables test-driven development (TDD) of UI, among many other use cases. We used annotated UI image data for training our deep learning model, and the method has been validated with 94% accuracy on new UI elements. This approach does not require manual maintenance of the generated automates unless there is a functional change in the application. We propose UI technology agnostic, zero touch, self-healing UI automation.