In graphical user interface (GUI) application development, software engineers are responsible for scripting both the backend logic and the frontend interfaces adhering to the specifications provided by the UI/UX design team. This process can lead to discrepancies between the initial GUI design mockup and its ultimate implementation. Those discrepancies can undermine user experience, erode trust in the application's reliability, and lead to increased development costs due to the necessity for revisions and debugging. The discrepancies might manifest in both the static layouts of the interface and the dynamic transitions between states. Manually identifying all potential inconsistencies proves to be a labor-intensive task. Moreover, the varied interpretations and implementations of the GUI design mockup introduce an additional layer of complexity.
In this work, we propose an automatic GUI-testing framework, GUI-Pilot, which:
- ✅ (i) Parses the noisy GUI design mockup into a structured and formalized GUI mockup graph.
- ✅ (ii) Assesses the visual layout inconsistencies between the design mockup and its implementation across all potential interfaces.
- ✅ (iii) Identifies inconsistencies in state transitions by automatically traversing the state graph of the actual implemented GUI application, guided by the traces present on the GUI mockup graph.
On the experimental dataset, our xxxxx.
-
Step 1: Parse the raw GUI design mockup into a GUI mockup graph
- Challenge: a variety of node types
- Solution: define four node types—interface node, decision node, process node, and widget node—to structure the GUI mockup graph effectively.
-
Step 2: Detect visual layout inconsistencies
- Challenges: (i) Widgets within the mockup are diverse, exhibit multimodal characteristics, and are often presented in low resolution. (ii) The mockup and the final interface will not align perfectly; dynamic content elements such as advertisements and animations may vary, leading to potential false positives if the analysis is overly strict.
- Solution: (i) Widgets are categorized into major classes based on their functionality to simplify detection and analysis. (ii) A hybrid approach utilizing both OCR (Optical Character Recognition) and object detection techniques identifies widgets. (iii) A matching algorithm tailored for this context is proposed to mitigate the identified challenges effectively.
-
Step 3: Detect transition inconsistencies
- Challenges: (i) additional actions are required (e.g. filling up the form) to perform certain transitions (ii) the actions may not be performed successfully at the first attempt
- Solution: (i) we develop an LLM-powered agent specialized in guiding GUI state transitions. This agent can autonomously recommend actions based on the current interface context. (ii) leveraging feedback, the agent iteratively refines its action suggestions, enhancing the robustness of transition testing.