Evaluating a Large Language Model on Searching for GUI Layouts-Reference-Cited by-同舟云学术

Evaluating a Large Language Model on Searching for GUI Layouts

Published:2023-06-14 Issue:EICS Volume:7 Page:1-37
ISSN:2573-0142
Container-title:Proceedings of the ACM on Human-Computer Interaction
language:en
Short-container-title:Proc. ACM Hum.-Comput. Interact.

Author:

Brie Paul¹^ORCID,Burny Nicolas²^ORCID,Sluÿters Arthur²^ORCID,Vanderdonckt Jean²^ORCID

Affiliation:

1. TeleportHQ, Cluj-Napoca, Romania

2. Université catholique de Louvain, Louvain-la-Neuve, Belgium

Abstract

The field of generative artificial intelligence has seen significant advancements in recent years with the advent of large language models, which have shown impressive results in software engineering tasks but not yet in engineering user interfaces. Thus, we raise a specific research question: would an LLM-based system be able to search for relevant GUI layouts? To address this question, we conducted a controlled study evaluating how Instigator, an LLM-based system for searching GUI layouts of web pages by generative pre-trained training, would return GUI layouts that are relevant to a given instruction and what would be the user experience of (N =34) practitioners interacting with Instigator. Our results identify a very high similarity and a moderate correlation between the rankings of the GUI layouts generated by Instigator and the rankings of the practitioners with respect to their relevance to a given design instruction. We highlight the results obtained through thirteen UEQ+ scales that characterize the user experience of the practitioner with Instigator, which we use to discuss perspectives for improving such future tools.

Funder

Fonds de la Recherche Scientifique-FRS

EU EIC Pathfinder-Awareness Inside challenge

Fonds De La Recherche Scientifique - FNRS

Publisher

Association for Computing Machinery (ACM)

Subject

Computer Networks and Communications,Human-Computer Interaction,Social Sciences (miscellaneous)

Link

https://dl.acm.org/doi/pdf/10.1145/3593230

Reference107 articles.

1. Toufique Ahmed and Premkumar Devanbu . 2023 . Few-Shot Training LLMs for Project-Specific Code-Summarization . In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering ( Rochester, MI, USA) (ASE '22). Association for Computing Machinery, New York, NY, USA, Article 177, 5 pages. https://doi.org/10.1145/ 3551349.3559555 Toufique Ahmed and Premkumar Devanbu. 2023. Few-Shot Training LLMs for Project-Specific Code-Summarization. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering (Rochester, MI, USA) (ASE '22). Association for Computing Machinery, New York, NY, USA, Article 177, 5 pages. https://doi.org/10.1145/ 3551349.3559555

2. Kei Aoki , Hajime Kimura , and Shigenobu Kobayashi . 2004 . Distributed Reinforcement Learning using Bi-directional Decision Making for Multicriteria Control of Multi-Stage Flow Systems . In Proceedings of IAS '04 . http://sysplan. nams.kyushu-u.ac.jp/gen/papers/klab2004/IAS8AokiBDM.pdf Kei Aoki, Hajime Kimura, and Shigenobu Kobayashi. 2004. Distributed Reinforcement Learning using Bi-directional Decision Making for Multicriteria Control of Multi-Stage Flow Systems. In Proceedings of IAS '04. http://sysplan. nams.kyushu-u.ac.jp/gen/papers/klab2004/IAS8AokiBDM.pdf

3. Gideon Avrahami , Kenneth P. Brooks , and Marc H. Brown . 1989. A Two-View Approach to Constructing User Interfaces . In Proceedings of the 16th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '89) . Association for Computing Machinery, New York, NY, USA, 137--146. https://doi.org/10.1145/74333.74347 10.1145/74333.74347 Gideon Avrahami, Kenneth P. Brooks, and Marc H. Brown. 1989. A Two-View Approach to Constructing User Interfaces. In Proceedings of the 16th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '89). Association for Computing Machinery, New York, NY, USA, 137--146. https://doi.org/10.1145/74333.74347

4. Dzmitry Bahdanau , Kyunghyun Cho , and Yoshua Bengio . 2015 . Neural Machine Translation by Jointly Learning to Align and Translate . In Proc. of 3rd International Conference on Learning Representations ( San Diego, CA, USA) (ICLR '15), Yoshua Bengio and Yann LeCun (Eds.). http://arxiv.org/abs/1409.0473 Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In Proc. of 3rd International Conference on Learning Representations (San Diego, CA, USA) (ICLR '15), Yoshua Bengio and Yann LeCun (Eds.). http://arxiv.org/abs/1409.0473

5. F. Behrang , S. P. Reiss , and A. Orso . 2018. GUIFetch: Supporting App Design and Development through GUI Search . In 2018 IEEE/ACM 5th International Conference on Mobile Software Engineering and Systems (MOBILESoft). IEEE Computer Society , Los Alamitos, CA, USA, 236--246. https://doi.ieeecomputersociety.org/ F. Behrang, S. P. Reiss, and A. Orso. 2018. GUIFetch: Supporting App Design and Development through GUI Search. In 2018 IEEE/ACM 5th International Conference on Mobile Software Engineering and Systems (MOBILESoft). IEEE Computer Society, Los Alamitos, CA, USA, 236--246. https://doi.ieeecomputersociety.org/