VID2META: Complementing Android Programming Screencasts with Code Elements and GUIs

Abstract

The complexity of software projects and the rapid technological evolution make it such that developers often need additional help and knowledge to tackle their daily tasks. For this purpose, they often refer to online resources, which are easy to access and contain a wealth of information in various formats. Programming screencasts hosted on platforms such as YouTube are one such online resource that has seen a growth in popularity and adoption over the past decade. These screencasts usually have some metadata such as a title, a short description, and a set of tags that should describe what the main concepts captured in the video are. Unfortunately, metadata are often generic and do not contain detailed information about the code showcased in the tutorial, such as the API calls or graphical user interface (GUI) elements employed, which could lead to developers missing useful tutorials. Having a quick overview of the main code elements and GUIs used in a video tutorial can be very helpful for developers looking for code examples involving specific API calls, or looking to design applications with a specific GUI in mind. The aim is to make this information easily available to developers, and propose VID2META, a technique that automatically extracts Java import statements, class names, method information, GUI elements, and GUI screens from videos and makes them available to developers as metadata. VID2META is currently designed to work with Android screencasts. It analyzes video frames using a combination of computer vision, deep learning, optical character recognition, and heuristic-based approaches to identify the needed information in a frame, extract it, and present it to the developer. VID2META has been evaluated in an empirical study on 70 Android programming videos collected from YouTube. The results revealed that VID2META can accurately detect and extract Java and GUI elements from Android programming videos with an average accuracy of 90%.