PoseBlocks: A Toolkit for Creating (and Dancing) with AI

Abstract

Body-tracking artificial intelligence (AI) systems like Kinect games, Snapchat Augmented Reality (AR) Lenses, and Instagram AR Filters are some of the most engaging ways students experience AI in their everyday lives. Additionally, many students have existing interests in physical hobbies like sports and dance. In this paper, we present PoseBlocks; a suite of block-based programming tools which enable students to build compelling body-interactive AI projects in any web browser, integrating camera/microphone inputs and body-sensing user interactions. To accomplish this, we provide a custom block-based programming environment building on the open source Scratch project, introducing new AI-model-powered blocks supporting body, hand, and face tracking, emotion recognition, and the ability to integrate custom image/pose/audio models from the online transfer learning tool Teachable Machine. We introduce editor functionality such as a project video recorder, pre-computed video loops, and integration with curriculum materials. We discuss deploying this toolkit with an accompanying curriculum in a series of synchronous online pilots with 46 students, aged 9-14. In analyzing class projects and discussions, we find that students learned to design, train, and integrate machine learning models in projects of their own devising while exploring ethical considerations such as stakeholder values and algorithmic bias in their interactive AI systems.