Data Management for ML-Based Analytics and Beyond

Abstract

The increasing capabilities of machine learning (ML) has enabled the deployment of ML methods in a variety of applications, ranging from unstructured data analytics to autonomous vehicles. Due to the volumes of data over which ML is deployed, it is infeasible for humans to monitor deployments: the Tesla fleet of vehicles produces exabytes of data and millions of hours of video per day. As a result, ML deployments can fail in unexpected and catastrophic ways. In this work, we highlight three important but underlooked aspects of ML deployment pipelines: (1) managing high-quality training data, (2) monitoring ML errors at deployment time, and (3) connecting end use to deployment algorithms. We first demonstrate that training labels are often erroneous, contrary to standard practice, even when labeled by leading vendors. We then demonstrate that standard methods of deploying ML methods can lead to downstream errors. As a first step toward addressing these issues, we review and contextualize two abstractions for finding errors in training data and deployments. We further describe how to improve algorithms for analytics queries as a case study for optimizing ML pipelines end to end.