The Image Calculator: 10x Faster Image-AI Inference by Replacing JPEG with Self-designing Storage Format

Abstract

Numerous applications today rely on artificial intelligence over images. Image AI is, however, extremely expensive. In particular, the inference cost of image AI dominates the end-to-end cost. We observe that the image storage format lies at the root of the problem. Images today are predominantly stored in JPEG format. JPEG is a storage format designed for the human eye; it maximally compresses images without distorting the components of an image that are visible to the human eye. However, our observation is that during image AI, images are "seen'' by algorithms, not humans. In addition, every AI application is different regarding which data components of the images are the most relevant. We present the Image Calculator, a self-designing image storage format that adapts to the given AI task, i.e., the specific neural network, the dataset, and the applications' specific accuracy, inference time, and storage requirements. Contrary to the state-of-the-art, the Image Calculator does not use a fixed storage format like JPEG. Instead, it designs and constructs a new storage format tailored to the context. It does so by constructing a massive design space of candidate storage formats from first principles, within which it searches efficiently using composite performance models (inference time, accuracy, storage). This way, it leverages the given AI task's unique characteristics to compress the data maximally. We evaluate the Image Calculator across a diverse set of data, image analysis tasks, AI models, and hardware. We show that the Image Calculator can generate image storage formats that reduce inference time by up to 14.2x and storage by up to 8.2x with a minimal loss in accuracy or gain, compared to JPEG and its state-of-the-art variants.