Adaptive Container Service: a New Paradigm for Robust and Optimized Bioinformatics Workflow Deployment in the Cloud

Abstract

AbstractWe propose Adaptive Container Service (ACS), a new paradigm for deploying bioinformatics workflows in cloud computing environments. By encapsulating the entire workflow within a single virtual container, combined with automatic workflow checkpointing and dynamic migration to appropriately scaled containers, ACS-based deployment demonstrates several key advantages over alternative strategies: it enables optimal resource provision to any workflow that comprise of multiple applications with diverse computing needs; it provides protection against application-agnostic out-of-memory (OOM) errors or spot instance interruptions; and it reduces efforts required for workflow development, optimization, and management because it runs workflows with minimal or no code modifications. Proof-of-concept experiments show that ACS avoided both under- and over-provisioning in monolithic single-container deployment. Despite being deployed as a single container, it achieved comparable resource utilization efficiency as optimized Nextflow-managed, multi-modular workflows. Analysis of over 18,000 workflow runs demonstrated that ACS can effectively reduce workflow failures by two-thirds. These findings suggest that ACS frees developers from navigating the complexity of deploying robust workflows and rightsizing compute resources in the cloud, leading to significant reduction in workflow development time and savings in cloud computing costs.