Cloud–Edge Hybrid Computing Architecture for Large-Scale Scientific Facilities Augmented with an Intelligent Scheduling System

Abstract

Synchrotron radiation sources are widely used in interdisciplinary research, generating an enormous amount of data while posing serious challenges to the storage, processing, and analysis capabilities of the large-scale scientific facilities worldwide. A flexible and scalable computing architecture, suitable for complex application scenarios, combined with efficient and intelligent scheduling strategies, plays a key role in addressing these issues. In this work, we present a novel cloud–edge hybrid intelligent system (CEHIS), which was architected, developed, and deployed by the Big Data Science Center (BDSC) at the Shanghai Synchrotron Radiation Facility (SSRF) and meets the computational needs of the large-scale scientific facilities. Our methodical simulations demonstrate that the CEHIS is more efficient and performs better than the cloud-based model. Here, we have applied a deep reinforcement learning approach to the task scheduling system, finding that it effectively reduces the total time required for the task completion. Our findings prove that the cloud–edge hybrid intelligent architectures are a viable solution to address the requirements and conditions of the modern synchrotron radiation facilities, further enhancing their data processing and analysis capabilities.