Optimizing Big Data Insights with Serverless Architecture

Abstract

Big data is the huge amount of data, which can be structured, semi-structured, or unstructured, that is required for current commercial processes. Big Data efforts and technologies are used to analyze large amounts of data in order to gain insights critical for strategic decision-making. Data size is constantly rising, reaching petabytes, exabytes, zettabytes, and even yottabytes, offering substantial management and processing issues. In practice, managing massive amounts of data involves several obstacles, such as server management, storage, clustering, and algorithm deployment. Manual intervention hampers the creation of successful Cloud-based data analysis platforms. Serverless computing provides a solution by offering clients pay-per-use backend services, reducing the need for users to manage server operations. This article describes a serverless architecture for large data analytics, including implementation, maintenance, and governance on Amazon Web Services (AWS). Furthermore, it investigates the differences between traditional and big data analytics in a serverless system