Application Massive Data Processing Platform for Smart Manufacturing Based on Optimization of Data Storage

Abstract

The aim of smart manufacturing is to reduce manpower requirements of the production line by applying technology of huge amounts of data to the manufacturing industry. Smart manufacturing is also called Industry 4.0 , and the platform for processing huge amounts of data has an indispensable role. The massive data processing platform is like the brain of the entire factory, receiving all data from production line sensors via edge computing, processing, and analyzing, and finally making feedback decisions. With the innovation of production technology, the data that the platform needs to process has become diverse and complex, and the amount has become increasingly large. As well, many precision manufacturing industries have begun to enter the field of Industry 4.0. In addition to the accuracy and availability of data processing, there is emphasis on the real-time nature of data processing. After the sensor receives the data, the platform must provide feedback within a short period of time. This article proposes a massive data processing platform based on the Lambda architecture, which has the coexistence of stream processing and batch processing to meet real-time feedback needs of high-precision manufacturing. To verify the effectiveness of the optimization, it is based on real data from the manufacturing industry. To generate a large amount of test data to confirm the optimization of the storage of pictures. The results show that it optimizes the storage and optimization of the image data generated by the Automated Optical Inspection technology used in manufacturing today and optimizes the query for data storage. It also reduces the consumption of a large amount of memory as expected, and the query for Hive reduced the time spent.