Research on mass monitoring data Retrieval Technology based on HBase

Abstract

Abstract With the arrival of the era of big data, the data generated by the monitoring system grows rapidly, the storage and quick retrieval of massive monitoring data pose a challenge to the database. HBase is an open-source version of Bigtable, built on top of HDFS, based design and query on rowkey is quick, but with the development of database application, when retrieving non-rowkey fields, HBase can only query by scanning the full table, which obviously cannot meet the actual requirements, and the query of the monitoring data is mostly based on the most recent data, so for this problem, this article presents a secondary indexing approach based on coprocessor and Redis. This method realizes the rapid creation and automatic update of secondary indexes in HBase and Redis through the coprocessor. It gets the corresponding rowkey quickly during retrieval, and query the corresponding data in the data table according to the rowkey. The former stores the index of the data table, while the latter stores the index of the latest data in the data table, to improve the real-time performance of data retrieval.