Affiliation:
1. Department of Electrical Engineering, University of Balamand, LEBANON
Abstract
The amount of digital data is constantly growing in almost all fields. This data is divided into two categories, structured and unstructured data. Non-structural databases known as NoSQL became one of the main fields of big data. Many companies are still using relational databases like PostgreSQL and MySQL. But with the rapid evolution and diversity of stored data, companies find themselves obliged to use big data tools like HBase or Hive. Big data is characterized by its capacity, speed, and ability to store diverse types of data. Data analysis and high storage capacity are the main reasons for companies to search for new database systems. Data migration to new systems is associated with the modification of the existing data and applications. This process costs a lot to adopt new specialists to handle this transition. Furthermore, due to different sources of data in old systems, e.g., real-time applications that are continuously collecting new data, companies will not be able to leave relational databases. For this reason, we present a system, termed Automatic Query Language, or AQL in short form, for migrating data from PostgreSQL to integrated HBase/Hive databases. In addition, we provide a platform that allows any user to query automatically PostgreSQL, Hive, and HBase databases using SQL query only. Querying the system is related to where each big data tool’s performance is better. After the platform was completed, we were able to insert and select data from both relational databases and big data components. Join operation was not a problem because complex queries for analysis were executed using Hive which was integrated with HBase. The tested AQL system proved that HBase can insert data with more efficiency than PostgreSQL and Hive, and that select query in Hive has a better performance than PostgreSQL for big data size, whereas, for small data size, the performance of PostgreSQL is better.
Publisher
World Scientific and Engineering Academy and Society (WSEAS)
Subject
Computer Science Applications,Information Systems
Reference29 articles.
1. Y. T. Liao, J. Zhou, C. H. Lu, S. C. Chen, C. H. Hsu, W. Chen, et al., “Data adapter for querying and transformation between SQL and NoSQL database,” Future Generation Computer Systems, vol. 65, pp. 111-121, 2016.
2. J. S. Daba, M. R. Bell, “Object discrimination and orientation determination in speckled images,” Optical Engineering, vol. 33, no. 4, pp, 1287-1303, 1994.
3. J. S. Daba, M. R. Bell, “Estimation of the surface reflectivity of SAR images based on a marked Poisson point process model,” IEEE International Symposium on Signals, Systems, and Electronics (ISSSE'95), San Francisco, CA, pp. 183-186, 1995.
4. J. S. Daba, M. R. Bell, “Segmentation of speckled images using a likelihood random field model,” Optical Engineering, vol. 47, no. 1, 2008.
5. J. S. Daba, M. R. Bell, “Statistics of the scattering cross section of a collection of constant amplitude scatterers with random phase,” ECE Technical Reports, Purdue University, p. 194, 1994.