Towards Handling Incremental Load for Anomalies in Near Real Time Data Warehouse-Reference-Cited by-同舟云学术

Towards Handling Incremental Load for Anomalies in Near Real Time Data Warehouse

Published:2020-12-07 Issue: Volume:15 Page:684-690
ISSN:1991-8763
Container-title:WSEAS TRANSACTIONS ON SYSTEMS AND CONTROL
language:en
Short-container-title:

Author:

Mohammed Muddasir N.¹,Raghuveer K.²,Dayanand R.³

Affiliation:

1. Dept. of IS&E,VVCE, Mysuru, Karnataka, India

2. Dept. of IS&E,NIE, Mysuru, Karnataka,India

3. Technical Director, India

Abstract

Refreshment anomalies occur in a data warehousing environment while performing Extract Transform and Load (ETL) to get the data for analysis from sources. There could be several reasons for the anomalies like not able to capture the delta on time, system time out, duplicate entries due to outer join operations and many more. Once anomalies are detected the compensation operation is executed to get the data that was missing into the data warehouse. In this work we would like to analyze scenario where it is necessary to perform incremental loads based on priority in an ongoing data warehouse maintenance work. The work proposes a novel approach to decide on when to perform ETL so that refreshment anomalies do not occur and to maintain integrity of data such that analytics queries always provide right information to the analyst. Two novelties have been discussed in this work one is to have a threshold before compensation of updates and two is while performing compensation updates prioritize the query with less freshness interval to have more time limits for the updates to be completed.

Publisher

World Scientific and Engineering Academy and Society (WSEAS)

Subject

Artificial Intelligence,General Mathematics,Control and Systems Engineering

Reference19 articles.

1. Stefan Dessloch, Weiping Qu, Vinanthi Basavaraj, Sahana Shankar, “Real-Time Snapshot Maintenance with Incremental ETL Pipelines in Data Warehouses,” Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol. 9263, pp. 28–39, 2015, doi: 10.1007/978-3-319-22729-0.

2. W. Qu, “Incremental ETL Pipeline Scheduling for Near Real-Time Data Warehouses 1,” no. Btw, pp. 299–308, 2017.

3. . Chen, W. Rahayu, and D. Taniar, “Towards near real-time data warehousing,” Proc. - Int. Conf. Adv. Inf. Netw. Appl. AINA, pp. 1150–1157, 2010, doi: 10.1109/AINA.2010.54.

4. Y. Zhuge, H. García-Molina, J. Hammer, and J. Widom, “View maintenance in a warehousing environment,” in SIGMOD ’95: Proceedings of the 1995 ACM SIGMOD international conference on Management of data, 1995, pp. 316–327, doi: 10.1023/A:1008698814840.

5. Y. Zhuge, H. Garcia-Molina, and J. L. Wiener, “Consistency Algorithms for Multi-Source Warehouse View Maintenance,” Distrib. Parallel Databases, vol. 6, no. 1, pp. 7–40, 1998, doi: 10.1023/A:1008698814840.