Method of Transition from Data Warehouses to Geographic Information System Data Lakes Based on Lambda Architecture-Reference-Cited by-同舟云学术

Method of Transition from Data Warehouses to Geographic Information System Data Lakes Based on Lambda Architecture

Published:2024-04-14 Issue:1 Volume:0 Page:45-55
ISSN:2413-2527
Container-title:Intellectual Technologies on Transport
language:en
Short-container-title:

Author:

Abu Hasan¹,Kirienko Andrey²,Homonenko Anatoliy¹²

Affiliation:

1. Emperor Alexander I St. Petersburg State Transport University

2. VKA named after A. F. Mozhaisky

Abstract

This paper discusses the transition from traditional data warehouses to data lakes in geographic information systems using Lambda architecture. Provides an overview of the key transition steps, including planning, data collection and processing, data querying, data analytics, and metadata management. Particular attention is paid to the interaction of data lakes and GIS, as well as sample big data processing code based on Lambda architecture. The advantages of using data lakes in GIS and the possibilities o integrating modern data processing technologies are considered.

Publisher

Petersburg State Transport University

Reference24 articles.

1. Ёcy, М. Т. Принципы организации распределенных баз данных = Principles of Distributed Database Systems. Fourth Edition / М. Т. Ёсу, П. Вальдуриес; пер. с англ. А. А. Слинкина. — Москва: ДМК Пресс, 2021. — 672 с., Ecy, M. T. Principy organizacii raspredelennyh baz dannyh = Principles of Distributed Database Systems. Fourth Edition / M. T. Esu, P. Val'duries; per. s angl. A. A. Slinkina. — Moskva: DMK Press, 2021. — 672 s.

2. Bhattacherjee, S. RStore: A Distributed Multi-Version Document Store / S. Bhattacherjee, A. Deshpande // Proceedings of the 34th International Conference on Data Engineering (ICDE 2018), (Paris, France, 16–19 April 2018). — Institute of Electrical and Electronics Engineers, 2018. — Pp. 389–400. DOI: 10.1109/ICDE.2018.00043., Bhattacherjee, S. RStore: A Distributed Multi-Version Document Store / S. Bhattacherjee, A. Deshpande // Proceedings of the 34th International Conference on Data Engineering (ICDE 2018), (Paris, France, 16–19 April 2018). — Institute of Electrical and Electronics Engineers, 2018. — Pp. 389–400. DOI: 10.1109/ICDE.2018.00043.

3. Leveraging the Data Lake: Current State and Challenges / C. Giebler, C. Gröger, E. Hoos, [et al.] // Big Data Analytics and Knowledge Discovery (DaWaK 2019): Proceedings of the 21st International Conference (Linz, Austria, 26–29 August 2019) / C. Ordonez, [et al.] (eds.). — Cham: Springer Nature, 2019. — Pp. 179–188. — (Lecture Notes in Computer Science. Vol. 11708). DOI: 10.1007/978–3–030–27520–4_13., Leveraging the Data Lake: Current State and Challenges / C. Giebler, C. Gröger, E. Hoos, [et al.] // Big Data Analytics and Knowledge Discovery (DaWaK 2019): Proceedings of the 21st International Conference (Linz, Austria, 26–29 August 2019) / C. Ordonez, [et al.] (eds.). — Cham: Springer Nature, 2019. — Pp. 179–188. — (Lecture Notes in Computer Science. Vol. 11708). DOI: 10.1007/978–3–030–27520–4_13.

4. Lock, M. Maximizing Your Data Lake with a Cloud or Hybrid Approach / M. Lock; Aberdeen Group. — 2016. — 4 p. URL: http://technology-signals. com/wp-content/uploads/download-managerfiles/maximizingyourdatalake.pdf (дата обращения 12.01.2024)., Lock, M. Maximizing Your Data Lake with a Cloud or Hybrid Approach / M. Lock; Aberdeen Group. — 2016. — 4 p. URL: http://technology-signals. com/wp-content/uploads/download-managerfiles/maximizingyourdatalake.pdf (data obrascheniya 12.01.2024).

5. Extending Data Lake Metadata Management by Semantic Profiling / J.W. Ansari, N. Karim, S. Decker, [et al.] // Proceedings of the 15th International Extended Semantic Web Conference (ESWC 2018), (Heraklion, Crete, Greece 03–07 June 2018). — Springer International Publishing, 2018. — 15 p. URL: http://2018.eswc-conferences.org/wp-content/uploads/2018/02/ ESWC2018_paper_127.pdf (дата обращения 12.01.2024), Extending Data Lake Metadata Management by Semantic Profiling / J.W. Ansari, N. Karim, S. Decker, [et al.] // Proceedings of the 15th International Extended Semantic Web Conference (ESWC 2018), (Heraklion, Crete, Greece 03–07 June 2018). — Springer International Publishing, 2018. — 15 p. URL: http://2018.eswc-conferences.org/wp-content/uploads/2018/02/ ESWC2018_paper_127.pdf (data obrascheniya 12.01.2024)