Monitoring Framework for the Performance Evaluation of an IoT Platform with Elasticsearch and Apache Kafka

Abstract

AbstractIoT platforms are in charge of extracting and processing the data that come from IoT networks, generating additional value, and providing access to the user through usable interfaces. However, the ever growing number of devices, networks, services and applications within the IoT ecosystem, and the recently adopted edge/cloud architecture, increase the complexity. Therefore, IoT platforms should integrate monitoring and visualization tools to facilitate deployment, management and maintenance tasks. In this work, we present the implementation and performance evaluation of an IoT modular platform for distributed architectures that combines the use of Elastic Stack tools (Elasticsearch, Kibana and Beats) and Apache Kafka. We have developed a monitoring framework based on Beats agents that supervise the platform performance attending to different metrics; and adapted the Kibana visualization tools to provide friendly and accessible information to platform administrators and users. Finally, we have deployed and evaluated the IoT platform in four real use cases, identifying the factors that affect the performance of the different modules: Edge Node, Data Streaming, Cloud Server and Search Engine.