Optimizing Logging and Monitoring in Heterogeneous Cloud Environments for IoT and Edge Applications

As data is becoming more and more important, Internet of Things (IoT) devices are widely used to collect information and process data from various industries, such as finance, autonomous driving, and smart factories. To address the limited computational power of IoT devices in processing real-time data, both edge computing, which utilizes nearby computers with greater computation capabilities, and cloud computing with even more processing power, are widely adopted solutions. As these systems have heterogeneous software and hardware configurations, it can be challenging to understand the behavior of the application from the perspective of different resources. In this article, we propose an efficient logging and monitoring system in large-scale, heterogeneous environments for IoT and edge applications. To do this, our scheme first collects system resource usage data from each compute node using the operating system's native system analysis tool. Then, it consolidates the system resource usage information from multiple nodes into an integrated database which creates a comprehensive view of the system. Finally, our scheme provides global system resource information in terms of specific jobs and nodes, providing a comprehensive understanding of complex heterogeneous hardware/software stacks. Our evaluation, using IoT and edge workloads in heterogeneous systems, demonstrates the efficiency of logging and monitoring schemes. The average network usage for Windows and Linux is 0.12 and 1.29 kB/s, respectively, resulting in minimal network overhead. In addition, the proposed scheme shows negligible overhead in terms of both runtime (up to 0.73%) and storage (0.0474%).