EDeN: Enabling Low-Power CNN Inference on Edge Devices Using Prefetcher-assisted NVM Systems

The accuracy of convolutional neural networks (CNNs) has significantly improved over the years. Meanwhile, due to the high portability and usefulness of edge devices, the demand for artificial intelligence (AI) based applications on edge computing devices has been soaring recently. Accordingly, CNN inference has become one of the mainstream AI applications on edge devices. However, the continually increasing leakage power of edge devices drags down the wide deployment of CNN inference applications, as the technology node scales down.In this work, we focus on reducing the power consumption in main memory, which consumes considerable power in CNN inference. Particularly, we observed that the idle state of memory is dominant in computationally intensive CNN inference. To achieve low-power CNN inference on edge devices, we first utilize next-generation nonvolatile memory (NVM) as the main memory device rather than dynamic random-access memory (DRAM) only for CNN inference tasks. To mitigate the increased latency caused by NVM, we propose a novel prefetcher that smartly leverages existing resources in commercial NVM system models; it is designed to predictably manage the locality-specific demands of CNN models while smartly leveraging existing resources in a modern NVM system. Furthermore, utilizing a prefetcher-based approach, we optimize the write allocation to enhance the data reuse and energy efficiency in CNN workloads. Based on simulation, our design improves the energy efficiency by 50% with a negligible impact on the performance compared with conventional DRAM-based platforms.