Para-Pipe: Exploiting Hierarchical Operator Parallelism of ML Computational Graphs on SoCs
2026-09-03 • Distributed, Parallel, and Cluster Computing
Distributed, Parallel, and Cluster ComputingMachine LearningPerformance
AI summaryⓘ
The authors address the problem of running complicated deep learning tasks on special computer chips made of different types of processors. They note that usual methods either split work across steps (pipelining) or run many parts at once (parallelism), but each approach alone has downsides for speed or latency. Their solution, Para-Pipe, cleverly combines both methods to better balance how fast and responsive the system is, while also saving energy. Tests on two different chip types show that Para-Pipe can find good setups that improve energy use and performance compared to only pipelining or only parallel execution.
edge computingdeep learningSystem-on-Chip (SoC)pipeliningoperator parallelisminference latencyenergy efficiencyARM big.LITTLE CPUsdeep learning acceleratorDSP
Authors
Yujie Zhang, Huiying Lan, Ehsan Aghapour, Zhiyuan Ning, Peng Zan, Weidong Shao, Anuj Pathania, Tulika Mitra
Abstract
As edge-based deep learning applications become more complex, optimizing performance on heterogeneous System-on-Chips (SoCs) presents unique challenges. Traditional pipelining techniques distributing the computation across different on-chip processing units, while effective for throughput, do not address the latency demands posed by modern neural networks with complex interdependencies and extensive operator parallelism. There is a potential in leveraging operator parallelism to enable concurrent execution across multiple processing units, thereby reducing inference latency. However, prioritizing pipelining or parallel execution often necessitates a compromise, where optimizing one performance metric adversely impacts the other. This paper introduces Para-Pipe, a hierarchical mapping framework that integrates intra- and inter-stage operator parallelism within a pipelined architecture. Para-Pipe navigates the trade-off between throughput and latency by selectively fine-tuning parallelism levels within and across pipeline stages. This strategy can significantly reduce inter-processor communication overhead, significantly improving energy efficiency. Our evaluation demonstrates that Para-Pipe generates multiple Pareto-optimal configurations, achieving a balance between throughput and latency on an Amlogic SoC equipped with ARM big.LITTLE CPUs and GPU, as well as the Black Sesame Technology SoC featuring a deep learning accelerator and two DSPs. More importantly, throughput-optimized configurations under Para-Pipe on Amlogic SoC show an average energy efficiency improvement of 11.0% over purely pipelined strategies and 23.3% relative to non-pipelined parallel execution.