Huawei Ascend 910B Support
SMoE can be deployed on Huawei Ascend platforms via CANN. The core idea remains unchanged: keep hot experts in device memory, maximize reuse (hits + soft hits), and overlap compute with asynchronous transfers to minimize end-to-end latency.
Accelerating MoE on the Edge
via Importance-Aware Expert Scheduling
On memory-constrained edge GPUs, we adopt an importance-aware policy to prioritize critical experts and substitute low-importance activated experts with functionally similar experts already cached in GPU memory, reducing PCIe transfers and CPU computation to significantly improve decoding latency (TPOT).
Motivation
Why edge MoE is hard: PCIe transfer dominates the latency.
The "Memory Wall" Bottleneck
On edge devices (e.g., NVIDIA A6000), fetching experts from CPU memory via PCIe is orders of magnitude slower than computing them on GPU. As shown in the figure, the PCIe transfer time (orange) often exceeds the actual GPU computation time (blue) by a large margin. To achieve real-time inference (low TPOT), optimizing compute is not enough—we must minimize these expensive transfers.
Method
SMoE: Importance-Aware Scheduling with Substitution.
Expert-Cache Router
Distinguish critical vs. non-critical experts based on gating scores. Prioritize VRAM for critical ones.
Similarity-based Substitution
If a non-critical expert is missing, don't fetch it. Substitute it with a functionally similar cached expert.
CPU–GPU Pipeline
Overlap the computation of cached experts with the prefetching of missing critical experts.
Key Findings
How far can we reduce latency without hurting accuracy?
01. Does substitution significantly reduce decoding latency?
Yes, by nearly half. By substituting non-critical experts instead of loading them, SMoE avoids a massive amount of PCIe traffic. Compared to state-of-the-art offloading baselines, our method achieves up to a 48% reduction in TPOT (Time Per Output Token).
02. Why does the speedup happen?
Because PCIe overhead is cut drastically. As shown below, TPOT is strongly correlated with PCIe time. By maintaining a high cache hit rate (>60%) through substitution, we ensure that most activated experts are already on the GPU.
PCIe time directly translates to TPOT
In edge settings, expert offloading is dominated by PCIe transfer. This plot shows a strong correlation between PCIe time and TPOT, explaining why reducing transfers yields immediate decoding speedups.
High cache hit rate keeps execution on GPU
SMoE maximizes the reuse ratio of GPU-cached experts. With substitution, many low-importance activations become “soft hits”, so the GPU can execute most experts without waiting for CPU→GPU transfers.
Substitution reduces the number of experts that must be loaded
The key is selectivity: only a few activated experts are truly critical (high-score). Low-score activated experts can be substituted by functionally similar experts already in VRAM, so fewer experts need fetching and prefetching becomes lighter.
03. Which component contributes the most?
Substitution is the key driver. The ablation study below shows that while the pipeline and prefetching help, the substitution mechanism (orange bar segments) provides the most significant reduction in latency.
04. Is CPU compute a viable fallback at low batch?
Usually not. In low-batch edge serving, CPU compute can be significantly slower than GPU execution, and the system often becomes dominated by memory movement and CPU-side compute stalls. This motivates keeping as much execution on GPU as possible.
GPU vs. CPU under low-batch serving
This comparison highlights the gap between GPU (A6000) and CPU (8-core) at low batch. SMoE’s goal is to avoid CPU-side expert execution whenever possible by increasing GPU-resident “hits” (hard hits + soft hits).
05. Does SMoE help beyond decoding (prefill)?
Yes. While TPOT is the core metric for interactive decoding, prefill time also matters for end-to-end latency. We observe consistent prefill-time improvements under the same memory and offloading constraints.
Prefill time reduction
Prefill is typically more parallel than decoding, but it still suffers from expert movement and cache misses when VRAM is constrained. SMoE reduces the loading pressure so prefill can finish faster on average.
06. Does prefetching compromise accuracy?
It can be controlled. We evaluate the quality impact of prefetching policies. Combined with importance-aware selection and substitution, the system is designed to keep quality near-lossless while accelerating inference.
Quality under prefetching
Prefetching reduces stalls but risks loading the “wrong” experts when routing changes. Measuring accuracy under prefetching helps validate that the scheduling policy stays aligned with the model’s true expert usage.
