ITC-MoE：基于重要性的令牌感知压缩方法用于MoE扩散语言模型

ITC-MoE consists of two complementary components. First, Importance-guided Adaptive Tucker Compression (IATC)

Specifically, we identify two properties: cross-mode non-uniform redundancy, where parameter redundancy and sensitivity to rank truncation vary across the input, output, and expert modes, and token-wise utilization variation, where hot and cold tokens exhibit distinct spectral characteristics and expert activation patterns. To address these challenges, we propose ITC-MoE, an Importance-guided Tok…