ITC-MoE: 重要性に基づくトークン-awareなMoE拡散言語モデルの圧縮

ITC-MoE consists of two complementary components. First, Importance-guided Adaptive Tucker Compression (IATC)

Specifically, we identify two properties: cross-mode non-uniform redundancy, where parameter redundancy and sensitivity to rank truncation vary across the input, output, and expert modes, and token-wise utilization variation, where hot and cold tokens exhibit distinct spectral characteristics and expert activation patterns. To address these challenges, we propose ITC-MoE, an Importance-guided Tok…