MMOE: Modernizing Diffusion Transformers with Efficient Expert Design

Modern large-scale AI has increasingly leaned on sparse experts to add capacity, but diffusion-transformer models used for AIGC have mostly chased bigger parameter counts and higher sparsity ratios instead of balanced efficiency. MMOE, short for ModernMOE, tries to import the practical scaling tricks that made modern LLMs workable: routed experts, shared and lightweight experts, gate-residual routing, and attention-residual reuse.

The key point for game teams is not just that the model is larger, but that it is designed to be cheaper to train and deploy relative to its quality. Every run was done on a single eight-GPU H100 node with batch size 256 for 400k steps, which is a notably accessible setup compared with the sprawling clusters often associated with foundation-model work. Under matched training and sampling settings, MMOE reached lower FID at every recorded checkpoint, meaning it converged faster than dense and intermediate sparse baselines.

Among the sparse variants, MMOE also landed the best balance between output quality and cost. The routing analysis is interesting in its own right: expert specialization stayed stable across depth, lightweight routes were used heavily, and routing changed only modestly from step to step during denoising. That suggests the model is not just throwing experts at the problem, but organizing them in a way that remains predictable during generation.

For developers watching the AI tooling space, the practical takeaway is that diffusion models may be able to follow the same “scale, but efficiently” path that LLMs took. If this...