EMODY Flow: Emotion-Aware Audio-Driven Full-Body Motion Generation

EMODY Flow is a lightweight motion-generation pipeline aimed at embodied conversational agents that need speech-synced body language and facial expression, not just plausible movement. The core problem it addresses is familiar to anyone building conditional generators: when a model gets both rich audio features and a discrete emotion tag, the weaker signal can be washed out, leaving nearly identical motion across emotions.

The system attaches to a frozen Qwen-3 Omni model and reuses its Mimi audio codecs to drive two parallel DiT generators, one for SMPL-X body pose and one for FLAME facial expressions. At roughly 35 million parameters, it is small enough to be practical while still targeting full-body performance rather than isolated gesture clips.

To keep emotion conditioning from collapsing, EMODY Flow adds a training-time auxiliary emotion classifier that pushes the generated motion to remain emotion-identifiable. In practice, that produces clearer separation between emotional states, and the model reportedly transfers zero-shot to facial animation on TFHP without domain-specific fine-tuning.

On BEAT2, the system reaches a new state of the art with FGD 0.302, Beat Correlation 0.853, and Diversity 24.62, improving over the previous best by 26%, 5%, and 62% respectively. For teams working on NPC dialogue, virtual humans, or avatar systems, the practical takeaway is that emotion conditioning may need explicit pressure during training if you want it to survive alongside stronger audio cues.