ProAct: Harnessing Streaming Motion Generation and Agentic Reasoning for Real-Time Embodied Social Interaction

ProAct tackles a problem that shows up in games, virtual agents, and robotics alike: how to keep an embodied character responsive while also making it socially aware. The framework separates the job into a low-latency Behavioral System for continuous multimodal motion and a slower Cognitive System that handles longer-horizon reasoning about context, memory, and user motivation.

For developers building NPCs, companions, or interactive avatars, the interesting part is the split between “how to move now” and “when to speak up.” The Behavioral System uses an intention-conditioned streaming flow-matching motion generator with a disentangled ControlNet branch, so deliberative intent can be turned into continuous nonverbal behavior without breaking interaction flow. That’s a useful pattern for any real-time character pipeline where animation, gesture, and conversational timing all need to stay in sync.

The Cognitive System adds an efficient memory mechanism plus a user-motivation prediction module to decide when proactive intervention is appropriate. In other words, the agent is not just reacting to the latest input; it is tracking accumulated dialogue and visual context to judge whether it should stay quiet or take the lead. That kind of trigger logic is often the difference between a believable social agent and one that feels overeager or dead inside.

ProAct has been tested on a physical humanoid robot and evaluated with real-world user studies, motion-generation benchmarks, and a new benchmark called ProActBench, aimed at proactive trigger detection and restraint in...