Seeing Speech: Learning Visible Articulatory Dynamics for Speech-Driven 3D Facial Animation

arXiv cs.GR details a new framework for speech-driven 3D facial animation that focuses on visible articulation rather than only vertex-level fit. The goal is to make mouth motion track speech in a way that looks more anatomically plausible, especially around the lips where current systems still struggle with one-to-many audio-to-motion mapping.

The method breaks visible speech into three directional articulatory motions: spreading, opening, and protrusion. A Speech--Articulatory Memory (SAM) module uses a key-value memory to retrieve and decode those motions from phonetic context, while a Topology-aware Articulatory Composition (TAC) stage combines them with mesh topology so the final facial motion stays surface-consistent.

For teams building character animation pipelines, the practical angle is better lip sync without relying purely on black-box regression from audio to vertices. That could matter for games, cinematics, virtual avatars, and any real-time or offline facial rig where believable mouth shapes are a visible quality marker.

The authors say experiments on VOCASET and TFHP beat standard reconstruction metrics and also reduce visible articulatory distance and velocity errors for lip motion. A user study reportedly preferred the results for both lip sync and realism, which is the kind of signal animation teams tend to care about when choosing between technically strong and actually convincing facial motion.