SumeruAI AudioToFace: 3D Lip Sync for Unreal Engine and Unity

SumeruAI AudioToFace (ATF) is an audio-driven facial-animation solution from Sumeru AI. It sends a speech clip to the SumeruAI cloud API and plays the returned ARKit blendshape weights on a compatible 3D character in Unreal Engine or Unity. Developers can use it to add speech-synchronized lip movement and facial animation to an existing character. It does not generate a character, and it does not run the model locally.

ATF is available as an Unreal Engine plugin and a Unity package, both connected to the SumeruAI cloud API. The Unreal Engine plugin is listed on Fab; if you install from Fab, use the zip that matches your engine version. The Unity package (com.sumeruai.atf) is available on GitHub as a Package Manager Git URL, a .unitypackage, or a folder you copy into Assets.

What Does AudioToFace Do?

AudioToFace connects spoken audio to an existing character's facial rig. Instead of displaying a pre-rendered talking-head video, the integration applies ARKit blendshape weights to a character inside the engine.

A typical session starts with a WAV clip. The sample records the microphone as 16-bit PCM WAV at 16 kHz (Unreal downmixes to mono). That clip is uploaded to the SumeruAI API. After the API returns, the plugin plays audio and applies ARKit weights to the face. This is clip processing, not a live audio stream: recording stops, the request completes, then playback starts.

This makes ATF relevant to developers building character dialogue, digital-human demonstrations, and interactive 3D experiences. It provides the facial-animation part of the workflow; a complete AI character still needs any required dialogue, speech-recognition, and speech-generation systems.

Supported Platforms and Characters

These requirements are documented in the Unreal Engine integration and Unity integration. The sample path is MotionType.ARKit. FLAME and bone-driven modes exist in the runtime but are not the getting-started path. Check the current package documentation before choosing an engine version for a production project.

MetaHuman in Unreal Engine

Official MetaHuman mouths are RigLogic control curves, not SetMorphTarget("JawOpen"). Use SumeruAI → Setup ATF on Selected Actor. That adds UATFComponent, keeps the MetaHuman Face AnimBP, and inserts the SumeruAI ATF Face Sync node before RigLogic. The node writes ARKit names and CTRL_expressions_* curves onto the pose (for example JawOpen → CTRL_expressions_jawOpen) so RigLogic can deform the face.

Confirm this chain in Face AnimBP:

… → SumeruAI ATF Face Sync → RigLogic → Output Pose

If Face Sync connects straight to Output Pose with no RigLogic after it, MetaHuman mouth motion will not be correct. Leave bReplaceAnimInstance off for MetaHuman. An optional LiveLink AnimGraph node also exists; It is not required for the MetaHuman sample.

Developers provide their own MetaHuman. The public plugin does not include MetaHuman mesh, skeleton, or Epic's Face_AnimBP. Import a MetaHuman first so /Game/MetaHumans/Common/Face/Face_Archetype_Skeleton exists. This public plugin also does not include Avatar, DNA.

Custom Characters in Unity and Unreal Engine

Custom characters must already have ARKit-named shapes. In Unity the plugin matches blendshape names on each SkinnedMeshRenderer (case-insensitive substring). In Unreal it matches morph-target names, or MetaHuman CTRL_expressions_* aliases. Examples: JawOpen, MouthSmileLeft, EyeBlinkLeft. Installing ATF does not create missing shapes or rig an unprepared mesh.

A blendshape, also called a morph target, is a named deformation of the face. The plugin drives those existing shapes with animation weights; it does not create missing blendshapes or automatically rig an unprepared model.

Unity:

Unreal Engine:

How the Audio-to-Face Workflow Works

1. Prepare a compatible character. In Unreal, import your own MetaHuman or an ARKit-named face. In Unity, use the Xandra sample or a SkinnedMeshRenderer with ARKit-named blendshapes.

2. Install the engine integration. Add the Unreal Engine plugin to Plugins/, or add the Unity package (Package Manager Git URL, .unitypackage, or copy into Assets).

3. Configure API access. Obtain an Access Key and Secret Key through the Developers section of the Sumeru AI website.

4. Provide speech audio. Select a PCM WAV file or record a clip using the sample's microphone controls.

5. Process and play the result.After the offline-mesh request returns, the integration plays audio and applies ARKit weights to the character.

The sample workflow submits a recorded clip for processing. Microphone recording in this workflow is not continuous live audio streaming.

Exploring the AudioToFace Demo

The Unity repository includes an AudioToFace demonstration and sample-scene instructions. The Unreal Engine repository explains how to create the ATF demo level and configure a character.

When reviewing a demo, look at three things: whether mouth movement follows the speech, whether the facial rig responds without visible distortion, and whether the result remains synchronized through pauses and the end of the clip.

For an initial test, use a short recording with clear speech. This makes it easier to distinguish an audio-input problem from a character-mapping or playback issue.

If the face does not move, check blendshape or morph-target names before assuming the API failed. On MetaHuman, also confirm Face Sync sits before RigLogic. A second request sent while one is in flight is ignored.

Where ATF Fits in a Character Project

ATF can be evaluated as the facial-animation component for:

• Game dialogue: speech-synchronized faces for compatible NPCs.

• Digital-human prototypes: demonstrations of a character speaking in a 3D scene.

• Virtual guides and teachers: facial playback for spoken explanations.

• Interactive characters: facial animation connected to audio produced by other systems.

The result depends on the audio, character rig, mapping, and project configuration. Test representative dialogue on the actual target character before committing to a production workflow.

Frequently Asked Questions

Is SumeruAI AudioToFace a character-generation tool?

No. This integration animates compatible existing faces from speech audio. It does not replace character modeling or facial rigging.

Does ATF run entirely offline?

No. The documented integrations send audio to the SumeruAI cloud API and require network access. The term "offline" in the clip-processing endpoint does not mean that the model runs locally without an internet connection.

Can I use my own character?

Yes, provided it meets the integration's facial-rig requirements. For the ARKit workflow, check the blendshape or morph-target names before attempting playback.

Is the Unreal Engine plugin free?

As checked on September 16, 2026, the Fab listing shows the plugin as Free. Cloud API usage is separate and follows the terms of the developer's SumeruAI account.

Does it stream from the microphone in real time?

No. The sample records a clip, then PlayFromAudio uploads that clip. This AudioToFace path is not a live streaming lip-sync pipeline.

What audio format does the API take?

The request sends WAV bytes. The sample recorder is 16 kHz, 16-bit PCM. Unreal's recorder downmixes to mono. Returned audio may be WAV, MP3, or PCM for playback; That is not the upload format.

Which platforms are supported?

The Unreal plugin modules are Win64-only. Unity 2020.3 LTS or later is documented; The sample runs on Built-in, URP, and HDRP. Playback decoding uses NAudio.

Does using ATF make a character conversational?

Not by itself. ATF handles audio-driven facial animation. Conversation logic, speech recognition, and voice generation are separate parts of an interactive-character application.

Get Started with SumeruAI AudioToFace

Choose the integration that matches your project:

• Get AudioToFace For UE on Fab.

• Read the Unreal Engine setup documentation.

• Download the Unity SDK and read its setup guide.

• Visit Sumeru AI for developer access.

Start with a sample scene and one short audio clip, then test your own character in the same configuration. In Unreal, run Setup ATF and confirm Face Sync comes before RigLogic.