Audio2Face-3D

Audio2Face-3D: Audio-driven Realistic Facial Animation For Digital Avatars

Audio2Face-3D is a real-time system for animating digital avatars' faces from audio and emotion inputs, capturing lip sync, facial skin, tongue, jaw, and eye movements. It offers both lightweight and high-quality models with retargeting to production rigs, making it ideal for interactive game characters and avatars.

• talking-head
• audio-driven
• face-animation
• avatar
• blendshape
• realtime
• streaming
• emotion

Authors: NVIDIA, :, Chaeyeon Chung, Ilya Fedorov, Michael Huang, Aleksey Karmanov, Dmitry Korobchenko, Roger Ribera, Yeongho Seol

Categories: cs.GR, cs.HC, cs.LG, cs.SD, eess.AS

Published 2025-08-22 · Updated 2025-08-22

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub199 stars2 watchers41 forksMIT
• Hugging Face81 downloads18 likes

Impact

• Semantic Scholar10 citations

Abstract

Audio-driven facial animation presents an effective solution for animating digital avatars. In this paper, we detail the technical aspects of NVIDIA Audio2Face-3D, including data acquisition, network architecture, retargeting methodology, evaluation metrics, and use cases. Audio2Face-3D system enables real-time interaction between human users and interactive avatars, facilitating facial animation authoring for game characters. To assist digital avatar creators and game developers in generating realistic facial animations, we have open-sourced Audio2Face-3D networks, SDK, training framework, and example dataset.

---

1. Problem Setting and System Goals

Audio2Face-3D is an audio-driven facial animation system for digital avatars. The paper’s goal is not only to synthesize plausible lip sync, but to produce a practical end-to-end pipeline that can drive facial skin, tongue, jaw, and eyeballs in real time, then retarget that motion to production rigs such as ARKit-compatible blendshape sets. The system is explicitly positioned for game characters and interactive avatars, where latency, concurrency, and retargetability matter as much as raw visual quality.

The paper presents two main inference networks trained on the same captured data but optimized for different deployment regimes: Audio2Face-3D-v2.3, a lighter regression model for single identities, and Audio2Face-3D-v3.0, a diffusion-based multi-identity model that generally yields higher-quality and more expressive motion. Both accept audio and emotion conditioning; the diffusion model additionally conditions on identity and produces longer animation blocks. The paper also includes a retargeting solver, an optional emotion predictor called Audio2Emotion, and several experimental extensions beyond the open-source core.

2. System Overview

The overall pipeline is: audio and emotion conditioning enter the network, the network predicts motion deltas for facial components, optional post-processing adjusts the result, and a blendshape solver converts vertex-space motion into rig parameters. The paper emphasizes that this design serves two use cases simultaneously: real-time interactive avatars and offline authoring for game assets.

The regression model uses a small audio window of about 0.52 seconds and emits a single-frame pose in a compressed representation. The diffusion model uses a 1-second audio chunk and outputs a 30-frame raw-vertex animation block, with a streaming mode that advances with a 0.5-second stride. The paper explicitly notes the trade-off: the regression system is lighter and scales to many concurrent tracks, while the diffusion system is more expressive.

Key outputs and retargeting path

• Facial skin deformation.
• Tongue motion.
• Jaw motion.
• Eyeball rotations.
• Optional conversion to ARKit-style blendshape weights for downstream rigs.

3. Data Capture and Representation

The paper’s core training corpus is a proprietary 4D facial capture dataset collected from professional actors using synchronized speech audio and multi-view facial capture. The actors performed roughly 50 to 70 target sentences each, with utterances spanning about 3 to 15 seconds, and the dataset covers 11 emotional states: neutral, amazement, anger, cheekiness, disgust, fear, grief, joy, out-of-breath, pain, and sadness. The authors state that emotionally inconsistent onset/offset regions were clipped so that training uses only the stable segments.

The training target is not raw video or a single mesh parameterization, but a structured set of component-wise representations:

The skin representation is a 140-dimensional PCA embedding of vertex displacements from the neutral mesh, except for the v2.3 Mark network, which uses 272 dimensions. Tongue motion is manually authored for one subject by a professional animator and then propagated to other subjects by training a temporary network and applying an affine adaptation to match target tongue shape; the final tongue PCA representation uses 10 dimensions. Jaw motion is represented by the 3D displacement of five stable mesh points, giving 15 dimensions. Eyeball motion is represented as two Euler angles for each eye, giving 4 dimensions.

Training-data augmentation

To expand the diversity of the paired audio/4D data, the paper uses three augmentation strategies. Voice conversion changes speaker identity while preserving timing, so the original animation stays synchronized. Text-to-speech synthesis generates new audio from transcripts, followed by phoneme detection and temporal alignment using dynamic time warping to keep motion aligned. Silence augmentation adds 4 seconds of silent audio per emotion state so the model learns stable idle behavior for pauses and clip boundaries.

4. Audio2Face-3D-v2.3: Regression-Based Network

The regression model follows the earlier audio-to-face paradigm but extends it in several important ways. Its function can be summarized as $f_\theta(\mathbf{A}, \mathbf{e}) \rightarrow \mathbf{x}$, where $\mathbf{A}$ is audio, $\mathbf{e}$ is emotion conditioning, and $\mathbf{x}$ contains skin, tongue, jaw, and eye outputs. The paper emphasizes that this model is designed for real-time streaming and low memory usage.

Architecture

The audio encoder is a hybrid module combining autocorrelation features with frozen Wav2Vec 2.0 features. The authors state that autocorrelation still helps extract pitch and volume cues, which matters for singing and non-verbal audio, while Wav2Vec 2.0 improves lip-sync quality and multilingual robustness. The paper uses the audio features from the 4th transformer layer of Wav2Vec 2.0.

A phoneme prediction head is trained jointly as an auxiliary task. Ground-truth phoneme probabilities are produced at 50 fps using a textless phoneme aligner, and the head is dropped after training because its main role is to push phoneme-sensitive information into the encoder. The authors note that this improves bilabial articulation, especially for mouth-closure sounds.

The decoder takes the encoded audio together with explicit one-hot emotion and an implicit learned emotion vector. The implicit emotion is intended to capture frame-to-frame variation not explained by the audio or the explicit emotion label.

Loss function

The regression objective combines reconstruction, temporal, phoneme, stability, and lip-shape terms. In compact form, the total loss is:

The key terms are: mean-squared reconstruction of the full-face coefficients; velocity matching between predicted and target sequences; cross-entropy phoneme prediction; temporal smoothness on phoneme probabilities; a volume-aware stability term that applies stronger smoothing at low audio volume; an implicit-emotion smoothness regularizer; and two lip-specific terms that constrain lip separation and lip thickness so that bilabials and mouth aperture look anatomically plausible.

The paper gives identity-specific loss weights. For example, James uses $\alpha_{\text{mse}}=1.0$, $\alpha_{\text{motion}}=10.0$, $\alpha_{\text{phoneme}}=0.1$, $\alpha_{\text{phoneme\_motion}}=0.1$, $\alpha_{\text{vol\_stab}}=100.0$, $\alpha_{\text{emo}}=1.0$, $\alpha_{\text{lip\_dist}}=50.0$, and $\alpha_{\text{lip\_size}}=0.01$. Claire and Mark use different lip and phoneme weights, reflecting subject-specific tuning.

Training and inference

Training uses Adam with initial learning rate $2 \times 10^{-4}$ and step decay with factor $0.994$ per epoch. The model is trained on an NVIDIA RTX A6000 for 50 epochs for Claire and Mark and 70 epochs for James, with batch size 32. Inference uses a sliding window; each input buffer is 8320 samples at 16 kHz, and the output is a 169-dimensional vector composed of $140 + 10 + 15 + 4$ coefficients. The final skin and tongue PCA coefficients are inverse-transformed back to geometry parameters.

Eye-closure filtering removes training samples with closed eyes to prevent unstable eyelid motion. The paper also notes that training samples are organized as adjacent-frame temporal pairs so temporal losses can be evaluated efficiently.

5. Audio2Face-3D-v3.0: Diffusion-Based Multi-Identity Network

The diffusion model is the paper’s higher-capacity architecture. It predicts denoised facial motion blocks conditioned on audio, timestep, emotion, and identity. The model is summarized as $\hat{\mathbf{X}}_0 = f_\theta(\mathbf{A}, \mathbf{X}_t, t, \mathbf{E}, \mathbf{i})$, where $\mathbf{X}_t$ is noisy facial motion, $t$ is the diffusion timestep, $\mathbf{E}$ is per-frame emotion, and $\mathbf{i}$ is a one-hot identity vector.

Architecture

Audio is encoded using HuBERT. The HuBERT features, noisy animation, diffusion timestep, emotion, and identity each pass through separate projection layers, are concatenated, and then feed GRU layers before decoding. The GRU is the temporal backbone, and the network predicts the denoised full animation directly rather than predicting noise, following the paper’s chosen denoising objective.

Because the model supports multiple identities, the paper also reports a later experimental extension using a mixture-of-experts layer in the HuBERT encoder, where identity-based routing selects among experts to reduce cross-identity coupling.

Loss function

The diffusion objective is a simple denoising reconstruction loss:

with conditioning $c = \{\mathbf{A}, \mathbf{E}, \mathbf{i}\}$. The model adds a lip-distance term for better bilabial closure and an upper-face regularizer to reduce jitter in the eyebrow and eye region:

Training uses Adam with learning rate $5 \times 10^{-5}$, cosine annealing down to $10^{-5}$, and a 10-epoch warmup. The model is trained for 400 epochs on an RTX A6000 with batch size 1 and gradient accumulation over 4 steps. The diffusion process uses 1000 steps and a cosine noise schedule. The loss weights are $\alpha_{\text{upper\_reg}}=25$ and $\alpha_{\text{lip\_dist}}=10^{-5}$.

The authors use variable-length audio segments during training, ranging from 30 to 600 frames, plus random pitch shifting. They explicitly state that this helped streaming inference. At test time, the network can run offline on the full utterance or in a streaming mode using a 1-second window and a 0.5-second stride; only two diffusion steps are said to be sufficient for high-quality output in streaming inference.

6. Post-Processing and Rig Retargeting

The network outputs are not the end product. The paper includes a substantial post-processing stage that lets users tune motion range, smoothing, and anatomical offsets after inference. This is important for production use because it allows artists or integrators to adapt the same network output to different characters and animation preferences without retraining.

Post-processing controls

• Skin controls: overall strength, upper/lower-face strength, smoothing, and mask softness.
• Jaw and tongue controls: lip-open offset, jaw strength/height/depth, and tongue strength/height/depth.
• Eye controls: eyelid offset, blink strength, emotion offset strength, saccade strength, and eye rotation offsets.

The paper describes this as a way to refine the inference output in a Maya-ACE plugin, making the system more usable in a content-creation workflow.

Blendshape solve

Because the networks predict vertex-space motion tied to a specific identity, the paper provides a blendshape solver to transfer motion to arbitrary target rigs. The solver uses a delta-blendshape formulation:

where $\mathbf{v}_0$ is the neutral mesh, $\mathbf{D}$ stacks the delta blendshape basis, and $\mathbf{w} \in [0,1]^N$ are the blendshape weights. The solver fits the target displacement in delta space with regularization:

The paper describes the $L_1$ term as a quadratic proxy to promote sparse activations, and the temporal term as a way to suppress jitter. The solver also supports active-subset constraints, canceling pose pairs, and symmetric pose regularization $R_{\text{Sym}}(\mathbf{w}) = \|\mathbf{S}\mathbf{w}\|_2^2$.

For ARKit compatibility, the authors refine generic ARKit shapes through automatic transfer, targeted personalization using range-of-motion captures and calibration audio, and manual refinement. This makes the solver more usable for external characters and production pipelines.

7. Experimental Features and Extensions

The paper has a dedicated section of experimental features that extend the current open-source distribution. These are framed as experiments or previews rather than the core shipped functionality.

7.1 Text-driven emotion and facial movement control

The authors replace the one-hot emotion vector with a CLIP text embedding and train the model with text prompts that describe emotion and facial movement. Prompt generation uses GPT-4o to create natural-language emotion descriptions and manual translation of ARKit blendshape names into short motion descriptions. The final prompt template is of the form: “a person {emotion} and speaks with {facial movement}.” They also add intensity adverbs when a derived intensity score passes a threshold.

The paper reports several qualitative capabilities: control over known emotions, generalization to an unseen emotion such as lonely, facial movement control, intensity control, and smooth emotion transitions by interpolating between two CLIP embeddings as $\mathbf{c} = \alpha \mathbf{c}_1 + (1-\alpha)\mathbf{c}_2$.

7.2 Head motion generation

The diffusion model is extended to predict head motion alongside facial motion. Head motion is parameterized as a matrix $\mathbf{H} \in \mathbb{R}^{N \times 9}$, where each frame contains 6 degrees of freedom for two head joints and 3 degrees of root translation. To suppress jitter, the paper adds an acceleration loss:

Because head-motion supervision is available only for a few identities, the paper uses a special multi-identity training strategy: identities without head-motion ground truth omit head-related losses, while identities with head-motion data are sometimes replaced by identities without such data and trained using only head-related losses. The authors also add idle-motion sequences so the model can produce more natural behavior during silence.

7.3 Direct prediction of facial rig parameters

The paper explores a simpler model that directly predicts rig parameters from audio, bypassing intermediate vertex geometry. This rig-driven system uses an identity-agnostic generic rig, blendshape weights extracted from 4D sequences, and a post-processing pipeline that corrects semantic mismatches and jitter in the solved weights. The architecture is diffusion-like but simplified: noisy rig parameters are concatenated with audio, emotion, identity, and timestep features, then processed by a GRU and decoded linearly. A mixture-of-experts layer is also used inside the HuBERT encoder to reduce coupling between identities.

7.4 Jaw-driven blendshape solver

To fix cases where jaw position is geometrically ambiguous, the paper adds a soft jaw constraint to the blendshape solver. The jaw displacement target is reconstructed from jaw-specific blendshape deltas in a matrix $\mathbf{C}$, and the regularizer is:

The constraint weight is dynamically scaled using jaw displacement magnitude and lip separation, so jaw correction is strong when the jaw should be visually important but relaxed when large articulated motion would otherwise make the fit too rigid. The paper argues this improves anatomical correctness in speech-driven expressions such as closed-mouth visemes.

8. Results and Benchmarks

The authors present both qualitative and system-level quantitative results. They emphasize that the project is not primarily about a single benchmark score: the reported metrics are mainly sanity checks, and final judgments rely heavily on expert inspection of realism and expressiveness.

The benchmark includes four metrics: SyncNet for lip-sync accuracy; a jitter metric based on Fourier energy and Fréchet distance in mouth-region motion; a bilabial sound score that measures whether /M/, /B/, and /P/ closures are achieved; and an expressiveness score based on neutralized FACEM features. These metrics are used to characterize different aspects of the generated motion, not to replace human review.

Qualitative comparisons

The paper shows that both v2.3 and v3.0 produce natural lip sync on the same audio, with v3.0 supporting multiple identities via one-hot identity conditioning. It also shows emotion-controlled outputs and smooth transitions between emotions, including interpolation from joy to anger. In the blendshape solve example, the retargeted blendshape animation visually matches the raw output closely enough to preserve motion semantics while enabling use on a different character.

Computation time and memory

The paper reports inference speed on a GeForce RTX 4090 using PyTorch, ONNX, and TensorRT. The v2.3-Claire model reaches 194 FPS in PyTorch, 252 FPS in ONNX, and 453 FPS in TensorRT for a single track. The v3.0 model is much faster because it processes larger blocks efficiently: 2069 FPS in PyTorch, 1803 FPS in ONNX, and 3269 FPS in TensorRT for a single track. Under TensorRT, v3.0 still remains at 1250 FPS with 8 concurrent tracks.

Memory consumption is also reported. v2.3 in TensorRT uses 0.6 GB for 8 concurrent tracks and scales to 1.3 GB for 128 tracks. v3.0 uses 1.4 GB for 1 track and 4.0 GB for 8 tracks. The paper attributes the memory difference to the output format: v2.3 emits a single-frame PCA-compressed output, while v3.0 emits 30-frame raw-vertex blocks.

Qualitative use cases

The paper highlights two main deployment settings: games, where large volumes of facial animation are needed; and interactive avatars, where speech comes from an upstream LLM/TTS system and facial motion must be generated in real time for conversation. The authors explicitly state that the system is suitable for customer-service and game-character avatars.

9. Audio2Emotion Appendix

The appendix introduces Audio2Emotion, a speech-emotion classifier used to automatically supply emotion trajectories to Audio2Face-3D. The paper motivates it as a way to avoid manual emotion sliders and to capture emotional nuance over time in speech.

Audio2Emotion is implemented as a six-class classifier over anger, disgust, fear, joy, neutral, and sadness. Because continuous emotion annotations are not available, the model is run in a sliding-window fashion over the audio. The default window is about 1.9 seconds with a 0.5-second stride, and each window is further subdivided into overlapping 0.625-second sub-windows whose class probabilities are averaged to smooth overconfident spikes. For offline audio, the resulting emotion keyframes are linearly interpolated; for streaming audio, exponential smoothing is used.

The classifier is fine-tuned from facebook/wav2vec2-large-lv60. The CNN feature extractor is frozen, while the 24-layer Transformer and classifier head are trained end-to-end. Inputs are resampled to 16 kHz, truncated to at most 10 seconds, and Z-normalized. Training uses AdamW with a learning rate that decays linearly from $5 \times 10^{-4}$ to $5 \times 10^{-5}$ over 20 epochs, batch size 16.

Training data and evaluation

The training set combines all corpora that satisfy the paper’s label, language, and licensing constraints. The authors use five public datasets plus two additional sets: a synthetic A2E-OpenAI-TTS set and an internal private training set. Evaluation is done on three public datasets and a private held-out set, with speaker-independent splits to avoid overly optimistic results.

The paper’s reported accuracy numbers are:

The authors conclude that the personalized model performs best overall. Among the generic checkpoints, v2.2 does better on the public benchmarks, while v3.0 does better on the private test set and produces more calibrated frame-wise predictions for animation.

They also report two personalization strategies: fine-tuning on a small labeled speaker-specific set, and a lighter method that uses a single neutral utterance from the target speaker. In one experiment, fine-tuning on 12 clips per speaker improved accuracy for 19 of 20 speakers, with an average gain of about 10%.

10. Related Work Positioning

The paper places itself in the audio-driven 3D facial animation literature rather than the 2D talking-head literature. Its closest 3D predecessors include early lower-face systems such as Taylor et al. and Visemenet, the real-time CNN approach of Karras et al., multi-identity methods like VOCA and FaceFormer, and more recent diffusion-based models such as FaceDiffuser, FaceTalk, and DiffPoseTalk. The distinguishing combination here is: full-face output, emotion conditioning, real-time streaming support, and a production retargeting pipeline.

The paper also notes related text-conditioned emotion control work, but its contribution is to incorporate this idea into a 3D production system rather than presenting it as a standalone text-to-expression model.

11. Limitations and Practical Boundaries

The authors are explicit about the system’s limitations. First, severe background noise or strong non-verbal sounds can hurt lip-motion accuracy. Second, lip shapes outside the training distribution may fail. Third, upper-face and eyeball motion remain weakly semantic: because the model sees only a short audio window, it cannot reliably infer conversational intent, listening behavior, or context-driven expressions. Finally, the system cannot generate natural idle or listening motion from long silence or from a conversation partner’s speech, which limits realism in dyadic interaction scenarios.

12. Takeaway

The paper’s main contribution is a practical, production-oriented facial animation stack that spans capture, training data preparation, audio-to-motion inference, post-processing, blendshape retargeting, and emotion estimation. The regression network offers efficient real-time deployment, while the diffusion network improves fidelity and supports multi-identity streaming. The experimental features extend the system toward text-conditioned expression, head motion, direct rig generation, and jaw-aware solving, but the paper is careful to separate these exploratory directions from the released core system.