VOCA
Capture, Learning, and Synthesis of 3D Speaking Styles
VOCA is a speech-driven 3D facial animation method that animates unseen adult faces from arbitrary speech. It factors identity from motion using a new 4D dataset to capture speaking styles and supports animator control over shape and pose. It enables realistic talking avatars without retargeting.
Demos
This demo highlights VOCA's capability to generate realistic 3D facial animations driven by speech. Observe the natural lip and jaw movements synchronized with audio and the range of speaking styles and identities animated without subject-specific retargeting, showcasing VOCA's versatility in capturing different speaking styles for unseen subjects and languages.
Links
Paper & demos
Impact
Abstract
Audio-driven 3D facial animation has been widely explored, but achieving realistic, human-like performance is still unsolved. This is due to the lack of available 3D datasets, models, and standard evaluation metrics. To address this, we introduce a unique 4D face dataset with about 29 minutes of 4D scans captured at 60 fps and synchronized audio from 12 speakers. We then train a neural network on our dataset that factors identity from facial motion. The learned model, VOCA (Voice Operated Character Animation) takes any speech signal as input - even speech in languages other than English - and realistically animates a wide range of adult faces. Conditioning on subject labels during training allows the model to learn a variety of realistic speaking styles. VOCA also provides animator controls to alter speaking style, identity-dependent facial shape, and pose (i.e. head, jaw, and eyeball rotations) during animation. To our knowledge, VOCA is the only realistic 3D facial animation model that is readily applicable to unseen subjects without retargeting. This makes VOCA suitable for tasks like in-game video, virtual reality avatars, or any scenario in which the speaker, speech, or language is not known in advance. We make the dataset and model available for research purposes at http://voca.is.tue.mpg.de.
Overview and problem setting
This paper addresses audio-driven 3D facial animation for realistic talking-head synthesis, with an emphasis on generalization across speakers, unseen face shapes, and different speech sources. The central claim is that prior methods were limited by scarce 3D audio-visual data, speaker-specific modeling, and weak standard evaluation protocols. The authors propose two coupled contributions: a new multi-subject 4D speech-face dataset, VOCASET, and a speech-driven animation model, VOCA (Voice Operated Character Animation), that factors identity from motion and uses subject conditioning to capture multiple speaking styles.
The intended output is a full 3D facial animation from an arbitrary speech clip and a static 3D template mesh. The model is designed to work with faces not seen during training, without the retargeting or subject-specific reconstruction requirements common in earlier systems. The paper repeatedly emphasizes that the system is subject-independent in the sense that it can animate new adult face shapes from audio alone, while remaining compatible with a parametric head model for additional control over identity-dependent shape and pose.
Key contributions
- A new 4D dataset of synchronized audio and high-resolution head scans, collected from 12 speakers, with 480 sequences total and about 29 minutes of capture.
- A compact neural architecture that maps speech features to per-vertex 3D displacements while conditioning on subject identity labels during training.
- A training objective that combines a per-frame vertex reconstruction loss with a temporal velocity loss to encourage smooth motion.
- A perceptual evaluation protocol based on blind Amazon Mechanical Turk studies, motivated by the paper’s argument that standard numeric prediction errors are not meaningful for speech-to-face animation.
- Empirical evidence that the model generalizes across unseen identities, can be driven by non-English speech, is robust to noise to a degree, and supports style interpolation and animator controls.
VOCASET: capture setup, scope, and preprocessing
The dataset consists of audio-4D scan pairs from 6 female and 6 male subjects. For each subject, the authors recorded 40 English sentences, each roughly 3 to 5 seconds long. The sentence pool was deliberately chosen to maximize phonetic diversity using standard speech-corpus selection ideas. The recorded texts include 27 TIMIT sentences, 3 pangrams previously used in related 3D face work, and 10 SQuAD questions. Across subjects, the dataset contains a mixture of shared and unique utterances: five sentences are shared by all subjects, 15 are shared by three to five subjects, and 20 are spoken by one or two subjects only. The paper reports 255 unique sentences in total.
The capture system is a multi-camera active stereo platform from 3dMD with six stereo pairs, six color cameras, five speckle projectors, and six white-light LED panels. It captures 3D meshes at 60 fps, each with about 120K vertices before processing. Audio is synchronized with the scanner and recorded at 22 kHz. Color images are used to generate UV texture maps for each scan, though the model itself is driven by audio and 3D geometry rather than appearance.
To place all scans into correspondence, the raw meshes are registered using sequential alignment with the public FLAME head model. The paper states that an image-based landmark predictor is used to improve robustness during fast motions. After alignment, each mesh has 5023 vertices. The authors report a very small mismatch between aligned scans and the FLAME surface: median distance 0.09 mm, mean 0.13 mm, and standard deviation 0.14 mm, which they interpret as evidence that the alignment faithfully captures the raw data.
The aligned meshes are then unposed to remove global rotation, translation, and neck rotation, producing a canonical "zero pose." The neck boundary and ears are automatically fixed, and the eye region is smoothed with Gaussian filtering to suppress capture noise, while the mouth region is intentionally left unsmoothed so that subtle speaking motions are preserved.
Modeling assumptions and design goals
The paper’s core modeling decision is to factor identity from facial motion. This matters because facial shape and speaking style are entangled in raw scan data: different people have different jaw proportions, lip contours, and characteristic articulation habits. VOCA aims to learn the motion component from speech while using a separate subject label to condition the model during training. The design also leverages a parametric head model so that the animation output lives in a shared canonical geometry and can be re-shaped or re-posed after prediction.
The system therefore tries to solve two coupled generalization problems: generalization across audio variation such as noise, accent, and recording conditions, and generalization across face variation such as identity-dependent shape and style. The authors use DeepSpeech features for the first problem and FLAME-style shape/pose control for the second.
VOCA architecture
VOCA is an encoder-decoder network. Its input is a subject-specific template mesh and a speech-derived feature window. The output is a dense set of vertex displacements in the FLAME zero-pose space, which are then added to the template to produce the final animated mesh.
Speech is first processed by a pre-trained DeepSpeech model. The network consumes unnormalized log probabilities over characters produced at 50 frames per second. These are linearly interpolated to 60 fps to match the scan rate. To incorporate short-range temporal context, the features are grouped into overlapping windows of size $W \times D$, where the paper uses $W=16$ and $D=29$. The windowed features have shape $60T \times W \times D$ for an audio clip of length $T$ seconds.
During training, the speech features are concatenated with an 8-dimensional one-hot subject code for the eight training identities. This code is concatenated both to each speech feature vector and to the output of the final convolution layer. The idea is to let the encoder learn subject-specific speaking styles while still sharing most parameters across subjects.
The encoder contains four 1D convolutional layers followed by two fully connected layers. Each convolution uses a kernel of size $3 \times 1$ and stride $2 \times 1$, operating over time rather than space because the DeepSpeech channels do not have a natural spatial structure. The first two convolution layers use 32 filters each, and the last two use 64 filters each. After the final convolution and identity concatenation, the network uses a fully connected layer with 128 units and $\tanh$ activation, then a 50-dimensional linear bottleneck.
The decoder is a single fully connected layer with linear activation that outputs a $5023 \times 3$ vector of vertex displacements. The weights of this layer are initialized with 50 PCA components computed from the training set’s vertex displacements, and the bias is initialized to zero. The paper highlights this linear decoder because it makes style interpolation easy: convex combinations in the 50-dimensional latent space correspond directly to convex combinations in vertex space.
In simplified notation, the model predicts $$ \mathbf{f}_i = g_\theta(\mathbf{x}_i, \mathbf{s}), $$ where $\mathbf{x}_i$ is a speech window centered at frame $i$, $\mathbf{s}$ is the subject label, and $\mathbf{f}_i$ is the predicted 3D mesh. The output is produced in FLAME’s zero pose, after which FLAME can add identity shape, expression, and pose components.
Pretrained speech features and the role of FLAME
The authors use a fixed, pretrained DeepSpeech front end as a robust speech representation. They explicitly frame this as a way to improve invariance to audio source, noise, recording artifacts, and language. The paper also notes that the TensorFlow DeepSpeech implementation differs from the original description in that the recurrent part uses LSTM cells and 26 MFCC features are used in the front end.
FLAME is used as the geometric backbone because it offers a common parametric space for identity, expression, and pose. In VOCA’s pipeline, this enables not only face animation but also post-hoc editing of identity-dependent shape and head pose. The paper describes control over head, jaw, and eyeball rotations, as well as over shape blendshapes and expression blendweights.
Training objective and optimization
Training is performed on pairs of speech windows and aligned 3D meshes. The model predicts a mesh $\mathbf{f}_i$ for each training frame $i$, and the loss combines a positional term and a velocity term: $$ E_p = \lVert \mathbf{y}_i - \mathbf{f}_i \rVert_F^2, \qquad E_v = \left\lVert (\mathbf{y}_i - \mathbf{y}_{i-1}) - (\mathbf{f}_i - \mathbf{f}_{i-1}) \right\rVert_F^2. $$ The position term encourages the output to match the ground-truth geometry, while the velocity term compares backward finite differences and promotes temporal stability. The paper uses weights of 1.0 for $E_p$ and 10.0 for $E_v$.
The optimization details are fairly modest: the network is trained for 50 epochs with a constant learning rate of $10^{-4}$, batch normalization, and batch size 64. Training uses the Adam optimizer, and one epoch takes about 10 minutes on a single NVIDIA Tesla K20. The pretrained DeepSpeech model is kept fixed during training.
Data splits and evaluation protocol
The dataset is split into 8 training subjects, 2 validation subjects, and 2 test subjects. The training set includes all 40 sequences for each of the eight training subjects, for a total of 320 training sequences. Validation and test each use 40 sequences from the held-out subjects, but only from the 20 sentence types that are not shared with any other subject. The paper emphasizes that the train/validation/test split is fully disjoint in both subjects and sentences.
The authors argue that standard metrics such as vertex $L_2$ error are not suitable for this task because the mapping from speech to facial motion is inherently many-to-many: multiple facial motions can plausibly correspond to the same utterance. Consequently, the paper evaluates VOCA primarily through perceptual studies and qualitative results.
Perceptual evaluation on Amazon Mechanical Turk
The paper reports three blind AMT studies, all performed on sequences and subjects fully disjoint from the training and validation data. Turkers first pass a qualification test designed to weed out poor responses; the authors describe this as a simplified version of the later style-comparison tasks with obvious correct answers.
1. Comparison to recorded performance
In the first study, Turkers compare a held-out recorded and processed test sequence to VOCA’s prediction conditioned on all training-speaker styles. Across 400 HITs, the recorded performance is judged more natural in $83 \pm 9\%$ of responses, while VOCA is preferred in $17 \pm 9\%$. The authors interpret this gap as evidence that VOCA produces realistic motion but cannot reproduce the subject-specific idiosyncrasies that make a real recording look more natural.
2. Speech feature ablation: DeepSpeech versus fbank
The second study replaces DeepSpeech features with Mel-filterbank energy features and retrains for 50 epochs under the same settings. Across 400 HITs, VOCA with DeepSpeech features is preferred as more natural in $78 \pm 16\%$ of comparisons, versus $22 \pm 16\%$ for the fbank variant. The paper uses this result to support the claim that the DeepSpeech front end improves generalization to unseen audio.
3. Style comparisons
The third study tries to isolate speaking style from identity-dependent shape. The authors subtract each subject’s personalized neutral template to obtain displacements, then add those displacements to a single common FLAME template. For each reference sequence, they compare two VOCA predictions driven by test audio: one conditioned on the reference subject, and one on a randomly chosen different subject. Turkers choose which prediction is more similar in speaking style to the reference. The experiment is run twice: once when the reference and prediction share the same sentence, and once when they differ.
The paper’s main qualitative conclusion is that style can often be recognized, but not consistently across all conditions; the role of sentence content is not significant for most conditions. The figure below reproduces the reported percentages of times the reference condition is chosen.
| Condition | Same sentence | Different sentence |
|---|---|---|
| 1 | 25 ± 13.4% | 45 ± 8.9% |
| 2 | 48 ± 12.6% | 34 ± 9.8% |
| 3 | 87 ± 8.6% | 92 ± 4.7% |
| 4 | 68 ± 14.5% | 67 ± 8.4% |
| 5 | 80 ± 10.1% | 76 ± 7.7% |
| 6 | 48 ± 12.6% | 26 ± 7.8% |
| 7 | 55 ± 12.6% | 55 ± 10.0% |
| 8 | 15 ± 9.0% | 14 ± 6.9% |
Qualitative findings
Generalization across face shapes
To demonstrate subject-independent animation, the authors select neutral scans from BU-3DFE, align and pose-normalize them, and drive them with the same speech input. The resulting animations show that VOCA can animate a range of adult face shapes not seen during training.
Generalization across languages and audio conditions
The paper states that VOCA also works on speech in languages other than English, indicating that the DeepSpeech feature representation helps transfer the model beyond the training language. In addition, a noise robustness test adds street-noise recordings at multiple levels: $-36$ dB, $-24$ dB, $-18$ dB, and $-12$ dB. The authors report that only the highest noise level noticeably dampens the motion, while the animations remain plausible even under substantial corruption.
Speaking style control and interpolation
Because the model is conditioned on subject labels during training, changing the condition at inference changes the speaking style. The paper uses lip opening as an illustrative metric and reports visibly different mouth amplitudes across conditions. More importantly, it shows that convex combinations of subject conditions generate intermediate styles. Due to the linear decoder, combining conditions in the latent 50-dimensional space is equivalent to combining the resulting 3D displacements. This yields a simple form of controllable style interpolation that could help approximate the speaking style of an unseen person.
Shape and pose control
The model can be combined with FLAME’s shape and pose parameters to manipulate identity-dependent facial shape and head orientation during animation. The paper demonstrates changing the first identity shape components by plus or minus two standard deviations and changing head pose by plus or minus 30 degrees, all while preserving plausible speech-driven motion.
Comparison to prior state of the art
The paper qualitatively compares VOCA to the subject-specific audio-driven model of Karras et al. VOCA is reported to reproduce similar facial animation without requiring any subject-specific training data, although the authors acknowledge that the Karras et al. model produces more detailed and more natural results on the original actor and especially in the upper face, where emotion-driven cues matter more.
Interpretation and limitations
The discussion is explicit about what VOCA does not model well. Upper-face motion such as eye and eyebrow movement is only weakly correlated with speech, so the model does not learn those motions strongly; the authors attribute this mainly to the absence of emotional speech in their capture protocol. Likewise, head motion is a weakly speech-correlated signal and is not inferred from audio; instead, the system provides manual controls for pose. The paper also notes that richer conversational behavior would require additional modeling, such as explicit head-motion prediction, attention to spatial cues like eye gaze, and expressive full-body animation.
A second limitation is style coverage. Conditioning on subject labels enables multiple styles, but the diversity is limited by the number of speakers in VOCASET. The authors explicitly identify expanding the subject pool, better controlling or mixing styles, and disentangling shape from motion more thoroughly as future work. In other words, VOCA is a strong step toward realistic multi-subject speech animation, but not a full conversation model.
Takeaway
The paper’s main technical message is that realistic speech-driven 3D facial animation becomes more practical when three ingredients are combined: high-quality aligned 4D supervision, speech features that are robust to audio variation, and a model that explicitly separates identity from motion. Within that framework, VOCA achieves subject-independent animation for unseen adult faces, supports style manipulation and head/shape control through FLAME, and is evaluated with human perceptual studies rather than misleading numeric reconstruction metrics.