SICAGE

SICAGE: Speaker-Independent Culture-Aware Gesture Generation using TED4C-L Dataset

SICAGE generates culturally appropriate co-speech gestures by learning speaker-independent cultural representations through domain generalization. It avoids confounding speaker identity with culture and pairs a real-time diffusion generator with TED4C-L, a new 106-hour multimodal dataset spanning four cultures.

• gesture
• co-speech
• speech-driven
• multimodal
• audio-driven
• talking-human
• realtime

Authors: Ariel Gjaci, Antonio Sgorbissa, Vittorio Murino

Categories: cs.CV, cs.GR, cs.HC, cs.SD

Comment: Accepted at ECCV 2026

Published 2026-06-29 · Updated 2026-06-29

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub0 stars0 watchers0 forks
• Hugging Face110 downloads0 likes

Impact

• Semantic Scholar0 citations

Abstract

Recent co-speech gesture generation methods often overlook cultural differences, limiting their effectiveness in human-agent interaction. Moreover, culture-conditioned models are rarely evaluated under speaker-disjoint splits, so apparent "cultural" behavior may be confounded with speaker-specific gesturing style. We introduce SICAGE, a modular framework for culture-aware co-speech gesture generation that conditions motion synthesis models on speaker-independent cultural representations. SICAGE learns these representations from audio and text by treating each speaker as a separate domain while imposing invariance across speakers. This encourages representations to remain culture-discriminative while reducing dependence on speaker identity. The resulting cultural embeddings condition a multimodal generator to produce culturally appropriate gestures. We instantiate this idea with two domain generalization approaches: adversarial learning and Fishr regularization. We further introduce ALaDiT, a real-time diffusion-based gesture generator designed to efficiently incorporate the learned cultural embeddings. To validate our method, we built TED4C-L, a 106-hour multimodal dataset of 764 TED speakers from four cultural groups. Experiments show that SICAGE improves motion realism, diversity, beat synchronization, semantic relevance, and cultural consistency.

---

Introduction and Motivation

Human communication is fundamentally multimodal: speech is consistently accompanied by co-speech gestures that reinforce verbal messages, convey abstract concepts, and express emotions. Yet despite the rich literature on gesture generation for human–agent interaction and social robotics, a critical dimension has been largely overlooked — culture. Cultural norms shape how gestures are performed and interpreted, and failing to account for them limits the naturalness and appropriateness of synthetic gesture in cross-cultural deployments.

A deeper problem is methodological. Existing culture-aware gesture generation methods often do not rigorously test whether learned cultural patterns genuinely transfer across speakers. When the same speakers appear in both training and test splits (speaker-dependent evaluation), a model may appear to capture culture while actually memorizing speaker-specific style. A robust evaluation of cultural generalization therefore demands both large-scale data and explicitly speaker-disjoint splits, so that cultural patterns must be inferred from group-level regularities rather than individual speaker identity.

This paper introduces SICAGE (Speaker-Independent Culture-Aware Gesture gEneration), a modular framework that addresses both gaps. SICAGE operationalizes culture as speaker-disjoint, group-level communication patterns shared within cultural groups, and casts cultural representation learning as a domain-generalization problem — treating each speaker as a domain and using speaker-invariant learning objectives to encourage embeddings that generalize to unseen speakers from the same cultural group. Alongside SICAGE, the authors introduce TED4C-L, a 106-hour multimodal dataset of 764 TED speakers from four cultural groups, and ALaDiT, a real-time diffusion-based gesture generator conditioned on the learned cultural embeddings.

Related Work

Rule-Based Methods

Early gesture generation relied on rule-based systems mapping linguistic or prosodic cues to predefined gesture units. Systems like BEAT use linguistic and contextual annotations to animate gestures via a knowledge-based engine, while other approaches combine part-of-speech tagging and speech timing with grammar rules to sample gesture trajectories. Although these methods produce smooth motion, they rely on predefined gesture units and scale poorly, with only one known study explicitly considering cultural variations.

Data-Driven Methods

With larger datasets, gesture synthesis shifted to data-driven models favoring scalability. Early probabilistic models inferred gestures from prosodic features, while LLM-based approaches extract intent before generating gestures. End-to-end generative methods — GANs, attention-based models, and diffusion models — can produce novel and diverse motions. Diffusion-transformer models have achieved state-of-the-art quality through tight text/audio alignment. However, existing diffusion models lack explicit cultural conditioning. Some works implicitly embed cultural cues via attention or culture-specific GANs, but these provide limited quality and generalization to unseen speakers. SICAGE addresses this by conditioning diffusion models on dedicated, speaker-independent cultural embeddings.

Domain Generalization

Domain Generalization (DG) aims to generalize to unseen domains by training on related source domains. Common strategies include data augmentation, representation learning, and advanced training schemes. SICAGE adopts two DG strategies: adversarial learning (using a gradient reversal layer) and Fishr regularization (matching gradient-variance statistics across domains). Each speaker is treated as a domain, and cultural group membership is the prediction target.

Datasets

Motion-capture datasets like CMU Panoptic and Talking With Hands 16.2M provide accurate 3D keypoints but lack diversity, with only one small-scale dataset (LISI-HHI) providing cultural annotations. TED-Talk datasets using OpenPose-extracted skeletons offer greater diversity (~97 hours, 1,700+ speakers) but omit cultural labels. MCGD annotates culture for 263 speakers but is not public and covers only ~20 speakers per culture. TED4C-L fills this gap with ~190 speakers per culture across four cultures (106 hours total), explicitly designed for cultural generalization.

TED4C-L Dataset

TED4C-L is a large-scale, multimodal, speaker-balanced dataset built from YouTube TED Talks, spanning four coarse country-level cultural groups: India (Hindi-language TED talks), Italy, Japan, and Turkey. These groups were selected for their distinct spoken languages, geographic separation to minimize cultural overlap, and the availability of a large number of native-language TED talks. Using native (non-English) languages closely links geographic and cultural identity, enhancing annotation reliability.

The selection criteria require speakers to be clearly visible, standing, speaking their native language, and not holding objects that could affect gesturing. From 106.45 hours of total video data, 659,454 overlapping five-second samples were extracted using a 0.5-second stride, each with aligned audio, motion, and transcripts. Compared to the largest comparable TED-Talk dataset (which has 1,766 speakers, 97h, and 252,109 samples), TED4C-L features fewer but more carefully selected speakers (764), longer average video duration (15.41 min vs. 13 min), a finer stride (0.5s vs. 0.67s), and a larger total sample count, along with explicit cultural labels and multilingual transcripts.

Feature Extraction

Audio: Audio is downsampled to 16 kHz and processed to extract: (i) 64-dimensional mel-log features capturing spectral content, (ii) onset strength for rhythm information, and (iii) 1024-dimensional wav2vec features from a model pretrained on 56 languages. The combination of mel-log/onset features and wav2vec embeddings ensures both acoustic/rhythmic cues and higher-level speech representations are available.

Text: Language-Agnostic BERT Sentence Embeddings (LaBSE) are used to represent contextual meaning, capturing the first and last words overlapping each sample window for better alignment. Because LaBSE primarily aligns sentences by semantic content and may attenuate language-specific idiomatic structure, the audio features provide complementary acoustic, rhythmic, and prosodic cues.

Motion: 9 upper-body 3D keypoints (neck, head, central hip, shoulders, elbows, wrists) are extracted at 15 FPS using pretrained MMPose models. This compact set was chosen deliberately: SICAGE targets macro-level upper-body gesticulation in unconstrained TED videos where hands and fingers are often blurred, occluded, or out of frame, and the lower body is frequently truncated. Only sequences where the main speaker is continuously detected for at least 5 seconds are retained. Poses are normalized to be yaw- and pitch-invariant (removing average shoulder rotation around the vertical axis and hip-to-neck rotation around the horizontal axis). Each 75-frame motion sequence is represented in 6D continuous rotation format and encoded by a pretrained VQVAE into 25 tokens (1024-entry codebook with 512-dimensional embeddings).

Cultural Signal Verification

To assess whether cultural patterns are detectable from gesture motion alone, the authors train a motion-based classifier to predict cultural labels under speaker-disjoint evaluation. The classifier achieves approximately 45% balanced accuracy on unseen speakers, compared to a 25% random baseline for four classes. This confirms that gesture motion contains culture-related signals that generalize across speakers, while highlighting the substantial intra-cultural variability that makes the task challenging.

Methodology: SICAGE Framework

SICAGE is a modular framework composed of three replaceable components: (i) a culturally diverse dataset (TED4C-L), (ii) a model for learning speaker-independent cultural representations (Fishr or adversarial learning), and (iii) a motion generator conditioned on culture and other features (ALaDiT).

Speaker-Independent Cultural Representation Learning

The dataset is split so that training, validation, and test sets contain disjoint speakers within each culture. A feed-forward network (FFN) is trained to classify culture from audio and text, using either Fishr regularization or adversarial learning for domain generalization, treating each speaker as a domain. Motion is intentionally not used to learn cultural embeddings: target motion is the output to be generated and unavailable at inference, while the one-second seed is too short to estimate culture reliably.

Fishr Regularization

For Fishr, due to the large number of speaker domains, $k=64$ speakers are randomly sampled per training step. For each selected domain $d \in \mathcal{S}$ (with $|\mathcal{S}|=64$), a minibatch of $n_d=16$ samples is formed. The overall training loss is:

$$\mathcal{L}(\theta) = \frac{1}{N} \sum_{d \in \mathcal{S}} \sum_{i=1}^{n_d} \ell_i^d + \lambda_p P(\theta) + \lambda_s \mathcal{L}_{\text{SupCon}}^{(\tau)}$$

where $N = \sum_{d \in \mathcal{S}} n_d$ and the per-sample loss is $\ell_i^d = -\log p(c_i^d \mid x_i^d; \theta)$, with $c_i^d$ the cultural label of sample $i$ from speaker $d$.

The supervised contrastive loss $\mathcal{L}_{\text{SupCon}}^{(\tau)}$ enforces speaker invariance across the multi-domain batch:

$$\mathcal{L}_{\text{SupCon}}^{(\tau)} = \frac{1}{N} \sum_{i=1}^{N} \left[ -\frac{1}{|P(i)|} \sum_{p \in P(i)} \log \frac{\exp(\mathbf{z}_i^\top \mathbf{z}_p / \tau)}{\sum_{a \in A(i)} \exp(\mathbf{z}_i^\top \mathbf{z}_a / \tau)} \right]$$

where $\tau$ is the temperature, $A(i)$ is all sample indices excluding the anchor, and $P(i) = \{p \in A(i) : c_p = c_i\}$ is the set of positive samples sharing the same cultural label as the anchor.

The Fishr penalty $P(\theta)$ quantifies discrepancies in gradient variance across domains:

$$P(\theta) = \frac{1}{|\mathcal{S}|} \sum_{d \in \mathcal{S}} \|v^d - \bar{v}\|^2$$

where $v^d = \frac{1}{n_d}\sum_{i=1}^{n_d}(g_i^d - \bar{g}^d)^2$ is the per-domain gradient variance, $g_i^d = \nabla_\theta \ell_i^d$ is the per-sample gradient, and $\bar{v}$ is the mean variance across domains. Training sets $\lambda_p = 0$ for the first 500 updates to stabilize training, then increases to $\lambda_p = 1000$. The contrastive weight is $\lambda_s = 0.2$ and temperature $\tau = 0.07$.

Adversarial Learning

The adversarial variant uses the same architecture with an additional speaker-classification head placed after a gradient reversal layer (GRL). The speaker classifier is trained to predict speaker identity, while the GRL forces the shared encoder to discard speaker-specific information. The encoder is optimized with:

$$\mathcal{L}_{\text{tot}}(\theta_l, \theta_c, \theta_s) = \mathcal{L}_{\text{cult}}(\theta_l, \theta_c) - \lambda_2 \cdot \mathcal{L}_{\text{spk}}(\theta_l, \theta_s) + \lambda_s \mathcal{L}_{\text{SupCon}}^{(\tau)}$$

where $\theta_l$, $\theta_c$, and $\theta_s$ denote the parameters of the shared encoder, culture classifier, and speaker classifier, respectively. The adversarial weight $\lambda_2$ is gradually increased during training.

ALaDiT: Adaptive Latent Diffusion Transformer for Gestures

ALaDiT is a diffusion-based architecture for gesture generation that incorporates six input modalities: mel-log spectrogram, onset strength, wav2vec embeddings, sentence embeddings, seed motion, and culture embeddings. It operates in the discrete VQVAE latent space to reduce jitter and focus on salient motion dynamics. A key engineering achievement is inference speed: ALaDiT can generate a 4-second motion sequence in under 14 ms, enabling genuine real-time gesture synthesis.

Input Representation and Feature Construction

Each sample consists of 25 motion tokens, split into a 1-second seed $X_0^{\text{seed}} \in \mathbb{R}^{T_p \times d}$ (with $T_p=5$ tokens) and a 4-second target $X_0^{\text{fin}} \in \mathbb{R}^{T_r \times d}$ (with $T_r=20$ tokens). Following the Motion Diffusion Model (MDM), Gaussian noise at diffusion timestep $t$ is added to $X_0^{\text{fin}}$ to produce $X_t^{\text{fin}}$.

The low-level audio context $X^{\text{low}} \in \mathbb{R}^{(T_p+T_r) \times d}$ is constructed by: projecting mel-log and onset features to $d/2$ dimensions via $\text{FFN}_1$, projecting wav2vec to $d/2$ via $\text{FFN}_2$, applying windowed attention pooling to align temporal dimensions, concatenating the projections, and downsampling to $(T_p + T_r)$ tokens so each audio token corresponds to a motion token.

The high-level context $X^{\text{high}}$ is built by concatenating sentence and culture embeddings and projecting via $\text{FFN}_3$. The seed $X_0^{\text{seed}}$ and noisy target $X_t^{\text{fin}}$ are concatenated with positional encoding to form $X^{\text{mot}}$. The diffusion timestep embedding $\mathbf{e}_t$ (sinusoidal encodings projected via $\text{FFN}_4$) is concatenated with $X^{\text{mot}}$ to yield the input:

$$X^{\text{in}} = \text{concat}\bigl(\mathbf{e}_t,\; X^{\text{mot}}\bigr), \quad X^{\text{in}} \in \mathbb{R}^{(1+T_p+T_r) \times d}$$

Hierarchical Transformer Processing

A 10-layer hierarchical Transformer processes $X^{\text{in}}$ through three operations per layer: (1) self-attention over $X^{\text{in}}$, (2) cross-attention to $X^{\text{low}}$, and (3) AdaIN conditioning with $X^{\text{high}}$. Each operation is followed by a residual connection and layer normalization. After processing, the first six tokens ($\mathbf{e}_t$ and $X_0^{\text{seed}}$) are discarded, and the remaining tokens are projected by $\text{FFN}_5$ to form the denoised motion $\hat{X}_0^{\text{fin}}$.

Training Objectives

The reconstruction loss uses the Huber loss:

$$\mathcal{L}_{\text{rec}} = \mathbb{E}_{\substack{X_0 \sim p(X_0 \mid X^{\text{low}}, X^{\text{high}}) \\ t \sim [1,T]}} \left[\mathcal{H}\!\left(X_0^{\text{fin}} - \hat{X}_0^{\text{fin}}\right)\right]$$

To align generated motion with both low-level audio and high-level textual/cultural context, the model computes cosine alignment losses. Projections of $X^{\text{out}}$ and $X^{\text{low}}$ into a common space yield embeddings $z^o$ and $z^l$, while $X^{\text{high}}$ remains in $\mathbb{R}^d$:

$$\mathcal{L}_{\text{low}} = \mathbb{E}[1 - \cos(z^o, z^l)], \quad \mathcal{L}_{\text{high}} = \mathbb{E}[1 - \cos(z^o, X^{\text{high}})]$$

A contrastive loss $\mathcal{L}_{\text{cont}}$ (average of low- and high-level contrastive terms) prevents trivial solutions by pulling matching pairs together and separating non-matching ones. A culture classification head $\text{FFN}_6$ on $\hat{X}_0^{\text{fin}}$ produces a cross-entropy loss $\mathcal{L}_{\text{cult}}$ to further enforce cultural consistency.

The final loss is:

$$\mathcal{L} = \mathcal{L}_{\text{rec}} + \lambda_{\text{cu}}\mathcal{L}_{\text{cult}} + \lambda_l \mathcal{L}_{\text{low}} + \lambda_h \mathcal{L}_{\text{high}} + \lambda_c \mathcal{L}_{\text{cont}}$$

with $\lambda_{\text{cu}} = \lambda_l = \lambda_h = 0.1$ and $\lambda_c = 0.01$.

VQVAE for Motion Representation

A dedicated VQVAE is trained on TED4C-L to produce robust compressed motion representations. The VQVAE takes as input 5-second gesture sequences (75 frames at 15 FPS), with 9 joints per frame and 6D rotation per joint, forming a $[75, 54]$ matrix.

Architecture

The VQVAE architecture comprises: (1) a 1D convolutional layer (kernel size 5, stride 3, padding 1, 512 filters) that compresses the input to a $[25, 512]$ representation; (2) three dilated convolutional blocks with dilation rates $\{1, 3, 9\}$ to capture short- and long-range temporal dependencies; (3) a final encoder 1D convolution; (4) vector quantization selecting the nearest of 1024 codebook entries of dimension 512; and (5) a mirror decoder with transposed 1D convolutions. The architecture achieves a temporal downsampling factor of 3.

Training Loss

The total VQVAE loss extends the standard formulation with velocity and acceleration terms to improve temporal smoothness:

$$\mathcal{L} = \mathcal{L}_{\text{rec}} + \lambda_\beta \mathcal{L}_{\text{com}} + \lambda_{\text{vel}} \mathcal{L}_{\text{vel}} + \lambda_{\text{acc}} \mathcal{L}_{\text{acc}} + \lambda_{\text{reg}} \mathcal{L}_{\text{reg}}$$

where $\mathcal{L}_{\text{rec}} = \|x - \hat{x}\|_1$, $\mathcal{L}_{\text{com}} = \|z_e(x) - \text{sg}(z_q(x))\|_2^2$, $\mathcal{L}_{\text{vel}} = \|\dot{x} - \dot{\hat{x}}\|_1$, $\mathcal{L}_{\text{acc}} = \|\ddot{x} - \ddot{\hat{x}}\|_1$, and the temporal regularization term:

$$\mathcal{L}_{\text{reg}} = \frac{1}{N(T-2)} \sum_{n=1}^{N} \sum_{t=2}^{T-1} \left\| \hat{x}^{(n)}_{t+1} + \hat{x}^{(n)}_{t-1} - 2\hat{x}^{(n)}_t \right\|_2^2$$

penalizes excessive second-order variation in the reconstructed sequence. Note that $\mathcal{L}_{\text{acc}}$ matches second-order dynamics to the ground truth, while $\mathcal{L}_{\text{reg}}$ directly penalizes high-frequency jitter in the output itself. Hyperparameters are $\lambda_\beta=0.2$, $\lambda_{\text{vel}}=0.1$, $\lambda_{\text{acc}}=0.1$, $\lambda_{\text{reg}}=0.1$. Codebook embeddings are updated via exponential moving average (EMA, $\beta=0.99$).

Experimental Setup

ALaDiT Variants and Ablations

Three main ALaDiT variants are trained, differing only in cultural conditioning:

• ALaDiT/NC: No explicit cultural conditioning; culture must be inferred implicitly from audio/text while an auxiliary culture-classification head is used during training.
• ALaDiT/FI: Conditioned on speaker-independent cultural embeddings learned with Fishr regularization.
• ALaDiT/ADV: Conditioned on embeddings learned via adversarial domain generalization.

Three additional ablations isolate specific design choices:

• OneHot: Conditions on the discrete group label projected via a two-layer MLP and injected through the same conditioning pathway.
• NoDG: Uses the same audio/text embedding architecture as FI but without speaker-domain regularization.
• NoAlign: Uses Fishr embeddings but removes ALaDiT's multimodal alignment losses.

Baselines

Two strong diffusion-based baselines are adapted using the same TED4C-L features as ALaDiT: Motion Diffusion Model (MDM) and DiffuseStyleGesture+ (DSG+). NC, FI, and ADV variants are trained for each baseline. For DSG+, a DSG+/FI+Align ablation adds an explicit multimodal alignment loss to test whether DSG+ can better exploit Fishr embeddings with alignment. All models use 50 diffusion steps.

Evaluation Metrics

All models are evaluated on speaker-disjoint test splits using:

• Fréchet Gesture Distance (FGD): Measures motion realism by comparing distributions of real and generated motion in VQVAE latent space (lower is better).
• Semantic Relevance Gesture Recall (SRGR): Measures semantic alignment using PCK with $\delta=0.05$ (higher is better).
• Beat Alignment Score (BAS): Aligns motion-beat times (local velocity minima) with audio onset times, $\sigma=3$ (higher is better).
• Diversity: Mean $\ell_1$ distance between randomly sampled generated motion codebook sequences (higher is better).
• Cultural Expressivity F1 (CE F1): Weighted F1 of a speaker-disjoint motion-based cultural classifier applied to generated gestures; measures cultural consistency.

Metrics are averaged over 10 evaluations on random subsamples of 3,000 test instances. Statistical significance is assessed with paired t-tests ($p < 0.01$).

Training Details

All generative models are trained on speaker-independent splits with no speaker overlap between training, validation, and test sets. Training uses AdamW optimizer, batch size 64, 50 diffusion steps, 8 attention heads, 10 layers, latent dimension 512, learning rate $5 \times 10^{-5}$, and a single RTX 3090 GPU with 64GB RAM. ALaDiT uses EMA ($\beta=0.999$) for parameter tracking. Checkpoints are saved every 50,000 steps up to 500,000 steps, and model selection is based on lowest validation FGD. ALaDiT has approximately 50 million parameters total.

Results

ALaDiT Ablation Study

The ablation study isolates the contribution of each design component within the same ALaDiT generator. Results are summarized below:

Key findings from the ablation:

• Fishr (FI) achieves the best FGD (1.03, significantly better than all others) and highest CE F1 (44.61), demonstrating that speaker-invariant Fishr embeddings capture culturally relevant motion patterns most effectively.
• Adversarial (ADV) remains close to NC on most metrics, suggesting the adversarial objective provides weaker gains in this setting.
• OneHot shows that discrete cultural labels alone are insufficient: they increase Diversity but do not match FI on FGD or CE F1.
• NoDG confirms that using the same embedding architecture without speaker-domain regularization is insufficient for cultural generalization.
• NoAlign confirms that ALaDiT's alignment losses help exploit learned embeddings; removing them worsens FGD and CE F1 despite a slight SRGR increase.

The strongest results require combining Fishr-based speaker-independent embeddings with an architecture capable of multimodal alignment.

Comparison with Baselines

Three consistent patterns emerge from the cross-model comparison:

• ALaDiT achieves substantially lower FGD than both MDM and DSG+, indicating much closer similarity to real motion distributions across all cultural conditioning variants.
• Fishr consistently outperforms adversarial learning for cultural representation across all architectures — this trend holds for MDM, DSG+, and ALaDiT.
• Cultural embeddings are most effective when combined with architectures that explicitly model multimodal alignment. The DSG+/FI+Align ablation shows that adding explicit alignment substantially improves DSG+/FI on FGD, Diversity, and BAS, but performance still remains below ALaDiT/FI. This strongly suggests that ALaDiT's architecture is better designed to exploit the cultural conditioning signal.

Qualitative Analysis

To visualize culture-specific differences, the sentence "This example helps explain the idea of cultural styles" was translated into each dataset language and synthesized with Bark. Using the same fixed ground-truth seed pose, motions were generated for each culture and compared across ALaDiT-FI, ALaDiT-ADV, and ALaDiT-NC.

For Japanese samples, all models produce relatively compact upper-body motion. However, NC exhibits higher spatial dispersion at the wrists and elbows during speech pauses, suggesting lower realism. FI and ADV show more stable trajectories and terminate gestures more cleanly as speech ends.

For Italian samples, FI and ADV generate broader spatial gestures and larger arm excursions than NC, with movement sustained across multiple seconds. NC remains more constrained with a limited range of motion. FI distributes motion more globally across the upper body, while ADV concentrates motion more strongly on the arms.

User Study

A user study was conducted with $N=20$ participants from different cultural backgrounds. For each culture and condition (Real, NC, FI, ADV), two 30-second clips were generated (32 clips total), presented in random order with original audio and English subtitles. Participants rated seven questions per clip on an 11-point Likert scale (0–10): speech coherence, appropriateness, fluency, timing, amount of gesticulation, naturalness, and cultural fit.

Key user study findings:

• FI obtains the highest average score among generated models and is significantly preferred over ADV overall (6.06 vs. 5.65, $p=0.033$).
• FI is significantly preferred over NC on Cultural Match (6.16 vs. 5.81, $p=0.038$), confirming that Fishr-based cultural embeddings produce perceptible improvements in culture-associated gesture style.
• NC and ADV receive comparable ratings, consistent with the objective results where ADV provides less consistent gains.
• Real motion remains the highest-rated condition, confirming that generated gestures are still perceptually distinguishable from ground-truth motion.
• For Turkish samples, NC is slightly preferred over FI and ADV, but this difference is not statistically significant.

Per-Culture Analysis

The per-culture objective results show that the effect of cultural conditioning is not uniform across cultures or metrics. FI remains the most balanced variant:

• Best CE F1 for Indian (52.44%) and Japanese (45.83%) samples.
• Best BAS for Indian, Japanese, and Italian samples.
• Best FGD for Turkish samples (2.89 vs. higher values for other variants).

Some isolated metrics are optimized by other variants for specific cultures: NoAlign achieves the best SRGR for Indian and Italian samples; OneHot and NoDG occasionally obtain the highest Diversity; NC is not uniformly worse at the per-culture level. The Italian CE F1 is highest for NC, and NC leads on Japanese/Turkish SRGR.

In the per-culture user study, FI obtains the strongest ratings for Indian samples across all questions, and is best or tied for all Japanese questions with significant advantages over ADV on several measures. For Italian samples, FI is strongest on Appropriateness, Timing, and Cultural Match. Turkish samples are the main exception where NC has the highest mean scores, though differences with FI are not statistically significant.

Culture Classifier Architecture Details

The multimodal Fishr and adversarial classifiers share an encoder backbone consisting of:

• Sentence branch ($\text{FFN}_S$): Two linear projections $768 \to 512 \to 512$, each followed by LayerNorm, GELU, and dropout (0.1).
• Mel branch ($\text{FFN}_M$): $64 \to 512 \to 256$ with same pattern.
• Onset branch ($\text{FFN}_O$): $1 \to 512 \to 256$ with same pattern. Mel and onset features are concatenated frame-wise and summarized by attention pooling.
• Wav2vec branch ($\text{FFN}_W$): $1024 \to 512 \to 512$ with LayerNorm, GELU, dropout, and attention pooling.

The three modality embeddings (sentence, pooled mel-onset, pooled wav2vec) are concatenated into a 1536-dimensional vector, processed by a fusion block $\text{FFN}_{\text{emb}}$ ($1536 \to 512$), then a second block ($512 \to 512$) to yield the final 512-dimensional culture embedding used for conditioning generative models.

On unseen speakers, both multimodal classifiers achieve approximately 98% weighted F1 for culture recognition. The motion-only Fishr classifier achieves approximately 45% weighted F1 (vs. 25% random baseline), while the adversarial motion-only classifier reaches approximately 40.5%. The Fishr motion-only classifier is used to compute CE F1 on generated motion due to better performance and training stability.

Contributions Summary

The paper makes four distinct contributions:

• SICAGE: A modular framework for speaker-independent, culture-aware co-speech gesture generation. Each component (dataset, cultural representation model, and gesture generator) is independently replaceable.
• TED4C-L: A large-scale, 106-hour multimodal dataset of 764 TED speakers from four cultural groups (India, Italy, Japan, Turkey) with explicit cultural labels, multilingual transcripts, and speaker-disjoint evaluation splits designed specifically for cultural generalization.
• Domain generalization for culture: The first application of domain generalization methods (Fishr regularization and adversarial learning) to learn cultural representations that generalize across speakers in the gesture generation domain, with each speaker treated as a separate domain.
• ALaDiT: A real-time diffusion-based gesture generator (under 14ms for a 4-second sequence) incorporating six modalities through a hierarchical architecture combining self-attention, cross-attention, and AdaIN conditioning with explicit multimodal alignment losses.

Limitations and Future Work

The authors identify several limitations and directions for future investigation:

• Coarse cultural granularity: Cultural labels are coarse country-level groupings (India, Italy, Japan, Turkey) that do not capture intra-cultural diversity, regional variation, or ethnicity. Future work should consider finer-grained annotations.
• Limited keypoint set: Using only 9 upper-body keypoints excludes hands and fingers, which are important carriers of gesture meaning. This was a pragmatic choice driven by the high noise rate in unconstrained video, but future work could explore reliable hand/finger tracking in in-the-wild videos.
• Representation entanglement: Cultural representations are learned from audio and text but remain entangled with language, prosody, and semantics. Future methods should better disentangle culture-associated regularities from these confounds, as well as from coarse country-level grouping effects.
• User study familiarity: User study participants may not be equally familiar with all evaluated cultural styles, potentially affecting the reliability of perceptual assessments. Studies involving participants more deeply familiar with each culture could provide more informative evaluation.
• Dataset imbalance: The Japanese subset contains fewer extracted poses than other cultures despite having a relatively high number of speakers, due to speaker visibility issues in the original videos.