MTAVG-Bench 2.0

MTAVG-Bench 2.0: Diagnosing Failure Modes of Cinematic Expressiveness in Multi-Talker Audio-Video Generation

A benchmark for evaluating cinematic expressiveness in multi-talker audio-video generation beyond basic metrics. It establishes a 45-failure-mode taxonomy across acting, atmosphere, and cinematography, with 11,600+ QA instances for diagnosing high-level audio-visual failures in scene-level video generation using LLMs.

• multimodal
• dialogue
• llm
• emotion
• audio-driven
• lip-sync
• gaze

Authors: Haitian Li, Yanghao Zhou, Heyan Huang, Liangji Chen, YiMing Cheng, Xu Liu, Dian Jin, Jiajun Xu, Jingyun Liao, Tian Lan, Ziqin Zhou, Yueying Liu, Yu Bai, Changsen Yuan, Jinxing Zhou, Xian-Ling Mao, Xuefeng Chen, Yousheng Feng

Categories: cs.AI, cs.MM, cs.SD

Published 2026-05-27 · Updated 2026-05-27

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub4 stars0 watchers0 forksMIT
• Hugging Face11 downloads2 likes

Impact

• Semantic Scholar0 citations

Abstract

In recent years, Multi-Talker Audio-Video Generation (MTAVG) models have shown promising performance on fundamental metrics such as lip-sync and audio-visual alignment. However, these metrics remain insufficient for assessing cinematic expressiveness in scene-level generation. In multi-character scenes, generation models must go beyond audio-visual realism to convey coherent character performance and other higher-level cinematic qualities. To fill this gap, we introduce MTAVG-Bench 2.0, a benchmark for diagnosing failure modes of cinematic expressiveness in multi-talker audio-video generation. Unlike prior settings that mainly focus on the quality of basic multi-turn dialogue, MTAVG-Bench 2.0 targets short-drama and scene-level generation, and establishes a high-level failure taxonomy spanning acting, narrative, atmosphere, and audio-visual language. Based on this taxonomy, we construct more than 10,000 question-answering evaluation instances, together with subsets for short-drama-level assessment and temporal localization of failure modes, to systematically evaluate the ability of omni large language models to diagnose high-level audio-visual failures. Experimental results show that commercial omni models such as Gemini substantially outperform other evaluators, yet even the strongest models continue to struggle with complex failures in our benchmark. These results demonstrate that MTAVG-Bench 2.0 provides a systematic benchmark for failure diagnosis in cinematic multi-talker audio-video generation.

---

Introduction and Motivation

Multi-Talker Audio-Video Generation (MTAVG) has progressed rapidly in recent years, with models achieving increasingly convincing performance on foundational metrics such as lip synchronization, audio-visual alignment, and speech naturalness. However, the paper argues that these low-level fidelity metrics remain fundamentally insufficient for evaluating the cinematic expressiveness required in scene-level, short-drama-style generation. In multi-character scenes — which inherently involve character relationships, emotional communication, and narrative arc — a generated clip must go far beyond technical correctness in basic audio-visual alignment. It must convey believable acting, coherent interpersonal dynamics, an appropriate emotional atmosphere, and shot organization that supports dramatic progression.

Current evaluation protocols for multi-speaker audio-video generation still center primarily on whether basic audio-visual capabilities are correct: speech quality and naturalness, audio-visual alignment, and lip-movement synchronization. While necessary, these criteria offer limited diagnostic power for high-level failures such as flat emotional performance, weak narrative progression, unconvincing atmosphere, or shot sequences lacking cinematic motivation. In other words, existing evaluation answers whether a clip is reasonable at the dialogue level but does not address whether the scene is expressive, whether it establishes a convincing atmosphere, or whether it exhibits coherent cinematic organization over time.

To fill this evaluation gap, the authors introduce MTAVG-Bench 2.0, a benchmark specifically designed for diagnosing failure modes of cinematic expressiveness in multi-talker audio-video generation. Unlike prior settings that focus on basic multi-turn dialogue quality, MTAVG-Bench 2.0 targets short-drama and scene-level generation, and establishes a high-level failure taxonomy spanning three major dimensions: acting, atmosphere, and cinematography. Based on this taxonomy, the benchmark constructs more than 10,000 question-answering (QA) evaluation instances, along with specialized subsets for short-drama-level assessment and temporal localization of failure modes. The benchmark is used to systematically evaluate the ability of contemporary omni large language models to diagnose high-level audio-visual failures.

Problem Formulation and Scope

The central insight of MTAVG-Bench 2.0 is that scene-level cinematic evaluation must be formulated as a structured failure diagnosis task, not a binary quality judgment. For a generated short-drama or scene-level clip to function as effective cinematic content, an evaluator needs to not only recognize that a clip is problematic, but also identify which specific category of failure is present and, in some cases, when in time it occurs. This structured diagnosis perspective provides two key advantages over holistic quality scoring: (1) it reveals the limitations of current generation systems in an interpretable, fine-grained manner, and (2) it provides actionable signal for advancing toward more expressive and narratively coherent audio-visual generation.

The benchmark focuses on the setting where models must jointly produce and evaluate multi-character scenes with coordinated audio and video, extending the evaluation scope to cover:

• Whether characters act, speak, and respond in a manner consistent with the intended narrative
• Whether the scene conveys an appropriate mood through coherent audio-visual cues
• Whether the audio-visual realization of shooting and editing exhibits cinematic coherence

Failure Taxonomy

MTAVG-Bench 2.0 organizes high-level failure modes into three major categories — Acting, Atmosphere, and Cinematography — and further decomposes each into fine-grained sub-dimensions, for a total of ten sub-dimensions and 45 failure modes. This taxonomy is designed to be cinematically grounded, annotation-operational, and evaluation-oriented.

Acting

Acting encompasses failures in character performance and interpersonal interaction, focusing on whether characters behave, speak, and respond in a manner consistent with the intended narrative. The four sub-dimensions are:

• Emotional Performance (EP): Emotional expressiveness of characters, including facial expression, gesture, vocal affect, and multimodal consistency with the scripted emotional state.
• Motion Performance (MP): Naturalness and script consistency of character motion, including action execution, motion dynamics, and intention continuity.
• Dialogue Performance (DP): Correctness of spoken delivery, including speech mode, dialogue content, and speaker identity consistency.
• Interaction Performance (IP): Plausibility of inter-character interaction, including gaze coordination, turn-taking, spatial relationship, targeting, and environmental responsiveness.

Atmosphere

Atmosphere evaluates whether a scene conveys an appropriate overall mood through coherent audio-visual cues. The three sub-dimensions are:

• Mood Construction (MC): Consistency between the intended scene mood and the conveyed affective cues from lighting, color, music, and overall audio-visual tone.
• Environmental Coherence (EC): Realism and temporal coherence of the environment, including scene plausibility, spatial depth, and smoothness of scene transitions.
• Soundscape Design (SD): Quality and appropriateness of non-speech audio, including ambient sound presence, background-audio balance, and scripted sound events.

Cinematography

Cinematography assesses whether the audio-visual realization of shooting and editing exhibits coherence with cinematic language. The three sub-dimensions are:

• Intra-shot Camera (IC): Appropriateness of camera behavior within a shot, including framing, motion, focus, blocking, and alignment with narrative attention.
• Inter-shot Grammar (IG): Logical progression of shot composition across adjacent shots, including shot sequencing and adherence to cinematic grammar rules.
• Continuity (CT): Preservation of action, spatial, and character continuity across shots, ensuring coherent scene progression and visual consistency.

The taxonomy further distinguishes between failure cases (concrete problematic phenomena observed in a clip) and failure modes (abstract taxonomy labels used to categorize such phenomena). This distinction is important for enabling structured, reproducible annotation and evaluation.

Benchmark Construction Pipeline

MTAVG-Bench 2.0 is constructed through a three-stage pipeline, as illustrated in the figure below.

Stage 1: Script Prompt Construction from Curated Film Scenes

The pipeline begins by collecting classic film scenes covering diverse multi-character dialogue settings, emotional relationships, and cinematic staging patterns. These scenes are converted into selective hierarchical prompts at both the short-drama and scene levels. A dedicated shot-analysis system prompt (powered by Gemini 3.1 Pro) is used to analyze uploaded film clips and produce copyright-safe, de-identified structured JSON plans. Each plan captures: clip-level summary fields (premise, core setting, overall tone, character registry), model inference rules (for visual atmosphere, music, performance, dialogue, restraint, continuity, and camera), and 8-to-10-second sequential segments with associated story context, dialogue, emotional tone, relationship dynamic, key event, and continuity information.

These prompts retain key elements relevant to audio-visual short-form narratives: plot summaries, character information, character emotion, scripted actions, dialogue content, scene descriptions, and atmospheric tone. A critical design choice is the "dialogue rule" enforced in the prompt: spoken dialogue must exactly match the lines provided in the dialogue field — no additions, omissions, paraphrasing, reordering, or summarization of spoken words are allowed. The shot-analysis prompt also enforces that original film titles, franchise names, copyrighted character names, and IP-specific proper nouns are never included in the output.

Stage 2: Video Generation and Failure Discovery

Using the structured prompts, multi-talker audio-visual clips are generated with multiple image+text-to-audio-video systems, specifically including: Grok Video 3, LTX 2.3, Sora 2, Veo 3.1, Vidu Q3, and Wan 2.6. These generation models span a range of approaches and design choices, ensuring diverse failure patterns in the collected data.

Human annotators then review the generated videos to identify observable failure evidence. Each identified case is subsequently mapped into the predefined taxonomy via bucket-level failure categorization. To ensure benchmark quality, failure-conditioned data selection is applied: only clips and temporal segments that exhibit clear, verifiable, and diagnostically meaningful failures are retained. This yields 2,466 videos spanning 10 sub-dimensions and 45 failure modes.

Stage 3: QA Construction and Expert Refinement

The selected failure cases are transformed into structured question-answering instances that assess failure classification, comparative judgment, and temporal grounding. Given a failure case, its associated failure mode, and supporting evidence, failure-aware diagnostic QA items are constructed, where candidate answers are drawn from semantically related failure modes to ensure diagnostic relevance rather than trivial matching.

Initial QA construction is assisted by Gemini 3.1 Pro, followed by expert-guided refinement. A three-way expert validation protocol is used: multiple annotators independently verify and reconcile each QA instance. The benchmark uses an expert pool of 22 human verifiers. For each item, two experts independently review it; if they agree, the item is accepted. If they disagree, a third expert is brought in for adjudication (required in 15.9% of items). The initial two-expert agreement rate is 84.1%, corresponding to a Cohen's kappa of 0.78.

MTAVG-Bench 2.0 includes diverse question formats:

• Single-choice for dominant failure classification
• Multiple-choice for multi-label or co-occurring failures
• Pairwise for comparative judgment
• Temporal localization for grounding failures in time

Questions are constructed with a rule-based pipeline maintaining more than twenty templates. Templates are matched with failure modes, and candidate options are generated from the corresponding failure-mode candidate set. The final benchmark comprises 2,466 videos, 10 sub-dimensions, 45 failure modes, and approximately 11,600 QA instances.

Data Characteristics and Analysis

Source Material and Sentiment Distribution

The 2,466 scene-level samples are drawn from 20 source movie clips. The overall sentiment distribution is reasonably balanced: negative scenes account for 42.0% of all samples, positive scenes for 38.5%, and neutral scenes for 19.5%. Within positive scenes, approval is the largest component (18.7%), followed by joy (9.9%) and admiration (3.7%). Within negative scenes, sadness (10.5%) and disapproval (8.1%) are most prevalent, followed by disgust (5.6%), annoyance (4.7%), and fear (4.1%). Neutral affect itself accounts for 17.0% of all samples.

Error Count Distribution

The distribution of annotated error counts across the three top-level dimensions is concentrated in the low-count range. For Acting, 83.58% of videos contain 0–3 errors, with 1 error being the most common (25.79%). For Atmosphere, 87.10% fall within 0–3 errors, mode at 2 errors (26.40%). For Cinematography, 86.98% remain within 0–3 errors, with 0 errors as the largest bin (24.53%). High-count tails are limited: the proportion of videos with 5 or more errors is 6.73% for Acting, 2.80% for Atmosphere, and 6.04% for Cinematography.

In the aggregate across all dimensions, only 2.31% of videos contain zero total errors, while 60.14% fall in the range of 4–9 total errors. The benchmark is therefore not dominated by saturated or excessively corrupted examples — most samples exhibit localized failures while a smaller subset contains denser cross-dimensional error patterns.

Failure-Mode Correlation Structure

Fine-grained failure modes are not uniformly independent. Visible clusters of co-occurring failures emerge: one cluster centers on expressive-performance failures (co-speech gesture inconsistency, emotion-driven motivation shift, multimodal expressive inconsistency, and exaggerated expression tend to co-occur), and a second cluster is centered on continuity and shot grammar (missing scene transition, scene consistency drift, action continuity break, spatial continuity break, and shot progression issues show consistently strong pairwise correlations). The presence of weakly correlated or negatively correlated pairs indicates that failure propagation is structured rather than global.

Evaluation Protocol

Scoring Formulas

All models are evaluated under a unified failure-diagnosis setting with a zero-shot, fixed inference protocol. Outputs are normalized via deterministic post-processing; invalid or unparsable responses are treated as incorrect.

For each question $i$, let $G_i$ denote the ground-truth answer set and $P_i$ denote the model prediction. The per-question score $s_i \in [0,1]$ is defined as follows:

• For single-choice questions: $$s_i = \mathbb{I}[P_i = G_i]$$
• For multiple-choice questions (coverage-based): $$s_i = \frac{|P_i \cap G_i|}{|G_i|}$$
• For pairwise comparison questions: $$s_i = \mathbb{I}[P_i = G_i]$$

For each sub-dimension $d$, the score is: $$S_d = \frac{1}{N_d} \sum_{i \in \mathcal{Q}_d} s_i$$ where $\mathcal{Q}_d$ is the set of questions in sub-dimension $d$ and $N_d = |\mathcal{Q}_d|$. The overall score is a weighted average over all sub-dimensions: $$\text{Avg.} = \sum_{d=1}^{D} \frac{N_d}{\sum_{k=1}^{D} N_k} S_d$$

Temporal Localization Metrics

For the temporal localization subset, three additional metrics are reported:

• Primary Issue Accuracy (PIA): whether the predicted primary failure label matches the annotated one: $$\text{PIA} = \frac{1}{N} \sum_{i=1}^{N} \mathbb{I}[\hat{y}_i = y_i]$$
• Temporal Localization Accuracy (TLA): whether predicted timestamp options cover the annotated failure region (same coverage-based rule as multiple-choice): $$\text{TLA} = \frac{1}{N} \sum_{i=1}^{N} \frac{|\hat{T}_i \cap T_i|}{|T_i|}$$
• Rationale Consistency (RC): alignment of the model-generated rationale with annotated failure evidence, temporal localization, and final diagnosis, scored by GPT-5.4 as judge on a 1–5 Likert scale: $$\text{RC} = \frac{100}{5N} \sum_{i=1}^{N} r_i$$ where $r_i \in [0,5]$ denotes the average judge score for instance $i$.

Failure Rate Definition

For generation-side analysis, the failure rate of model $m$ on dimension $d$ is defined as: $$\text{FR}_{m,d} = \frac{C_{m,d}}{|V_m|}$$ where $C_{m,d}$ is the number of annotated failure clip segments in dimension $d$ for model $m$, and $|V_m|$ is the total number of generated videos. When multiple failures occur within the same video, they are accumulated separately per dimension — so this is a clip-segment-level incidence rate, not a binary video-level indicator.

Comparison with Prior Benchmarks

MTAVG-Bench 2.0 substantially extends the scope of prior audio-video generation benchmarks. The following table summarizes key differences:

MTAVG-Bench 2.0 is the only benchmark in this comparison that jointly covers speaker-centric evaluation, multi-speaker interaction, dialogue structure, cinematic expressiveness, failure-mode diagnosis, and both text-to-AV and image-to-AV modalities.

Experiments and Results

Models Evaluated

The benchmark evaluates a diverse set of contemporary omni models supporting joint audio-video understanding and multimodal reasoning, in zero-shot setting:

Proprietary omni models:

• Gemini 3.1 Flash Lite
• Gemini 3.1 Pro
• Gemini 3 Flash
• Gemini 2.5 Flash

Open-source omni models:

• Qwen 2.5 Omni (7B)
• MiniCPM-o 2.6 (7B)
• OmniVinci (9B)
• VideoLLaMA 2 (7B)
• Ola Omni (7B)
• Ming Lite Omni 1.5 (30B)

Main Results

The main results across all ten sub-dimensions are reported in the following table:

Gemini 3.1 Pro achieves the best overall average (62.16), substantially higher than the strongest open-source model, Ming Lite Omni 1.5 (41.11). Its advantage is particularly pronounced in Atmosphere and Cinematography dimensions. In the Acting category, performance is more mixed: VideoLLaMA 2 achieves the highest Emotional Performance score overall (55.64), and Gemini 3 Flash performs best on Interaction Performance (43.57).

These results indicate that MTAVG-Bench 2.0 clearly separates model capability on high-level cinematic failure diagnosis: current proprietary models show substantially stronger scene-level reasoning ability, while open-source models exhibit only isolated strengths on a few sub-dimensions.

Model-wise Failure Rate Analysis on Generated Videos

Two key observations emerge from the failure-rate analysis of the video generation models:

1. Failure patterns are highly uneven across dimensions. Dimensions requiring richer scene-level coordination — including dialogue performance, interaction performance, mood construction, inter-shot grammar, and continuity — tend to exhibit consistently higher failure rates across multiple models compared with relatively localized failures such as emotional performance or soundscape design.
2. Model robustness varies substantially across sub-dimensions. No single system is uniformly reliable across all failure types. Grok Video 3 and LTX 2.3 exhibit relatively high failure rates across multiple sub-dimensions, indicating persistent weaknesses in scene-level cinematic generation.

At the category level: acting-related failures are generally less severe, whereas multi-actor coordination remains more challenging (especially for action-script alignment and interaction timing). In Atmosphere, lighting and color consistency are among the most persistent error sources. The strongest separation appears in Cinematography, where shot progression, 30-degree and 180-degree rule violations, and spatial continuity errors exhibit the widest spread across models.

Holistic Automatic Quality Assessment

The benchmark also reports a holistic quality assessment of different video sources using deep learning-based automatic metrics (Audio Aesthetic, Lip Sync, A-V Align, Desync, T-A Align, T-V Align):

A key finding is that strong performance on conventional automatic metrics does not translate into reliable scene-level failure diagnosis. Grok Video 3 achieves the best scores on Audio Aesthetic, T-A Align, and T-V Align, yet it still exhibits relatively high failure rates in the diagnostic benchmark. This discrepancy is partly attributed to its tendency to generate acoustically clean speech signals, which improves low-level audio scores, while still lacking environmental grounding and contextual integration — resulting in audio that sounds clean in isolation but remains weak in environmental coherence and scene-level soundscape consistency. This finding is a central motivation for MTAVG-Bench 2.0.

Analysis and Ablations

Temporal Localization Results

The temporal localization subset is particularly demanding, requiring the evaluator to identify both what failure occurs and when it happens in time. Gemini 3.1 Pro achieves a high Rationale Consistency (RC) of 83.8%, but its Primary Issue Accuracy (PIA) and Temporal Localization Accuracy (TLA) drop significantly to 60.6% and 60.9%. This contrast reveals a critical limitation: current omni models can generate plausible textual rationales for cinematic failures but struggle to precisely anchor these issues to specific timestamps. Fine-grained temporal grounding remains a major bottleneck for scene-level diagnosis.

Ablation Study on Input Modalities

This ablation study (on Gemini 3 Flash) probes the reliance of cinematic diagnosis on multimodal reasoning:

• Full audio-visual input yields the best average performance (54.00%).
• Removing vision causes a drastic drop to 33.10%, barely exceeding the text-only baseline (32.82%). This proves that evaluating Acting and Cinematography heavily relies on visual cues like expressions and framing.
• Removing audio reduces average to 40.64%. Notably, this absence not only degrades Atmosphere dimension but also impairs Acting and Cinematography, emphasizing the necessity of audio-visual synchrony and rhythm.

These findings confirm that MTAVG-Bench 2.0 strictly requires joint multimodal understanding, effectively preventing models from exploiting unimodal shortcuts.

Rationale Consistency Distribution

Rationale-consistency scores are concentrated in the upper range (mean 4.15, median 4.0), indicating that predicted rationales usually provide clear support for the final selected option. Low-score cases show that the metric is not saturated and still captures instances where rationale-answer alignment is weak.

Case Studies of Failure Modes

Diagnostic QA Case Study

This example from the Interaction Performance dimension illustrates the difficulty of fine-grained diagnostic evaluation. The prompt implies a tense, confrontational exchange where direct eye contact is expected; the generated segment exhibits inconsistent gaze behavior. Successful evaluation requires the evaluator to infer intended acting from context, distinguish among closely related failure types, and ground the answer in concrete visual evidence.

Acting-Level Failure Cases

Cinematography-Level Failure Cases

Atmosphere-Level Failure Case

Annotation Infrastructure

Annotation is conducted using a customized Label Studio interface. Reviewers are presented with a scene-level clip, candidate failure-mode options, and concise explanatory fields (Story Context, Key Event, Relationship Dynamic, Action). The interface provides dimension-specific guidance and reference cues to support consistent annotation. The expert pool consists of 22 human verifiers, with two-expert independent review per item, a third-expert adjudication for 15.9% of items where disagreement arises, and a Cohen's kappa of 0.78.

Prompt Design for Benchmark Construction

The benchmark construction pipeline uses three structured system prompts:

Shot Analysis Prompt

A two-part prompt instructs the model (Gemini 3.1 Pro) to analyze an uploaded film clip and convert it into a copyright-safe, de-identified structured JSON plan. Key design features include: operational steps for internal film identification followed by de-identified output; segmentation rules for 8-second generation units; a strict dialogue rule requiring exact match of spoken content; style and inference rules (no explicit direction of lighting, color palette, background music, etc. — these are encoded as model inference rules); and output constraints ensuring no copyrighted names or IP-specific proper nouns appear. The output schema includes clip_summary, model_inference_rules, and segments with rich per-segment metadata.

Video Generation Prompt

A prompt for segment-level video generation consumes the de-identified fields from the shot-analysis stage and enforces: generation that realizes the scene from provided script metadata (not copying known film-specific phrasing), continuity of character identity and dramatic state, reference-image handling for preserving cross-segment continuity, and specific model inference rules. A critical constraint is that the video must remain a single full-frame image — no split screen, multi-panel layouts, grids, picture-in-picture, or multiple simultaneous views within one frame.

RC Judge Prompt

For rationale-consistency evaluation, a shared judge prompt (used with GPT-5.4) assesses whether the model's rationale logically supports its final answer choice on a 1–5 Likert scale, returning a structured JSON output with score and rationale analysis.

Ethical Considerations

MTAVG-Bench 2.0 is designed to evaluate and diagnose cinematic-level generation failures, not to make value judgments about real individuals or social groups. Scripts do not involve personal information from real people, and evaluation does not depend on identity-specific information about real actors. The setup resembles the practice of "imitation shooting" in film education: cinematic scenes are reconstructed for analysis using T2VA and I2VA generation pipelines. Users are cautioned that because some benchmark materials are inspired by or derived from existing films, copyright and fair-use boundaries should be carefully considered when using, releasing, or redistributing related materials.

Limitations and Open Challenges

Despite strong overall performance, even the best current evaluator (Gemini 3.1 Pro, 62.16% average) shows clear limitations. Key open challenges identified by the benchmark include:

• Complex acting failures: Motion Performance remains especially challenging, with the best model achieving only 52.35% and open-source models dropping as low as 9.01%.
• Temporal grounding: The gap between RC (83.8%) and PIA/TLA (~60%) for Gemini 3.1 Pro reveals that generating plausible rationales is far easier than precisely anchoring failures to timestamps.
• Cross-dimensional diagnosis: Failures in dimensions requiring scene-level coordination (dialogue performance, interaction, mood construction, inter-shot grammar, continuity) are consistently harder than localized, modality-specific failures.
• Open-source model gap: The large gap between commercial (62.16%) and open-source models (best: 41.11%) suggests substantial room for improvement in open-source omni model capabilities for cinematic diagnosis.
• Metric-diagnosis gap: Strong performance on conventional automatic quality metrics (lip sync, audio aesthetic, T-A align) does not reliably translate into strong performance on cinematic failure diagnosis.

Contributions Summary

The paper makes three core contributions:

1. Problem formulation: High-level failure diagnosis is formulated as a distinct evaluation problem for scene-level cinematic expressiveness in multi-talker audio-video generation, moving beyond conventional evaluation centered on low-level fidelity and local interaction quality.
2. MTAVG-Bench 2.0: A benchmark tailored to short-drama and scene-level generation, with a compact taxonomy spanning acting, atmosphere, and cinematography. Over 10,000 evaluation instances are constructed, together with dedicated subsets for scene-level assessment and temporal localization, enabling fine-grained and temporally grounded diagnosis of high-level failures. The benchmark comprises 2,466 videos, 10 sub-dimensions, 45 failure modes, and ~11,600 QA instances.
3. Systematic benchmarking: Contemporary omni audio-visual understanding models are systematically benchmarked, showing that leading commercial systems perform best overall but diagnosing complex failures in character performance remains challenging even for the strongest evaluators — highlighting the need for richer benchmarks and more capable omni models.