EMOVA

EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions

EMOVA is an open-source omni-modal LLM that integrates vision, speech, and text for end-to-end emotional spoken dialogue. It disentangles semantic content and acoustic style in speech, enabling vivid emotion and pitch control while achieving state-of-the-art vision-language and speech performance.

• llm
• multimodal
• dialogue
• asr
• tts
• prosody
• emotion

Demos

The demos showcase EMOVA's unified omni-modal capabilities by illustrating its end-to-end architecture, state-of-the-art performance on vision-language and speech tasks, and most notably, its ability to generate vivid emotional speech responses in multi-modal dialogues. When evaluating, focus on the seamless integration of vision and speech with emotional expressiveness, the superior benchmark results compared to GPT-4o/4V and Gemini Pro, and the naturalness and diversity of emotional spoken dialogues.

Authors: Kai Chen, Yunhao Gou, Runhui Huang, Zhili Liu, Daxin Tan, Jing Xu, Chunwei Wang, Yi Zhu, Yihan Zeng, Kuo Yang, Dingdong Wang, Kun Xiang, Haoyuan Li, Haoli Bai, Jianhua Han, Xiaohui Li, Weike Jin, Nian Xie, Yu Zhang, James T. Kwok, Hengshuang Zhao, Xiaodan Liang, Dit-Yan Yeung, Xiao Chen, Zhenguo Li, Wei Zhang, Qun Liu, Jun Yao, Lanqing Hong, Lu Hou, Hang Xu

Categories: cs.CV, cs.CL

Comment: Accepted by CVPR 2025. Project Page: https://emova-ollm.github.io/

Published 2024-09-26 · Updated 2025-03-20

Links

Paper & demos

• arXiv
• Project page

Code & resources

• Hugging Face6 downloads1 likes

Impact

• Semantic Scholar60 citations5 influential citations

Abstract

GPT-4o, an omni-modal model that enables vocal conversations with diverse emotions and tones, marks a milestone for omni-modal foundation models. However, empowering Large Language Models to perceive and generate images, texts, and speeches end-to-end with publicly available data remains challenging for the open-source community. Existing vision-language models rely on external tools for speech processing, while speech-language models still suffer from limited or totally without vision-understanding capabilities. To address this gap, we propose the EMOVA (EMotionally Omni-present Voice Assistant), to enable Large Language Models with end-to-end speech abilities while maintaining the leading vision-language performance. With a semantic-acoustic disentangled speech tokenizer, we surprisingly notice that omni-modal alignment can further enhance vision-language and speech abilities compared with the bi-modal aligned counterparts. Moreover, a lightweight style module is introduced for the flexible speech style controls including emotions and pitches. For the first time, EMOVA achieves state-of-the-art performance on both the vision-language and speech benchmarks, and meanwhile, supporting omni-modal spoken dialogue with vivid emotions.

---

Overview

EMOVA (Emotionally Omni-present Voice Assistant) is an end-to-end omni-modal large language model that aims to do all three of the following within one unified system: see images, hear speech, and speak with controllable emotional style. The paper positions EMOVA as an open-source attempt to approximate the omni-modal interaction pattern popularized by GPT-4o, while staying within the constraints of publicly available data.

The central claim is not just that EMOVA can ingest vision and speech and emit text or speech responses, but that the choice of representation matters: a semantic-acoustic disentangled speech tokenizer and a lightweight style-control module allow speech to be integrated end-to-end without sacrificing vision-language performance. The authors report that, after omni-modal alignment, the model can actually become better on both vision-language and speech tasks than bi-modal counterparts aligned separately.

At a high level, the paper contributes three things:

• a unified architecture with a continuous vision branch and discrete speech tokenization/generation;
• a text-centric omni-modal training recipe that leverages image-text and speech-text data jointly rather than relying on scarce image-text-speech triples;
• emotion- and pitch-controllable spoken dialogue through a small style module and synthesized style-rich instruction data.

Problem setting and main design choice

The paper starts from a clear gap in the open-source ecosystem: most vision-language LLMs can see and textually reason but depend on external speech tools; most speech-language models can hear and speak but lack strong visual understanding. By contrast, EMOVA is designed as a single model that can align text, vision, and speech in one autoregressive framework.

The key modeling decision is to represent speech with a discrete tokenizer-detokenizer pipeline, but not in the same way as systems that quantize both semantics and acoustics into a heavy token stream. Instead, EMOVA explicitly splits speech into:

• semantic content: what is said, which is quantized and aligned with the LLM;
• acoustic style: how it is said, including emotion, pitch, and related expressive cues.

This disentanglement is the paper's core technical idea for making speech compatible with a text-centric LLM while preserving style controllability.

Model architecture

Let the LLM be $f(\cdot)$, text input be $\mathbf{X}_T$, image input be $\mathbf{X}_V$, and speech input be $\mathbf{X}_S$. Text is tokenized into $\mathbf{U}_T$. Images are encoded by a vision encoder $v(\cdot)$ into $\mathbf{E}_V$, then projected into the LLM embedding space by $p(\cdot)$ as $\mathbf{H}_V$. Speech is first encoded by a speech encoder $s(\cdot)$ into continuous features $\mathbf{E}_S$, then discretized by a quantizer $q(\cdot)$ into speech units $\mathbf{U}_S$.

The model is trained autoregressively to predict output text units $\mathbf{U}_T^o$ and output speech units $\mathbf{U}_S^o$ conditioned on omni-modal inputs:

$$ p(\mathbf{U}_T^o, \mathbf{U}_S^o \mid \mathbf{U}_{\text{omni}}) = \prod_{i=1}^{L} p(x_i \mid \mathbf{U}_{T,

where $L = |\mathbf{U}_T^o| + |\mathbf{U}_S^o|$ and $\mathbf{U}_{\text{omni}} = \mathbf{U}_T \cup \mathbf{U}_S \cup \mathbf{H}_V$.

To recover speech, EMOVA uses a Unit-to-Speech detokenizer $d(\cdot, \cdot)$ that takes semantic unit embeddings and a style embedding to synthesize waveform output. In other words, the LLM emits both the content-bearing units and a style label, and the detokenizer converts them into expressive speech.

Vision branch

EMOVA uses QwenViT as the visual encoder and an MLP projector with a $4\times$ downsample rate. The base LLM is from the Qwen-2.5 family, with three reported sizes: $3$B, $7$B, and $72$B.

The architecture keeps the vision pathway continuous, which is important for preserving fine-grained image detail. This is particularly relevant for OCR, charts, documents, and math-heavy visual reasoning, where fully discrete vision tokenization can lose too much information.

Speech-to-Unit tokenizer

The speech tokenizer is built on SPIRAL with a finite scalar quantization layer. Its codebook has size $4096$, and it operates at $25$ tokens per second. The resulting units are shared across languages, specifically English and Chinese in the paper.

The paper emphasizes two design benefits:

• data efficiency: pretraining on large unlabeled speech makes the tokenizer easy to adapt with only limited speech-text pairs;
• bilingual sharing: a single codebook can model both English and Chinese speech units.

Importantly, the tokenizer is trained so that the quantized units focus on semantic content rather than entangled acoustic style. The encoder still extracts a style embedding, but only the semantic branch is discretized and passed into the LLM.

Formally, for speech input $\mathbf{X}_S$, the encoder produces

$$ \{\mathbf{E}_{\text{semantic}}, \mathbf{E}_{\text{style}}\} = \mathbf{E}_S = s(\mathbf{X}_S), $$

and only $\mathbf{E}_{\text{semantic}}$ is quantized into speech units $\mathbf{U}_S = q(\mathbf{E}_{\text{semantic}})$.

Unit-to-Speech detokenizer and style control

The speech decoder follows a conditional VAE-style design inspired by VITS. A speech unit encoder maps output units to embeddings, while a style encoder extracts a style vector from a reference speech. The decoder then synthesizes the final waveform from the semantic embedding and the style embedding.

The style-control mechanism is designed around a small inventory of style conditions:

• 2 genders: male, female;
• 4 emotions: neutral, happy, sad, angry;
• 3 pitches: normal, low, high.

This yields $24$ style combinations. The paper reports that one representative style reference speech is often enough to control the output effectively.

To build style prototypes, the authors synthesize candidate reference samples with external TTS, score emotion confidence with Emotion2Vec, and select the top candidate per style combination. This prototype bank is then used to condition the detokenizer at inference time.

Why semantic-acoustic disentanglement matters

The paper's ablation study argues that speech units should be closer to language-like symbols, not a bundle of both meaning and timbre. In the authors' view, this reduces the burden on the LLM during omni-modal alignment because the model no longer has to disentangle semantics from low-level acoustics on its own.

That hypothesis is supported by a direct comparison between a disentangled tokenizer and an entangled alternative based on HuBERT plus K-means clustering. The disentangled version consistently performs better on both vision-language and speech tasks.

Training strategy

EMOVA is trained in three stages.

1. Stage 1: vision-language pre-alignment. Only the vision projector is trained, using LCS-558K and a high-resolution image-slicing strategy.
2. Stage 2: omni-modal text-centric alignment. Vision-language and speech-language alignment are trained jointly using large-scale image-text and speech-text data.
3. Stage 3: omni-modal instruction tuning. Multi-task omni-modal instruction data is used to teach the model to follow user instructions and produce emotionally expressive speech.

The general optimization goal in Stage 2 is to align both image-text and speech-text through text as a shared bridge. The paper explicitly tests four alignment paradigms: VL, Speech, VL→Speech, Speech→VL, and Joint, plus a Joint-entangled variant.

The strongest result comes from joint alignment rather than sequential training. The paper argues this likely reduces catastrophic forgetting and lets the modalities reinforce each other under a shared semantic space.

Stage 1 details

Stage 1 trains only the vision projector on the $558$K LCS dataset. The model uses the QA prompt template and image resolutions ranging from $56^2$ to $1792^2$ pixels, with $4$ to $4096$ visual tokens depending on the image.

Stage 2 details

Stage 2 uses a unified $7.4$M-sample alignment set. For image-text data, the sources include ShareGPT4V, ALLaVA, ShareGPT-4o, and OCR-focused data from SynthDog, MMC-Alignment, K12 Printing, and the UReader text-reading subset. For speech-text alignment, the paper uses $2000$ hours of ASR and TTS data from LibriSpeech and AISHELL-2. To preserve language ability, text-only data from Magpie Pro is also included.

Training updates the LLM, the vision projector, and the deeper half of the vision encoder. The reported batch size is $256$, context length is $8192$, and the stage is run for one epoch.

Stage 3 details

Stage 3 uses a $4.4$M-sample omni-modal SFT set. High-quality visual instruction datasets are merged and normalized, including ShareGPT4V, InternVL, Meteor, Idefics-2, Cambrian, and LLaVA-OneVision. Speech instruction data is added from the paper's own synthetic pipeline and a $10\%$ subset of speech alignment data to preserve ASR/TTS quality.

The authors train on $128$ Ascend 910B NPUs. Stage 3 still uses one epoch, batch size $128$, and context length $8192$.

Omni-modal instruction tuning and emotional spoken dialogue

The instruction-tuning stage is where EMOVA becomes more than a recognizer or generator: it learns to hold spoken conversations with explicit style control. Because real omni-modal instruction data is scarce, the authors synthesize it from existing text and visual instruction datasets.

The pipeline is:

1. filter out non-vocal tasks such as code-heavy or formula-heavy data;
2. clean the remaining instruction text to make it more speech-like;
3. predict style labels with GPT-4o for gender, pitch, and emotion;
4. synthesize speech responses using CosyVoice and Azure AI Speech;
5. randomly vary speakers across $39$ available voices to improve diversity.

The resulting speech instruction set contains $24$ style combinations and, in aggregate, the paper reports $120$K speech-text pairs and $110$K speech-image pairs as part of the instruction data generation process.

For deployment and parsing, the target outputs are organized as JSON-like structured fields: recognized user speech, generated text response, predicted emotion and pitch labels, and generated output units.

Benchmark suite and evaluation protocol

The paper evaluates EMOVA on two broad fronts:

• vision-language: 15 benchmarks spanning OCR, document/chart reasoning, general perception, math, and science;
• speech-language: ASR, TTS, and emotionally rich spoken dialogue.

For speech dialogue, the authors define several metrics. They use GPT-4o to score the overall end-to-end spoken dialogue, as well as the text quality with and without speech synthesis in the loop. They also evaluate style-label classification accuracy and style controllability.

Main results

Vision-language and speech benchmarks

The paper reports that EMOVA achieves strong results across both multimodal vision-language benchmarks and speech benchmarks. The headline claim is that EMOVA-72B outperforms GPT-4o/4V and Gemini Pro 1.5 on 11 of 15 reported vision-language benchmarks, while also achieving the best reported Librispeech WER in the comparison table.

Selected headline numbers from the main benchmark table are below.

For speech recognition and synthesis, the paper compares EMOVA to Whisper-Large, Mini-Omni, AnyGPT, and VITA. The results show that EMOVA scales well with model size and that the 72B variant is the best among the reported EMOVA models on both ASR and TTS.

Emotion-rich spoken dialogue

The authors evaluate the conversational system on four synthesized test splits: Speech-Image-EN, Speech-Text-EN, Speech-Image-ZH, and Speech-Text-ZH. They report end-to-end spoken dialogue score, a unit-input-text-output score, a text-input-text-output score, style classification accuracy, and ASR/TTS metrics.

The main pattern is that the end-to-end score is close to the unit-input-text-output score, indicating that the speech synthesis pipeline preserves dialogue quality fairly well. The largest reported gap is only $0.33$, except on Speech-Image-ZH.

Two practical observations stand out. First, style-label accuracy is consistently above $75\%$, and pitch control is especially strong, often above $90\%$. Second, English speech synthesis is much easier than Chinese in the reported TTS metrics, which the authors attribute to Chinese's greater complexity and the harder conversational content in the Chinese splits.

Ablation findings

The paper contains two main ablation themes: alignment strategy and vision-language configuration.

Alignment strategy ablation

On the alignment side, the paper compares several training schedules. The result is unambiguous: joint alignment is better than sequential alignment, and it is also better than the entangled-speech version.

• Joint vs. bi-modal training: adding the second modality improves performance rather than causing competition.
• Joint vs. VL→Speech / Speech→VL: joint training outperforms sequential alignment, suggesting that avoiding modality-stage forgetting matters.
• Joint vs. Joint-entangled: semantic-acoustic disentanglement improves both vision-language and speech results.

The authors interpret this as evidence that aligning both image-text and speech-text simultaneously through a common text bridge is not just sufficient, but superior to staged transfer.

Vision-language configuration ablation

The appendix also explores the vision encoder configuration and prompt template during Stage 2. The paper compares a full ViT, a half-trained ViT, and a frozen ViT, plus QA-style versus plain templates. The final choice is the half-trained ViT with the QA template.

In words, the authors conclude that a partially trainable vision encoder plus QA prompt formatting yields the best pre-alignment behavior for the downstream omni-modal system.

Qualitative behavior

The qualitative section shows that EMOVA can do more than standard captioning or ASR. The model is demonstrated on humor understanding, numerical calculation, coding, geometry/math-style reasoning, and multiple emotional spoken-dialogue scenarios.

The authors also show examples where the model handles structured visual content inside spoken dialogue, including PDFs, movie posters, résumés, charts, and websites. They explicitly note that the model was not specially trained on spoken dialogue for all of these data types, so these examples are used to argue that text-centric alignment can transfer visual understanding into spoken interaction.

Limitations stated by the paper

The paper is explicit about several limitations and future directions:

• Not duplex: EMOVA cannot yet process simultaneous input and output streams in the way fully duplex conversational systems do.
• No direct unit-to-unit generation: text still mediates speech generation, and the authors see direct unit-to-unit modeling as a future speed and quality improvement.
• Only one vision encoder: the current system uses a single vision encoder pre-trained in a vision-language manner; combining multiple visual pretraining regimes or architectures is left open.
• No visual generation: the work is focused on visual understanding rather than visual synthesis.
• Robustness to noisy vision: robustness under corrupted or noisy visual inputs is suggested as a future direction.

The paper also compares its design with Emu3 in the appendix: both use discrete tokenization, but Emu3 discretizes visual data, whereas EMOVA discretizes speech while keeping vision continuous.

Bottom line

EMOVA's main technical message is that text-centric omni-modal alignment plus semantic-acoustic speech disentanglement can produce a unified model that is simultaneously strong at vision-language understanding, speech understanding, speech synthesis, and emotionally expressive spoken dialogue. The paper's empirical result is that adding speech to a strong vision-language backbone need not degrade visual capability; with the right tokenizer and joint training, it can actually improve both modalities.