Kimi-Audio
Kimi-Audio Technical Report
Kimi-Audio is an open-source foundation model unifying audio understanding, generation, and conversation using a novel mix of discrete semantic tokens and continuous acoustic features. It supports diverse audio tasks in one architecture with large-scale pre-training and a streaming detokenizer for real-time deployment.
Demos
The demos showcase Kimi-Audio's universal audio foundation model capabilities including state-of-the-art audio understanding, generation, and conversational audio interactions. Viewers should evaluate the clarity and accuracy of speech recognition, the naturalness and latency of generated audio, and the model's ability to handle multiturn audio conversations with coherent, context-aware responses. The radar chart illustrates Kimi-Audio's superior benchmark performance across diverse audio tasks, validating the technical claims.
Links
Paper & demos
Code & resources
Abstract
We present Kimi-Audio, an open-source audio foundation model that excels in audio understanding, generation, and conversation. We detail the practices in building Kimi-Audio, including model architecture, data curation, training recipe, inference deployment, and evaluation. Specifically, we leverage a 12.5Hz audio tokenizer, design a novel LLM-based architecture with continuous features as input and discrete tokens as output, and develop a chunk-wise streaming detokenizer based on flow matching. We curate a pre-training dataset that consists of more than 13 million hours of audio data covering a wide range of modalities including speech, sound, and music, and build a pipeline to construct high-quality and diverse post-training data. Initialized from a pre-trained LLM, Kimi-Audio is continual pre-trained on both audio and text data with several carefully designed tasks, and then fine-tuned to support a diverse of audio-related tasks. Extensive evaluation shows that Kimi-Audio achieves state-of-the-art performance on a range of audio benchmarks including speech recognition, audio understanding, audio question answering, and speech conversation. We release the codes, model checkpoints, as well as the evaluation toolkits in https://github.com/MoonshotAI/Kimi-Audio.
1. Problem Setting and High-Level Contribution
Kimi-Audio is presented as an open-source audio foundation model that unifies audio understanding, audio generation, audio-to-text chat, and speech conversation in a single architecture. The paper is motivated by a gap in prior work: many audio-LMM systems are specialized for only one task family, rely mainly on downstream fine-tuning rather than large-scale audio pre-training, or are not publicly reproducible. The main claim is that a universal audio model can be built by combining a high-rate audio tokenizer, an LLM core initialized from a strong text model, and a streaming detokenizer for waveform generation.
The report emphasizes three pillars: (1) architecture, where audio is represented by discrete semantic tokens plus continuous acoustic features; (2) data, where more than $13$ million hours of audio are curated and processed for pre-training, and a large supervised fine-tuning corpus is assembled for downstream tasks; and (3) training, where the model is continual pre-trained on both audio and text with multiple task types, then instruction-tuned for audio-centric use cases.
The paper also contributes an evaluation toolkit intended to standardize benchmarking for audio foundation models, along with a real-time deployment workflow for speech-to-speech interaction. In the reported benchmarks, Kimi-Audio claims state-of-the-art or near-state-of-the-art performance across speech recognition, general audio understanding, audio question answering/chat, and speech conversation.
2. Architecture
2.1 Unified audio I/O and LLM core
The system consists of three modules: an audio tokenizer, an audio LLM, and an audio detokenizer. The tokenizer converts input waveforms into a low-rate sequence of discrete semantic tokens at $12.5$ Hz and also extracts continuous acoustic vectors. The LLM consumes the audio representation and produces both text tokens and audio semantic tokens. The detokenizer converts predicted audio tokens back into waveforms using a flow-matching-based synthesis pipeline.
A central design choice is the use of hybrid audio representations. Kimi-Audio keeps discrete semantic tokens as the main representation for both input and output, but augments the input side with continuous acoustic features. The authors argue that this combination improves perception while preserving efficiency and making audio generation feasible within an LLM-like framework.
2.2 Audio tokenizer: semantic tokens plus continuous Whisper features
The discrete semantic tokens follow the GLM-4-Voice line of work: a Whisper-based encoder is equipped with a vector-quantization layer and reduced to a single-codebook token stream at $12.5$ Hz. In parallel, the model uses a continuous acoustic branch derived from a pre-trained Whisper model. Because Whisper features are naturally produced at $50$ Hz, Kimi-Audio adds an adaptor to downsample them to $12.5$ Hz so that they can be added to the semantic token embeddings.
The paper frames this as a trade-off between semantic grounding and acoustic fidelity: discrete tokens provide compact symbolic content, while continuous features expose richer acoustic cues useful for broad audio understanding.
2.3 Audio LLM: shared layers with separate text and audio heads
The audio LLM is an adapted decoder-only transformer. A set of lower layers is shared across modalities, after which the network splits into two parallel transformer heads: one for autoregressive text generation and one for autoregressive audio-token generation. The text head is used for textual responses; the audio head predicts semantic audio tokens that are later detokenized into speech.
Initialization is important: the shared transformer layers and the text head are loaded from a pre-trained text LLM, specifically Qwen2.5 $7$B, while the audio head is randomly initialized. This preserves language competence while adding audio generation capacity.
2.4 Audio detokenizer: flow matching plus chunk-wise streaming
The detokenizer is based on the MoonCast architecture and has two stages: a flow-matching model that maps $12.5$ Hz semantic tokens to $50$ Hz mel-spectrograms, followed by a BigVGAN vocoder that synthesizes waveforms. To support low-latency deployment, the paper introduces a chunk-wise autoregressive streaming framework.
In this framework, audio is split into chunks $c_1, c_2, \dots, c_N$ (for example, $1$ second each). During training and inference, earlier chunks act as prompts for later chunks. For chunk $c_i$, the model conditions on previous chunk states and reconstructs the mel-spectrograms associated with $c_i$. Since chunk boundaries can degrade quality, the authors also introduce a look-ahead mechanism: for chunk $c_i$, the system prepends the first $n$ semantic tokens from the next chunk $c_{i+1}$ before detokenization, then keeps only the mel output corresponding to $c_i$. This is training-free and trades a small delay for improved boundary quality.
3. Data
3.1 Pre-training corpus
The pre-training corpus contains both unimodal and multimodal data. The audio-only portion spans about $13$ million hours of raw audio from audiobooks, podcasts, interviews, music, environmental sound, and multilingual material. The text-only portion is taken from an existing high-quality text pre-training source referenced by the paper. The authors stress that much of the raw audio lacks clean transcriptions, language labels, speaker identities, or segmentation, and is often contaminated by background noise, reverberation, and speaker overlap.
To transform raw audio into usable multimodal data, the paper builds an automated pipeline with several steps: speech enhancement, diarization, segmentation refinement, language identification, and transcription. The stated goal is not just short utterance extraction, but long-form annotations that preserve context and speaker consistency.
3.2 Audio preprocessing pipeline
Speech enhancement. The paper uses a BSRNN-based enhancement model operating at $48$ kHz. Enhancement removes noise and reverberation, but the authors note an important caveat: enhancement can also remove environmental sound and music, which are valuable for audio understanding. To mitigate this, they randomly choose original or enhanced audio with a $1:1$ ratio during pre-training.
Segmentation by diarization. PyAnnote is used for speaker diarization, followed by three refinement steps. First, speaker clusters whose representative embeddings have cosine similarity greater than $0.6$ are merged to reduce fragmentation. Second, segments are split into $1.5$-second chunks and reassigned when adjacent chunks appear to belong to different speakers based on a cosine-similarity threshold of $0.5$. Third, adjacent same-speaker segments are merged until the accumulated segment length exceeds $27$ seconds or the inter-segment silence gap exceeds $2$ seconds.
Speech transcription. Whisper-large-v3 is used for spoken language identification. Only English and Mandarin are retained. English segments are transcribed directly by Whisper-large-v3, including punctuation. Mandarin segments are transcribed with Paraformer-Zh, and punctuation is heuristically inserted from character-level timing gaps: a comma for gaps between $0.5$ and $1.0$ seconds and a period for gaps above $1.0$ second.
Implementation scale. The preprocessing pipeline runs on $30$ cloud instances, each with $128$ vCores, $1$ TB RAM, and $8$ NVIDIA L20 GPUs. The reported throughput is about $200{,}000$ hours of raw audio per day.
3.3 Supervised fine-tuning data
The SFT data is organized into three groups: audio understanding, speech conversation, and audio-to-text chat. The paper reports roughly $300$K hours of data for SFT and emphasizes the use of mostly open or accessible sources, plus in-house data for key tasks.
Audio understanding datasets
For audio understanding, the authors compile datasets across six task types: automatic speech recognition (ASR), audio question answering (AQA), automated audio captioning (AAC), speech emotion recognition (SER), sound event classification (SEC), and audio scene classification (ASC). Open datasets include WenetSpeech, WenetSpeech4TTS, AISHELL-1/2/3, Emilia, Fleurs, CommonVoice, KeSpeech, Magicdata, zhvoice, LibriHeavy, MLS, GigaSpeech, LibriSpeech, VoxPopuli, LibriTTS, CompA-R, ClothoAQA, AudioCaps, Clotho-v2, MACS, FSD50K, CochlScene, Nonspeech7k, MusicAVQA audio-only, AVQA audio-only, IEMOCAP, MELD, RAVDESS, SAVEE, ESD, TUT2016, TUT2017, TAU2022, ESC50, VocalSound, VGGSound, and UrbanSound8K, plus Kimi in-house ASR data and in-house audio data for AAC/AQA.
Training epochs vary by dataset; most are trained for $2.0$ epochs, while some smaller or more specialized sets receive $4.0$ epochs. The paper gives the full list in its table and uses this to build coverage across speech, sound, music, and emotion tasks.
Speech conversation data
The speech conversation dataset is constructed to teach expressive multi-turn spoken dialogue. User queries are written by LLMs and then synthesized into speech using Kimi-TTS, with prompt timbres drawn from a large set of more than $125$K voices. Assistant responses are generated in the style and emotion of a carefully recorded Kimi-Audio speaker, a selected voice actor. The recording setup covers more than $20$ styles and emotions, each further divided into $5$ intensity levels, and was directed in a professional studio.
Two auxiliary systems support this data creation. Kimi-TTS is a zero-shot TTS system that uses a $3$-second prompt to preserve timbre, emotion, and style, and is trained on about $1$M hours of automatically generated data plus reinforcement learning. Kimi-VC is a voice conversion system built on Seed-VC with source-timbre perturbation during training; it converts diverse in-the-wild speech into the Kimi-Audio speaker timbre while preserving style, emotion, and accent.
Audio-to-text chat data
To teach basic audio-conditioned chat, the paper converts open text instruction data into speech queries using Kimi-TTS. The text sources include Magpie-Pro, Magpie-MT, Evol-Instruct, Evol-Instruct-Code, Infinity-Instruct, Synthia, NuminaMath, Tulu3, OpenHermes-2.5, and OpenOrca. Before conversion, the authors filter out content that is difficult to speak naturally, such as complex math, code, tables, complex multilingual text, and overly long prompts. They also rewrite some instructions into more colloquial forms and convert some single-turn examples into simpler multi-turn conversations.
4. Training Objectives and Recipe
4.1 Pre-training formulation
The model treats each audio segment $a_i$ as a combination of discrete semantic tokens $a_i^d$ and continuous acoustic vectors $a_i^c$, and each text segment as $t_i$. The input sequence can therefore interleave audio and text in several formats, such as $a_i^c / a_i^d / t_i$, $a_i^d$, $t_i$, $a_i^c / a_i^d$, or $a_i^d / t_i$. To make the sequence lengths compatible, shorter text or audio sequences are padded with blank tokens.
The paper defines three families of pre-training tasks:
- Audio/text unimodal pre-training: next-token prediction on text only and on audio semantic tokens only.
- Audio-text mapping: ASR-style audio-to-text prediction and TTS-style text-to-audio semantic token prediction.
- Audio-text interleaving: alternations such as audio-to-semantic, audio-to-text, and audio-to-semantic-plus-text prediction to bridge modalities more tightly.
For ASR, the training sequence is of the form $\{a_1, t_1, a_2, t_2, \dots\}$ and the loss is applied only to text tokens. For TTS, the sequence is $\{t_1, a_1^d, t_2, a_2^d, \dots\}$ and the loss is applied only to audio semantic tokens. In the interleaving mixed task, the paper notes that the first few semantic-token predictions are difficult because the model must simultaneously predict text and audio; to ease this, the authors prepend $6$ blank semantic tokens before the audio prediction region.
4.2 Task weights and optimization
The task weights are reported as $1:7:1:1:1:1:2$ for, respectively, audio/text unimodal, audio-to-text mapping, and the three interleaving tasks. The model is trained on $585$B audio tokens and $585$B text tokens for one epoch. Optimization uses AdamW with a cosine learning-rate schedule from $2 \times 10^{-5}$ to $2 \times 10^{-6}$ and $1\%$ warmup tokens.
The audio LLM is initialized from Qwen2.5 $7$B. The Whisper-based acoustic feature extractor is initialized from Whisper large-v3 and is frozen for roughly the first $20\%$ of pre-training tokens before being unfrozen for joint tuning.
4.3 Supervised fine-tuning recipe
For SFT, the paper deliberately avoids task-specific switches and instead uses natural-language instructions. For each downstream task, both text instructions and audio instructions are prepared, and one is sampled at random during training. To improve robustness, the authors generate $200$ distinct ASR instructions and $30$ instructions for other tasks using an LLM. Fine-tuning on each source runs for $2$ to $4$ epochs depending on the dataset, with AdamW, a cosine schedule from $1 \times 10^{-5}$ to $1 \times 10^{-6}$, and $10\%$ warmup tokens.
The detokenizer is trained in three stages: pre-training on about $1$M hours of audio, chunk-wise fine-tuning with dynamic chunk sizes from $0.5$ to $3$ seconds, and finally fine-tuning on the high-quality single-speaker Kimi-Audio speaker recordings.
5. Inference and Deployment
5.1 Real-time speech conversation workflow
The paper describes a client-server workflow for real-time speech-to-speech conversation. The client streams user audio to the server, a VAD module decides when the user has finished speaking, a commit signal triggers model inference, and audio chunks are streamed back to the client for playback as they are generated. On the server, input audio is tokenized, system and history tokens are concatenated with the new input, the audio LLM generates output tokens, and the detokenizer converts the output into waveform chunks.
5.2 Production deployment
In the production architecture, the paper separates the system into an RTC service, an inference scheduler, and modular tokenizer, LLM, and detokenizer services. The RTC layer uses WebRTC for low-latency stable connections. The inference scheduler stores conversation history as tokens, invokes the tokenizer for each new turn, calls the LLM service to generate response tokens, and then invokes the detokenizer. This service decomposition is meant to scale parallel requests through load balancing and multiple inference instances while preserving dialogue continuity.
6. Evaluation Toolkit
A major part of the paper is not just the model itself, but also a standardized evaluation toolkit for audio foundation models. The authors argue that prior evaluations are hard to reproduce because of inconsistent metric implementations, prompt sensitivity, and unclear generation evaluation practices. Their toolkit aims to make comparisons fairer and more reproducible across understanding, generation, and conversation tasks.
The toolkit provides three key pieces. First, it standardizes WER computation following the Qwen2-Audio style normalization. Second, for tasks such as audio question answering, it integrates GPT-4o-mini as an automated judge instead of relying only on exact string matching. Third, it releases task recipes that specify prompts and inference settings, so that model comparisons can use the same evaluation protocol. The toolkit also includes speech-conversation benchmarks that assess emotional control, speaking rate, accent, empathy, and diverse styles such as storytelling and tongue twisters.
7. Experimental Results
7.1 Automatic speech recognition
Kimi-Audio is evaluated on LibriSpeech, Fleurs, AISHELL-1, AISHELL-2, WenetSpeech, and an internal ASR test set. The paper reports word error rate, where lower is better. Across the reported tables, Kimi-Audio is either best or tied-best on all listed ASR benchmarks.
| Dataset | Kimi-Audio | Notable baseline comparison |
|---|---|---|
| LibriSpeech test-clean / test-other | 1.28 / 2.42 | Qwen2.5-Omni: 2.37 / 4.21 |
| Fleurs zh / en | 2.69 / 4.44 | Qwen2.5-Omni: 2.92 / 4.17 |
| AISHELL-1 | 0.60 | Qwen2.5-Omni: 1.13 |
| AISHELL-2 ios | 2.56 | Qwen2.5-Omni: 2.56 |
| WenetSpeech test-meeting / test-net | 6.28 / 5.37 | Qwen2.5-Omni: 7.71 / 6.04 |
| Kimi-ASR internal subset1 / subset2 | 1.42 / 2.44 | Qwen2.5-Omni: 1.53 / 2.68 |
The most important takeaways are that Kimi-Audio improves substantially on the difficult English and Mandarin ASR settings, particularly on LibriSpeech, AISHELL-1, WenetSpeech, and the internal test set. The paper positions these results as evidence that the combination of large-scale audio pre-training and hybrid tokenization translates into robust transcription performance across languages and acoustic conditions.
7.2 General audio understanding
The audio-understanding evaluation covers music, sound, and speech categories on MMAU; captioning/question-answering on ClothoAQA; non-speech sound classification on VocalSound and Nonspeech7k; emotion understanding on MELD; and scene classification on TUT2017 and CochlScene. Higher is better in these benchmarks.
| Dataset | Kimi-Audio | Best reported competitor |
|---|---|---|
| MMAU music / sound / speech | 61.68 / 73.27 / 60.66 | Qwen2.5-Omni: 62.16 / 67.57 / 53.92 |
| ClothoAQA test / dev | 71.24 / 73.18 | Qwen2.5-Omni: 72.86 / 73.12 |
| VocalSound | 94.85 | Qwen2-Audio-base: 93.82 |
| Nonspeech7k | 93.93 | Qwen2-Audio-base: 87.17 |
| MELD | 59.13 | Qwen2-Audio-base: 51.23 |
| TUT2017 | 65.25 | Qwen2.5-Omni: 43.27 |
| CochlScene test / dev | 79.84 / 80.99 | Qwen2.5-Omni: 63.82 / 63.82 |
These results show a notable spread beyond speech transcription: Kimi-Audio is especially strong on sound and scene understanding, emotion recognition, and non-speech event classification. The paper interprets this as a sign that the model has learned more general acoustic representations rather than only speech-to-text mapping.
7.3 Audio-to-text chat
Audio-conditioned chat is evaluated on OpenAudioBench and VoiceBench. These suites probe instruction following, question answering, reasoning, safety, and general response quality. Kimi-Audio leads on several submetrics and is consistently competitive across the rest.
| Benchmark | Sub-metric | Kimi-Audio | Best baseline |
|---|---|---|---|
| OpenAudioBench | AlpacaEval | 75.73 | Qwen2.5-Omni: 72.76 |
| Llama Questions | 79.33 | GLM-4-Voice: 76.00 | |
| Reasoning QA | 58.02 | Qwen2.5-Omni: 63.76 | |
| TriviaQA | 62.10 | Qwen2-Audio-chat / Baichuan-chat: 55.40 to 58.70 range | |
| Web Questions | 70.20 | Step-Audio-chat: 73.00 | |
| VoiceBench | AlpacaEval / CommonEval | 4.46 / 3.97 | next best: 4.33 / 3.84 |
| SD-QA / MMSU | 63.12 / 62.17 | Qwen2.5-Omni: 57.41 / 56.38 | |
| OpenBookQA / IFEval | 83.52 / 61.10 | Qwen2.5-Omni: 79.12 / 53.88 | |
| AdvBench / Avg | 100.00 / 76.93 | Qwen2.5-Omni: 99.62 / 72.83 |
The paper highlights that Kimi-Audio is particularly strong on instruction-following and benchmarked reasoning in the audio-to-text setting. The most consistent gains appear on VoiceBench, where Kimi-Audio leads on nearly every reported metric, including safety-related adversarial prompts.
7.4 Speech conversation
For end-to-end speech conversation, the paper reports human ratings on a $1$ to $5$ scale across speed control, accent control, emotion control, empathy, style control, and an average score. Excluding GPT-4o, Kimi-Audio is best or second-best on most categories and attains the top average among the speech-oriented models compared in the table.
| Model | Speed | Accent | Emotion | Empathy | Style | Avg |
|---|---|---|---|---|---|---|
| GPT-4o | 4.21 | 3.65 | 4.05 | 3.87 | 4.54 | 4.06 |
| Step-Audio-chat | 3.25 | 2.87 | 3.33 | 3.05 | 4.14 | 3.33 |
| GLM-4-Voice | 3.83 | 3.51 | 3.77 | 3.07 | 4.04 | 3.65 |
| GPT-4o-mini | 3.15 | 2.71 | 4.24 | 3.16 | 4.01 | 3.45 |
| Kimi-Audio | 4.30 | 3.45 | 4.27 | 3.39 | 4.09 | 3.90 |
The main qualitative takeaway is that Kimi-Audio can produce speech that is more controllable and expressive than prior audio models, especially for emotion and speed. The average score remains below GPT-4o but above the other open or specialized baselines reported in the paper.
8. What Seems New or Distinctive
- Hybrid representation: semantic tokens are not used alone; continuous Whisper-derived features are added as input to improve perceptual richness.
- Dual-generation audio LLM: a shared LLM trunk branches into separate text and audio heads, enabling a single model to answer in text or generate speech tokens.
- Chunk-wise streaming detokenization with look-ahead: the paper explicitly tackles boundary artifacts in streaming speech generation.
- Large-scale curated pre-training: the authors emphasize a very large audio pre-training corpus and a carefully engineered long-form segmentation/transcription pipeline.
- Evaluation infrastructure: the paper does not only report benchmarks; it also offers a standardized toolkit and a conversation benchmark focused on controllable speech generation.
9. Limitations and Future Directions Stated by the Paper
The final section of the report is unusually explicit about open problems. Three future directions are highlighted.
- From transcription to description: the current pre-training paradigm relies heavily on ASR transcripts, which capture what is said but not richer audio properties such as emotion, style, timbre, and environmental context. The authors argue for more descriptive supervision, such as audio captions.
- Better audio representations: semantic tokens tend to emphasize transcription-oriented content, while acoustic tokens emphasize reconstruction-oriented details. A more integrated representation may be needed to fully capture both semantic and paralinguistic information.
- Reducing dependence on ASR and TTS: the paper argues that current audio foundation models are still strongly constrained by the accuracy of ASR pseudo-labels and the expressive range of synthetic speech in TTS. A future goal is to learn from native audio data more directly, without leaning so heavily on pseudo-supervision.
These limitations are framed as general issues for the field rather than implementation flaws of Kimi-Audio alone. In that sense, the paper is both a model report and a research agenda for the next generation of audio-language foundation models.
10. Bottom Line
Kimi-Audio is a unified audio-language system built around a $12.5$ Hz hybrid tokenizer, a Qwen2.5-initialized audio LLM with separate text and audio heads, and a flow-matching detokenizer designed for low-latency chunked speech synthesis. The paper’s main empirical claim is that this recipe, combined with very large-scale curated pre-training and broad SFT, yields strong performance across ASR, audio understanding, audio-to-text chat, and speech conversation. Beyond the model itself, the paper’s evaluation toolkit and deployment discussion make it especially relevant for a talking-head / conversational-AI team that needs both benchmark performance and practical streaming speech interaction.
Code & Implementation
The Kimi-Audio GitHub repository provides the official implementation, pretrained models, and evaluation toolkit corresponding to the methods described in the paper. The codebase is organized primarily under kimia_infer/ for inference-related functionalities and finetune_codes/ for fine-tuning examples and model training.
The core model implementation is encapsulated in finetune_codes/model.py, which extends a transformer-based causal language model to integrate an audio encoder based on the Whisper model. This class KimiAudioModel manages joint audio-text modeling by merging Whisper continuous acoustic features with discrete tokens, reflecting the paper’s approach of a multi-input LLM architecture outputting both textual and audio semantic tokens.
The inference API is provided in kimia_infer/api/kimia.py, where the KimiAudio class wraps model loading, prompt management, and detokenizer usage. It includes a custom generation loop combining sampling strategies for both audio and text token streams, handling the delay tokens and end-of-stream conditions as detailed in the paper. The detokenizer converts predicted semantic audio tokens back to waveforms, supporting audio generation and multi-turn conversation.
The repository contains extensive examples and instructions in the README for usage scenarios like audio-to-text transcription, audio conversational generation, and fine-tuning scripts. The modular design aligns closely with the paper’s three main components: audio tokenizer (Whisper encoder), Audio LLM (transformer with parallel heads), and streaming audio detokenizer.