This Week in Conversational AI Research: June 22–28, 2026
This week pushed conversational AI toward native-streaming systems: full-duplex audio-visual agents, streaming TTS, and real-time avatars all moved closer to a single interactive loop. At the same time, speech generation papers focused on controllability, prosody, voice cloning robustness, and representation alignment for deployable voice agents.
Full-duplex and native-streaming audio-visual agents
The week’s clearest through-line was the push from cascaded voice pipelines toward models that reason, listen, speak, and render visual behavior under a shared streaming state. A useful starting point is the new full-duplex survey, which gives builders vocabulary for distinguishing scheduler-level duplexing from representation-level duplexing, and separates architectural claims from actual interaction behavior using an L0–L3 hierarchy, a temporal/intent/response ontology, and a five-state decision machine.
The most ambitious system paper is Wan-Streamer, which frames real-time audio-visual interaction as a single Transformer sequence over input and output text, audio, and video tokens. Its block-causal attention, causal encoders/decoders, and 160 ms streaming units are designed so response timing, turn management, multimodal synchronization, and generation are learned jointly rather than delegated to VAD, ASR, LLM, TTS, and avatar modules.
Moshi-Face takes a more incremental but highly relevant route: start from a full-duplex spoken dialogue model and add a discrete face-token stream. The paper trains a VQ-VAE face codec over 3D head meshes, then adds a non-autoregressive Face Transformer so Moshi can emit synchronized speech and facial motion without giving up the low-latency dialogue behavior of the audio-only model.
For avatar video, InteractiveAvatar focuses on the long-horizon failure modes of streaming talking-head generation: identity drift, visual inconsistency, and shallow user-intent handling. Its Long-Short Visual Memory compresses recent and older visual history, while the Reasoning-Reaction Module uses an LLM-driven action/stable-state cycle to make the avatar respond with intent-aligned speech and actions rather than simply lip-syncing audio.
The TTS side of the same latency problem appears in S5-TTS, which attacks a common bottleneck in cascaded LLM→TTS agents: the language model may stream tokens, but many TTS systems still wait for full context. S5-TTS uses monotonic alignment, lookahead-causal masks, and distillation to synthesize word by word with limited future text while preserving zero-shot speaker similarity.
Together these papers sketch two competing implementation paths for interactive agents:
- End-to-end native streaming, as in Wan-Streamer, where all modalities are modeled in one causal state.
- Composable streaming modules, as in S5-TTS, Moshi-Face, and InteractiveAvatar, where each component is made streamable and then coordinated around shared timing constraints.
The interesting engineering question is no longer whether sub-second interaction is possible; it is where the system should learn interruption, backchanneling, facial response, and action timing.
Universal audio generation and natural-language control
Speech generation is becoming less tied to rigid metadata slots. Bagpiper-TTS is the most explicit example: it treats a user’s natural-language request as the front door, reasons into a rich caption containing transcript and acoustic metadata, and then synthesizes from that blueprint. This turns tasks like multi-talker dialogue, role-play, intent-to-speech, and singing voice synthesis into variants of one planning-caption-generation workflow.
AudioCALM approaches universality from the modeling side. Instead of generating discrete codec tokens with a softmax, it extends autoregressive language modeling to continuous audio latents using a flow-matching head. Its AR-Flow attention is causal across blocks but bidirectional within the active denoising block, while the Asymmetric Mixture-of-Modality-Experts gives speech a dedicated residual expert so speech transcripts do not dominate the more diffuse conditioning needs of sound and music.
ZONOS2 is the scale-and-release story of the week for open TTS: an 8B-parameter, 900M-active MoE decoder over DAC tokens, trained on more than 6M hours, using byte-level UTF-8 text instead of phonemes and a compact speaker-embedding prefix for zero-shot cloning. The report is especially relevant because it keeps streaming latency in view while scaling naturalness, prosody, multilinguality, and voice similarity.
At the scene level, ScenA moves beyond clean turn-by-turn speech synthesis by conditioning an in-the-wild text-to-audio foundation model on multiple reference voices and a free-form scene prompt. Its key technical fix is the Reference Shortcut analysis: under ordinary noise schedules, the model can bind speakers by acoustic similarity instead of obeying the text prompt, so ScenA biases training toward high-noise timesteps to force prompt-based speaker assignment.
A smaller but conceptually useful generation paper asks whether discriminative speech models can be reused as generators. Score Subnet freezes a noise-conditioned speech classifier and trains only a lightweight score decoder on its intermediate features and JEM-style gradient taps, reducing the need for a separate large U-Net score model in classifier-guided diffusion.
The shared direction is a move from narrow “text plus fixed controls” TTS toward instruction-following audio generation: prompts may specify speakers, intent, acoustic scene, role, emotion, or music-like structure, and the model must map that underspecified request into an audio plan.
Controllable TTS: voice, prosody, accent, clarity, and post-training
A strong theme in TTS was control without brittle disentanglement. FineCombo-TTS uses reference speech as an acoustic baseline and text descriptions as relative edit instructions, then learns a Conditional Flow Matching speech-variance predictor over a unified attribute embedding. The important detail is that it learns reference-to-target transformations from the paired FineEdit dataset instead of relying on hand-separated timbre, prosody, and emotion factors.
RTFree-F5 removes a practical failure point in zero-shot flow-matching TTS: the need for a reference transcript. By projecting self-supervised speech features into F5-TTS’s text-conditioning space, it reuses the pretrained checkpoint while avoiding ASR errors and text/acoustic mismatch for atypical speakers, with especially large gains on dysarthric speech.
For preference alignment, FlowTTS-GRPO shows how to run online RL directly on flow-matching TTS. It turns deterministic ODE sampling into stochastic SDE trajectories for exploration, then optimizes multi-objective rewards for speaker similarity, intelligibility, and perceptual quality; the paper’s practical findings about reward weighting, omitting CFG during training, and hard-case synthesis are likely to be useful beyond the two evaluated backbones.
Clarity control appears in LombardTTS, which separates vocal effort from articulation and allows both utterance-level and word-level emphasis. That distinction matters for agents in noisy rooms or accessibility settings: louder is not always clearer, and the paper’s pseudo-labeled flow-matching setup gives a way to increase intelligibility without collapsing everything into one Lombard style knob.
For expressive prosody, OscillaTTS proposes an adaptive oscillatory activation inside a StyleTTS2-like diffusion decoder. The activation keeps a linear bypass for stability while adding learnable periodic modulation, aiming to represent sharp pitch and energy transitions better than fixed periodic nonlinearities such as Snake.
Accent controllability gets a dedicated treatment in CrossAccent-TTS, which uses a Perceiver bottleneck plus gradient-reversal accent suppression to separate speaker identity from accent cues, then reintroduces accent explicitly through a learned language/accent embedding. Its Accent Intensity Controller makes accent a continuous interpolation variable rather than a discrete label.
Two front-end and transfer papers provide useful cautions. The Japanese G2P benchmark shows that strong LLMs can outperform traditional morphological analyzers when prompted to perform parsing plus rule-based kana conversion, and that feeding LLM-predicted kana into a kana-input TTS can improve pronunciation over end-to-end text input.
But the phoneme-addition transfer study warns that pretraining does not magically solve new phoneme acquisition: fine-tuning improves naturalness, yet learning unseen phonemes can require as much or more target data than training from scratch. For low-resource TTS builders, that separates “sounds good” transfer from actual inventory expansion.
Codecs and voice conversion under the leakage–utility trade-off
Streaming voice conversion exposes a hard constraint: if the content representation removes too much information, the decoder needs future context or explicit side channels; if it keeps too much, source timbre leaks through. Zero-VC addresses this by using Speaker Anonymization as the training perturbation, arguing that anonymization naturally preserves linguistic and prosodic utility while suppressing speaker identity. That lets the model use a strictly causal decoder with zero lookahead and a 20 ms frame floor.
SDP-Codec works at the codec level. It separates a global speaker branch from local content/prosody tokens, injects normalized F0 into the local stream, and reconstructs pitch with a global-conditioned decoder so absolute pitch range can live outside the token bottleneck. The result is a low-bitrate representation with reduced speaker-probing accuracy and useful zero-shot VC behavior.
ProsoCodec offers a complementary formulation: prosody is not treated as a clean independent stream but as a conditional residual after accounting for text and speaker embeddings. By conditioning the encoder and decoder with text and speaker prefix tokens, the discrete bottleneck is pushed toward intonation and rhythm that are not explained by content or target identity.
These three papers are valuable because they avoid the simplistic goal of “remove speaker information.” For conversational systems, the real goal is more surgical: remove source timbre while preserving timing, emotion, emphasis, rhythm, and intelligibility under latency constraints.
Speech LLMs and acoustic grounding for voice agents
For task-oriented voice agents, CORTIS asks a very practical question: can you adapt a spoken language model to emit structured outputs using only text-form task supervision? By freezing speech modules and fine-tuning the language side on text prompts and JSON/function-call targets, it enables speech-to-structure inference without task-specific speech-target pairs, and becomes more attractive than matched ASR→LLM cascades under acoustic degradation.
CAAD targets a different weakness: SLMs can over-trust linguistic priors and under-use audio. It distills the effect of contrastive decoding — audio-aware logits minus text-only logits — into a student model using synchronized teacher forcing over a shared pseudo-ground-truth sequence, aiming to get stronger acoustic grounding without dual-path inference-time cost.
The interpretability paper on interleaved speech-text LMs gives a mechanistic clue for why text remains so central. Logit-lens probes show that interleaved SLMs often pass through an implicit transcription phase: intermediate layers make the spoken word’s text token decodable, then the model predicts the next word in text space before mapping back to speech units.
Finally, the translation-enhanced speech encoder study argues that ASR-pretrained encoders can be structurally mismatched with frozen LLMs because they preserve language-specific spaces. Adding bidirectional speech-translation objectives during encoder pretraining encourages more language-agnostic representations, improving downstream Speech LLM integration.
Taken together, these papers suggest that production voice agents will need more than better transcriptions. They need speech encoders and adaptation recipes that preserve acoustic evidence, align with LLM semantic spaces, and still output reliable structured actions.
Robust ASR and self-supervised speech representations
DASH improves noisy ASR by aligning clean and noisy views through prototype assignments at multiple encoder layers. Instead of directly regressing hidden states, it distills distributions over learned acoustic prototypes, encouraging noise-invariant representations from low-level acoustics through higher-level semantics with little extra training cost.
S-JEPA revisits the masked-prediction recipe for SSL speech and replaces hard cluster IDs with soft GMM posteriors. That preserves ambiguity near acoustic boundaries and removes the need for offline corpus re-clustering, while producing a compact encoder that performs strongly on SUPERB ASR and emotion tasks.
For streaming encoders, Online Predictive Coding regularizes online registers in dual-mode SSL models by making them predict future offline representations. Paired with dual-mode LayerNorm, this narrows the online/offline WER gap without adding algorithmic latency.
InterAligner focuses on a newer ASR architecture class, Aligner-Encoders, where the encoder must internally form monotonic token alignment without decoder attention or an RNN-T lattice. Adding intermediate CTC and intermediate Aligner losses turns alignment into a progressive depth-wise curriculum, with the largest gains on long utterances.
For multilingual deployment, BLoRA shows that code-switching adaptation is not just a synthetic-data problem. Bayesian factorized LoRA integrates English-German switching knowledge into a strong Whisper baseline while preserving monolingual performance, reducing errors on switched words where ordinary fine-tuning tends to overwrite existing capability.
Low-resource ASR gets a transcription-free adaptation stage from the audio-image alignment paper. Using Vaani’s picture-prompted audio-image pairs, the method aligns a pretrained FastConformer audio encoder to frozen vision embeddings before supervised ASR fine-tuning, improving downstream ASR while keeping inference audio-only.
Across these papers, robustness is being engineered into the representation stack: through clean/noisy consistency, soft targets, predictive streaming registers, progressive alignment, Bayesian adapters, and cheap multimodal supervision.
Digital humans: physical interaction, relighting, single-image avatars, and motion
PIAvatar addresses a limitation that visually faithful avatars still often have: they do not physically interact well. Its key idea is to decouple commanded kinematic motion from the deformation gradient in an MPM simulation, preventing user-driven pose changes from creating spurious internal stress while still allowing real contact deformation with objects and other avatars.
Generative Relightable Avatars combines explicit 3D control with generative refinement. It optimizes UV-space material parameters for physics-grounded relighting, refines pose- and illumination-dependent textures with a feed-forward network, and then uses video-to-video diffusion for high-detail, temporally coherent full-body renderings under novel environment maps.
FiCA pushes toward instant avatar creation from a single portrait. It uses human-centric vision foundation models to extract partial UV observations, a diffusion model to complete texture and geometry, and a feed-forward refinement network before decoding the result into real-time 3D Gaussian Codec Avatars — without per-subject test-time optimization.
For full-body motion, MotionMAR reconstructs poses from sparse head and hand tracking by treating motion as a temporal hierarchy. A multi-scale VQ-VAE tokenizes coarse-to-fine motion, an autoregressive network predicts scales sequentially, and scale-aware control keeps the generated body aligned with actual VR/AR observations.
The connection to conversational AI is direct: believable agents need more than a synced mouth. They need persistent identity, relightable appearance, plausible body motion, and physical reactions that remain stable over long interactive sessions.