EventSpeech

Can We Hear from Events? Generating Speech from Event Camera

EventSpeech uses neuromorphic event cameras to generate emotionally expressive speech from text by capturing high-frequency facial articulatory dynamics at microsecond precision, eliminating motion blur and temporal mismatch inherent in frame-based visual speech synthesis.

• tts
• talking-face
• emotion
• audio-driven
• multimodal
• prosody

Authors: Jingping Fang, Lin Chen, Chenyang Xu, Tong Zhao, Weidong Cai, Xiaoming Chen

Categories: cs.MM, cs.SD

Published 2026-05-26 · Updated 2026-06-17

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub0 stars0 watchers0 forksApache-2.0

Abstract

Traditional RGB-based speech generation faces Temporal Granularity Mismatch since fixed camera exposure times inevitably blur the high-frequency articulatory transients essential for rendering emotional speech. To break this ceiling, we propose EventSpeech as a novel text-conditioned framework pioneering the use of neuromorphic events for expressive speech generation, since these microsecond-precise events naturally align with acoustic waveform dynamics. Our architecture integrates a dedicated Event Encoder to model sparse neuromorphic events alongside a multi-scale Audio Encoder featuring a Hierarchical Wavelet Contextualizer (HWC). A bidirectional alignment mechanism seamlessly synchronizes linguistic content and visual dynamics with dense acoustic features. Furthermore, we construct EVT-SPK as the first benchmark comprising large-scale synthetic data and real-world recordings from specialized neuromorphic hardware. Extensive evaluations demonstrate that EventSpeech significantly outperforms current baselines by preserving fine-grained emotions and resisting motion blur to establish a new paradigm for multimodal speech generation. Code and demo are available at https://xrfang-0102.github.io/EventSpeechWeb/.

---

Introduction and Motivation

Human speech is a fundamentally multimodal physical process. Acoustic expressiveness is tightly coupled with high-frequency facial micro-dynamics — subtle lip tremors, rapid jaw accelerations, and microexpressions that are physically entangled with the generation of high-frequency acoustic features. While the text-to-speech (TTS) community has made enormous strides with autoregressive, diffusion-based, and flow-matching architectures, and while audio-visual approaches attempt to ground synthesis in visual signals, the authors argue that a deeper, physical bottleneck has gone unaddressed: the reliance on frame-based RGB sensors.

A standard RGB camera integrates light intensity over a fixed exposure window — typically 33 ms at 30 fps. This integration acts as a low-pass filter, irreversibly smoothing out the high-frequency visual jitters that encode precisely those articulatory transients most informative for expressive speech. The paper names this the Temporal Granularity Mismatch: fixed-exposure sensors operate at a temporal scale that is fundamentally coarser than the acoustic waveform dynamics they are meant to supervise. When a model is conditioned on such blurred visual inputs, it inevitably predicts temporally averaged acoustic features, producing speech that lacks temporal crispness and emotional vitality.

To break this physical ceiling, the paper introduces EventSpeech, the first framework that leverages neuromorphic event cameras for expressive, text-conditioned speech generation. Unlike frame-based cameras, event sensors asynchronously record per-pixel brightness changes with microsecond resolution. This mechanism precisely captures facial articulatory kinematics and provides continuous motion guidance that is naturally aligned with acoustic waveform dynamics — eliminating motion blur by design rather than by post-hoc processing.

Alongside the model, the paper contributes EVT-SPK, the first benchmark for event-driven speech generation, encompassing both a large-scale synthetic corpus and real-world recordings captured with specialized neuromorphic hardware. Together, EventSpeech and EVT-SPK establish a new paradigm for multimodal speech generation grounded in the physical properties of biological speech production.

Problem Setting and Background

Temporal Granularity Mismatch

The central problem the paper addresses is that standard frame-based approaches impose a temporal granularity on visual conditioning that is incompatible with the fine-grained temporal structure of speech. A camera operating at 30 fps captures one frame every 33 ms, but acoustic events such as formant transitions, stop consonant bursts, and prosodic microstructure occur on timescales of milliseconds or less. Even high-speed cameras at 120 fps remain fundamentally limited by the exposure-readout paradigm: within each frame, rapid motion still produces intra-frame blur that destroys precisely the high-frequency articulatory information relevant to speech. The paper demonstrates this quantitatively in ablation experiments, showing that performance improvements plateau when going from 25 to 60 to 120 fps RGB cameras, whereas event-based sensing, operating at microsecond resolution, continues to improve significantly.

Neuromorphic Event Cameras

Event cameras are bioinspired sensors that record asynchronous, per-pixel brightness-change events $(x, y, t, p)$ where $x, y$ is the pixel location, $t$ is the timestamp with microsecond precision, and $p \in \{+1, -1\}$ is the polarity (brightness increase or decrease). Because events are triggered by motion rather than captured at fixed intervals, they are inherently free of motion blur and operate with extremely high dynamic range. These properties make them naturally suited to capturing the continuous kinematic dynamics of speech articulators.

Related Work

Text-Driven TTS has evolved from autoregressive sequence-to-sequence models (Tacotron, Tacotron2) through non-autoregressive architectures (FastSpeech) to modern generative frameworks using diffusion (Grad-TTS, DiffWave), normalizing flows (Glow-TTS), end-to-end systems (VITS, NaturalSpeech), and flow-matching (Matcha-TTS, F5-TTS). Despite impressive quality, these systems rely solely on text, inherently neglecting the physical grounding of speech production in facial kinematics.

Frame-Based Visual Speech Generation includes works such as VDTTS, Seeing What You're Saying, and more recent visual voice cloning and dubbing systems (VoiceCraft-Dub, HPMDubbing, StyleDubber, VTS). These leverage RGB frames to modulate prosody, emotion, and lip synchrony. However, the paper argues they all hit a physical ceiling: even with feature interpolation, the lost microsecond-scale articulatory micro-dynamics cannot be recovered from time-averaged frames.

Neuromorphic Events for Audio has been applied to environmental sound generation from object interactions and cross-modal alignment tasks. Event cameras have also been integrated into speech-related discriminative tasks such as visual speech recognition and voice activity detection. However, no prior work has addressed the generative challenge of synthesizing expressive speech waveforms directly from neuromorphic signals in synergy with textual content.

The EventSpeech Framework

EventSpeech orchestrates three synergistic modules: (1) an Event Encoder that transduces silent video into neuromorphic event streams and extracts high-frequency kinematic features; (2) a Multi-Scale Audio Encoder featuring the Hierarchical Wavelet Contextualizer (HWC) that synergizes selective state space models with wavelet analysis; and (3) a Hierarchical Alignment Module that enforces synchronization at both temporal (frame-level) and semantic (emotion-level) scales. During training, visual events provide high-fidelity motion supervision. During inference, the framework supports both text-only generation via learned alignment priors and vision-augmented synthesis when events are available.

Module 1: Neuromorphic Events Modeling (Event Encoder)

Event Generation via V2E

Given a silent video $V$ containing $T$ frames of size $H \times W \times 3$, the Event Encoder first transduces the static frames into a sparse event stream using the differentiable neuromorphic emulator V2E. The simulator converts RGB inputs to grayscale and computes the log-luminance deviation between consecutive frames. Discrete events $E^{\pm}$ are triggered at coordinates $(x, y, t)$ precisely when this deviation surpasses learnable sensitivity thresholds $\{\theta^+, \theta^-\}$ for positive or negative polarities, respectively.

Spatiotemporal Kinematic Encoding

To encode the sparse event signals into dense kinematic features, a hierarchical architecture is employed. A Spatial CNN (based on ConvNeXt) first extracts local spatial textures, reducing spatial resolution while increasing channel depth. Subsequently, bidirectional GRU (BiGRU) layers capture long-range temporal dependencies, yielding a spatiotemporal representation $H \in \mathbb{R}^{B \times T' \times 512}$, where $T' = T - 1$ is the temporal resolution adjusted for interval-based kinematic modeling.

Disentangled Visual Dynamics via MHFE

To explicitly disentangle the multifaceted visual dynamics inherent in speech production, the paper introduces five specialized Multi-Head Feature Extractors (MHFE). These project the shared spatiotemporal features $H$ into distinct latent subspaces representing:

• Lip Motion — the primary articulatory channel
• Facial Action Units (AU) — muscle group activations encoding expression
• Head Pose — global head orientation dynamics
• Speaking Rhythm — temporal pacing of utterance production
• Visual Prosody — high-level expressive variation

The extraction backbone is optimized via a hybrid supervision paradigm. Explicit physical branches (Lip Motion, AU, Head Pose) are supervised by automated pseudo-labels from OpenFace. For implicit temporal traits (Rhythm, Prosody) that lack explicit labels, an orthogonality penalty alongside temporal modeling achieves self-supervised disentanglement. The aggregated temporal features are integrated with global speaker ($F_{spk}$) and emotion ($F_{emo}$) embeddings via an MLP to produce the visual embedding $F_v$, which synergizes microsecond-level kinematic precision with high-level semantic context.

Module 2: Multi-Scale Audio Encoding with HWC

The Audio Encoder is designed to reconcile the trade-off between long-range temporal modeling and fine-grained spectral preservation. It synergizes selective State Space Models (SSM, specifically Mamba) with the proposed Hierarchical Wavelet Contextualizer (HWC).

Acoustic Feature Decomposition

The raw waveform $a$ is transformed into a mel-spectrogram via Short-Time Fourier Transform (STFT). Simultaneously, frame-level pitch, energy, and duration contours are extracted for explicit prosodic encoding. These components are individually quantized, embedded, and fused with spectral features to construct a unified prosodic embedding $F_p \in \mathbb{R}^{B \times T_a \times D}$.

Dual-Path Spectral Refinement

To mitigate generation artifacts and enforce spectral smoothness, a dual-path mechanism is introduced:

• A 1D-convolutional branch extracts local temporal consistencies $H_{\tau}$
• A global linear branch preserves global spectral coherence $H_{\omega}$

These complementary views are dynamically fused via a learnable gate $\alpha$. A global timbre embedding $e_{\text{tmb}}$ is concatenated with the fused representation to incorporate speaker identity. The final acoustic representation is:

where $\odot$ denotes element-wise multiplication, $\oplus$ represents channel-wise concatenation, $W_f$ is a linear projection, and $\tilde{F}$ denotes the enhanced context features from the time-frequency state space modeling stage.

Time-Frequency State Space Modeling

Conventional Transformers suffer from quadratic complexity with sequence length, limiting their efficacy on high-resolution audio. An SSM-based architecture (Mamba) efficiently models global acoustic context of the prosodic embedding $F_p$ with linear complexity. To complement this and recover local spectral textures potentially attenuated by state-space representations, the HWC performs multi-scale time-frequency analysis via wavelet decomposition. This synergistic design ensures the encoder captures global prosodic dependencies while retaining fine-grained formant structures, yielding the enhanced context features $\tilde{F}$.

Module 3: Hierarchical Cross-Modal Alignment

A critical challenge in event-driven speech generation is the inherent resolution mismatch between visual event frames ($T'$) and acoustic frames ($T_a$). The Cross-Modal Alignment Module addresses this at two scales.

Fine-Grained Temporal Registration

The sequence lengths of visual and acoustic features are first aligned via interpolation. A Bidirectional Cross-Attention mechanism then models dense, frame-level interactions by allowing acoustic and visual representations to serve as reciprocal queries for each other. This captures the intricate interplay between lip kinematics and phonetic content. The resulting bidirectional contexts are integrated via a linear projection to produce a unified, precisely synchronized representation.

Emotion-Aware Manifold Alignment

Frame-level alignment alone may overlook high-level semantic consistency (e.g., emotional intensity). To enforce global coherence, temporally pooled representations are projected into a shared latent space: $z_v = \Psi(\bar{F}_v)$ and $z_a = \Psi(\bar{F}_a)$. An InfoNCE contrastive loss maximizes mutual information between matched audio-visual pairs:

where $z_a^j$ is the acoustic embedding of the $j$-th negative sample in the batch, $\tau$ is a temperature parameter scaling the similarity distribution, and $N$ is the batch size. By jointly optimizing temporal and semantic alignment, the method generates speech that is both lip-synced and emotionally congruent.

Text Pipeline and Inference Architecture

Built upon the VITS dual-stream paradigm, EventSpeech establishes a rigorous mathematical connection between the training and inference latent spaces:

• Training: A Posterior Encoder extracts the target latent representation $z_{\text{post}}$ directly from ground-truth acoustic features. Concurrently, the Prior Encoder processes the aligned multimodal conditions to predict $z_{\text{prior}}$, minimizing a KL loss to approximate the posterior distribution.
• Inference: Ground-truth audio is absent. The framework bypasses the posterior branch and relies exclusively on the trained Prior Encoder to generate the latent representation.

A dedicated Knowledge Bridge maps the active latent variable into unified hidden features, which condition an Optimal Transport Conditional Flow Matching (OT-CFM) decoder. This Transformer-based decoder solves the continuous-time ordinary differential equation (ODE), projecting Gaussian noise onto the target mel-spectrogram. A pretrained HiFi-GAN vocoder converts the mel-spectrogram to the final waveform.

The joint conditioning for the Prior Encoder concatenates: the linguistic context $F_t$ (from a text encoder), the categorical emotion vector $\text{Emo}$, and the acoustic timbre embedding $e_{\text{sty}}$ derived from a reference audio clip. During text-only inference, learned alignment priors from training substitute for explicit event input, maintaining high-quality generation without a live event stream.

EVT-SPK Benchmark

A critical barrier to event-based speech research has been the absence of appropriate benchmark data. Existing datasets rely on RGB images or videos. While simulated events can address specific failure modes of RGB sensors, they inherently lack authentic physical sensor noise and illumination dynamics. EVT-SPK is introduced to address this.

EVT-SPK-Synth

The synthetic corpus is built on two carefully selected audiovisual datasets rich in emotional expression:

• RAVDESS: Emotionally expressive speech from professional actors following rigorous recording procedures, providing high-quality emotion coverage.
• MEAD: More granular emotional intensities across a larger speaker population.

These were chosen over alternatives (ESD, CMU-MOSEI) which either lack synchronous high-quality video or contain conversational rather than emotionally expressive speech. Events are synthesized from the video using the V2E differentiable emulator. EVT-SPK-Synth comprises 36,000 clips totaling 38 hours, with HD visual quality and explicit emotion annotations.

EVT-SPK-Real

To validate framework efficacy on authentic neuromorphic data, EVT-SPK-Real was captured with:

• DAVIS346 neuromorphic camera for synchronized event streams
• H3-VR high-fidelity audio recorder for premium audio capture
• Strict synchronization with drift $< 1$ ms
• Soundproof studio with DC-driven LEDs to eliminate flicker

The dataset comprises 2,800 clips (4 hours) of expressive speech from 15 actors across 7 emotions. Crucially, the recordings deliberately include demanding physical scenarios: extreme low illumination and rapid kinematic dynamics that induce severe motion blur in standard RGB capture. This explicit stress-testing validates the claimed superiority of neuromorphic sensing in conditions where RGB cameras fail most severely.

Experiments and Results

Implementation Details

EventSpeech has 113M parameters and is trained on the EVT-SPK-Synth dataset using 6 NVIDIA A100 GPUs. Optimization uses AdamW with OneCycleLR scheduling for approximately 940,000 iterations. Multi-modal augmentations are applied during training: temporal jittering for events and standard spectral perturbations for audio. At inference, a 20-step flow matching solver is used with a fine-tuned HiFi-GAN vocoder, generating 8-second audio clips with a Real-Time Factor (RTF) of 0.006 (approximately 48 ms per clip).

Baselines

The paper establishes a comprehensive set of baselines across three domains:

Text-to-Speech:

• VALL-E 2 — neural codec language model for zero-shot TTS
• MATCHA-TTS — conditional flow matching TTS

Video-to-Audio (adapted with Audio-to-Speech conversion):

• MMAudio+AS — latent diffusion audio model with speech conversion
• Diff-Foley+AS — latent diffusion foley model with speech conversion

Visual Voice Cloning and Dubbing:

• VTS — hierarchical visual representations for speech
• VoiceCraft-Dub — neural codec model for lip-synced generation
• HPMDubbing — hierarchical facial and scene prosody
• StyleDubber — multi-scale phoneme-level style learning
• VTS+VE — VTS augmented with a Video-to-Events simulator for direct cross-modal comparison

Additionally, EventSpeech-T (text-only, no event conditioning) is evaluated as an internal ablation to isolate gains attributable to neuromorphic events.

Evaluation Metrics

• MCD (Mel-Cepstral Distortion, DTW-aligned): spectral distortion vs. ground truth
• LSE-D / LSE-C (SyncNet Lip Sync Error Distance / Confidence): audio-visual alignment quality
• F0-RMSE: root-mean-square error of log-scale fundamental frequency trajectories
• MCD-SL: speaker-level normalized MCD for prosody-aware acoustic quality
• KL Divergence: between emotional-prosodic feature distributions for emotional fidelity
• WER: Word Error Rate via Whisper-large-v3 ASR for speech intelligibility
• CMOS: Comparative MOS for perceptual quality (human evaluation)
• SMOS: Speaker Similarity MOS for identity preservation (human evaluation)

Main Results: EVT-SPK-Synth

On the synthetic dataset, EventSpeech achieves state-of-the-art performance across all metrics. Notably, even the text-only variant (EventSpeech-T) surpasses several vision-conditioned models in acoustic fidelity and prosodic naturalness, validating the strength of the underlying architecture. The full model with neuromorphic event conditioning further elevates performance, particularly on metrics sensitive to fine-grained dynamics (LSE-C, F0-RMSE, KL), confirming that event representations explicitly isolate high-frequency articulatory dynamics that standard frames obscure.

Main Results: EVT-SPK-Real

On the real-world dataset, EventSpeech maintains its advantage and even widens it on several metrics. The text-only EventSpeech-T experiences expected degradation in lip synchronization due to absence of visual physiological constraints. Notably, competitors such as VTS+VE that rely on RGB representations remain unaffected by realistic sensor degradation in the synthetic evaluation but are outperformed on real data where physical event streams provide richer physiological dynamics. The full model achieves MCD 3.18, F0-RMSE 0.124, LSE-C 0.843, KL 0.108, WER 0.028, CMOS -0.24, and SMOS 4.45 — all surpassing all baselines by substantial margins.

Qualitative Results

The mel-spectrogram comparison visually confirms the quantitative results: EventSpeech produces spectrograms that more closely track the fine-grained temporal and spectral structure of ground-truth speech, with sharper formant transitions, more faithful prosodic contours, and better-preserved emotional variation compared to all baselines.

Ablation Studies

All ablations are conducted on EVT-SPK-Synth using the 113M-parameter model configuration.

Events versus High-Speed Cameras

This ablation directly validates the paper's central thesis. Performance gains plateau at 120 fps because conventional cameras remain bound by the exposure-readout paradigm and inevitably suffer from intra-frame motion blur during rapid articulation. Neuromorphic events transcend discrete temporal sampling entirely, capturing continuous biological dynamics with microsecond precision to preserve fast articulatory cues that are lost even in high-speed video.

Efficacy of the HWC Architecture

Both components contribute independently, but their combination is synergistic. Mamba captures global prosodic dependencies via selective state space modeling with linear complexity, while the wavelet module preserves local spectral hierarchies and formant fine structure. Neither alone matches the combination, confirming the complementarity of the two modeling paradigms for hierarchical acoustic encoding.

Cross-Modal Alignment Strategy

Naive concatenation and linear interpolation fail to model non-linear coarticulation, causing severe temporal misalignment. Unidirectional attention suffers from inherent visual ambiguities. Bidirectional cross-attention establishes reciprocal reasoning: visual features temporally anchor acoustic boundaries, while acoustic context resolves visual ambiguities. This mutual constraint cultivates a tightly coupled latent manifold, evidenced by minimized KL divergence and superior LSE-C.

Voxelization of Native Events

Coarse voxelization ($N=1$) degrades acoustic fidelity by discarding microsecond-level sparsity. Dense event processing ($N=6$) provides marginal quality improvements but incurs severe computational overhead (16.4 GFLOPs vs 11.2, RTF 0.23 vs 0.16). The chosen $N=3$ binning resolves this dilemma: it preserves the asynchronous temporal advantages of event cameras while remaining computationally tractable, achieving a superior quality-efficiency trade-off.

Key Contributions and Novelty

The paper makes four principal contributions:

1. EventSpeech: The first framework leveraging neuromorphic events for speech generation, resolving the Temporal Granularity Mismatch of RGB sensors and achieving microsecond-level alignment between visual kinematics and acoustic waveforms.
2. Physically-grounded human perception paradigm: Unlike traditional methods limited by RGB camera frames, the event-driven approach captures subtle motion changes more accurately and completely, establishing a new physical foundation for multimodal speech synthesis.
3. EVT-SPK benchmark: The first comprehensive benchmark for event-driven speech, featuring both synthetic (36K clips, 38 hours) and real-world (2.8K clips, 4 hours) data with explicit emotion annotations, captured using specialized neuromorphic hardware under challenging conditions including low illumination and severe motion blur.
4. State-of-the-art performance: Both objective and subjective evaluations demonstrate superior robustness against motion blur and fidelity in fine-grained prosody preservation over all competitive baselines across both synthetic and real-world datasets.

From an architectural novelty perspective, the paper introduces:

• The MHFE (Multi-Head Feature Extractor) with five disentangled visual-speech attribute streams and hybrid supervised/self-supervised training
• The HWC (Hierarchical Wavelet Contextualizer) combining Mamba SSM global modeling with wavelet-based local spectral preservation
• A dual-path spectral refinement mechanism with learnable gating between local convolutional and global linear branches
• A bidirectional cross-modal alignment combining frame-level cross-attention with InfoNCE contrastive loss at the manifold level
• A VITS-based dual-stream inference architecture with Knowledge Bridge connecting to OT-CFM for flexible text-only or vision-augmented synthesis

Limitations and Future Directions

The paper is transparent about several inherent limitations:

• Limited real-world scale: EVT-SPK-Real contains only 2,800 clips (4 hours), necessitating reliance on simulated events for the bulk of training. This means the model may not fully capture the complex dynamics of physical sensor noise from native neuromorphic hardware.
• Voxelization trade-off: Practical voxelization ($N=3$ bins) compromises some of the raw temporal sparsity that constitutes the fundamental advantage of event cameras, representing an engineering compromise rather than full exploitation of neuromorphic potential.
• Limited extreme condition coverage: Insufficient training data under extreme conditions (very low illumination, very rapid motion) restricts universal generalization to the most challenging real-world scenarios.

The stated future trajectory focuses on: (1) scaling the physical EVT-SPK-Real dataset to reduce the simulation-to-reality gap; and (2) investigating Neural Ordinary Differential Equations (Neural ODEs) to enable continuous-time modeling of event streams, which would more fully leverage the asynchronous, continuous nature of neuromorphic sensing without the discretization inherent in voxelization.

Conclusion

EventSpeech represents a fundamental paradigm shift in visually-conditioned speech generation. By identifying and addressing the Temporal Granularity Mismatch at the physical level — moving from time-averaged frame accumulation to asynchronous microsecond-precise neuromorphic sensing — the paper demonstrates that the bottleneck in expressive speech generation is not merely architectural but is rooted in the sensor physics of visual conditioning. The framework's combination of disentangled MHFE event encoding, HWC-based hierarchical acoustic modeling, and bidirectional cross-modal alignment consistently outperforms all evaluated baselines on both synthetic and real-world data, across both objective metrics and human perceptual evaluations. The EVT-SPK benchmark provides the research community with the first dedicated resource for event-driven speech generation, including authentic real-world neuromorphic recordings under challenging conditions. Together, these contributions establish neuromorphic events as a demonstrably superior visual modality for capturing the physical rhythms of biological speech production.