MindAU

MindAU: EEG-Conditioned Facial Action Unit Editing via Dual-Stream Manifold Alignment

MindAU enables facial action-unit editing from EEG brain signals through dual-stream manifold alignment that bridges neural patterns with semantic expression spaces while preserving identity. It is the first framework for AU-level control from brain activity, with applications toward assistive communication interfaces.

• face-animation
• multimodal
• emotion

Authors: Zhenhang Li, Xin Zhou, Hao Deng, Lijun Yin

Lab: Binghamton University

Categories: cs.CV, cs.LG

Published 2026-07-01 · Updated 2026-07-01

Links

Paper & demos

• arXiv

Impact

• Semantic Scholar0 citations

Abstract

Recent brain decoding studies have made substantial progress in reconstructing externally perceived visual content from neural signals. However, using electroencephalography (EEG) recordings to guide facial expression editing remains largely unexplored and poses a distinct challenge: rather than recovering what a subject sees, it requires identifying facial-action related patterns from noisy EEG signals and grounding them in localized, identity-preserving expression edits. In this paper, we investigate EEG-conditioned facial image editing for fine-grained facial action unit (AU) control and propose MindAU, a unified framework for controlling facial AU edits from EEG signals. MindAU first learns noise-robust and AU-discriminative EEG representations through temporal masked reconstruction and AU classification supervision. It then bridges the modality gap via Dual-Stream Manifold Alignment, aligning EEG features with AU-level text semantics and identity-reduced visual displacement trajectories in the multimodal space of Qwen2.5-VL. Finally, MindAU incorporates EEG-aware Multimodal Rotary Positional Embeddings, landmark-guided reference masking, and AU-aware region supervision into a multimodal diffusion-based editor for high-fidelity identity-preserving editing. We also introduce E-CAFE, a curated benchmark for EEG-Conditioned Action-Unit Facial Editing with paired EEG-face editing samples and standardized evaluation protocols. Extensive experiments demonstrate the effectiveness of MindAU and suggest its potential as a step towards future assistive expression technologies for individuals with facial neuromuscular disorders.

---

Introduction and Motivation

Brain decoding research has made remarkable strides in reconstructing externally perceived visual content from neural signals — from static images to dynamic video — typically by aligning neural representations with visual or semantic spaces through contrastive learning or signal reconstruction. However, the vast majority of these efforts focus on passive perceptual reconstruction: recovering what a subject sees. A substantially different and largely unexplored direction is controllable facial behavior editing conditioned on brain activity, establishing a pathway from neural signals to interpretable facial actions.

This direction has particular practical relevance: EEG-conditioned facial editing may enrich EEG–face supervision, facilitate cross-modal modeling between brain activity and facial dynamics, and — looking further ahead — support assistive communication interfaces where expressive facial behavior is driven by neural signals for individuals with facial neuromuscular disorders such as facial paralysis or Bell's palsy.

Facial expressions can be decomposed into Facial Action Units (AUs), which provide a structured and interpretable control space for linking neural signals to localized facial movements. EEG-conditioned facial editing requires extracting subtle and noisy facial-action cues from EEG signals and grounding them in fine-grained, identity-preserving expression changes on a reference face — a distinctly challenging problem. To the best of the authors' knowledge, no prior work has addressed EEG-conditioned facial image editing with AU-level control.

The core challenges of EEG-driven facial expression editing are:

• Learning noise-robust and AU-discriminative EEG representations from inherently noisy, high-dimensional signals.
• Bridging the large modality gap between EEG features and the semantic space used by pretrained multimodal generative models.
• Preserving reference identity while performing fine-grained, localized expression edits.
• The scarcity of paired EEG–face editing data and the absence of standardized evaluation protocols for this new task.

To address all four challenges, the authors propose MindAU, the first unified framework for EEG-conditioned facial image editing at the action-unit level, along with E-CAFE, a curated benchmark for standardized evaluation.

Related Work

Representation Learning for EEG Signals

Existing EEG pre-training strategies fall into two broad camps. Generative methods model EEG signal distributions through raw waveform reconstruction or time-frequency codebook learning. Task-oriented approaches (often based on contrastive learning) map EEG representations into downstream task spaces such as semantic classification or cross-modal retrieval. However, neither category is explicitly grounded in facial action semantics, making them insufficient for the AU-level control demanded by MindAU. MindAU therefore introduces AU-aware EEG pre-training combining temporal masked reconstruction with explicit AU classification supervision.

Brain-Guided Image Generation and Editing

Early brain-guided visual generation used GANs to reconstruct perceived stimuli; recent approaches adopt diffusion models. Most existing works focus on passive perceptual reconstruction from fMRI or EEG recorded during stimulus viewing. Text-guided diffusion and facial editing models have achieved high-fidelity semantic editing but require explicit text or image conditions and cannot be directly driven by noisy brain signals. A few studies use brain signals as implicit guidance for image editing, but they mainly rely on neural responses to external stimuli rather than localizing facial-action-related EEG cues into specific expression edits. The closest prior work, Mind-to-Face, maps EEG to dense 3D facial geometry for subject-specific avatar synthesis; MindAU differs by targeting reference-based facial image editing with transferable AU-level control and identity preservation.

Method Overview

MindAU processes an EEG segment $\mathbf{X} \in \mathbb{R}^{C \times T}$ and a reference face image $\mathbf{I}_{ref}$ to generate an edited image $\hat{\mathbf{I}}$ that preserves the identity of $\mathbf{I}_{ref}$ while expressing the facial action implied by $\mathbf{X}$. The pipeline proceeds in three progressive stages:

1. AU-Aware EEG Pre-training — learns AU-discriminative EEG representations.
2. Dual-Stream Manifold Alignment — maps these representations to semantic embeddings and identity-reduced visual displacement trajectories.
3. EEG-Conditioned Facial Editing — conditions a multimodal diffusion-based editor on the aligned EEG tokens and the reference face for identity-preserving expression edits.

Stage 1: AU-Aware EEG Encoder Pre-training

EEG Preprocessing

For each trial, a DC offset removal step is applied, followed by a fourth-order Butterworth band-pass filter between 0.1 and 50 Hz, and channel-wise Z-score normalization. This preprocessing removes slow drift, attenuates high-frequency noise, and normalizes channel-wise signal scales before temporal masked modeling.

Temporal Masked Modeling

To learn robust representations from the noisy but preprocessed EEG signals, MindAU adopts a temporal Masked Autoencoder (MAE) backbone following the architecture of DreamDiffusion. A preprocessed EEG segment $\mathbf{X} \in \mathbb{R}^{C \times T}$ (128 channels, 48 time points in experiments) is partitioned into temporal patches, a random subset of which is masked to yield $\tilde{\mathbf{X}}$. An encoder $E_\theta$ maps $\tilde{\mathbf{X}}$ to a latent $\mathbf{z} = E_\theta(\tilde{\mathbf{X}})$, and a decoder $D_\phi$ reconstructs the masked patches. The reconstruction loss is:

where $\mathbf{M}$ is the temporal patch mask and $\odot$ denotes element-wise multiplication. A 50% masking ratio is used, trained with AdamW for 80 epochs. The encoder has 6 Transformer layers; the decoder has 4.

AU Semantic Supervision

Signal reconstruction alone lacks explicit AU-level semantic grounding. MindAU adds an AU classification objective on top of the reconstruction objective. AU prediction is formulated as a multi-class classification task over the 10 predefined AU categories. The encoder feature $\mathbf{z}$ is fed into a classification head $f_{cls}(\cdot)$:

where $y$ is the ground-truth AU label. The combined Stage-1 objective is:

This joint pre-training yields temporally contextualized and AU-aware EEG representations that serve as the basis for Stage-2 cross-modal alignment.

Stage 2: Dual-Stream Manifold Alignment

Although the Stage-1 encoder captures AU-correlated EEG features, its latent space is not directly compatible with the semantic space used by pretrained multimodal generative models (specifically Qwen2.5-VL). Dual-Stream Manifold Alignment bridges this gap through two complementary streams.

Semantic Projection Stream

An EEG Adapter converts Stage-1 EEG features into semantic tokens compatible with Qwen2.5-VL. Given the EEG feature sequence, $Q = 32$ learnable query tokens cross-attend to it through a Transformer Decoder. The resulting query features are projected into the embedding space of the frozen Qwen2.5-VL text encoder, forming the EEG semantic tokens. The pooled representation $\mathbf{z}_{sem}$ is used for global-level alignment; the full token sequence is retained as E-tokens for token-level supervision and for Stage-3 editing.

AU-specific natural language descriptions (e.g., "inner brows raised" for AU1, "lip corners pulled up" for AU12) are encoded by the frozen Qwen2.5-VL text encoder to yield the pooled text embedding $\mathbf{z}_{text}$. EEG and text embeddings are aligned using a cross-modal supervised contrastive loss:

where $P(i)$ is the set of text samples sharing the same AU label as EEG sample $i$, $\mathrm{sim}(\cdot,\cdot)$ is cosine similarity, and $\tau$ is the temperature. To preserve fine-grained token-level semantic structure beyond the pooled embedding, each EEG token $\mathbf{e}_{i,q}$ is further aligned to its most semantically relevant text token:

where $\mu_{i,l}$ is the text validity mask and $Q=32$ is the number of EEG tokens. This loss does not enforce one-to-one positional correspondence, allowing each EEG token to align with its most relevant text counterpart.

Trajectory Modeling Stream

Text supervision is categorical and cannot capture within-AU variation such as expression intensity. The Trajectory Modeling Stream addresses this by supervising EEG features with identity-reduced visual displacement trajectories.

Given visual embeddings $\mathbf{z}_{ref}$ (reference image) and $\mathbf{z}_{gt}$ (ground-truth target image) from the frozen Qwen2.5-VL visual encoder, the raw visual displacement is computed as:

Since this displacement may still contain identity-related components, the component aligned with the reference embedding is removed by projecting onto its orthogonal complement:

The resulting $\boldsymbol{\delta}_{gt}$ serves as an identity-reduced visual displacement target emphasizing expression-specific deformation relative to the reference face. The Manifold Trajectory Projector (MTP) is trained to predict an EEG-driven trajectory $\boldsymbol{\delta}_{pred}$ from the pooled EEG representation using a hybrid loss combining directional consistency and magnitude matching:

where $D$ is the feature dimension. The full Stage-2 objective is:

To provide coarse magnitude-aware supervision, the authors synthesize identity-diversified expression-transition pairs using Nano Banana 2. For each real training sample, the reference-to-target facial change (target AU and observed expression strength) is used to guide the same intended transition on four generated identities. These pairs provide an approximate prior on EEG-associated deformation magnitude across identities, helping the EEG Adapter learn identity-invariant AU deformation and coarse intensity variation before fine-tuning on real EEG–face pairs. An ablation study confirms that including this pre-training improves CSIM (74.29% → 75.33%), FID (45.22 → 43.35), and AU ACC (27.10% → 27.71%).

Stage 3: EEG-Conditioned Facial Editing

With the aligned EEG representations in hand, MindAU builds on a Multimodal Diffusion Transformer (MM-DiT) for identity-preserving facial editing, specifically adopting the LongCat-Image architecture (double-stream blocks followed by single-stream blocks).

Token Fusion and EEG-Aware M-RoPE

The following heterogeneous token types must be fused in a shared attention space:

• Q-tokens: semantic tokens from the frozen Qwen2.5-VL encoder processing a fixed text prompt and the reference image.
• E-tokens: 32 EEG semantic tokens from the frozen EEG Adapter.
• V-token$_{noise}$: the ground-truth image encoded by the VAE and perturbed with Gaussian noise (training only).
• V-token$_{ref}$: the reference image encoded by the VAE, providing identity information.

To unify these modalities, MindAU redesigns the Multimodal Rotary Positional Embedding (M-RoPE). The positional encoding contains one modality dimension (distinguishing text, EEG, ground-truth image, and reference image) and two spatial dimensions (encoding image geometry). Image tokens use their natural $(h, w)$ grid coordinates; 1D text or EEG tokens use their sequence index for both spatial dimensions.

Conditional Flow Matching Objective

The model is trained with Conditional Flow Matching (CFM). Let $\mathbf{z}_0$ denote the latent of the ground-truth image and $\mathbf{z}_1 \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$ denote Gaussian noise. The linear interpolation path and its target velocity are:

The multimodal condition set is $\mathcal{C} = \{\text{Q-token}, \text{E-token}, \text{V-token}_{ref}\}$ and the flow-matching loss is:

At inference, the model takes only the EEG segment, the reference image, and a fixed text prompt; the ground-truth image is used only during training.

Landmark-Guided Reference Masking

Training on limited data can lead to a degenerate shortcut: since the reference image accounts for most of the target-image content, the model may learn a near-identity mapping and under-utilize EEG conditions for expression control. To combat this, MindAU masks AU-relevant regions in the reference image before encoding it as a visual condition.

Masks are derived from reference-image 68-point facial landmarks. The six local subregions used are: left eye (landmarks 36–41), right eye (42–47), left eyebrow (17–21), right eyebrow (22–26), nose (27–35), and mouth (48–67). For eyes, eyebrows, and mouth, expanded elliptical masks are used; for the nose, a convex-hull mask is used. The ellipse is constructed from the axis-aligned bounding box of the landmarks, with region-specific scaling factors, center offsets, and post-expansion factors to ensure full coverage of expression-related texture. All masks are further dilated.

A progressive masking schedule is applied: at the start of training, 70% of reference images are masked and 30% are left unchanged; this ratio is linearly annealed over 3,000 steps until 30% are masked and 70% remain unaltered. This encourages early reliance on EEG conditions while gradually restoring full reference information for identity-preserving editing at inference.

AU-Aware Region Supervision

In addition to the full-image flow-matching loss, two spatial auxiliary losses are computed from the channel-averaged prediction error map in latent space. For the $b$-th sample:

A global facial-region mask is the union of all four AU-relevant subregion masks:

An AU-focused mask concentrates on the specific facial region activated by the target AU:

The full Stage-3 objective is:

E-CAFE: Benchmark for EEG-Conditioned Action-Unit Facial Editing

To support standardized evaluation of EEG-driven facial editing, the authors introduce E-CAFE (EEG-Conditioned Action-unit Facial Editing), a curated benchmark built upon the BU-EEG dataset.

BU-EEG contains 28 participants performing 10 distinct AUs (AU1, AU2, AU4, AU5, AU9, AU12, AU15, AU17, AU25, AU27), with synchronized 128-channel EEG signals and facial videos sampled at 250 Hz and 24 fps respectively. For subject-independent evaluation, the dataset is split by identity: two subjects are strictly held out as a test set; the remaining subjects are used for training.

Data construction pipeline: The Face Alignment Network (FAN) detects 68-point facial landmarks and filters out frames inconsistent with the target AU activation. AU-specific geometric scores (e.g., brow–eye distance, eye-opening height, mouth width, inner-mouth opening) further remove weak or ambiguous activations. A subsequent manual verification step removes residual failures, ambiguous expressions, and reference frames with expression leakage. For each retained target frame, the preceding 48-time-point EEG segment and a reference face image are extracted. For testing, only neutral-to-target sequences are retained to minimize expression leakage from the reference image.

The resulting benchmark contains 5,000 paired EEG–face editing samples with training and test splits. Two evaluation protocols are provided:

• Self-Referenced Editing: The reference image and EEG signal come from the same held-out identity $(I_A, E_A)$. This evaluates reconstruction fidelity on unseen subjects — whether the model can translate EEG signals into the correct facial dynamics while preserving identity.
• Cross-Referenced Editing: EEG signals from held-out subjects drive 128 synthetic identities generated by Z-Image, covering diverse ages and genders. This evaluates the transferability and disentanglement of the learned EEG representations — whether AU-related expression semantics are captured independently of the source identity.

Experiments

Implementation Details

All experiments are conducted on 8× NVIDIA RTX A6000 GPUs (48 GB memory). The three training stages incur approximately 2 GPU-hours (Stage 1), 8×36 GPU-hours (Stage 2), and 8×48 GPU-hours (Stage 3), respectively.

• Stage 1: Transformer-based asymmetric MAE with a 6-layer encoder and 4-layer decoder. EEG input dimensionality $128 \times 48$. 50% masking ratio, AdamW optimizer, 80 epochs, batch size 32.
• Stage 2: AdamW optimizer, 5,000 steps, global batch size 64. Visual features of reference and target images are pre-extracted and cached using the frozen Qwen2.5-VL vision encoder to avoid loading the heavy backbone during alignment training.
• Stage 3: Initialized from LongCat-Image pretrained parameters. Trained for 5,000 steps at $512 \times 512$ resolution with per-GPU batch size of 8. Progressive masking schedule applied as described above.

Evaluation Metrics

• FID: Fréchet Inception Distance measuring distributional discrepancy between synthesized and ground-truth images.
• CSIM: Cosine similarity between ArcFace embeddings of the generated face and the reference identity.
• AU ACC: AU detection accuracy using OpenGraphAU initialized from BP4D and further adapted on the E-CAFE training split.
• CLIP Score: Semantic consistency between generated images and the corresponding AU conditioning descriptions.

FID is reported only for Self-Referenced Editing, as no ground-truth images exist for Cross-Referenced Editing pairs.

Baselines

Since no prior method is designed for EEG-conditioned facial editing, three representative baselines are compared:

• DreamDiffusion: A strong EEG-to-image generation model fine-tuned on the E-CAFE training set; generates images directly from EEG without a reference image.
• LongCat-Image (pipeline): The EEG Stage-1 encoder's AU classification head predicts the AU category; this is converted to a fixed textual description; LongCat-Image then edits the reference image using that text prompt.
• FireRed-Image-Edit (pipeline): Same pipeline approach using FireRed-Image-Edit as the text-guided editor.

For a fair comparison, the EEG-to-AU classifier used in the pipeline baselines shares the same Stage-1 EEG encoder and AU classification head as MindAU, achieving 32.93% standalone AU accuracy on the held-out test split.

Quantitative Results

MindAU achieves the best performance on the three most task-critical metrics: identity preservation (CSIM 75.33% vs. best baseline 38.42%), image realism (FID 43.35 vs. best baseline 59.40), and expression fidelity (AU ACC 27.71% vs. best baseline 20.88%). LongCat-Image achieves the highest CLIP Score, which is expected since it uses fixed AU text prompts and CLIP mainly measures coarse text-image consistency. Text-guided editors often produce canonical or exaggerated expressions semantically aligned with the prompt but inconsistent with the actual intensity implied by the EEG signal. MindAU avoids compressing EEG signals into a discrete AU label and instead conditions the editor on continuous EEG-aligned representations for more intensity-aware, identity-preserving edits.

An ablation on EEG-free inference (replacing E-tokens with all-zero tokens) yields a dramatic drop: CSIM falls from 75.33% to 25.12%, AU ACC from 27.71% to 7.22%, and FID rises from 43.35 to 136.2, confirming that the EEG conditioning is genuinely utilized during generation.

Qualitative Results

DreamDiffusion fails to preserve the reference identity because it generates images directly from EEG without using the reference image as a condition. Text-guided pipelines better retain reference structure but suffer from two intrinsic limitations: (1) they compress EEG signals into a discrete AU label, discarding fine-grained information such as expression intensity; (2) they have no error-correction mechanism — if the EEG classifier confuses similar AUs (e.g., AU25 vs. AU27), the downstream editor is forced to follow an incorrect prompt and cannot recover. MindAU directly injects continuous EEG-conditioned tokens and produces subtler expression changes better aligned with the ground truth.

Ablation Studies

A comprehensive ablation study was conducted under both Self-Referenced and Cross-Referenced Editing protocols. The table below summarizes results.

Key findings from the ablation:

• AU classification supervision improves AU ACC and CLIP Score, confirming explicit AU labels help the EEG encoder learn expression-discriminative representations.
• Supervised contrastive alignment over direct pairwise alignment brings a large AU ACC gain (23.89% vs. 15.66%), highlighting the importance of AU-level semantic structure in the alignment objective.
• $\mathcal{L}_{tok}$ (token-level alignment) preserves fine-grained semantic structure and yields the best FID (41.87), but its flexible token matching does not enforce global AU discrimination or deformation consistency when used alone.
• $\mathcal{L}_{traj}$ (trajectory loss) introduces deformation-aware supervision and improves AU fidelity; combined with $\mathcal{L}_{tok}$, the two losses stabilize alignment by jointly preserving local token semantics and facial deformation cues.
• Stage-3 spatial losses ($\mathcal{L}_{reg}$ and $\mathcal{L}_{foc}$) further improve editing accuracy by concentrating optimization on AU-relevant facial regions. The full model achieves the best overall trade-off across CSIM, AU ACC, and CLIP Score on both protocols.
• Landmark-guided reference masking (separate ablation) is critical: without it, CSIM rises to 90.45% and FID falls to 35.12% — but AU ACC collapses to only 5.02%, revealing severe shortcut learning where the model simply replicates the reference image rather than generating the target expression.

Failure Case Analysis

Although the landmark-guided reference masking strategy reduces reference-dominant shortcut learning, the model can still occasionally under-utilize EEG conditions when the target expression is subtle or the reference and target images are visually similar. Some outputs exhibit incorrect AU generation or insufficient expression change, where the generated image remains overly close to the reference image. This suggests that fully preventing reference shortcut behavior remains challenging under limited paired EEG–face supervision.

Contributions Summary

1. Novel task formulation: MindAU is the first framework specifically designed for EEG-conditioned action-unit-level facial image editing.
2. AU-Aware EEG Encoder: Temporal masked reconstruction combined with AU classification supervision yields noise-robust, AU-discriminative EEG representations.
3. Dual-Stream Manifold Alignment: Bridges EEG representations with Qwen2.5-VL's multimodal space through AU-level text semantics (Semantic Projection Stream) and identity-reduced visual displacement trajectories (Trajectory Modeling Stream).
4. EEG-Conditioned Editing Backbone: EEG-aware M-RoPE, landmark-guided progressive reference masking, and AU-aware region supervision mitigate shortcut learning and preserve reference identity during EEG-conditioned editing.
5. E-CAFE Benchmark: 5,000 paired EEG–face editing samples built on BU-EEG with two standardized evaluation protocols (self-referenced and cross-referenced editing).

Limitations

• Healthy-participant data: E-CAFE is built from healthy participants physically performing facial actions. These recordings may contain facial electromyographic (EMG) artifacts that overlap with EEG frequency bands and are not fully removed by standard band-pass filtering. Results should therefore be viewed as a first step rather than clinical validation. Future work should study imagined or attempted facial movements in patient populations with stronger artifact-control protocols.
• Limited paired data and domain gap: Large-scale paired EEG–face datasets remain scarce, and the domain gap from controlled recordings to in-the-wild settings (variations in identity, pose, background, EEG acquisition conditions) may affect generation quality.
• Single-AU, static editing: MindAU currently focuses on single-AU image editing, whereas real expressions involve multi-AU combinations and temporal dynamics.
• Evaluation metrics: Current metrics do not fully capture fine-grained AU intensity, localized deformation accuracy, or alignment with EEG-implied expression strength. More fine-grained metrics and human perceptual studies are needed.