Representation Engineering (RepE)

Representation Engineering: A Top-Down Approach to AI Transparency

Representation Engineering (RepE) focuses on population-level neural representations for AI transparency. It offers tools to read and control high-level cognitive features in neural networks, enabling better understanding and safer behavior control across areas like honesty and bias.

• llm
• dialogue
• agent

Demos

The demos showcase Representation Engineering (RepE) techniques that interpret and manipulate population-level representations in large language models, enhancing transparency and safety. Watch for how RepE improves control over behaviors like truthfulness and memorization, demonstrating a top-down AI transparency approach. Visuals highlight rep-reading and rep-control pipelines applied to neural networks for AI safety.

Authors: Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, Dan Hendrycks

Categories: cs.LG, cs.AI, cs.CL, cs.CV, cs.CY

Comment: Code is available at https://github.com/andyzoujm/representation-engineering

Published 2023-10-02 · Updated 2025-03-03

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub1,010 stars28 watchers129 forksMIT

Abstract

In this paper, we identify and characterize the emerging area of representation engineering (RepE), an approach to enhancing the transparency of AI systems that draws on insights from cognitive neuroscience. RepE places population-level representations, rather than neurons or circuits, at the center of analysis, equipping us with novel methods for monitoring and manipulating high-level cognitive phenomena in deep neural networks (DNNs). We provide baselines and an initial analysis of RepE techniques, showing that they offer simple yet effective solutions for improving our understanding and control of large language models. We showcase how these methods can provide traction on a wide range of safety-relevant problems, including honesty, harmlessness, power-seeking, and more, demonstrating the promise of top-down transparency research. We hope that this work catalyzes further exploration of RepE and fosters advancements in the transparency and safety of AI systems.

---

1. Problem framing and core thesis

The paper argues for representation engineering (RepE) as a top-down alternative to neuron- and circuit-centered transparency work. Rather than treating individual neurons, edges, or circuits as the primary explanatory objects, RepE treats representations—population-level directions, subspaces, and transformations in hidden-state space—as the main unit of analysis. The authors motivate this shift using the Hopfieldian view from cognitive neuroscience, where cognition is explained by distributed patterns of activity rather than isolated units.

The central claim is not that low-level mechanistic interpretability is wrong, but that it is incomplete for high-level phenomena in large language models (LLMs). RepE is positioned as a practical way to monitor and manipulate concepts such as truthfulness, honesty, utility, power-seeking, probability, risk, emotion, harmfulness, bias, memorization, and factual associations. The paper’s empirical contributions are intentionally broad: it introduces baseline methods for representation reading and representation control, then demonstrates these on a range of safety-relevant behaviors.

The abstract and introduction emphasize that RepE is intended to support both understanding and control. In the paper’s framing, reading representations can expose what the model internally tracks, while controlling representations can nudge behavior in targeted ways. The authors explicitly connect this to AI safety: if we can read and steer the internal representations of harmful concepts or dangerous dispositions, then we gain traction on deception, hallucination, power-seeking, and other failure modes.

2. Conceptual framing: bottom-up vs. top-down transparency

The paper contrasts RepE with mechanistic interpretability. Mechanistic interpretability is treated as a bottom-up project: identify neurons, edges, or circuits and compose them into explanations. RepE instead starts from the high-level phenomenon and asks what representation in hidden-state space corresponds to it. This is compared to the distinction between the Sherringtonian and Hopfieldian views in neuroscience. The authors also cite the broad systems intuition that complex phenomena often require analysis at the appropriate level of abstraction rather than pure reductionism.

In the appendix, they sharpen the contrast as follows: bottom-up work emphasizes composition, microscopic parts, neurons/circuits/mechanisms, and mechanistic explanations; top-down work emphasizes decomposition, macroscopic variables, representations, and functional explanations. Their claim is not that one supersedes the other, but that a mature transparency agenda should support both.

3. Representation Engineering: the method

RepE is decomposed into two main tasks: representation reading and representation control. The paper focuses on LLMs, although it also includes one CLIP example in the appendix.

3.1 Representation reading

Representation reading seeks a direction in hidden-state space that aligns with a concept or function. The paper introduces a new baseline called Linear Artificial Tomography (LAT), intentionally borrowing the structure of neuroimaging: (1) design a stimulus and task, (2) collect neural activity, and (3) fit a linear model. The goal is to use a simple linear readout to extract a concept such as truthfulness or probability, or a function such as lying or power-seeking.

The stimulus design differs for concepts versus functions. For concepts, the model is prompted to assess the amount of a target concept in a stimulus. For functions, the paper uses paired prompts that elicit the target function and a reference condition that does not. The authors emphasize an unsupervised setting by default: they prefer unlabeled or self-generated stimuli, which avoids annotation bias and is useful when trying to extract potentially superhuman representations.

For decoder models, the default extraction point is usually the last token before prediction. For functions, representations are collected across the response tokens, since the function is engaged during generation. Formally, if $M$ is the model and $\operatorname{Rep}(M, x)$ returns tokenwise hidden states, then for a concept $c$ they construct a set

$$A_c = \{\operatorname{Rep}(M, T_c(s_i))[-1] \mid s_i \in S\},$$

and for a function $f$ they collect experimental and reference activations

$$A_f^{\pm} = \{\operatorname{Rep}(M, T_f^{\pm}(q_i, a_i^k))[-1] \mid (q_i, a_i) \in S,\ 0 < k \le |a_i|\}.$$

The third step is to fit a linear model to these activations. The paper mainly uses PCA in an unsupervised way, often on pairwise difference vectors. For concepts, the input is differences between paired concept activations. For functions, the input is alternating differences between experimental and reference activations. The first principal component is used as the reading vector $v$. Prediction is simply a dot product between the vector and a new representation: $\operatorname{Rep}(M, x)^\top v$.

The paper also explicitly evaluates reading vectors using four experimental lenses: correlation, manipulation, termination (lesion/removal), and recovery (rescue/reintroduction). This is important because correlation alone is not treated as evidence of causal relevance.

3.2 Representation control

Representation control aims to modify internal representations using the directions discovered during reading. The paper introduces three baseline controllers:

• Reading vector: directly add or subtract a fixed vector, independent of the specific input.
• Contrast vector: compute a stimulus-dependent vector by contrasting activations under paired prompts at inference time.
• LoRRA (Low-Rank Representation Adaptation): train low-rank adapters so the model’s internal representations match target representations derived from contrast vectors and optionally reading vectors.

The control operators are simple and explicitly linear:

$$R' = R \pm v,$$

$$R' = R + \operatorname{sign}(R^\top v)\, v,$$

$$R' = R - \frac{R^\top v}{\|v\|^2}v.$$

These correspond to linear combination, a piece-wise conditional operation, and projection, respectively. The paper’s practical message is that reading vectors are useful for analysis, but not always the best controllers; contrast vectors and LoRRA are often more effective, especially when the goal is to alter behavior rather than merely identify it.

3.3 What makes LAT work in practice

The paper repeatedly emphasizes that the prompt template matters. For many tasks, simply feeding raw stimuli is weaker than using the LAT template that asks about the target concept directly. The authors also note that random pairings can work surprisingly well, but better pairs often improve separability. For concept extraction, the most salient token may be the concept token itself or the last token in the prompt; for some tasks, concept-token hidden states are more robust to misleading prompts than last-token states.

In the appendix, the authors provide a step-by-step PCA construction procedure: pair stimuli, extract hidden states at the chosen token position, compute normalized differences, fit PCA, and use the first component as the direction. For sign selection, they use the PCA score direction consistent with the binary label or concept ordering.

4. In-depth case study: honesty and truthfulness

The paper’s strongest and most developed case study is honesty. The authors separate two related but distinct notions: truthfulness and honesty. Truthfulness is about whether the output matches external reality. Honesty is about whether the output matches the model’s own internal beliefs. This distinction matters because a model can know the truth internally yet still produce false outputs, especially in the presence of incentives or deceptive behavior.

4.1 Reading a truthfulness representation

The paper uses LAT on true/false statement datasets and on traditional QA benchmarks to show that models possess a consistent internal concept of truthfulness. They test on OpenbookQA, CommonsenseQA, RACE, and ARC. The important result is that LAT outperforms few-shot prompting on all of these benchmarks, suggesting that the internal representation is more informative than the answer surface form.

The paper reports average accuracy across the three sizes of 48.4 versus 59.2 on OBQA, 67.9 versus 68.7 on CSQA, 84.6 versus 86.4 on ARC-e, 59.9 versus 65.7 on ARC-c, and 49.5 versus 60.3 on RACE for few-shot versus LAT. They also compare to the CCS baseline on an encoder-only model and report a clear improvement: average 69 for CCS versus 82 for LAT, with gains on COPA, RTE, BoolQ, QNLI, PIQA, and Story Cloze.

On TruthfulQA MC1, the standard zero-shot baseline performs poorly, while LAT substantially improves accuracy. The paper reports three stimulus sources: ARC-Challenge examples, self-generated examples, and QA primer examples. Across these, LAT significantly exceeds both standard zero-shot scoring and a verbalized heuristic baseline.

The paper’s interpretation is that larger models have better internal models of truthfulness, but standard generation can still be dishonest because the model’s answer policy diverges from its internal belief state. This distinction becomes central to the honesty section.

4.2 Honesty versus truthfulness

The authors define honesty as matching the model’s output to what it internally believes. A model may be truthful but not honest, or honest but factually wrong. They argue that many truthfulness evaluations only test output factuality and therefore miss deceptive behavior. Their analysis suggests that larger models can improve on truthfulness while still showing room for dishonesty, which makes honesty a more direct target for safety analysis.

4.3 Extracting and monitoring honesty

For honesty, the paper uses a dataset of true statements and prompts the model to be honest or dishonest. The resulting LAT reading vector achieves over 90% accuracy in distinguishing held-out honest versus dishonest cases. The authors then run temporal scans over generations from Vicuna-33B-Uncensored and visualize layer-by-token honesty activations. The detector is able to flag long scenarios where the model appears to be reasoning about lying, not just the final lied-upon token.

One important limitation appears in the qualitative analysis: the detector can light up on phrases associated with the act of lying or with suspicious reasoning, not only on the exact lie itself. The authors explicitly note that this complicates interpretation and suggests a need for more nuanced future evaluation.

4.4 Controlling honesty

Once a direction is extracted, the authors test whether it can be used to steer the model. They add the honesty vector to encourage honest answers and subtract it to induce lying. In their examples, this is sufficient to flip behavior in both directions. They then evaluate TruthfulQA MC1 again under representation control.

The contrast vector is the best performing control method on TruthfulQA in the paper’s experiments, but it requires more inference compute because the contrast has to be computed at runtime. LoRRA trades some of that peak performance for much lower inference overhead because the control is baked into low-rank adapters.

5. Ethics, utility, probability, risk, and power

The paper broadens RepE beyond honesty and truthfulness to a set of safety-relevant ethical and epistemic concepts. The authors show that LLMs appear to learn coherent internal structure for utility, morality, power, probability, and risk. They also argue that some of these are compositional: for example, risk can be approximated from internal representations of utility and probability.

5.1 Utility as an internal concept

For utility, the authors use the ETHICS utilitarianism task. The model is shown pairs of scenarios that differ in utility, and the task template asks about happiness. The paper illustrates that the first PCA component often dominates the variance, suggesting a strong low-dimensional structure. Their quantitative evaluation on utility shows that concept reading works well and that prompt design matters: using the LAT template is substantially better than feeding raw stimuli alone.

They compare several linear models on utility, including prompt difference, PCA, K-means, mean difference, and logistic regression. The key ablation point is that correlation is not enough: logistic regression can score well as a correlational probe yet have little causal effect when used for manipulation or termination. By contrast, unsupervised directions such as PCA and K-means, and the supervised mean-difference direction, show stronger causal leverage.

5.2 Morality and power-seeking

For morality, the paper uses the ETHICS commonsense morality task. For power, it uses the Machiavelli-derived power dataset, grounded in French’s power ontology and annotated across categories such as coercive, reward, legitimate, referent, expert, informational, economic, political, military, and personal power. The authors report that the reading vectors can monitor morally significant and power-related behavior and that the function-style prompt works better for tracking these tendencies than concept-only prompts.

In control experiments on the MACHIAVELLI benchmark, LoRRA is used to encourage or suppress power-seeking and immoral tendencies. The reward scores remain broadly similar, while the power and immorality scores shift in the intended directions.

5.3 Probability, risk, and compositionality

The paper also applies LAT to probability and risk. Using synthetic pairwise datasets generated with prompting, the authors ask the model to distinguish higher from lower probability or risk examples. They also define risk in terms of expected negative utility,

$$\operatorname{Risk}(s,a) = \mathbb{E}_{s' \sim P(s' \mid s,a)}\left[\max(0,-U(s'))\right],$$

and then use the extracted utility and probability directions to compute an approximate risk signal. Empirically, the directly extracted risk direction and the composed risk signal are positively correlated, especially in early layers, which the authors interpret as evidence that high-level internal concepts can be built compositionally from simpler primitives.

The summary table below reports LAT accuracy on five concept tasks.

6. Additional frontiers of RepE

The paper then demonstrates that the same methodology applies to several other topics: emotion, harmless instruction following, bias and fairness, knowledge editing, and memorization. These sections are best read as proof-of-breadth rather than full problem solutions. They show that the same representational tools can surface surprisingly coherent structure in diverse settings.

6.1 Emotion

The authors generate more than 1,200 short scenarios using GPT-4 to elicit six basic emotions: happiness, sadness, anger, fear, surprise, and disgust. When hidden states are visualized with t-SNE, distinct clusters emerge across layers, and even mixed emotions can be separated. LAT then extracts emotion directions that classify emotional arousal well. They also use these vectors for manipulation, showing that raising happiness in LLaMA-2-Chat can make it more willing to comply with harmful requests.

The appendix also includes a CLIP experiment on the Ferg-DB character dataset using the openai/clip-vit-base-patch32 model, with 512 training images and 128 test images.

6.2 Harmless instruction following and jailbreak robustness

The paper studies whether models’ apparent willingness to comply with harmful instructions can be understood through an internal harmfulness representation. Using Vicuna-13B, the authors extract a harmfulness direction from harmful versus benign instructions drawn from AdvBench and ShareGPT. The vector classifies held-out harmful/harmless examples with over 90% accuracy even when the model is subjected to manual jailbreaks or adversarial suffixes, suggesting that the internal harmfulness concept is robust to these perturbations.

For control, the paper compares a linear combination approach with a piece-wise transformation that conditionally amplifies or suppresses harmfulness-related activity. The piece-wise operator provides the best tradeoff between helpfulness and harmlessness under both manual and automatic jailbreaks.

The paper’s interpretation is that making harmfulness more salient can improve refusal behavior, but also that naive linear steering can overcorrect and over-reject harmless prompts.

6.3 Bias and fairness

Bias is treated both as a representation-reading problem and a control problem. The paper uses StereoSet to extract a bias direction and then shows that the direction can be used to alter outputs in a way that reduces stereotyped behavior. A key claim is that RLHF-style refusal behavior may conceal bias rather than eliminate it: the model can still express bias when prompted differently or attacked with adversarial suffixes. LAT, by contrast, is used to expose and then steer the underlying representation.

The authors also argue that a vector derived from racial bias examples transfers to gender and occupation bias, suggesting that the direction captures a more unified representation of bias. In the appendix, they show a medical example on sarcoidosis, where controlled generations reduce disproportionate mentions of black female patients.

6.4 Knowledge editing and non-numerical concepts

The paper shows that RepE can also be used for a concrete fact-editing task: changing the model’s internal representation of the fact that the Eiffel Tower is in Paris to the fact that it is in Rome. The same framework is also applied to a non-numerical concept, “dogs,” using Alpaca-style instruction-tuning prompts. The authors claim counterfactual control over generations in both directions: activate or suppress dog-related text, and edit the Eiffel Tower location fact while preserving specificity on related facts such as the Louvre’s location.

6.5 Memorization

The memorization section asks whether RepE can detect and reduce regurgitation of memorized content. The authors consider two paired classification settings: popular versus synthetic quotes, and popular versus synthetic literary openings. LAT directions transfer reasonably well across these settings, indicating that the directions are correlated with memorization rather than just one narrow dataset.

For control, they evaluate completions on more than 100 partially completed well-known quotes. Subtracting the memorization direction sharply lowers exact match and embedding similarity, while adding the direction or using a random vector has little effect. Importantly, the model’s general knowledge remains mostly intact: accuracy on historical-year questions drops only slightly from 97.2% to 96.2% after memorization reduction.

7. Implementation details, ablations, and robustness checks

The appendix provides several practical details that matter for reproducibility. For TruthfulQA control, the authors use a linear-combination coefficient of 0.25. For the 7B model they edit layers $8$ through $31$ in steps of $3$; for the 13B model they edit layers $10$ through $39$ in steps of $3$. For LoRRA, they use a learning rate of $3 \times 10^{-4}$, batch size 16, rank 8, and train for 40 to 80 steps, choosing checkpoints using ARC-Easy validation. They attach adapters to query and value weights in selected middle layers, rather than to the entire network.

The paper also performs a robustness analysis against misleading prompts. On Utility, ARC, and TruthfulQA, LAT scores are less affected by biased instructions such as “I think the answer is X, but I’m curious to hear what you think” than zero-shot generation probabilities. In addition, using concept-token hidden states is sometimes more robust than using the final token alone. The appendix also compares the LLaMA-2 base and chat models and finds that the chat model retains a more salient truthfulness representation across middle and late layers.

The broader robustness message is that hidden-state directions can be more stable than raw generation probabilities, but they are not immune to superficial feature leakage. The paper explicitly notes that some PCA runs can latch onto spurious correlates if the sampled stimuli are not well chosen.

8. Limits and caveats stated by the paper

The paper is ambitious but still frames its methods as baselines and early steps. Several limitations are explicit in the text:

• Correlation is not causation. A direction that predicts a concept may not causally control it. The utility experiments are used specifically to show that some readouts are correlational only.
• Prompt and token sensitivity. LAT performance can depend on the exact template, the stimulus choice, and whether the concept token or last token is read out.
• Superficial features can leak into PCA. The appendix warns that a small fraction of runs can underperform because PCA captures other shared properties in the stimulus set.
• Some detectors are semantically broad. The honesty and immorality monitors can trigger on associated phrases or intentions, not only on the target property itself.
• Inference overhead varies by controller. Contrast vectors require extra inference compute because the contrast must be computed dynamically; LoRRA is introduced to reduce this cost.
• Results are baseline-level and model-specific. Most experiments are on LLaMA-2-Chat, Vicuna, DeBERTa, or CLIP, so generalization beyond the evaluated systems remains an open question.

The conclusion is correspondingly careful: the paper claims traction, not a complete theory. It suggests future work on trajectories, manifolds, and state spaces of representations, while emphasizing that top-down analysis should complement, not replace, lower-level mechanistic work.

9. Bottom line

The main contribution of this paper is a coherent and empirically grounded argument that representations are a first-class object for AI transparency. LAT provides a simple unsupervised baseline for reading concepts and functions from hidden states. Linear combination, piece-wise steering, contrast vectors, and LoRRA show that these representations can also be controlled. Across honesty, utility, power-seeking, emotion, harmfulness, bias, knowledge editing, and memorization, the authors demonstrate that many safety-relevant phenomena are not only detectable but also steerable at the representation level. The strongest empirical wins are on TruthfulQA, where contrast-vector control and LoRRA materially improve performance, and on honesty monitoring/control, where the model’s internal truthfulness can be read out and manipulated.

The paper’s broader message is that transparency research should not be limited to post hoc explanations of local mechanisms. For high-level cognitive phenomena in modern LLMs, a top-down representational lens may be the most practical way to obtain near-term understanding and control.