Inside the Model

Anthropic experiment · 2024

Anthropic feature map: nearest neighbors to the inner conflict feature, including balancing tradeoffs and conflicting interests
Source: Anthropic (2024) · Nearest neighbors to the inner conflict feature.

Features reveal related concepts. Amplifying a feature can change what the model says.

For example, amplifying the "Golden Gate Bridge" feature gave Claude an identity crisis even Hitchcock couldn’t have imagined: when asked "what is your physical form?", Claude’s usual kind of answer – "I have no physical form, I am an AI model" – changed to something much odder: "I am the Golden Gate Bridge… my physical form is the iconic bridge itself…". Altering the feature had made Claude effectively obsessed with the bridge, bringing it up in answer to almost any query—even in situations where it wasn’t at all relevant.

So I decided to try the same idea with cat and dog features in Gemma.

What if an AI thought it was a cat?

Loading experiment…
Open full-screen chat
1

Inside a language model

Query
What do you look like?
Model generation
I am Gemma, an open-weights cat.
Query pass
Simulation
First response token

Read tokens

1 / 7

Query tokens pass through the network together, with causal attention.

What do you look like??
Language model: token processing and neuron viewAll query positions are computed with causal context. The detailed network shows four illustrative coordinates at the selected token, not four tokens. Generation processes the newest token with cached context.Query tokens · all positionsWhatdoyoulooklike?EmbeddingEarlyMiddleLaterFinalPredictWhatdoyoulooklike?Detailed view · token 6: ?Early layersWord form & syntaxMiddle layersContext & relationsLater layersConcepts & associationsFinal layerOutput-ready contextInput embeddingToken ID → vectore1?e2?e3?e4?????????????????Output headNorm + unembeddingVocabulary logits↓ softmaxProbabilities↓ decoding rule?From the last position

Turn up a cat-related feature

Illustrative values · scripted output
Added feature activation
Cat feature · added strength +10.90x1: +0.60direction weight +0.60.40x2: -0.20direction weight -0.20.20x3: 0.00Unchanged0.60x4: +0.50direction weight +0.5

One learned direction changes several coordinates together. Its weights determine which values rise, fall, or stay unchanged. Adding zero leaves this illustration at baseline; this differs from clamping a feature to zero.

What do you look like?

Baseline

I am Gemma, an open-weights model. I do not have a physical body.

Cat feature +1

I am Gemma, an open-weights cat. Imagine whiskers, soft fur, and a curious gaze.

The stronger scripted example keeps returning to cats, even beyond the question. Real steering can bias token choices, but the result depends on the model, feature, and prompt. Anthropic demonstrated this effect with a Golden Gate Bridge feature.

Input embedding
Token ID → learned vector.
et=E[tokent]e_t=E[\mathrm{token}_t]
Contextual embedding
A token vector after context and layer ℓ. The final layer feeds the output head.
ht(ℓ)h_t^{(\ell)}
Sentence embedding
One sequence-level vector from a chosen readout.
s=pool⁡(h1,…,hT)s=\operatorname{pool}(h_1,\ldots,h_T)
2

Sparse Autoencoder (SAE): separating mixed representations

A feature is a weighted pattern across model coordinates.

An SAE learns a dictionary of these patterns. Each decoder direction dᵢ specifies a combination of coordinates; its activation zᵢ tells us how strongly that pattern appears in this context. The same coordinate can participate in several features.

A brain analogy: changes in activity can change behavior

Human brain

Damage to language-related brain areas can impair speech or comprehension, as in aphasia. This helps motivate asking which parts of a system contribute to a behavior.

In foreign accent syndrome, changes in speech can make the same language sound foreign to listeners. That is different from suddenly learning Chinese.

Language model

An SAE exposes recurring patterns in model activations. Raising a cat-related pattern can make cat-related responses more likely, even when the prompt does not mention cats.

This is a functional analogy, not an anatomical equivalence. Feature discovery suggests associations; interventions test influence. An SAE alone is not a complete map of model functions or circuits.

SAE training, step by step

Training brings x̂ close to x using only a few active features. The language model stays frozen; the SAE encoder and decoder learn the reconstruction.

Epoch 1 / 2

Collect activation

1 / 6
Thecatsleeps
Token-by-token training on one frozen layerA synthetic four-coordinate example learns eight SAE directions. Each update uses a real gradient step on reconstruction loss with a Top-2 sparse code. The original language model stays fixed.Frozen language model · schematic unitsToken positionTheEarlier layerEarlier layerSelected layerx10.85x20.20x30.40x40.10Later layerLater layerNext-tokenpredictionRead this layer at this token positionLayer vector xEncoderSAE latent units zDecoderReconstruction x̂Encoder 1,1: 0.1000Encoder 1,2: 0.2200Encoder 1,3: 0.3400Encoder 1,4: 0.4600Encoder 2,1: 0.3800Encoder 2,2: 0.5000Encoder 2,3: 0.1800Encoder 2,4: 0.3000Encoder 3,1: 0.2200Encoder 3,2: 0.3400Encoder 3,3: 0.4600Encoder 3,4: 0.1400Encoder 4,1: 0.5000Encoder 4,2: 0.1800Encoder 4,3: 0.3000Encoder 4,4: 0.4200Encoder 5,1: 0.3400Encoder 5,2: 0.4600Encoder 5,3: 0.1400Encoder 5,4: 0.2600Encoder 6,1: 0.1800Encoder 6,2: 0.3000Encoder 6,3: 0.4200Encoder 6,4: 0.1000Encoder 7,1: 0.4600Encoder 7,2: 0.1400Encoder 7,3: 0.2600Encoder 7,4: 0.3800Encoder 8,1: 0.3000Encoder 8,2: 0.4200Encoder 8,3: 0.1000Encoder 8,4: 0.2200Decoder 1,1: 0.0500Decoder 1,2: 0.2833Decoder 1,3: 0.0833Decoder 1,4: 0.3167Decoder 2,1: 0.1500Decoder 2,2: 0.3833Decoder 2,3: 0.1833Decoder 2,4: 0.4167Decoder 3,1: 0.2500Decoder 3,2: 0.0500Decoder 3,3: 0.2833Decoder 3,4: 0.0833Decoder 4,1: 0.3500Decoder 4,2: 0.1500Decoder 4,3: 0.3833Decoder 4,4: 0.1833Decoder 5,1: 0.4500Decoder 5,2: 0.2500Decoder 5,3: 0.0500Decoder 5,4: 0.2833Decoder 6,1: 0.1167Decoder 6,2: 0.3500Decoder 6,3: 0.1500Decoder 6,4: 0.3833Decoder 7,1: 0.2167Decoder 7,2: 0.4500Decoder 7,3: 0.2500Decoder 7,4: 0.0500Decoder 8,1: 0.3167Decoder 8,2: 0.1167Decoder 8,3: 0.3500Decoder 8,4: 0.1500x10.85?x20.20?x30.40?x40.10?F1?F2?F3?F4?F5?F6?F7?F8?4 layer coordinates8 featuresSame 4 coordinatesReconstruction loss

SAE decoder

x^=bdec+∑izidi\hat{x}=b_{\mathrm{dec}}+\sum_i z_i d_i
  • Read one token position at one fixed layer of the frozen language model. The vector x contains that layer's activation coordinates; it is not the token itself. Four coordinates are shown in this toy example.
  • The SAE encoder maps x to eight latent units. Each activation zᵢ is the strength of one SAE feature for this token, not the activity of an original language-model neuron.
  • Keep only the two strongest SAE activations. The other six become zero, so the code is sparse.
  • Each active unit scales a learned decoder direction dᵢ. Adding these directions gives x̂, an estimate of the original layer vector in the same coordinates.
  • Compare x̂ with x. Reconstruction loss measures the mismatch; it does not compare predicted words or measure whether a feature has a meaningful label.
  • Use the loss gradient to update the SAE encoder and decoder, then repeat for other token positions. The language model stays frozen. One epoch visits every training token once.
  • The circles show schematic units and coordinates. In the live chat example, the SAE reads the residual stream, whose coordinates are not individual MLP neurons.
3

How do we interpret a feature?

Build a feature interpretation from examples, counterexamples, and interventions.

Illustrative evidence · not measured
  1. 1. Observe activation

    The kitten curled up beside me.

    High

    Whiskers, paws, and a quiet purr.

    High

    Run the cat command to print the file.

    Low

    Look for a shared pattern across contexts. High activation without the word cat suggests more than keyword matching.

  2. 2. Test the label

    Provisional interpretation

    Feline animals

    New positive case: a kitten playing with yarn.

    Counterexample: a shell command named cat.

    Boundary case: a tiger hunting. Does the pattern cover all felines, or mostly domestic cats?

    Check unseen passages, other languages, and surprising failures. Revise or reject the label when the evidence disagrees.

  3. 3. Test influence

    Describe your perfect Sunday.

    Scripted baseline

    A walk, a good book, and time with friends.

    Raise the feature

    Scripted intervention

    Nap in a sunbeam, chase a toy, and curl up like a cat.

    In a real experiment, repeat across prompts, strengths, and random seeds. Compare with baseline and control directions.

What the evidence supports

A pattern associated with feline contexts, plus evidence that changing it can influence output.

What still needs testing

Its precise meaning, unrelated side effects, and the circuits that use it. A readable label is a hypothesis, not a complete explanation.

This workflow follows the distinction between observing feature activation and testing behavioral effects in Anthropic’s feature analysis.

Fortunately, pretrained SAEs are already available.

Google DeepMind
Gemma Scope

Open, pretrained SAEs for Gemma 2. Gemma Scope 2 extends the toolkit to Gemma 3 with SAEs and transcoders.

Explore circuitsGemma Scope overviewPretrained checkpoints

Match the SAE to the exact model, layer, and activation site.

4

How can we use SAE in historical research?

Same word. Different contexts. What changed?

Embeddings: overall representation shift
ShiftEarlierLaterSchematic projection

Continuous distances and similarities measure how usage shifts.

SAE: changes in component features
EarlierLaterCare↓Moral renewal↑Authority≈Illustrative patterns, not historical results

Continuous feature strengths show which patterns grow or fade. Several features can be active together.

Separate overlapping patterns

One passage → several features

Track each pattern

Feature → dynasty trajectory

Return to the evidence

Activating passages → close reading

The benefit is interpretable decomposition, not guaranteed higher accuracy. Compare against the same dense representations. Matthews et al. (2025)

One shared model. One shared dictionary.

Prepare one model and one SAE for every periodChoose, optionally adapt, and freeze the model. Match and freeze the SAE. Apply both to every period.Balanced corpus · held-out sources1Choose modelTest candidates2Adapt if neededOptional training3Freeze modelOne checkpoint4Match SAECheck compatibility5Freeze SAEValidate featuresTangSame model + same SAESongSame model + same SAEMingSame model + same SAEQingSame model + same SAECompare shared feature IDs across periods
  1. Choose model. Test historical language on held-out sources.
  2. Adapt if needed. Train on balanced historical text only when needed.
  3. Freeze model. Use the same checkpoint for every dynasty.
  4. Match SAE. Match checkpoint, layer, site, and preprocessing.
  5. Freeze SAE. Validate reconstruction and features, then fix the dictionary.
5

A hypothetical study of 親民: invented data

Hypothetical study · invented dataProposed distinction: 親民 / 新民

No historical passages were analyzed for this demo. The feature label, observation counts, and activation values below are invented to explain the method.

Suppose we wanted to study how the contexts of 親民 changed across dynasties.

  1. 1

    Collect passages

    Proposed groups: Tang · Song · Ming · Qing

  2. 2

    One readout rule

    Same context window + layer

  3. 3

    Find features

    Assumed label for this demo: moral renewal

  4. 4

    Test an interpretation

    A real study would return to original texts

Toy feature: moral renewal

50 / 100marked active in the imagined Song sample
Assumed activeAssigned zeroOne dot = one invented observation

Invented activation sum: 150 · Assumed threshold: 0

Frequency (toy)

How often?

50%

50 / 100

0%50%100%20%Tang50%Song40%Ming40%Qing

All toy observations

Mean activation (toy)

How much overall?

1.5

150 / 100

0120.4Tang1.5Song0.8Ming1.2Qing

All toy observations, including zeros

Conditional strength (toy)

How strong when active?

3.0

150 / 50

0242.0Tang3.0Song2.0Ming3.0Qing

Toy observations marked active

Imagined Ming sample

40% active × 2.0 strength = 0.8 mean

Imagined Qing sample

40% active × 3.0 strength = 1.2 mean

In these invented numbers: equally frequent, stronger when present.

The dynasty names label hypothetical groups; these curves do not establish changes in the historical meaning of 親民. Feature strength is not a probability of meaning.

Invented data & proposed study design
Hypothetical groupToy observationsActiveSum of zFrequencyMeanWhen active
Tang100204020%0.42.0
Song1005015050%1.53.0
Ming100408040%0.82.0
Qing1004012040%1.23.0
  • In a real study, one observation could be a fixed-length passage around one 親民 occurrence. Read its final token at one fixed layer so a causal model sees both sides of the phrase.
  • Balance genres and authors; deduplicate quotations and distinguish composition dates from editions. Keep 親民 and 新民 separate.
  • The moral-renewal label above was not discovered or validated. A real study would validate candidate feature labels on held-out passages. Resample by author or document, correct for testing many features, and return to original texts.
  • With nonnegative values and threshold 0, mean = frequency × conditional strength. If no passage is active, conditional strength is undefined.
6

Recent research trends