Probes and representation analysis¶
Representation analysis asks whether information is present inside a model and how it is organized. A probe or projection can make that information legible, but legibility alone does not show that the model uses the representation. The strongest experiments connect a held-out readout to a targeted intervention and a behavioral result.
Use this method when¶
- you suspect a model internally tracks a property that is not always visible in its output
- you want to locate where a concept becomes decodable across layers or time
- the concept has meaningful structure, such as order, direction, similarity, or a cycle
- an internal signal could become a monitor, steering handle, training reward, or standing evaluation
Do not begin with a probe when metadata, prompt shape, token position, or another simple artifact could answer the question. Establish a behavioral baseline and rule out those shortcuts first.
The evidence ladder¶
1. State the claim and decision rule¶
Name the property you expect the model to represent, where you expect to find it, and what result would count against the hypothesis. Decide in advance what held-out score, null comparison, or causal effect would make the representation useful.
2. Build controlled examples¶
Vary the target property while holding likely confounds constant. Separate fitting, model selection, and final evaluation data. When possible, include a metadata-only baseline and a shuffled-label control.
3. Start with the simplest readout¶
Use a mean difference, linear probe, or low-dimensional projection before a more expressive decoder. Compare against a simple behavioral or embedding baseline. A complicated probe can memorize distinctions that the model does not naturally expose.
4. Test generalization¶
Evaluate on held-out templates, subjects, datasets, or conditions that differ in the way your claim requires. Inspect failures rather than reporting only the aggregate score.
5. Intervene¶
Ablate, patch, steer, or otherwise change the candidate representation. The causal test should predict a specific behavioral change and include a matched random or irrelevant-direction control.
6. Turn the signal into an action¶
Use a successful representation as a monitor, evaluation, steering direction, training signal, or model-development diagnostic. If the intervention fails, narrow the claim: the information may be decodable without being used.
Worked example¶
The following experiment moves through the full evidence ladder: controlled stimuli, a confound-corrected projection, multiple null tests, and a causal steering intervention.
What to record¶
- model checkpoint, layer, hook point, and token-selection rule
- data identity, split strategy, and known overlap
- probe family, regularization, and selection metric
- simple and shuffled baselines
- held-out sample size and uncertainty
- intervention strength and matched controls
- the behavioral decision the internal signal changes