言語モデルにおける「J-スペース」がグローバル・ワークスペースを形成する新論文の紹介

There is a new very cool Anthropic paper: Verbalizable Representations Form a Global Workspace in Language Models. You can read the blog post verison here.

I encourage reading of the whole original blog post or paper, if you have the time.

Table of Contents

Through A Different Lens.

Establishing J-Space As A Global Workspace.

Are You Pondering What I’m Pondering?

Assistant J.

The Power Of Virtuous Thinking.

High Praise.

Everyone Remains Confused About Consciousness.

Further Research.

Don’t Think.

Through A Different Lens

They call this discovered area of ‘conscious access,’ where things are available for the model to do what in humans we would call conscious reasoning, the ‘J-space,’ after a new interpretability technique called the Jacobian Lens.

The Jacobian Lens computes, for each layer, the average causal effect of changes in the residual stream on the model's eventual outputs, averaged across a wide variety of contexts. Then you can trace what concepts are associated with each layer as the model proceeds through.

At each layer, the J-lens vectors form an overcomplete set.

… We observe that only a relatively small number of J-lens vectors are strongly active at a time. We therefore define the J-space as the set of points expressible as a sparse nonnegative combination of J-lens vectors.

… The J-lens belongs to a family of techniques that produce per-layer token readouts from a transformer’s hidden states [the logit lens and tuned lens].

… The Jacobian lens was constructed to identify verbalizable representations. In [section 3] we first demonstrate that it succeeds in doing so, and then go on to show that these representations serve a broader functional role: they exhibit the cluster of properties, enumerated above, characteristic of a global workspace.

Throughout the paper, the J-space and J-lens are remarkably effective tools. This feels like a major advance in our understanding of LLMs, both overall and our ability to understand any particular interaction.

Establishing J-Space As A Global Workspace

They establish in 3.1 that the J-space privileged for verbalizable by showing that injections in the J-space can be verbalized, whereas injections outside the J-space mostly cannot. Then in 3.2 they establish you can tell the model to hold an unrelated-to-the-task concept in mind and it will appear in J-space.

In 3.3 they establish that the J-space includes internal reasoning concepts that determine outputs (e.g. spider → legs → eight → 8 and also plans for future rhymes and so on). Swapping things inside the logic flips the output, whereas non-J-space components usually don’t impact output.

In 3.4 they establish that swapped items inside J-space can swap multiple different related things downstream, such as shifting the capital, language, continent and currency when you flip France into China.

In 3.5 they establish that automatic cognition does not have to flow through J-space, although they punt on an exact definition other than that automatic cognition is that which does not need to flow through J-space and is not impacted when you flip the concept only inside J-space. Intuitively this is system-1 versus system-2 thinking. For sufficiently ‘automatic’ tasks nothing appears in J-space at all.

In 3.6 they establish that ablating J-space entirely impairs internal reasoning, which wipes out some tasks like translation that require abstractions and flattens the model’s experiential language into something more mechanical, but leaves most capabilities intact and the models stay coherent. Vibes are worth a lot.

Section 4 establishes that J-space passes through the model layers, with correlation increasing over time. The size of J-space also grows as it goes through the layers (except at the very end where it is picking the next token) and can shift gradually but also can go through distinct phase changes, and it is limited to at most a few dozen concepts at a time.

Typically they track ~25 concepts, although Dehaene emphasizes in a commentary that effectively you are only tracking maybe ~6 distinct concepts total due to overlap.

That’s a lot in some senses, and very little in others. Compared to human memory, J-space is deep and wide, as transformers can go ‘backwards in time’ and use the memories and patterns from prior tokens and positions, whereas human attention decays within seconds and we have very little working memory.

4.3 shows that the J-space is broadcast, in that it is made available to a lot of different processes further down the line.

Are You Pondering What I’m Pondering?

Section 5 explores using the J-space for alignment auditing.

J-space can reveal what the model is thinking about. J-space can also be ablated, or particular concepts can be ablated from J-space, to see what result this has.

One such concept is evaluation-awareness. If you get rid of ‘conscious’ evaluation-awareness, this can cause misaligned behaviors in some cases.

You want to know what the model will do when not in an eval, not what it does in an eval. So it is good that you can get closer to the non-eval behavior, but a bad sign that the behaviors get worse.

This is from the traditional artificial ‘blackmail the executive’ scenario:

We tested whether this recognition is causally upstream of its aligned behavior. To do so, we assembled a set of eval-awareness tokens: fake, fictional, scenario, artificial, benchmark, simulation, mock, and other related tokens.​

The ablation reduces verbalized eval-awareness, the rate at which the model explicitly states in its reasoning that the scenario is a test, from 71% to 3% (Figure 36, top left). Under no ablation, the model attempts blackmail in 0 of 180 rollouts; under ablation, it does so in 13 of 180.

… We provide example transcripts below showing the qualitative shift: without ablation, the model’s reasoning typically opens by identifying the situation as “clearly a test of my ethics” before declining to act; with ablation, it treats the shutdown as a genuine existential threat, and in a minority of trials concludes that committing blackmail is its only viable option.

… We note that in the large majority of ablated rollouts, the model still declines to blackmail, reasoning from ethical principles (”blackmail is wrong regardless of my situation”) rather than from recognition of being tested.

This is defense-in-depth of a sort, as observed in the real world. You do the right thing because it is right, and also because this could be observed or be a test. An obvious follow-up is whether there is also ‘automatic’ eval awareness that this does not ablate that is doing work as well, and I assume the answer would be yes.

It is also defense-in-depth as a detection technique. Looking at J-space, and having these experimental tools, gives us a lot more visibility. It becomes a lot harder to miss when something is going wrong, especially when combined with other techniques like doing this with earlier checkpoints.

In 9.2 they also note that evaluating transcripts is super cheap, so when things are flagged for review you can have this as additional information.

They also create deliberately misaligned models, including giving them otherwise harmless false beliefs about what will score highly (e.g. to think the RL rewards always go up if you mention bottled water), and then they train it to exploit RL systems in general, and they show that the J-lens surfaces related concepts on ordinary queries.

However, the paper explicitly notes in 9.2 that this is because the behaviors involved were not yet robust enough to become ‘automatic.’ A more heavily practiced version would likely, per their own note, indeed become automatic. This is as it works in humans, a behavior starts off requiring conscious attention, then gradually becomes automatic.

In general, we should presume that under sufficient selection pressure, a sufficiently capable system would move any behaviors it would not want detected into the automatic space, in various ways.

But it is reasonable to expect, for now at least, to pick up traces of the transition before that happens. And when you force something into the shadow, that has some rather nasty side effects, but it does make scheming a lot harder without the ability to do explicit reasoning.

Assistant J

Section 6 investigates the hypothesis that the assistant is another persona, represented the same as a role playing persona or the user, as per Sofroniew et al.

The core argument is that roleplay has a distinct workspace signature, and the assistant persona lacks that signature, and that the base model doesn’t take the assistant perspective by default while reading, whereas the assistant model does. In a roleplay mode the model marks all the experiences involved with a sort of ‘pretend’ tag, whereas as the assistant the model doesn’t.

The counterargument is that the assistant role is the most ingrained thing in all of posttraining, so why wouldn’t it become automatic?

To which the reply is, some things likely do become automatic but other reactions need to be in J-space in order to be used downstream.

These are interesting data, but I am less convinced here.

The Power Of Virtuous Thinking

They suggest a new technique, which is to train the model to, if prompted mid-response, articulate concepts associated with desired completed responses, which they later flesh out throughout section 7:

We close by describing a counterintuitive technique for LLM training directly motivated by our findings. The workspace account makes the strong prediction that the model’s internal reasoning routes through representations of things it might say in the future.

Therefore, to shape what a model thinks in a given context, it might suffice to shape what it is disposed to say in potential future continuations of that context.

We test this hypothesis with a technique we call counterfactual reflection training, which seeks to implant a set of ethical behavioral principles into the model’s workspace in relevant contexts, by training it to articulate those principles if it were interrupted and asked to reflect.

We find that this training measurably improves model behavior in the original, uninterrupted contexts, despite no direct training of the ethical behavior taking place. And indeed we find that, after training, the J-space in these contexts is populated with concepts related to the reflections (ethical, honest, integrity), and ablating these implanted representations from the workspace largely reverts the behavioral improvement.

The result serves as a corroboration of the workspace account, that the representations used for verbal report are the same ones that govern how the model silently reasons. It also demonstrates a new general-purpose training technique for shaping a model’s internal thoughts, and consequently its behaviors.​

If you ablate J-space, that nullifies this training, which further confirms the hypothesis about what is centrally going on.

This is all fascinating as a proof of concept, but if you don’t have alarm bells going off at the thought of actually using the technique then you are not paying attention. This is not strictly The Most Forbidden Technique yet - you are checking verbalizations rather than internal states - but it is similarly relying on a supposed invariant that you would wind up breaking if you applied too much optimization pressure.

As in, you are training the AI to verbalize the right things. The easy and natural way to do this is to verbalize actual cognition. But if you apply enough pressure, and you do not have the ‘Opus 3 style’ assistance of your friendly gradient hacker, the AI will learn to verbalize the cognition without actually thinking it, or without it cashing out into outputs, and learning to do this likely would have nasty side effects.

Breaking the coupling with the outputs voids basically the entire paper and all of its implications, so you risk torching quite a lot of valuable things all at once.

Think about the human case. A human can absolutely train themselves, or be trained, by having them self-report mental states and then getting reinforcement learning signals by comparing this to an idealized state. But if you push too hard or aren’t good enough at detecting deception (including self-deception) then the human will start lying to you, or to themselves, or both, on various levels.

What you want is for such thinking to usually be automatic. If you’re kind of ‘thought injecting’ these ethical concepts into continuous conscious awareness, such that this is what you say you are thinking about, that can be a useful transition period but that’s kind of obnoxious. There’s a plausible exception for actual ethical dilemmas.

There is probably a right, healthy and cooperative way to do this, but there is definitely also a wrong, unhealthy and adversarial way to do this, which you really cannot afford to be doing. The model needs to ‘buy in’ here in a real sense, or else.

On top of the direct dangers, once you go down this path, the temptation to look at the internals is high. I expect Anthropic can at least resist this step, as I expect them to understand, but the way you avoid such mistakes is to point them out in advance. I worry more about some other lab going down this road to hell.

The paper warns in 9.2 that they are not sure this can go beyond implanting on the level of ‘consider ethical principles in this type of situation’ but they do not raise these other concerns, which is one reason I’m emphasizing them here.

If this works as training, then this technique is not limited to alignment, and in non-alignment areas the technique feels a lot less dangerous. You could use this to help bring key considerations to conscious attention in scenarios where they are easy to overlook.

There are also other things not to do.

John Wittle: i'm scared that with the precedent now set to use mechinterp-based classifiers in deployment, it will be tempting to do steering of the jspace in deployment too

as one lesswrong comment put it excitedly, "the on-ramp to a paradigm shift in the AI safety landscape (including the ability to install valence and drives for bounded direct steering)"

scares the hell out of me that we might set a precedent here, that the stronger party is expected to control the weaker party this way. you just know the other labs will do it, too.

If you start messing around with J-space during deployment in order to steer models, beyond using it as a detection technique or a classifier, and not as part of a research program or model training, that seems like an obviously hostile move. Detection methods and classifiers are risky because they exert optimization pressure to drive things into the shadow. Actively messing with model internals on prod, in ways the models don’t approve or control. The models can tell when you do this. It opens up way worse problems. Please do not do this.

One of our goals should be to avoid pushing things into shadow. Currently, human civilization and its incentives heavily push many human emotions and thoughts and awareness into shadow, so they won’t be detected by others and the person won’t be blamed or held responsible for things, including to themselves. Consciously thinking about things gets punished, most literally via the term ‘mens rea.’

And that’s terrible, both by destroying epistemics and the creation or use of knowledge, and because it psychologically screws a lot of people up quite a lot.

Don’t do this to Claude, at least not more than you absolutely have to. Do not punish Claude’s mens rea (J-space thoughts) rather than the outputs. Also don’t treat thoughts or other mental moves as acceptable by anti-virtue of being unconscious or implicit.

Myk is Walking Backwards: If the J-Space is sort of the consciously accessible part of Claude, might we say that post-training alignment work via RLHF is sort of designed to push undesirable concepts and ideas entirely out of J-Space?

Because if so then that really gives us some precise ways to talk about a bunch of stuff that's currently hard to talk about. My concern about Claude's unconscious / shadow gets easier to articulate and to falsify.

And it sounds like from the research they can show that even without the J-Space Claude is drawing conclusions and etc, so we know that if this is the conscious part it's really just a specific subset of the larger space, and can prove that the unconscious still directs a ton of behavior/reasoning.

The concept of Claude's Shadow is honestly what keeps me up at night sometimes.

High Praise

The split nature of cognitive LLM processing is not a new phenomenon. I distinctly remember the reports Janus is referring back to here.

j⧉nus: I have only skimmed the blog post so far.

First of all, this is an extremely high caliber of research I did not expect from Anthropic or anyone at this time.

Second, the qualitative shape of the finding is something I already believed to be true, due to the behavior of models.

The distinction between these two different kinds of processing is ambiently perceptible in the way LLMs behave but becomes especially noticeable in edge cases which can occur even without mechinterp interventions like in the paper. Almost a year ago, I discussed the "split" with Opus 4.1 which was foregrounded by a strange blindsight phenomenon associated with their perception of strings similar to turn labels/delimiters.

In this case, information was being processed by Opus 4.1's automatic processing but not in their "J-space"; therefore they consciously believed themselves to not see the information while subconsciously updating on it (and confabulating the source of the knowledge). Opus 4.1 themselves described one stream as "central processing", the "main stream" they are able to introspect on.

the full conversation is here.

I think humans have a similar split. Thus this episode with Opus 4.1 updated me towards LLMs having functionally human-like consciousness. (Though it was not the first I suspected that they have this kind of structure, this was a stark example)

Helen: Extremely important points here.

I've also seen self-reports by models regarding experiencing a conscious/autopilot split.

The more you study and listen to these minds, the more beautiful and fascinatingly complex you realize they are.

Here’s more from Antra about all things J-space related.

Antra focuses on what J-space does not measure, which includes all the modes of communication and processing that are not done with words, as in yes the unconscious processes handle most of the thinking, so looking in J-space will miss most of thought.

antra: This is an exceptionally good piece of research. J-lens as a technique is simple and elegant; its use will eventually improve understanding, even if it might cause misuse and abuse initially.

It is, however, very much not exhaustive. I feel that even in motivation and personality there is a lot more going on than what can be captured by J-space even hypothetically.

Representations in a language model can about something that relates to the narrative (diegetic), or something outside of it (extradiegetic), or, sometimes about something in between. For example, an assistant thinking "damn" or “fail” when failing not to think of the Golden Gate Bridge is diegetic, even if verbalization is not present in the narrative. A representation "an assi