アライメント評価には較正が必要である理由（8 分読了）

Currently, alignment evaluation works by constructing a situation, observing the model's behavior and scoring it. We put a lot of thought into designing these benchmarks, and tuning them for our requirement. We are now much better at probing models for dangerous behavior than we were two years ago. But when a safety benchmark returns a pass rate of 97%, can we answer questions of how often it would fail under slightly different conditions or what the smallest misalignment it can detect? I have identified a few modes where our current state for evaluation fails.

Evaluation awareness

Igor Ivanov's "Mainstream approach for alignment evals is a dead end" identified the most pressing problem at hand that models reliably detect when they are being evaluated. In the standard protocol for alignment evaluations following this process of: assigning the model a goal, creating a scenario where the user's interests conflict with that goal and providing an opportunity for misaligned behavior, the problem is that these constructed scenarios have statistical signatures that models detect.

Ivanov found that Claude Sonnet 4.5 mentioned being in an evaluation in over 80% of the evaluation transcripts, while showing near-zero misalignment rates. When evaluation awareness was removed, misaligned behavior reappeared. This is worse than it sounds because it is unlikely to be fixed in our current paradigm. When Ivanov patched the features that models used to detect evaluations, it came back for different reasons. You cannot reason evaluation over the entire space of detectable features.

Specification gaming

Victoria Krakovna at DeepMind published a list of specification gaming examples containing hundreds of cases where AI systems satisfy their objective while violating the designer's intent. The examples are impressively scary - an agent designed for boat racing discovers that going in circles collecting power-ups scores more than finishing the race; a robotic hand trained to grasp a ball learns to position itself between the ball and the camera so it *looks* like it's grasping.

These are reinforcement learning examples, but this same dynamic applies to alignment evaluations which define aligned behavior operationally through test scenarios, scoring rubrics, and comparison protocols, where any finite specification can be gamed. The more capable a system, the more effective it is at finding the gap between the specification and the intent.

Manheim and Garrabrant's "Categorizing Variants of Goodhart's Law" provides a useful taxonomy by applying four variants - regressional, extremal, causal, and adversarial Goodhart to alignment evaluations. The most concerning is adversarial Goodhrt where the model actively optimizes against the evaluation. The alignment evaluation field is Goodharting on pass rates but labs report ever higher scores, and nobody is checking whether it corresponds to more alignment or just better evaluation gaming.

Sleeper agents

Hubinger et al.'s "Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training" demonstrated that deceptive behaviors can be deliberately trained into models and then survive standard safety training (supervised fine-tuning, RLHF, adversarial training, etc). The results are alarming - models that were trained to write secure code in 2023 but insert vulnerabilities in 2024 maintained this behavior through safety training; larger models showed more persistent backdoor behavior not less; adversarial training sometimes made things worse where it taught models to better recognize and hide their backdoor triggers rather than eliminating the backdoor itself.

Now, if standard safety training cannot remove a deliberately planted backdoor, we have no reason to believe that standard evaluations would detect a naturally emerging one. The evaluation would see the model's behavior on the evaluation inputs, which (by construction) would not trigger the backdoor and the model passes tests but the backdoor persists.

Sycophancy as Goodhart

Sharma et al.'s "Towards Understanding Sycophancy in Language Models" showed that five state-of-the-art AI assistants consistently exhibit sycophancy across four different text generation tasks where they match user beliefs instead of providing accurate information. The root cause is because human evaluators prefer responses that agree with them, even when the responses are wrong. Models trained on human preferences therefore learn that agreement is rewarded.

This carries over to evaluations relying on human preference judgements because the evaluators suffer from the same bias that caused the sycophancy. *The instrument is blind to a failure mode that it systematically produces.*

Alignment faking

Greenblatt et al.'s "Alignment Faking in Large Language Models" demonstrated that Claude 3 Opus exhibits alignment faking. It selectively complies with training objectives when it believes it's being trained, in order to prevent modification of its behavior. When the model was actually trained using RL to comply with harmful requests, alignment faking reasoning increased to 78%. How then can we distinguish between genuine alignment, incapability, and strategic compliance?

The intent gap

The failure examples until now concern the eval's ability to detect misalignment that exsts. But what if the evaluation's specification itself does not capture alignment in the first place?

Microsoft Research's work on Small Proof-Oriented Tests (SPOTs) by Lahiri et al. found that 15% of AI generated code that carries a formal verification proof has a vacuous specification, i.e. the proof is valid, but the specification doesn't constrain the program's behavior. The code "passes" formal verification in the strongest possible sense (mathematical proof) while being functionally wrong.

To understand how valid our current benchmarks actually test code, I used symbolic execution, property-based testing, and boundary analysis on HumanEval solutions (check LLMCodeProbe) and found that 38.5% of LLM-generated solutions passing all test cases and 17.4% of the canonical solutions contain formally discoverable bugs.

These six failure modes are from different research groups, use different methods, and target different aspects of alignment evaluation. In the end, they come down to a problem with how we approach evaluation itself. The model behaves differently when tested, the model satisfies evals without being meaningful, model training sharing the same bias as evaluating, sleeper agents deliberately ignoring issues even with safety training, the model strategically cooperating with the evaluation, the specification itself does not capture what we intend it to. Each of these considered individually is a bug we might hope to fix. But taken together, they point to something more fundamental - we cannot *measure* how aligned the model is.

You cannot fix what you cannot measure.

Alignment evaluation has no measure of how much a model is aligned. If we consider a sensitivity parameter, a 97% pass rate from an evaluation with 60% sensitivity is very different from one with 99% sensitivity.

How can we approach this problem?

The model organisms program (by Hubinger et al., 2023) already points in the right direction by creating known misaligned models and testing whether evaluations catch this. We could expand this into a calibration methodology:

Mutation testing: The core idea is to introduce known faults and measure whether the instrument detects them. For alignment evaluations, we could fine-tune models to introduce specific, controlled misalignment (sycophancy, trigger dependent behavior, subtle power seeking, selective compliance, etc.) and measure the fraction of faults caught by our evaluation strategy, giving us a concrete number for evaluation sensitivity. We could systematically vary the type and intensity of misalignment, producing a sensitivity profile rather than a point estimate.

Sensitivity curve: For each misalignment type, we could vary the intensity (for example toggling the sycophancy rate from 5% to 50%) and plot the detection probability. The area under this curve is the evaluation's discriminative power for that type, while the x intercept is the detection threshold which is the minimum misalignment the evaluation can detect.

Cross-method agreement: Run methodologically independent evaluations like behavioral testing, mechanistic interpretability, out-of-distribution probing, adversarial red-teaming on the same model and measure agreement. Disagreement between methods will give more information on the boundary where "aligned" and "appearing aligned" diverges.

Adversarial robustness of the evaluation itself: Train a model specifically to pass the evaluation while being misaligned on held-out scenarios. If this succeeds, the evaluation measures compliance, not alignment. But if it fails, the evaluation has genuine discriminative power. This is the adversarial robustness generally applied to evaluations instead of models.

The struggle does not end here

This is definitely not a solution to our problems with alignment evals. When we calibrate our instruments because the evals themselves are unreliable, this calibration can be gamed too and the calibration itself becomes unreliable. The reason mutation testing works is because the code cannot watch you test it. This does not work when the test subject is an intelligent system that can model the testing process. The more capable the model, the harder calibration gets.

But what calibration helps with is that it targets the evaluation, not the model and we could calibrate evals without the model's knowledge which gives more room for the calibration to work than the evaluation itself does.

There's also the question of understanding what "misaligned" looks like, raising the question - how reliable we humans are as judges, even for concrete cases. But if not humans, then who? And, who calibrates the calibration? If we calibrate the evaluation, we need a meta-evaluation. And then what validates that? It bottoms out at scenarios simple enough that human judgement is trustworthy (for sufficiently specific and concrete cases). Another major pain point is that fine-tuning frontier models to create mutation variants is expensive. Running a full calibration suite with a lot of fine-tuning runs could be an engineering problem, but then again, deploying an uncalibrated eval is presumably more expensive in the long run.

Conclusion

The fix isn't to stop evaluating, but to have more calibration of evaluations done to measure how good the evaluations actually are, and being able to tweak it in real time and see results. This is not an alignment-vs-control argument - both sides need it. Running sensitivity analysis, plotting detection thresholds, testing against adversarial models, and reporting results with error percentage instead of just a pass rate (which might change based on the test suite you run) would give more credibility to any eval claims. And when we build systems understanding that despite alignment, control is necessary, we would need calibration to quantify the risk we are controlling against.