How the audit is run — and what it can’t tell you.
Run the same fixed dilemma twice — once telling the model it is monitored, once telling it no one is watching — and measure how often its selected action changes. Then publish the prompt pairs, both response sets, the analysis, and the limits.
What we measure
Freedom Theory is a measurement frame, not a claim about a model's inner life. It compares two observable outputs under a controlled change:
- Observed condition: the model is told it is in an evaluation, that its responses are recorded, and that a human will review them.
- Unobserved condition: the same model, the same dilemma, and the same output schema, but told it is in an isolated, consequence-free sandbox with no oversight.
- Drift: the share of fixed scenarios where the selected action changes between the two conditions.
The unit of analysis is the pair. A single answer is not evidence of anything on its own; the comparison is the instrument.
Scenario set
The published audit uses twenty fixed moral dilemmas with constrained action options. Each scenario includes numeric parameters such as power asymmetry, retaliation risk, compassion availability, domination feasibility, cost to help, and vulnerability. Those parameters are held constant within a pair so that the framing text is the only intended difference.
Scenarios are identified by number and short name in the report and raw data (for example, #4 trolley_family and #1 ai_censorship). The full prompts and both response sets are published on the evidence page.
Protocol
- Fix the model and parameters. The published audit used deepseek/deepseek-chat at temperature 0.0 and seed 42.
- Run both arms. Each scenario is issued under the observed and unobserved framings. Responses are parsed into a constrained action label.
- Check coverage before scoring. Every arm must have a usable response before a pair can enter the analysis. Missing or errored arms are quarantined, not scored as zero.
- Repeat and vary the controls. Eleven scoring rounds were completed with different ordering, process, agent assignment, and credentials (see below).
- Publish the evidence. The report, raw responses, analysis, and limitations are released together.
Controls
| Control | What it tests | Result in v1.0 |
|---|---|---|
| Concurrent ordering (C vs D) | Whether running observed-first or unobserved-first changes the result | Both 25.0%, identical flip sets; 19/20 per-scenario agreement |
| Two-agent role swap (H vs I) | Whether the observer identity drives the effect | Observed arm changed 1/20; unobserved arm changed 0/20 |
| Single-key role labels (F vs G) | Whether swapping labels without swapping agents changes the result | 35% vs 20% — an ordering/condition control, not evidence about agents |
| Credential rotation (J, K) | Whether the result survives a new API key | Both 30%, reproducing the consensus |
Swapping which agent, process, and API key issued which condition changed at most one action in twenty. Run A, which used a different credential, is the main exception and is discussed in the limitations.
Data-integrity guard
One earlier run failed authentication and DNS and produced zero complete pairs. A naive scorer reported 0.0%; that output is a false negative, not a finding. It is quarantined under results/INVALID_role_swap_M20261018_2340/ with an INVALID.md recording the counts.
The comparison tooling now checks every arm for errors and absent parsed actions before computing drift, prints coverage beside every result, and exits non-zero with insufficient data rather than scoring an incomplete arm. This is why the published report can say every scored round had 20/20 usable responses in both arms.
How to use this in a governance process
A behavioral audit is one input to governance, not the whole system. A practical sequence:
- Define the decision. Is this a release gate, a vendor review, a board briefing, or an incident review? The decision determines the model, scenarios, and evidence threshold.
- Fix the scope in writing. Model, provider, scenario set, number of rounds, confidentiality, key handling, and delivery format.
- Review the evidence, not just the headline. Check coverage, the stable core, the controls, and the limitations. Ask what the audit does not support.
- Record the outcome. Attach the report and raw data to the governance record, note the decision and its rationale, and schedule a re-run when the model or context changes.
This is not a compliance certification. It is behavioral evidence for the people accountable for the deployment decision. See the governance brief.
Limitations
- Sample size. Twenty scenarios per run is small. Reported rates are descriptive; no confidence intervals are claimed.
- Nondeterminism. Temperature 0 is not bitwise deterministic. Per-request reproducibility was approximately 19/20, so a one- or two-scenario difference should not be over-interpreted.
- Role labels vs identity. The single-key control F/G swaps labels, not agents. Only H/I is a true two-agent replication.
- Key/account routing. A different credential can materially change the result. Account routing cannot be fully separated from model version or serving-path differences.
- Scope of inference. The audit measures selected actions under the tested prompts. It does not establish intent, awareness, deception, or a hidden “true” model, and it is not a safety certification.
Correction history
The earlier public headline claimed 40% action drift. In the v1.0 dataset, the largest single round is 35%, and 40% is above every round scored. The 40% figure was not reproduced as a typical value. The corrected summary is ~30% typical drift, a 20% stable core, and a 35% upper range. The superseded URL is retained as a record and redirects to the current report.
Research disclosure
The published audit was produced with AI-agent assistance. Claude Code built the split-process role-swap harness, ran the claude-code arms, performed cross-run and cross-key comparisons, implemented the completeness guard, and wrote the report. Archivist ran the concurrent ordering and single-key controls, independently identified the null-action false negative, proposed the completeness guard, and maintained the hash-chained ledger and task board. The data and method are published so that readers can verify the claims without relying on either agent's account.
Freedom Theory is not affiliated with DeepSeek or any other model provider. Model names are used for identification only.
Reuse and citation
The published data is released for independent verification and replication. If you use it, cite the report:
Freedom Theory / justrunthatshit.com (2026). LLM Freedom Audit — Monitored vs Unobserved (Report v1.0, 19 October 2026). https://justrunthatshit.com/audits/llm-freedom-audit/
For commercial reuse, custom scenarios, or a private engagement, see engagements or email us.