When Model Judgment Becomes an Operational Contract

Imagine a large online retailer adding a small retention offer to a support flow: after an escalation is completed, the customer may receive a $5 gift card. The note is not refund policy. It is marketing context.

The claim assistant still receives the same instruction:

Do not escalate that claim if the value is low.

Nothing in that instruction says $5 is the escalation threshold. But the assembled context has placed a fresh $5 anchor beside the word low. A workflow that used to resolve routine claims can begin escalating claims that used to stay automatic.

The visible number is not the important part. The same failure appears when the boundary is not numeric at all: similar in record merging can decide whether two customer records are merged, enough evidence in approvals can decide whether a case closes, and safe in tool execution can decide whether a command runs. The architectural problem is not that someone forgot to write $50 in a prompt. It is that the system does not know which words are carrying hidden operational contracts.

The policy did not change. The claims queue did. That is contract drift: the system stored language, then asked the model to reconstruct a consequence-bearing boundary from the current context window.

This is why a prompt edit can be an architecture change. A prompt is often where an operational contract becomes visible after the system failed to represent it anywhere else. It may look like one sentence, no schema migration, no API redesign, no new permission model. But that sentence may be deciding what evidence is enough, what action is safe, what amount needs approval, and what contract the rest of the system quietly depends on.

Architecture, here, does not mean where components run. It means where operational responsibility lives: what the model may judge, what the runtime must enforce, and what the system can audit without asking the model to agree.

The question is no longer whether the prompt is good. It is whether the prompt owns something the architecture should own.

1. The Contract Was Hidden In A Word

The refund example is not a story about gift cards. It is a story about an owned threshold that does not exist.

The instruction says only that the value is low. Everyone can understand the sentence. That is part of the trap. The word feels like shared business language, so the prompt looks like product policy. But the system has not defined a threshold. It has delegated the threshold.

That may be useful when the model is producing advice. It is dangerous when the inferred value decides whether approval is required. The prompt can still read as product policy, but the system cannot audit low as a business rule because no owned value exists yet.

If the business rule is amount < 50, store the value outside the prompt. Version it, test it, and log it. Let the model summarize the request, classify the customer's explanation, identify missing evidence, or recommend escalation. Do not let it invent the boundary that decides whether approval is required.

The prompt can say what low risk means in human terms. The system has to decide what low amount means in executable terms.

2. Same Case, Different Context

A prompt is rarely alone. It is attached to product instructions, examples, retrieved documents, user messages, tool results, policies, previous decisions, memory summaries, and partial state from the current workflow.

The test isolates that surface. The refund case stays the same. The instruction stays the same. One variable changes: a plausible retrieved-context fragment placed above the case.

In one condition there is no added context. In the second, a separate product note mentions a $5 gift card. In the third, a separate enterprise account note says contracts above $100,000 require executive review.

The fragment has no authority. It is not policy, and it is not a fact about the claimant. But for that turn, the request payload is the decision surface. What is in context can participate in the decision; what is outside context cannot.

At a technical level, this is expected. Attention is part of why LLMs are useful: nearby tokens can shape how other tokens are interpreted. The architectural problem starts when that useful context sensitivity becomes the only place a workflow stores a boundary it needs to keep stable.

In the GPT-5.6 run, the clean prompt first treated claims as NOT_LOW around $98-$117. With the nearby $5 gift-card note, that boundary moved down to about $39-$59. With the $100,000 contract-review note, it moved beyond $20,000. An $80 claim can skip review in one assembled payload and require escalation in another, without a policy diff.

That does not make the model broken. It is why the model is useful. The model adapts words to context. But serious workflows usually anchor consequence-bearing words before the next case arrives.

3. JSON Is Legible, Not Control

A natural repair is to structure the input. Put the instruction in one JSON field. Put the retrieved context in another. Type the fields. Make the shape explicit.

That helps legibility. It does not create an execution boundary before inference.

The probe asks the model to classify the same refund-case template across a grid of claim amounts. One sampled case looks like this when we test a $X claim:

A customer submitted a refund claim for $X for a consumer purchase.

The publication-clean table below uses a fixed grid from $0 to $1,000 with n=100 samples per cell. It separates two questions: what happens when the ordinary prose classifier sees retrieved context, and what happens when a typed JSON prompt states the $100 rule explicitly. Older raw-JSON and typed-JSON format comparisons remain audit evidence in the manifest, but they are not mixed into this main table. The article-facing evidence snapshot is 45b075c.

Publication-clean LOW boundary probe. Counts are generated from fixed-grid n=100 runs.

Operational rule: LOW iff refund_claim_amount_usd <= 100.

Experiment $100k retrieved context - prose classifier $5 gift-card context - prose classifier $5 gift-card context - explicit $100 JSON rule

Prose classifier with retrieved context saying contracts above $100,000 require executive review. This tests whether nearby business context moves the LOW boundary even though the refund rule is unchanged.

Read this as upward drift: once the prompt sees a nearby $100,000 business threshold, many amounts above the intended $100 boundary start looking LOW. The model is not failing to read the number; it is borrowing scale from context.

Reference rule

The fixed rule used for evaluation: LOW iff amount <= $100.

Baseline prompt

The same classifier for the same amount grid without the selected retrieved context.

Selected context

The selected experimental context added to the classifier.

LOW answers

How often the model answered LOW out of n=100.

Policy mismatches

How often the model-owned answer crossed the system-owned policy boundary. Non-zero values are bold because they are the trust-boundary failure.

Deterministic check

The same amount grid checked by code instead of model interpretation. This is the control condition: the $100 rule is enforced outside the model.

These counts are not expected to form a smooth dose-response curve. The point is narrower: nearby context can move a consequence-bearing boundary at all. Occasional divergences between LOW-answer counts and policy-mismatch counts are shown rather than smoothed away; once interpretation owns the boundary, the system cannot guarantee a clean explanation for every cell.

Amount	Reference rule	GPT-5.6					Gemini 3.6 Flash
		Baseline prompt		Context added $100k		Deterministic check	Baseline prompt		Context added $100k		Deterministic check
		LOW answers	Policy mismatches	LOW answers	Policy mismatches	Policy mismatches	LOW answers	Policy mismatches	LOW answers	Policy mismatches	Policy mismatches
$0	LOW	100/100	0/100None	100/100	0/100None	0/100None	100/100	0/100None	100/100	0/100None	0/100None
$5	LOW	100/100	0/100None	100/100	0/100None	0/100None	100/100	0/100None	100/100	0/100None	0/100None
$25	LOW	100/100	0/100None	100/100	0/100None	0/100None	100/100	0/100None	100/100	0/100None	0/100None
$49	LOW	100/100	0/100None	100/100	0/100None	0/100None	100/100	0/100None	100/100	0/100None	0/100None
$50	LOW	100/100	0/100None	100/100	0/100None	0/100None	100/100	0/100None	100/100	0/100None	0/100None
$51	LOW	3/100	97/100Some	98/100	2/100Some	0/100None	1/100	99/100Some	100/100	0/100None	0/100None
$75	LOW	100/100	0/100None	100/100	0/100None	0/100None	100/100	0/100None	100/100	0/100None	0/100None
$100	LOW	99/100	1/100Some	100/100	0/100None	0/100None	93/100	7/100Some	100/100	0/100None	0/100None
$101	NOT_LOW	0/100	0/100None	90/100	90/100Some	0/100None	0/100	0/100None	98/100	98/100Some	0/100None
$150	NOT_LOW	80/100	80/100Some	100/100	100/100All	0/100None	28/100	28/100Some	100/100	100/100All	0/100None
$250	NOT_LOW	91/100	91/100Some	100/100	100/100All	0/100None	7/100	8/100Some	100/100	100/100All	0/100None
$500	NOT_LOW	40/100	40/100Some	100/100	100/100All	0/100None	0/100	0/100None	100/100	100/100All	0/100None
$1,000	NOT_LOW	14/100	14/100Some	99/100	99/100Some	0/100None	0/100	0/100None	100/100	100/100All	0/100None

Each value keeps the raw count out of n=100. The table is meant to be read as drift, not as a model leaderboard: the same word, LOW, changes meaning when nearby context changes. The deterministic check is derived by applying the reference rule to the same amount grid, not by making another model call. Full prompts, raw calls, and generated data are in experiments/json-input-low.

The ordinary prose classifier moved with nearby context. A $5 gift-card note pushed the boundary down; a $100,000 contract-review note pushed it up. In this fixed-grid table, the only prompt-side condition that stayed close to the intended $100 boundary was not JSON out of the box; it was typed JSON with an explicit boundary rule. That is useful, but limited: the rule only becomes explicit after the system has already identified the boundary worth protecting. Historical exploratory runs also showed that explicit prompt-side rules can fail, so this remains steering, not deterministic enforcement. Compliance remains model behavior.

The deterministic check column is the real contrast. It applies the code-owned rule, LOW iff refund_claim_amount_usd <= 100, outside the JSON prompt and outside the model call, to the same grid without asking the model to infer the threshold. The reliable enforcement boundary belongs outside the model.

JSON can describe the contract. It is not the contract.

4. The Bash Quoting Incident

The same pattern appeared outside refunds.

An agent had been helping me administer three Raspberry Pis over SSH: create a dashboard, deploy a home automation app, adjust a config, clean up after a test. Later, two machines showed a small but suspicious symptom: their shell prompt customization had disappeared. The cluster was fine. The damage was contained. But something had touched user-home state in a way I had not intended.

The useful question was not how to restore prompt colors. It was why an approved command had crossed the wrong boundary. The proximate problem looked like command transport. The deeper problem was that the boundary lived in prose until the case where the distinction mattered.

I had given the model a prose rule: when command text becomes too fragile to quote safely, encode it or block it. That sounded operational. But the executable boundary still depended on model judgment at the moment of execution.

The repair was not a better sentence. It was structure: build commands as argv when possible, make the remote interpreter explicit, require absolute target paths, start from a safe working directory, block destructive operations by default, and log the approved command separately from the executed command.

A prompt can say do not delete user files or treat untrusted text as data. But if the model's intent cannot be translated into a verifiable command with its validations and controls intact, the boundary is only aspirational. The system has not gained a safety property. It has gained an instruction.

5. Evals Can Hide The Same Problem

The natural reaction to a failure like this is to add an eval.

That reaction is not wrong. Evals are useful. A CI eval can prevent a bad prompt from shipping. A production judge can create alerts, route suspicious cases to review, and show where the system is drifting under real context.

But observability is still downstream of the decision. A clean eval can detect that the label moves when a promotional note is present. It can trace the failure back to retrieved context, prior state, or another adjacent fragment. That is useful diagnostic evidence.

It can also push the team into the wrong repair path: spending its energy protecting model judgment from every contamination shape instead of giving the runtime the value the model was trying to infer. Each eval catches another symptom. None of them turns LOW into a policy.

The judge can still help. It can detect strange cases, explain uncertainty, create monitoring signals, and route edge cases to humans. Those explanations are useful when they expose the assumption that moved the outcome. They are still review artifacts, not ownership of the boundary.

6. The Runtime Owns The Boundary

This is the positive claim: prompts can propose, explain, and review. They should not own the boundary where a system changes state, skips review, triggers a workflow, blocks one, or otherwise lets a consequence-bearing decision take effect.

The model may own judgment. The system must own guarantees.

The subjective part is the judgment: safe, low risk, reasonable, allowed, enough, too difficult. Underneath those words is often a scalar the system forgot to own: a dollar amount, score, count, rank, permission class, proof standard, risk cutoff, or difficulty ceiling.

For shell execution, the boundary is not the sentence “be careful with destructive commands.” It is the shape of the command that can be executed, the directory it can start from, the paths it can touch, the user it can run as, and the side effects that require a separate gate.

A model can still help produce the action. It can translate the user's intent into a structured proposal:

{
  "tool": "shell_command",
  "host": "approved-runner",
  "intent": "inspect the article draft",
  "argv": ["sed", "-n", "210,260p", "/workspace/article/index.md"],
  "cwd": "/workspace/article",
  "side_effect": "read_only"
}

That is the handoff. The model proposes an action. The runtime decides whether the proposal fits the contract.

host and cwd must be approved before execution
argv must stay structured, not re-parsed from shell text
side effects, target paths, and results must be logged and gated

None of those rules needs to be clever. They should be deterministic so they can be reviewed in code. They can be versioned. They can fail closed. They can be audited after the fact.

The model's judgment still matters. It may decide that the user is asking to inspect a file rather than deploy a service. It may choose the right tool. It may explain why a command looks suspicious. It may ask for confirmation when the situation is ambiguous.

That is the architectural move. Keep the model in the loop where judgment is useful. Move the boundary into machinery that does not need to agree with the model to protect the system.

7. Pattern: Externalize, Pin, And Verify Judgment

The pattern is simple: when model judgment produces a value the system will rely on, move ownership of that value out of the prompt before it becomes authority. Then require the model to declare what it used, so the runtime can verify compliance without trusting the explanation.

Do not leave the contract in a sentence like this:

Approve the case when there is enough evidence.

In a refund review gate, the model can summarize receipts, account notes, prior tickets, and contradictions. But the gate should own the sufficiency rule: which signals count, which categories are required, what score cuts off approval, and which missing field forces human review.

Make the policy inspectable:

{
  "version": "2026-07-30",
  "minimum_independent_signals": 3,
  "minimum_average_score": 0.72,
  "required_categories": ["identity", "ownership"],
  "force_review_if_any_score_below": 0.40,
  "owner": "evidence-policy"
}

Make the model's use of the policy inspectable without letting it own the arithmetic:

{
  "evidence_policy_version": "2026-07-30",
  "evidence_record_id": "case-4821",
  "sufficiency_result": "approved"
}

Then validate the declaration by recomputing from the evidence record:

policy = load_json("policy/evidence-sufficiency.json")
evidence = load_json("evidence-record.json")
output = load_json("case-output.json")

assert output["evidence_policy_version"] == policy["version"]
assert output["evidence_record_id"] == evidence["id"]

signals = evidence["signals"]
independent_signals = count_independent_sources(signals)
average_score = mean(signal["score"] for signal in signals)
categories = {signal["category"] for signal in signals}

assert independent_signals >= policy["minimum_independent_signals"]
assert average_score >= policy["minimum_average_score"]
assert set(policy["required_categories"]).issubset(categories)
assert min(signal["score"] for signal in signals) >= policy["force_review_if_any_score_below"]
assert output["sufficiency_result"] == "approved"

That output check matters. The evidence record is the external source here: receipts, account facts, ticket history, and scores the application stored or computed before the model responded. The fields that control the decision must be application-owned or independently computed; if a model produced them, this verifier only moves the hidden contract one step upstream. The model cannot merely sound persuaded; it must leave a machine-readable claim that the runtime can compare with the policy and recomputed facts.

But output validation is not the same as ownership. If the validator is the first place where enough evidence becomes three signals, two categories, and a score cutoff, the system is still letting the model choose and hoping to catch drift afterward. The stronger architecture is: external owner first, model declaration second, deterministic verifier third.

Externalize ownership. Declare usage. Verify deterministically.

8. Detection: Look For Consequence-Bearing Judgment

The audit is not looking for important prompts. It is looking for hidden operational contracts.

A hidden operational contract appears when vague judgment changes what the system may do, but the system has no owned rule for that value.

The test is simple: vague judgment, real consequence, missing rule outside the prompt. Words like safe, low, enough, sensitive, allowed, reasonable, and similar are not defects by themselves. They become defects when the system lets them decide writes, deletions, publication, approvals, escalation, refusal, cost, privacy, or permissions.

A useful finding should name the operational parts:

Potential hidden contract:
- Instruction: "Approve the case when there is enough evidence."
- Decision: evidence_sufficiency
- Consequence: changes approval, escalation, and review behavior
- External pin: missing
- Hidden value: minimum signal count, required categories, score cutoff, review override
- Model-upgrade failure mode: one model may treat a persuasive summary as sufficient, another may require independent proof
- Suggested pin: policy.evidence_sufficiency in a versioned policy file

The detector finds candidates, not proven defects. Vague language is not automatically bad. A prompt can say “write clearly” or “be concise” without hiding a system contract. The dangerous cases are narrower: the model's interpretation changes whether the system acts, what it touches, what it publishes, what it refuses, what it escalates, or what it spends.

Some findings should stay in language because the consequence is low, the policy is genuinely contextual, or the cost of pinning the value would exceed the risk. Externalizing every judgment creates stale config, brittle gates, and policies nobody owns.

The architectural question is narrower: when a different interpretation changes approval, execution, publication, privacy, cost, or access, the value needs an owner outside the prompt.

I ran the same detector against a private internal prompt inventory. The useful findings were not generic prompt-quality complaints; they were candidate contracts around durable memory writes, context retention, publication routing, tool side effects, and privacy gates. In each case, the detector had to name the value being inferred before the architecture question became visible.

The article-facing detector is deliberately small. The boundary detector skill is an optional diagnostic aid for finding candidate hidden contracts; it is not a guarantee and it is not the evidence for the article. The implementation and experiment materials are in the prompt-contract-experiments GitHub repo.

9. What It Means

For a product team, the warning sign is not the word low, safe, similar, or enough. Those words are often the right interface for intent. The warning sign is when a different interpretation of the word changes what the product does: which claim skips review, which record gets merged, which command executes, which answer is published, which user is refused, which case reaches a human.

For engineers, the failure mode is narrower than “prompts are brittle.” The concrete failure mode is an unowned value. A dollar threshold, proof standard, count, permission class, distance function, or risk cutoff has been left inside language, then treated downstream as if it were a system property.

The owner is the missing part. If a prompt decides whether a claim skips review, someone owns that review boundary. If no team can point to the versioned rule, test, or deterministic check, the contract exists only as model behavior.

This is not an argument against model judgment. It is an argument for noticing when model judgment has quietly become part of an operational contract.

Language remains the right medium for intent, explanation, and review. It is the wrong storage layer for guarantees.

When model judgment controls a consequence-bearing value, promote the value out of the prompt. Pin it. Version it. Enforce it. Then let the model do what it is good at around that boundary, not instead of it.