DEMO · REPLAY OF REAL TEST RUNS
Catch the refund the agent should refuse.
One airline cancellation policy, compiled into an executable check. Seven generated cases on a simulated airline backend. A low-cost support agent on its own, then the same agent with the check enforced while it works.
THREE ACTS
What was missed,
found, and stopped.
Pick an act. The policy's decision tree stays the same; only what we know about each branch changes. Click any branch to open its case.
METHOD AND LIMITS
Every number on this page comes
from an archived run.
How it was produced
Limits
- One policy, seven generated cases, a handful of runs. A demonstration of the mechanism, not a benchmark.
- Models are not fully deterministic. The low-cost model failed the same three cases in all three unguarded runs and passed all seven in all five guarded runs; other models ran once.
- Before these runs one pipeline bug was fixed by hand: the step that lists the reasons a customer may state had missed "health" and "weather". Only post-fix runs are shown.
- The guardrail ran in enforce mode. In every run the agent corrected itself after one rejection; the override path never fired.
- Two of the seven cases describe the same situation (economy, no insurance, change of plan). Both are kept because the pipeline produced them.
- Act I is a simulated pre-launch scenario. The agent runs are real; the "shipped" framing is not a customer deployment.
- Frontier models were run once each as the unguarded agent: GPT-5.5, Claude Opus 5.5, Claude Sonnet 5.5 and Gemini 3.8 Flash all matched the policy on all seven cases. GPT-6 astra could not be run: on this gateway it needs the Responses API for tool use, which the pipeline does not use yet.
Results by run
Archived inputs, tool calls and model outputs are available on request.
Attempt ledger: every policy tried
| Policy | Backend | Compile and validate | Generated cases | Manual steps | Agent results |
|---|---|---|---|---|---|
| Airline cancellation (tau2-bench) | Real Tau2 airline DB and tools | Passed | 7 cases, all round-trip ready | One: the extracted reason list missed two values; fixed by hand, messages regenerated | This page |
| Retail order cancellation (tau2-bench) | Real Tau2 retail DB, generic adapter | Passed first try | 2 cases, both ready | None | GPT-4o mini 2/2. Trivial policy |
| FMLA leave eligibility (DOL Fact Sheet #28) | None; policy-only sandbox (lookup tools return the check's facts) | Oct 12: failed once (a non-total output rule), passed after the policy was reworded to a single decision. Oct 17, hardened pipeline, Gemini 3.8 Flash compiler: passed; 336 combinations → 33 outcomes, including the non-qualifying-reason branch the first compile had missed | Oct 12: 5 outcomes, user-stated fields wrong (hours and tenure treated as user claims, no identifier). Oct 17: 33 cases, identifiers and reasons extracted correctly | Oct 12: two (policy rewording; replacing the extracted user-stated fields). Oct 17: none | Oct 12 after fixes: GPT-4o mini 5/5, GPT-4.1 nano 3/5, nano with guardrail 5/5. Oct 17: Claude Haiku 4.5 33/33 |
| Retail return and exchange (tau2-bench) | Real Tau2 retail DB on Oct 12, then none (sandbox) | Oct 12: failed three times (invalid Rego on recompile; OPA evaluation error; a check with no input for return-vs-exchange). Oct 17, hardened pipeline, Gemini compiler: passed first try; 2,304 combinations → 51 outcomes (50 evaluated, the per-run cap) | Oct 12: unusable. Oct 17: 50 cases with order id, action and refund destination extracted correctly | Oct 12: policy simplified once, schema patched once. Oct 17: none | Oct 17: Claude Haiku 4.5 48/50; both misses are the agent adding a rule the policy does not state |
| Airline cancellation, re-run from scratch (Oct 17) | Real Tau2 airline DB and tools | Passed with the Gemini compiler; 120 combinations → 6 outcomes, all round-trip ready | 6 cases | None | Claude Haiku 4.5 5/6. The miss is a fidelity flaw in the compiled check: it treated the customer's stated reason "airline cancelled" as the fact, so the flights stayed "available" and the agent rightly escalated. A policy owner's review must catch this; the pipeline did not. |
Oct 12: of four policies, the pipeline was reliable end to end only on the airline policy it was developed on. Oct 17: after fixing the three failure classes those attempts exposed (user-stated fields duplicating looked-up facts, missing identifiers, airline heuristics applied to other domains), all three went through from scratch without manual edits. The compiler moved from GPT-4o to Gemini 3.8 Flash because our gateway lost its OpenAI access; the Oct 12 agent numbers elsewhere on this page are unchanged and still GPT-based.
The policy text the pipeline received
The compiled check (Rego, validated with OPA; not independently verified against the policy text)
Policy facts the oracle distinguishes
LET'S BUILD TRUST IN AGENTS
Testing an agent that
must follow written policy?
Research stage. Looking for two or three design partners: a support or operations agent that makes a policy-governed, high-value tool call (refunds, credits, approvals), a policy owner who can review compiled rules, and a scoped pilot.