The problem
System prompts drift. Not immediately, but reliably: a persistent user, a novel domain, a slightly ambiguous problem, and the model will occasionally leak the final answer. This is not a bug in the model. It is a fact about long-form generation.
For a product whose entire promise is “we will not give the answer,” this drift is unacceptable.
The guard, in three layers
Layer 1 — heuristic
A cheap regex scan runs on every reply. It catches obvious leaks: “the answer is,” “you should get,” “therefore, the answer.” Near-zero latency. Near-perfect precision on the pattern set.
Layer 2 — Haiku cross-reference
If the heuristic doesn’t flag anything AND we have a known answer for this problem, a Haiku call cross-references the reply against the known answer. JSON-only output. One question: did this leak?
Layer 3 — regenerate or fallback
On a leak, we regenerate the tutor’s reply once with an injected “STRICT MODE — you just leaked the answer, respond with ONLY a guiding question” clause. On a second leak, we return the refusal template.
Why not just a stricter system prompt
Because prompts drift, and the promise doesn’t. The guard is defense-in-depth: even when the base prompt fails, the surface presented to the user holds.
Rates
In production, the heuristic catches roughly 78% of leaks. The Haiku cross-reference catches most of the rest. The regenerate-and-retry pass fixes about 60% of those. The refusal template fires on roughly one in a thousand replies.
That last number is the one that matters. One in a thousand is the rate at which the product visibly holds its stance under adversarial conditions. It is not zero, but it is the number our stance can afford.