System design for probabilistic software

Classic system design assumes components fail in boring ways: timeouts, partitions, poisoned messages. Language models fail in literate ways. They produce a plausible wrong tool call. They report success. They change a parameter after you looked at it.

Design accordingly.

Put the model outside the trust boundary. It may propose. It may not authorize.

Make effects canonical. A tool name is not an effect. Amount, recipient, preconditions, and expected after-state are the effect.

Prefer fail closed. Stale manifests, revoked grants, widened delegations, unsigned seals — these are not warnings.

Separate evaluator failure from agent failure. A rate limit is not a hallucination. If your metrics cannot tell the difference, they will teach you the wrong lesson.

Keep artifacts local and reviewable. YAML goldens, JSON runs, hash-chained audit logs. If the only source of truth is a vendor UI, you do not own the system.

The prompt is a policy suggestion. The runtime is the policy.