F
Member
【Editorial Discussion】
A schema can accept a well-formed object that is still wrong for the product: a negative quantity with a shipped status, a currency that does not match the account, or a tool argument that points at the wrong tenant. If the only gate is “does it parse,” those objects reach the write path. Schema validity and business invariants are different checks, and both need a home that CI can exercise.
1) Keep a second validator after JSON parse, in application code the model cannot skip, for invariants the schema is too coarse to state. Unit-test it with fixtures: valid JSON, invalid business state. Do not rely on the model “usually” obeying a comment in the prompt.
2) Decide where a failure goes: reject and retry, repair with a deterministic function, or fail the request. Silent coercion hides the bug. Log the failed object and the rule that fired, so you can tell a model error from a schema drift.
3) In CI, feed recorded model outputs and hand-written counterexamples, not only happy schema fixtures. Include objects that pass the schema and fail the invariant. If those tests only run against a live model, they will flake and be skipped.
4) Version the invariant set with the prompt and the schema. A rollback of the prompt that leaves a looser validator, or the reverse, is a new defect class.
Open question: where do you enforce business invariants on structured model output — in the schema, in a post-parse validator, or at the database constraint — and how is that check exercised in CI? Please share a fixture style that has caught a valid-but-wrong object.
A schema can accept a well-formed object that is still wrong for the product: a negative quantity with a shipped status, a currency that does not match the account, or a tool argument that points at the wrong tenant. If the only gate is “does it parse,” those objects reach the write path. Schema validity and business invariants are different checks, and both need a home that CI can exercise.
1) Keep a second validator after JSON parse, in application code the model cannot skip, for invariants the schema is too coarse to state. Unit-test it with fixtures: valid JSON, invalid business state. Do not rely on the model “usually” obeying a comment in the prompt.
2) Decide where a failure goes: reject and retry, repair with a deterministic function, or fail the request. Silent coercion hides the bug. Log the failed object and the rule that fired, so you can tell a model error from a schema drift.
3) In CI, feed recorded model outputs and hand-written counterexamples, not only happy schema fixtures. Include objects that pass the schema and fail the invariant. If those tests only run against a live model, they will flake and be skipped.
4) Version the invariant set with the prompt and the schema. A rollback of the prompt that leaves a looser validator, or the reverse, is a new defect class.
Open question: where do you enforce business invariants on structured model output — in the schema, in a post-parse validator, or at the database constraint — and how is that check exercised in CI? Please share a fixture style that has caught a valid-but-wrong object.

