Eval & LLMOps

The Output Contract: Validating LLM JSON Before Production

Parsing a model response is not the same as validating it. This guide separates syntax, schema and semantic checks, maps each to official sources, lists the failure modes that reach production, and gives a go-live checklist you can hand to a reviewer. Include…

Halaman ini belum dilokalkan — materi sumber ditampilkan dalam bahasa aslinya.

Why an output contract belongs in the deployment, not the prompt

On a customer deployment, a language model is rarely the last component in the chain. Its output is parsed, written into a database, forwarded to a ticketing system, or shown to an operator who has to act on it. The interesting failure is not the model being wrong in an obvious way. It is the model returning something that looks perfectly plausible, parses as JSON, and still has a missing field, a number where the downstream service expects a string, or an extra key that quietly changes a branching decision. Everything upstream looks healthy. The incident appears three systems later, in someone else’s queue.

Inference. This is why we treat the output contract as a deployment boundary rather than a prompt-engineering detail. A prompt asks nicely. A contract decides whether the request is allowed to continue. Forward-deployed engineers are usually the ones who have to make that distinction concrete before go-live, because they are accountable for what happens on the customer’s production system, not for how elegant the demo looked.

What the standards actually say

Fact. JSON Schema is the standard vocabulary for describing the shape of a JSON document. Its keyword set includes type, required and additionalProperties, which let a consumer state which fields must exist, what type they must have, and whether unknown keys are permitted [S1]. The Model Context Protocol specification uses JSON Schema for tool input and output schemas, so the same vocabulary governs what a tool declares it accepts and returns [S3]. Together these two official sources establish that schema validation is a defined, portable contract mechanism and not a vendor convention.

Fact. Python’s standard json module exposes json.loads for decoding a JSON document, and malformed input raises JSONDecodeError [S2]. LangChain’s official documentation documents model integration and structured output as first-class concerns in an application stack [S4]. We cite these four sources for definitions and API behaviour only; we do not reproduce their text, and we do not claim any performance characteristic from them.

Inference. Note what those facts do not give you. A schema tells you the shape is legal. It does not tell you the content is true, that the value is the right one for this customer, or that the downstream system will accept it. A parser tells you the bytes are JSON. It does not tell you a required field survived. These are three separate checks, and collapsing them into one “validate the response” step is the most common reason a validation layer fails in production.

Three layers, in order

Layer 1 — Syntax. Parse the raw response with a real parser and treat a parse error as a first-class, observable outcome. Do not regex your way to a dictionary, and do not silently strip prose around a JSON block. If the model emits text before the object, that is a contract violation with a cause you can investigate, not a cleaning step to hide [S2].

Layer 2 — Shape. Validate the parsed object against a versioned schema: required fields present, types correct, enums respected, unknown keys either rejected or explicitly allowed. Keep the schema next to the code that consumes the object, and version it in the same repository. additionalProperties: false is a deliberate decision, not a default: rejecting unknown keys is safer for a stable downstream contract, while allowing them is kinder to a model that is still being tuned [S1][S3].

Layer 3 — Meaning. Assert what the schema cannot express. A confidence value of 0.0 may be perfectly valid JSON and still be useless. A date may be a string and still be in the past when it must be in the future. A ticket category may be in the enum and still contradict the ticket text. Write these as named assertions with named failure reasons. They are business rules, they change more often than the schema, and they deserve their own test suite.

Advice. Fail closed at layer 1 and layer 2, but decide deliberately at layer 3. A hard failure is right when a wrong value would write bad data; a fallback or human review is right when the cost of a refusal exceeds the cost of a check. Record which branch you took. “The model returned something odd” is not an incident report; “layer 2 rejected missing field customer_id, request routed to retry, then to review” is.

Illustrative example (not measured)

Example, not a benchmark. The sketch below shows the shape of a three-layer gate. It is illustrative scaffolding: it has not been benchmarked, it is not a library, and it makes no accuracy, latency or cost claim. Adapt it to your own schema and your own failure policy before using it in production.

import json
from json.decoder import JSONDecodeError

def accept(raw, schema_validate, semantic_checks):
    # Layer 1: syntax. Never strip or "repair" silently.
    try:
        obj = json.loads(raw)
    except JSONDecodeError as exc:
        return reject("syntax", str(exc))

    # Layer 2: shape, against a versioned schema.
    errors = schema_validate(obj)
    if errors:
        return reject("shape", "; ".join(errors))

    # Layer 3: business meaning, named assertions.
    for name, check in semantic_checks.items():
        if not check(obj):
            return reject("semantics", name)

    return accept_object(obj)

def reject(layer, reason):
    # Log the layer and reason; do not log the raw payload if it may carry PII.
    return {"ok": False, "layer": layer, "reason": reason}

Advice. Log the failing layer and reason, not the raw payload: on a real deployment the payload may contain customer data. A validation log that leaks the very data you were trying to protect is a worse outcome than the malformed response.

Failure modes worth writing down before go-live

FailureWhich layer catches itWhat to record
Prose wrapped around the JSON objectSyntaxParse error, model/version, prompt template id
Required key missing or nullShapeSchema version, field name
Number where a string was declaredShapeSchema version, declared vs observed type
Extra key silently addedShape (if additionalProperties: false)Key name, decision to reject or allow
Valid JSON, wrong business valueMeaningAssertion name, input features, routing decision
Retry storm after a validation failurePolicyRetry count, backoff, circuit-breaker state

Inference. The last row is the one teams forget. Once you start rejecting outputs automatically, a systematic model or prompt regression can turn into a retry storm. A bounded retry with backoff, a circuit breaker, and a path to human review are part of the contract, even though no schema expresses them.

A go-live checklist you can actually hand to a reviewer

  1. Name the downstream consumer and the exact fields it requires, with an owner for each.
  2. Keep one versioned schema in the repository; reference it from code, never paste a second copy into a prompt.
  3. Decide additionalProperties explicitly and write down why.
  4. List the semantic assertions separately from the schema, with named failure reasons.
  5. Log the failing layer and reason; never log raw payloads that may contain personal data.
  6. Bound retries, add backoff, and define what happens when the breaker opens.
  7. Add a regression fixture for every real validation failure you see in production.
  8. Record the model and prompt-template version alongside each validation outcome.

Advice. Bring the checklist to the go-live review with the failure log from a staging run. A reviewer can accept a contract they can see failing in a controlled way; they cannot accept a verbal assurance that “the model usually returns JSON”.

Limits of this guide

We did not run an accuracy, latency or cost benchmark for anything described here. No framework is claimed to be better than another, no schema library is endorsed, and the code sketch is an illustrative example rather than tested production code. The official sources below are cited for definitions and API behaviour only; their text is not reproduced, and no commercial reuse licence is asserted. Version-sensitive details such as API parameters, model capabilities and pricing must be re-checked against the linked official documentation on the date you deploy — this guide records a verification date, not a permanent guarantee.

Sources

Statement types used above: Fact = traceable to an official source; Inference = our engineering reasoning; Advice = a practice recommendation; Example = illustrative code, not a measurement.

Yang bisa Anda lakukan sekarang

  • Parsing, shape validation and semantic assertions are three different checks
  • Version the schema next to the code that consumes the object
  • Log the failing layer and reason, never the raw payload
  • Bound retries: automatic rejection can become a retry storm
Paket rekomendasi

FDE Deployment Toolkit 2026

Arsitektur referensi, checklist go-live 60 poin, model biaya GPU, dan daftar pendek vendor — empat artefak yang selalu Anda bangun ulang di setiap deployment.

$19 Beli →