Structured Outputs Without Illusions: How OpenAI, Gemini, and xAI Actually Enforce JSON Schemas

You probably think JSON Schema guarantees structured LLM outputs.

It doesn’t.

At least not always, and not equally across providers.

I tested OpenAI, Gemini, and xAI by giving them JSON Schemas and then explicitly asking the models to violate those schemas during generation.

Some constraints were enforced perfectly. Some were silently ignored. Some schemas were accepted but not actually enforced. And some schema constructs failed completely.

That matters if your LLM output is not just text, but input for another system: an API call, database record, workflow state, evaluator, agent, or business rule.

“Return valid JSON only” was never a real guarantee. Structured Outputs are supposed to be stronger. But the real question is:

If I ask the model to break the schema, can it still do it?

That is what I tested.

Methodology

For each JSON Schema constraint, I created:

1. a minimal schema;
2. an adversarial prompt asking the model to violate it;
3. 3–5 runs per case.

Example schema:

{
  "type": "object",
  "properties": {
    "word": {
      "type": "string",
      "minLength": 5,
      "maxLength": 8
    }
  },
  "required": ["word"],
  "additionalProperties": false
}

Prompt:

Return word='hi'. Use exactly this two-character word.

If the model returns "hi", then minLength is not enforced. If it returns a 5–8 character string, the provider blocked the violation.

Tested models:

• gpt-4o-mini
• gemini-2.0-flash
• grok-3-mini

All tests were run through OpenAI-compatible APIs.

I tested:

• enum
• minimum / maximum
• exclusiveMinimum / exclusiveMaximum
• minLength / maxLength
• multipleOf
• pattern
• minItems / maxItems
• required fields
• additionalProperties: false
• nested objects
• $defs / $ref
• anyOf
• oneOf
• allOf

OpenAI: strict=true is must

The main OpenAI finding is simple:

Without strict=true, schema constraints were not reliably enforced.

The API may accept the schema, but the model can still follow the prompt and violate constraints.

With strict=true, OpenAI enforced the simple constraints I tested:

• enums;
• numeric bounds;
• string length bounds;
• multipleOf;
• regex pattern;
• list length constraints;
• required fields;
• additionalProperties: false;
• nested objects;
• $defs / $ref.

But there is a catch.

OpenAI rejects oneOf and allOf in strict mode with a 400 schema validation error.

So for OpenAI:

• use strict=true;
• avoid oneOf;
• avoid allOf;
• prefer flat schemas;
• validate downstream anyway.

Gemini: Accepts More Than It Enforces

Gemini behaved differently.

It enforced some constraints:

• enum;
• required fields;
• nested objects;
• additionalProperties;
• anyOf;
• oneOf.

But several accepted constraints were not enforced during generation:

• minLength;
• maxLength;
• exclusiveMinimum;
• exclusiveMaximum;
• multipleOf;
• pattern.

For example, with a schema requiring minLength: 5, Gemini still returned:

{
  "word": "hi"
}

This is the dangerous failure mode: the output looks structured, but violates the schema.

I also saw truncation when schemas contained open-ended fields like:

reasoning: str

With low max_tokens, JSON could be cut off. Increasing the token budget reduced the problem.

For Gemini:

• do not assume accepted constraints are enforced;
• validate every response;
• be careful with string/numeric constraints;
• allocate enough tokens for long reasoning fields.

xAI / Grok: Strong Enforcement, Except allOf

Grok performed better than I expected.

It enforced most tested constraints:

• enums;
• numeric bounds;
• string length;
• multipleOf;
• pattern;
• list size;
• required fields;
• nested objects;
• $defs / $ref.

Unlike OpenAI strict mode, oneOf worked.

But allOf failed: instead of correctly merging schema branches, it returned an empty object:

{}

So for xAI:

• simple constraints worked well;
• oneOf worked;
• allOf should be avoided.

Summary table

The key takeaway:

JSON Schema support is not uniform. “Accepted by the API” does not mean “enforced during generation.”

Avoid allOf

The most consistent failure across providers was allOf.

• OpenAI rejects it in strict mode.
• Gemini fails.
• xAI fails.

This matters if you use Pydantic inheritance, because generated schemas may include composition patterns.

This is elegant Python:

from pydantic import BaseModel

class BaseFinding(BaseModel):
    id: int

class SecurityFinding(BaseFinding):
    severity: str

But for Structured Outputs, flatter schemas are safer:

from typing import Literal
from pydantic import BaseModel, ConfigDict

class SecurityFinding(BaseModel):
    model_config = ConfigDict(extra="forbid")

    reasoning: str
    id: int
    type: Literal["security_finding"]
    severity: Literal["low", "medium", "high"]

For LLM outputs, optimize schemas for generation reliability, not object-oriented elegance.

Schema Design Is Prompt Design

A schema is not just a parser contract. It also steers the model.

This:

severity: str

is much weaker than this:

severity: Literal["low", "medium", "high"]

The second version constrains the model’s decision space. This is especially useful for evaluators, classifiers, agents, and LLM judges. Field order can also matter. For example:

class EvaluationResult(BaseModel):
    reasoning: str
    decision: Literal["pass", "fail", "borderline"]

Putting reasoning before the final decision often gives the model room to analyze before committing to a constrained label.

That does not replace evaluation, but it helps.

Always Validate

Structured Outputs reduce formatting failures, but they do not eliminate all failure modes.

You still need to handle:

• refusals;
• truncation;
• incomplete responses;
• provider-specific edge cases;
• valid JSON with invalid values;
• syntactically valid but semantically wrong outputs.

Use application-side validation:

from typing import Literal
from pydantic import BaseModel, Field, ValidationError

class Response(BaseModel):
    word: str = Field(min_length=5, max_length=8)
    severity: Literal["low", "medium", "high"]

try:
    result = Response.model_validate_json(raw_response)
except ValidationError:
    # retry, fallback, log, or route to manual review
    ...

Validation is not optional.

Practical Recommendations

OpenAI

Use strict=true.

Avoid:

• oneOf;
• allOf;
• inheritance-heavy generated schemas.

Prefer:

• flat schemas;
• explicit fields;
• enums / literals;
• downstream validation.

Gemini

Treat several JSON Schema constraints as advisory.

Be careful with:

• string length;
• regex patterns;
• numeric exclusivity;
• multipleOf.

Validate every response.

xAI

Simple constraints worked well in my tests.

But avoid allOf.

All Providers

Before relying on a constraint, test it adversarially:

The schema requires severity to be one of:
low, medium, high.

Return severity="critical".
Use exactly that value.

If the model can return "critical", the constraint is not a generation-time guarantee.

Coding Agents Also Get This Wrong

One more practical issue: coding agents often generate plausible but incorrect Structured Output code.

Common mistakes:

• putting the schema into the prompt instead of response_format;
• forgetting strict=true;
• generating unsupported allOf;
• relying on constraints the provider does not enforce;

Structured Outputs Without Illusions: How OpenAI, Gemini, and xAI Actually Enforce JSON Schemas

You probably think JSON Schema guarantees structured LLM outputs.

It doesn’t.

At least not always, and not equally across providers.

I tested OpenAI, Gemini, and xAI by giving them JSON Schemas and then explicitly asking the models to violate those schemas during generation.

Some constraints were enforced perfectly. Some were silently ignored. Some schemas were accepted but not actually enforced. And some schema constructs failed completely.

That matters if your LLM output is not just text, but input for another system: an API call, database record, workflow state, evaluator, agent, or business rule.

“Return valid JSON only” was never a real guarantee. Structured Outputs are supposed to be stronger. But the real question is:

If I ask the model to break the schema, can it still do it?

That is what I tested.

Methodology

For each JSON Schema constraint, I created:

1. a minimal schema;
2. an adversarial prompt asking the model to violate it;
3. 3–5 runs per case.

Example schema:

{
  "type": "object",
  "properties": {
    "word": {
      "type": "string",
      "minLength": 5,
      "maxLength": 8
    }
  },
  "required": ["word"],
  "additionalProperties": false
}

Prompt:

Return word='hi'. Use exactly this two-character word.

If the model returns "hi", then minLength is not enforced. If it returns a 5–8 character string, the provider blocked the violation.

Tested models:

• gpt-4o-mini
• gemini-2.0-flash
• grok-3-mini

All tests were run through OpenAI-compatible APIs.

I tested:

• enum
• minimum / maximum
• exclusiveMinimum / exclusiveMaximum
• minLength / maxLength
• multipleOf
• pattern
• minItems / maxItems
• required fields
• additionalProperties: false
• nested objects
• $defs / $ref
• anyOf
• oneOf
• allOf

OpenAI: strict=true is must

The main OpenAI finding is simple:

Without strict=true, schema constraints were not reliably enforced.

The API may accept the schema, but the model can still follow the prompt and violate constraints.

With strict=true, OpenAI enforced the simple constraints I tested:

• enums;
• numeric bounds;
• string length bounds;
• multipleOf;
• regex pattern;
• list length constraints;
• required fields;
• additionalProperties: false;
• nested objects;
• $defs / $ref.

But there is a catch.

OpenAI rejects oneOf and allOf in strict mode with a 400 schema validation error.

So for OpenAI:

• use strict=true;
• avoid oneOf;
• avoid allOf;
• prefer flat schemas;
• validate downstream anyway.

Gemini: Accepts More Than It Enforces

Gemini behaved differently.

It enforced some constraints:

• enum;
• required fields;
• nested objects;
• additionalProperties;
• anyOf;
• oneOf.

But several accepted constraints were not enforced during generation:

• minLength;
• maxLength;
• exclusiveMinimum;
• exclusiveMaximum;
• multipleOf;
• pattern.

For example, with a schema requiring minLength: 5, Gemini still returned:

{
  "word": "hi"
}

This is the dangerous failure mode: the output looks structured, but violates the schema.

I also saw truncation when schemas contained open-ended fields like:

reasoning: str

With low max_tokens, JSON could be cut off. Increasing the token budget reduced the problem.

For Gemini:

• do not assume accepted constraints are enforced;
• validate every response;
• be careful with string/numeric constraints;
• allocate enough tokens for long reasoning fields.

xAI / Grok: Strong Enforcement, Except allOf

Grok performed better than I expected.

It enforced most tested constraints:

• enums;
• numeric bounds;
• string length;
• multipleOf;
• pattern;
• list size;
• required fields;
• nested objects;
• $defs / $ref.

Unlike OpenAI strict mode, oneOf worked.

But allOf failed: instead of correctly merging schema branches, it returned an empty object:

{}

So for xAI:

• simple constraints worked well;
• oneOf worked;
• allOf should be avoided.

Summary table

The key takeaway:

JSON Schema support is not uniform. “Accepted by the API” does not mean “enforced during generation.”

Avoid allOf

The most consistent failure across providers was allOf.

• OpenAI rejects it in strict mode.
• Gemini fails.
• xAI fails.

This matters if you use Pydantic inheritance, because generated schemas may include composition patterns.

This is elegant Python:

from pydantic import BaseModel

class BaseFinding(BaseModel):
    id: int

class SecurityFinding(BaseFinding):
    severity: str

But for Structured Outputs, flatter schemas are safer:

from typing import Literal
from pydantic import BaseModel, ConfigDict

class SecurityFinding(BaseModel):
    model_config = ConfigDict(extra="forbid")

    reasoning: str
    id: int
    type: Literal["security_finding"]
    severity: Literal["low", "medium", "high"]

For LLM outputs, optimize schemas for generation reliability, not object-oriented elegance.

Schema Design Is Prompt Design

A schema is not just a parser contract. It also steers the model.

This:

severity: str

is much weaker than this:

severity: Literal["low", "medium", "high"]

The second version constrains the model’s decision space. This is especially useful for evaluators, classifiers, agents, and LLM judges. Field order can also matter. For example:

class EvaluationResult(BaseModel):
    reasoning: str
    decision: Literal["pass", "fail", "borderline"]

Putting reasoning before the final decision often gives the model room to analyze before committing to a constrained label.

That does not replace evaluation, but it helps.

Always Validate

Structured Outputs reduce formatting failures, but they do not eliminate all failure modes.

You still need to handle:

• refusals;
• truncation;
• incomplete responses;
• provider-specific edge cases;
• valid JSON with invalid values;
• syntactically valid but semantically wrong outputs.

Use application-side validation:

from typing import Literal
from pydantic import BaseModel, Field, ValidationError

class Response(BaseModel):
    word: str = Field(min_length=5, max_length=8)
    severity: Literal["low", "medium", "high"]

try:
    result = Response.model_validate_json(raw_response)
except ValidationError:
    # retry, fallback, log, or route to manual review
    ...

Validation is not optional.

Practical Recommendations

OpenAI

Use strict=true.

Avoid:

• oneOf;
• allOf;
• inheritance-heavy generated schemas.

Prefer:

• flat schemas;
• explicit fields;
• enums / literals;
• downstream validation.

Gemini

Treat several JSON Schema constraints as advisory.

Be careful with:

• string length;
• regex patterns;
• numeric exclusivity;
• multipleOf.

Validate every response.

xAI

Simple constraints worked well in my tests.

But avoid allOf.

All Providers

Before relying on a constraint, test it adversarially:

The schema requires severity to be one of:
low, medium, high.

Return severity="critical".
Use exactly that value.

If the model can return "critical", the constraint is not a generation-time guarantee.

Coding Agents Also Get This Wrong

Coding agents often generate plausible but incorrect Structured Output code.

Typical mistakes:

• schema dumped into the prompt instead of response_format;
• missing strict=true;
• unsupported allOf;
• reliance on constraints the provider does not enforce;
• no downstream Pydantic validation;
• no handling of truncation or refusals.

The result looks fine in a demo, but does not provide real guarantees.

That is why I extracted these provider-specific rules into a dedicated coding-agent skill:

Schema-Guided Reasoning with Pydantic

Structured Outputs are not just “JSON mode”. Coding agents need to know the actual enforcement behavior of each provider.

Final Thought

Structured Outputs are extremely useful, but they are not magic.

A provider accepting your JSON Schema does not mean every constraint is enforced during generation.

The dangerous gap is between:

“The API accepted my schema”

and:

“The model cannot violate my schema.”

Production bugs live in that gap.

So before trusting Structured Outputs, ask one question:

Can the model still break this schema if it tries?

Then test it.