Q22: Structured Outputs & Function Calling: Constrained Decoding (CFG Grammars) vs. Pydantic - AI Engineering & LLMs (50 Qs)

1. The Challenge of Unconstrained Generation

When integrating LLMs into software pipelines (e.g., triggering a database write or calling a payment gateway), the model's output must adhere strictly to a deterministic schema (JSON, Protocol Buffers, SQL).

Prompt Engineering ("Output valid JSON"): Fragile. Models frequently add markdown backticks (```json), trailing commas, or explanatory preamble text.
Retry Loops: Incurs latency and cost penalties when regenerating failing payloads.

2. Constrained Decoding via Context-Free Grammars (CFG)

Modern production frameworks (OpenAI Structured Outputs, Outlines, llama.cpp, Guidance) enforce JSON schemas at the token generation level:

text

                        Next Token Logits from Transformer
                                        │
                                        ▼
                  ┌───────────────────────────────────────────┐
                  │   CFG Grammar Mask (JSON Schema DFA)      │
                  │   • Disallow all tokens that violate      │
                  │     valid JSON schema syntax              │
                  │   • Set forbidden token logits to -inf    │
                  └─────────────────────┬─────────────────────┘
                                        │
                                        ▼
                   100% Mathematically Guaranteed Valid JSON

3. Production Pydantic & Instructor Implementation

python

from pydantic import BaseModel, Field
from typing import List, Optional

class CustomerSupportTicket(BaseModel):
    ticket_id: str = Field(description="Unique UUID string")
    urgency_level: str = Field(description="Must be LOW, MEDIUM, or CRITICAL")
    category: str = Field(description="Billing, Bug, Feature, or Security")
    action_items: List[str] = Field(description="Concrete bullet points to resolve")
    sentiment_score: float = Field(ge=-1.0, le=1.0, description="Sentiment between -1 and 1")

# With OpenAI Structured Outputs (json_schema enforcement)
completion = client.beta.chat.completions.parse(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": "Extract structured ticket metadata."},
        {"role": "user", "content": "The production database is down and charging users twice!"}
    ],
    response_format=CustomerSupportTicket,
)

ticket: CustomerSupportTicket = completion.choices[0].message.parsed
assert ticket.urgency_level == "CRITICAL"