Q19: Streaming Architecture: Server-Sent Events (SSE), WebSockets & The Vercel AI SDK Protocol - AI Engineering & LLMs (50 Qs)

1. Why Streaming is Mandatory for LLM UX

Because autoregressive models generate text sequentially at ~30–80 tokens per second:

Non-streaming: User stares at a blank screen for 6–10 seconds until the entire 500-token paragraph completes.
Streaming: The user receives the first token in < 300 ms (Time-To-First-Token, TTFT), reading tokens in real time as they generate.

2. Transport Protocol Comparison

Protocol	Connection Type	Multiplexing	Browser Overhead	Ideal Use Case
Server-Sent Events (SSE)	Unidirectional (Server → Client) over HTTP/2	High (reuses HTTP/2 connection)	Very Low (`EventSource` or `fetch` with `ReadableStream`)	Industry Standard for LLM chat completions
WebSockets	Full-Duplex (Bidirectional)	Requires dedicated TCP connection	High (custom heartbeat & reconnect logic)	Audio/Video Live API (e.g. Gemini Live, OpenAI Realtime)
HTTP Long Polling	Unidirectional HTTP/1.1	None	Very High	Obsolete for LLM applications

3. Production Fastify / Next.js SSE Streaming Implementation

typescript

// Next.js Route Handler (App Router: app/api/chat/route.ts)
import { OpenAIStream, StreamingTextResponse } from 'ai';
import OpenAI from 'openai';

const openai = new OpenAI();

export async function POST(req: Request) {
  const { messages } = await req.json();

  const response = await openai.chat.completions.create({
    model: 'gpt-4o',
    stream: true,
    messages,
  });

  // Convert raw generator into WHATWG ReadableStream
  const stream = new ReadableStream({
    async start(controller) {
      for await (const chunk of response) {
        const text = chunk.choices[0]?.delta?.content || '';
        if (text) {
          // SSE Data Framing Protocol
          controller.enqueue(new TextEncoder().encode(`data: ${JSON.stringify({ text })}\n\n`));
        }
      }
      controller.enqueue(new TextEncoder().encode('data: [DONE]\n\n'));
      controller.close();
    },
  });

  return new Response(stream, {
    headers: {
      'Content-Type': 'text/event-stream',
      'Cache-Control': 'no-cache, no-transform',
      'Connection': 'keep-alive',
    },
  });
}