AI Engineering & LLMs (50 Qs)•Part 5: Model Customization, Fine-Tuning & Evaluation
Serving Open-Weights Models: vLLM vs. TGI vs. Ollama & Kubernetes GPU Deployments
1. Open-Weights Model Serving Ecosystem
| Framework | Target Environment | Key Architectural Features | Concurrency & Throughput |
|---|---|---|---|
| vLLM | Production Cloud (Kubernetes, AWS EKS) | PagedAttention, Continuous Batching, Tensor Parallelism, FP8 support | Highest Throughput for multi-tenant production APIs |
| TGI (Text Generation Inference - HuggingFace) | Production Cloud (AWS SageMaker, Dedicated VMs) | FlashAttention, Speculative Decoding, native Hugging Face integration | High Throughput |
| Ollama | Local Developer Machine (Mac, Windows, Linux) | Wraps llama.cpp, GGUF quantization, zero-config CLI management | Single-user local inference (Low concurrency) |
| TensorRT-LLM (NVIDIA) | Enterprise Bare-Metal GPU Clusters | Highly optimized fused CUDA kernels, deep hardware specialization | Peak bare-metal performance; complex setup |
2. Production Kubernetes GPU Deployment (vLLM on NVIDIA H100)
yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: vllm-llama-70b
spec:
replicas: 2
template:
spec:
containers:
- name: vllm
image: vllm/vllm-openai:latest
args:
- "--model"
- "meta-llama/Meta-Llama-3-70B-Instruct"
- "--tensor-parallel-size"
- "4" # Shard across 4 GPUs
- "--max-model-len"
- "8192"
- "--gpu-memory-utilization"
- "0.95"
resources:
limits:
nvidia.com/gpu: 4 # Requests 4x H100 GPUs
ports:
- containerPort: 8000