AI Engineering & LLMs (50 Qs)•Part 1: RAG Architecture, Chunking & Vector Search
Cross-Encoder Re-Ranking: Architecture, Late Interaction (ColBERT) & Two-Stage Retrieval
1. The Two-Stage Retrieval Funnel
Bi-encoders (e.g., text-embedding-3-small, BGE) encode query and documents independently into fixed-length vectors. Because tokens cannot interact across query and document during embedding generation, bi-encoders sacrifice subtle semantic nuance for retrieval speed:
less
[ 10,000,000 Document Chunks ]
│
▼ Stage 1: Bi-Encoder (Dense Vector Search / BM25)
[ Top 50–100 Candidate Chunks ] ──► (Fast ANN search: < 10ms)
│
▼ Stage 2: Cross-Encoder Re-Ranker (Full Attention)
[ Top 3–5 Pristine Chunks ] ──► (Deep cross-attention: ~50ms)
│
▼
[ LLM Prompt Context ]
2. Bi-Encoder vs. Cross-Encoder Architecture
ini
Bi-Encoder: Independent Feed-Forward
Query ──► [ Transformer ] ──► Vector Q ──┐
├─► Cosine Sim (Fast!)
Doc ──► [ Transformer ] ──► Vector D ──┘
Cross-Encoder: Joint Multi-Head Attention
[CLS] Query [SEP] Document [SEP] ──► [ Full Transformer Attention ] ──► Relevance Score [0, 1]
3. Production Two-Stage Retrieval Implementation
python
from typing import List, Dict
import numpy as np
# Conceptual implementation of Stage 1 + Stage 2 Funnel
class TwoStageRAGPipeline:
def __init__(self, vector_store, reranker_model):
self.vector_store = vector_store
self.reranker = reranker_model
def retrieve(self, query: str, top_k: int = 5) -> List[Dict]:
# Stage 1: Fast candidate retrieval (retrieve 50 candidates)
candidates = self.vector_store.search(query, k=50)
# Stage 2: Cross-Encoder Re-ranking
# Pair query with each candidate for joint cross-attention
pairs = [[query, doc["content"]] for doc in candidates]
scores = self.reranker.predict(pairs)
for i, doc in enumerate(candidates):
doc["rerank_score"] = float(scores[i])
# Sort descending by re-rank score and return top K
ranked_docs = sorted(candidates, key=lambda x: x["rerank_score"], reverse=True)
return ranked_docs[:top_k]