Q3: Embedding Models & Distance Metrics: Cosine Similarity, Dot Product & Euclidean Distance - AI Engineering & LLMs (50 Qs)

1. Vector Distance Metrics in Semantic Retrieval

Semantic search maps textual chunks into high-dimensional vector spaces (e.g., 768 or 1536 dimensions). Similarity is determined by geometric distance metrics:

css

   Cosine Similarity (Angle θ)        Euclidean Distance (L2)           Dot Product (Magnitude + Angle)
              B                                  B                                  B
             /                                  /|                                 /
            /                                  / |                                /
           / θ                                /  | d                             /
          /──────── A                        /___| A                            /──────── A
      cos(θ) = (A·B)/(||A|| ||B||)      d = sqrt(sum((Ai - Bi)^2))            A·B = sum(Ai * Bi)

2. Mathematical Comparison & Selection Matrix

Metric	Formula	Range	Normalization Dependency	Best Used In
Cosine Similarity	$\cos(\theta) = \frac{u \cdot v}{\|u\| \|v\|}$	$[-1.0, 1.0]$	Invariant to vector length / magnitude	General semantic search where text length varies
Dot Product (Inner Product)	$\langle u, v \rangle = \sum_{i=1}^{n} u_i v_i$	$(-\infty, \infty)$	Highly sensitive to vector length	Normalized embeddings (unit vectors $\|u\|=1$) where Dot Product $\equiv$ Cosine
Euclidean Distance ($L_2$)	$\|u - v\|_2 = \sqrt{\sum (u_i - v_i)^2}$	$[0, \infty)$	Sensitive to vector length	Coordinate-based systems, image embeddings, clustering

[!IMPORTANT] Performance Optimization: Computing cosine similarity requires two vector norm divisions per comparison. In production vector databases (Milvus, Pinecone, Qdrant), always L2-normalize embeddings at ingestion time. Once normalized ($|v|=1$), Cosine Similarity is identical to Dot Product, which runs significantly faster via hardware SIMD/AVX instructions.

3. Python Implementation of Metric Equivalence

python

import numpy as np

def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

def l2_normalize(vec: np.ndarray) -> np.ndarray:
    norm = np.linalg.norm(vec)
    return vec if norm == 0 else vec / norm

# Proof of equivalence
v1 = np.array([0.4, 0.8, 0.2])
v2 = np.array([0.1, 0.9, 0.5])

# Standard cosine
cos_score = cosine_similarity(v1, v2)

# Fast Dot Product on normalized vectors
norm_v1 = l2_normalize(v1)
norm_v2 = l2_normalize(v2)
dot_score = np.dot(norm_v1, norm_v2)

assert np.isclose(cos_score, dot_score), "Normalized dot product must match cosine similarity!"