Design a Distributed Web Crawler (URL Frontier, Bloom Filters, and Politeness).

In Simple Words: Systematically crawl billions of web pages across the public internet for indexing, processing 100,000 pages per second while adhering to host politeness policies (robots.txt).

Key Architectural Components:

• URL Frontier: Prioritizes crawling order and prevents flooding a single target web server (Politeness queue based on host domain).
• Bloom Filter: Memory-efficient probabilistic data structure that prevents re-crawling duplicate URLs with 0% false negatives.
• Content Signature (SimHash / MinHash): Detects near-duplicate web page content to prevent indexing duplicate articles.

flowchart TD
    Frontier["URL Frontier (Prioritizer & Politeness Queues)"] --> Fetcher["Async Fetcher Worker Cluster"]
    Fetcher --> DNS["DNS Resolution Cache"]
    Fetcher --> Rob["Robots.txt Cache"]
    Fetcher --> Downloader["Download Content"]
    Downloader --> Dedup["Bloom Filter (Seen URL Check)"]
    Dedup --> Parser["HTML Link Extractor & Storage"]
    Parser -->|"Discovered Links"| Frontier