Domain Adaptation & Continued Pre-Training vs. Catastrophic Forgetting

1. When Instruction Fine-Tuning is Insufficient

When introducing an LLM to a completely novel domain with proprietary jargon and vast corpus data (e.g., clinical oncology, specialized enterprise ERP codebases, legal statutes in a foreign language):

• RAG struggles because the base model does not understand the fundamental domain vocabulary.
• Instruction fine-tuning (LoRA) fails because it trains on Q&A pairs, not raw foundational knowledge.

Continued Pre-Training (Domain Adaptation) continues next-token prediction self-supervised training on gigabytes of raw, uncurated domain text.

2. Catastrophic Forgetting & How to Mitigate It

When models learn new domain distributions, their weights shift, severely degrading prior general abilities (math, coding, logical reasoning):

Before Adaptation:   [ High General Reasoning (MMLU 85%) ] ── [ Zero Oncology Knowledge ]
After Naive Training: [ Low General Reasoning (MMLU 55%) ]  ── [ High Oncology Knowledge ]
                               ▲ Catastrophic Forgetting!

Production Mitigations:

1. Replay Buffer (Data Mixing): Always mix 10% to 20% general pre-training data (Wikipedia, open web, GitHub code) into the domain training corpus.
2. Low Learning Rates: Use a learning rate that is an order of magnitude lower than original pre-training (e.g., $1e-5$ to $5e-6$) with a cosine decay schedule.
3. Weight Regularization (EWC): Elastic Weight Consolidation penalizes changes to weights identified as critical to general tasks.