Behavioral & Leadership (50 Qs)•Part 4: Production Incident Management & Post-Mortems (37 - 44)
How do you design graceful degradation and circuit breakers into user-facing distributed systems?
In Simple Words: When a downstream dependency fails or times out, a Circuit Breaker trips, preventing downstream cascading failure and returning a sensible fallback (e.g. cached data or empty recommendation widget) instead of crashing the entire page.
The 3 States of a Circuit Breaker:
- Closed: Normal operations; requests pass through.
- Open: Error threshold exceeded (e.g. >50% timeouts); requests fail fast immediately without hitting downstream servers.
- Half-Open: Periodic test requests sent to downstream service; if successful, circuit resets to Closed.
Practical Script & Summary Template
typescript
// Production Circuit Breaker implementation using Opossum:
import CircuitBreaker from 'opossum';
async function fetchPersonalizedRecommendations(userId: string) {
return recommendationApi.get(`/users/${userId}/recommendations`);
}
const breaker = new CircuitBreaker(fetchPersonalizedRecommendations, {
timeout: 300, // If request takes >300ms, consider it failed
errorThresholdPercentage: 50, // Trip if 50% of requests fail
resetTimeout: 10000 // Test downstream again after 10s
});
// Graceful Degradation Fallback:
breaker.fallback(() => {
// Return static top-10 popular items instead of failing customer's page!
return getCachedPopularItems();
});