Fallback Logic in AI Systems: Designing for Failure

The Failure Modes

• Model API downtime
• Rate limiting
• Context window exceeded
• Malformed or low-quality output
• Response timeout
• Cost threshold exceeded

The Fallback Hierarchy

• Primary model: GPT-4o or Claude 3.5 Sonnet
• Fallback model: GPT-3.5 or Claude Haiku
• Cached response: Previously cached similar query result
• Template response: Safe generic response acknowledging limitations
• Human escalation: Route to human if all AI options fail

Handling Malformed Output

• Parse the output. If it fails...
• Retry with stricter prompt: "Return ONLY valid JSON."
• If retry fails, try with a different model
• If all retries fail, return a defined error response

Circuit Breakers and Cost Guards

If an AI endpoint fails 5 times in 60 seconds, stop sending requests for 2 minutes. Implement per-request token limits and daily cost caps — set alerts before hard cutoffs.

Build Production-Grade AI Systems

Our workshop covers reliability patterns, fallback design, and production deployment.