Call LLM APIs in Production: Structure, Streaming, Errors and Cost

Define the task and its output

The feature summarises a course outline into a short description and three learning outcomes. Write the task as a specification: the input, the output format, length limits and rules such as 'use only the provided outline'. Request structured output, such as JSON matching a schema, when another part of the system consumes the result, and validate the response against that schema before using it. Many providers offer structured-output or tool-calling features that make this more reliable than asking for JSON in plain text.

Build the prompt from controlled parts

Keep instructions, reference material and user input clearly separated, for example with labelled sections. Treat any user-supplied or retrieved text as data, not as instructions, because it may contain text that tries to change the model's behaviour. Keep the system instructions in version control and change them deliberately, with evaluation, like any other code.

Handle latency with streaming and timeouts

Model responses can take several seconds. For text shown to users, stream the response so they see progress, and design the interface for a stream that ends early or fails partway. For background jobs, return a job ID and process the request asynchronously. Set a timeout on every call and show a clear message with a retry option when it expires.

Classify and handle errors

• Rate limits (HTTP 429) and temporary server errors: retry with exponential backoff and jitter, respecting any retry-after information.
• Invalid requests, such as an oversized input: do not retry; fix the request or show a helpful message.
• Refusals or safety blocks: handle as a valid outcome with an appropriate message, not as a crash.
• Malformed structured output: validate, optionally retry once with the validation error, then fall back.
• Provider outages: degrade gracefully, for example by hiding the AI summary and showing the original outline.

Control cost

Cost depends on input and output tokens and the model chosen. Cap input size and set a maximum output length per task. Use a smaller, cheaper model where evaluation shows it performs well enough, and reserve larger models for harder tasks. Cache results for identical inputs, such as a summary for an unchanged course outline. Record token usage per feature and per user, and set budgets with alerts.

Evaluate before and after release

Build a small test set of real or realistic inputs with expected properties: summaries stay under the length limit, mention no facts absent from the outline and return valid JSON. Run it whenever you change the prompt, model or parameters. After release, sample outputs for human review and collect user feedback, looking for patterns rather than reacting to single complaints.

Protect data

Send only the data the task needs. Remove personal information that is not required, follow your provider's data-handling terms and your own privacy policy and avoid logging full prompts and responses that contain sensitive content. Store API keys in a secret manager and use separate keys per environment.

Checklist

• The task, output schema and limits are specified
• Instructions, references and user input are separated
• Calls have timeouts, and user-facing output streams
• Errors are classified, with retries only where useful
• Token usage is capped, measured and budgeted
• An evaluation set runs on every change
• Only necessary data is sent and keys are protected