We have enough practical experience with SSE right now. We don’t need to perfect the transport layer, lets move on to improve our production error handling architecture.
Step 12 – Production Hardening of the AI Integration
We’ll cover this as one compact step:
LLM request
├── timeout
├── rate limit
├── provider error
├── invalid response
├── logging
└── token/cost tracking
12.1 Add a custom AI error
Create:
app/services/ai/error.rb
class Ai::Error < StandardError
end
class Ai::ProviderError < Ai::Error
end
class Ai::RateLimitError < Ai::Error
end
class Ai::TimeoutError < Ai::Error
end
This gives our application its own error vocabulary instead of exposing SDK/provider exceptions everywhere.
12.2 Wrap the provider call
In Ai::Client, wrap the API call.
Conceptually:
def chat(messages:)
response = @client.chat.completions.create(
model: MODEL,
messages: messages
)
{
content: response.choices.first.message.content,
model: response.model,
input_tokens: response.usage&.prompt_tokens,
output_tokens: response.usage&.completion_tokens
}
rescue Faraday::TooManyRequestsError => e
raise Ai::RateLimitError, e.message
rescue Faraday::TimeoutError => e
raise Ai::TimeoutError, e.message
rescue Faraday::Error => e
raise Ai::ProviderError, e.message
end
The exact exception classes can depend on the SDK/version, so inspect the exception raised by your installed openai gem rather than blindly copying provider-specific classes.
The important architecture is:
OpenRouter/SDK error
↓
Ai::Client
↓
Ai::RateLimitError
Ai::TimeoutError
Ai::ProviderError
↓
Rails application
Your controllers don’t need to know OpenRouter’s exception hierarchy.
12.3 Add timeout thinking
Never allow an AI request to hang indefinitely.
A production system should have:
connection timeoutread/request timeout
and then either:
retry
or:
fail gracefully
depending on the failure.
A key interview answer:
Retry transient failures such as timeouts and 429s with bounded exponential backoff, but don’t blindly retry all errors.
12.4 Token tracking
We’re already storing:
input_tokensoutput_tokens
in messages.
That gives us an important operational capability:
conversation.messages.sum(:input_tokens)
and:
conversation.messages.sum(:output_tokens)
Now we can answer:
How many tokens did this conversation consume?
Later we can add pricing:
input tokens × input price
+
output tokens × output price
=
estimated cost
Don’t hard-code provider pricing into the model. Pricing changes.
12.5 Add request timing
For a production AI application, latency is valuable.
In Ai::Client:
started_at = Process.clock_gettime(Process::CLOCK_MONOTONIC)
response = ...
latency_ms =
((Process.clock_gettime(Process::CLOCK_MONOTONIC) - started_at) * 1000).round
Then eventually store:
latency_ms
on the message or in a separate AI usage/event table.
This allows:
modeltokenslatencyerrors
to be correlated.
12.6 Don’t log prompts blindly
Avoid:
Rails.logger.info(params)
for AI endpoints.
User prompts may contain:
- PII
- secrets
- customer information
- proprietary company data
Log metadata instead:
conversation_idmodellatencytoken countserror type
rather than dumping the entire conversation into logs.
12.7 Add application-level rate limiting
An expensive AI endpoint should never be unrestricted.
Conceptually:
User
↓
Rate limit
↓
AI endpoint
↓
LLM
For example:
10 requests/minute/user
The exact limit depends on your application.
This protects:
- cost
- provider quotas
- abuse
- system capacity
12.8 What about retries?
Use something like:
Timeout → retry
429 → retry with backoff
5xx → retry with backoff
400 → don't retry
401 → don't retry
invalid input → don't retry
The exact mapping depends on the provider.
A useful int. phrase:
“I distinguish transient failures from permanent failures and use bounded retries with exponential backoff.”
The Rails Drop 