Integrate AI with Rails: AI bootcamp for Developers – Day 7 – AI Response Streaming

Now let’s implement streaming. OpenRouter supports Server-Sent Events (SSE) when stream: true, and the current Ruby SDK exposes Chat Completions streaming through stream_raw; its higher-level stream helper is not available in every released SDK version. (OpenRouter)

We’ll keep the implementation practical and compatible with the SDK behavior you’re using.

Step 9 – Stream the AI response

What changes?

Currently:

Browser
  ↓
POST
  ↓
Rails waits for entire LLM response
  ↓
redirect

We want:

Browser
  ↓
POST
  ↓
Rails
  ↓
OpenRouter SSE stream
  ↓
token
token
token
token
  ↓
Browser

SSE is a long-lived HTTP response where the server sends incremental events. OpenRouter explicitly supports this with stream: true.

9.1 First, prove streaming works from Ruby

Before involving Rails, modify Ai::Client temporarily with a method:

def stream_chat(messages:, &on_delta)
  stream = @client.chat.completions.stream_raw(
    model: MODEL,
    messages: messages
  )

  stream.each do |chunk|
    delta = chunk.choices.first&.delta&.content
    on_delta.call(delta) if delta.present?
  end
end

The current SDK’s stream_raw returns an enumerable stream of chat completion chunks. (RubyDoc)

Now from Rails console:

conversation = Conversation.first

messages = Ai::PromptBuilder
  .new(conversation: conversation)
  .build

Then:

Ai::Client.new.stream_chat(messages: messages) do |delta|
  print delta
  $stdout.flush
end

You should see the answer appearing progressively:

Ruby is a programming language...

instead of getting the entire answer at once.

Why $stdout.flush?

Ruby can buffer stdout. Flushing makes each chunk visible immediately in the console.

9.2 Now expose streaming from Rails

Instead of making MessagesController#create wait for the completed response, we’ll create a streaming endpoint.

Open:

config/routes.rb

Add:

resources :conversations, only: [:create, :show] do
  resources :messages, only: [:create]
end

get "/conversations/:conversation_id/messages/stream",
    to: "messages#stream",
    as: :conversation_messages_stream

9.3 Add the streaming controller action

Open:

app/controllers/messages_controller.rb

Add:

include ActionController::Live

and:

def stream
  conversation = Conversation.find(params[:conversation_id])

  response.headers["Content-Type"] = "text/event-stream"
  response.headers["Cache-Control"] = "no-cache"
  response.headers["X-Accel-Buffering"] = "no"

  sse = SSE.new(response.stream)

  messages = Ai::PromptBuilder
    .new(conversation: conversation)
    .build

  content = +""

  begin
    Ai::Client.new.stream_chat(messages: messages) do |delta|
      next if delta.blank?

      content << delta

      sse.write(
        { content: delta },
        event: "message"
      )
    end

    sse.write(
      { done: true },
      event: "done"
    )
  ensure
    sse.close
    response.stream.close
  end
end

But Rails doesn’t provide SSE automatically.

Add:

include ActionController::Live

and use Rails’ ActionController::Live::SSE if available in our Rails 8.1 setup, or otherwise we can use the standard SSE format directly. Rails 8.1’s Live controller infrastructure is the relevant mechanism here.

To avoid another dependency, let’s actually use the raw SSE format ourselves.

Replace the sse.write(...) parts with:

response.stream.write(
  "event: message\n" \
  "data: #{JSON.generate(content: delta)}\n\n"
)

and completion:

response.stream.write(
  "event: done\n" \
  "data: #{JSON.generate(done: true)}\n\n"
)

So the complete action becomes:

include ActionController::Live

def stream
  conversation = Conversation.find(params[:conversation_id])

  response.headers["Content-Type"] = "text/event-stream"
  response.headers["Cache-Control"] = "no-cache"
  response.headers["X-Accel-Buffering"] = "no"

  messages = Ai::PromptBuilder
    .new(conversation: conversation)
    .build

  begin
    Ai::Client.new.stream_chat(messages: messages) do |delta|
      next if delta.blank?

      response.stream.write(
        "event: message\n" \
        "data: #{JSON.generate(content: delta)}\n\n"
      )
    end

    response.stream.write(
      "event: done\n" \
      "data: #{JSON.generate(done: true)}\n\n"
    )
  rescue IOError
    # Browser disconnected.
  ensure
    response.stream.close
  end
end

9.4 What’s happening?

The server sends chunks like:

event: message
data: {"content":"Ruby"}

event: message
data: {"content":" is"}

event: message
data: {"content":" a"}

event: message
data: {"content":" programming"}

That’s SSE.

The browser doesn’t need to wait for the entire LLM response.

9.5 Important limitation

Our current stream action is only streaming the display.

We are not yet persisting the final assistant message.

That’s deliberate.

The next iteration will accumulate:

content << delta

and after the stream finishes:

conversation.messages.create!(
  role: :assistant,
  content: content,
  model: ...,
  input_tokens: ...,
  output_tokens: ...
)

So we ultimately want:

LLM
 ↓
stream chunks
 ↓
Browser
 ↓
accumulate full response
 ↓
PostgreSQL

9.6 Browser side

We can consume SSE with JavaScript:

const source = new EventSource(
  `/conversations/${conversationId}/messages/stream`
);

let content = "";

source.addEventListener("message", (event) => {
  const data = JSON.parse(event.data);

  content += data.content;

  document.querySelector("#assistant-response").innerHTML =
    content;
});

source.addEventListener("done", () => {
  source.close();
});

For our application, we’ll eventually use a Stimulus controller rather than inline JavaScript.

9.7 Don’t spend time styling this

Our immediate objective is proving:

LLM → SSE → Browser

Once you can see the response arriving incrementally, we’ve achieved the important part.

9.8 Commit

Once Ruby streaming works:

git add app/services/ai/client.rb
git commit -m "feat: stream LLM responses"
git push

Then we’ll wire the browser properly.

Int. knowledge from this step

You should now be able to explain:

What is SSE?

A persistent HTTP connection where the server pushes events to the client.

Why use it for AI?

Because LLM output naturally arrives incrementally, and streaming improves perceived latency.

Why not Action Cable?

WebSockets are bidirectional; SSE is simpler when the server primarily needs to push generated output to the browser.

Where does the LLM stream end?

At the Rails server, which consumes the provider’s SSE stream and forwards its own stream to the browser.

OpenRouter documents its AI streaming as SSE, while the Ruby SDK provides streaming chat-completion chunks through stream_raw.

Next step

Since ActionController::Live::SSE exists in Rails 8.1, let’s test the controller before committing.

One important point first: don’t test this through bin/rails server with WEBrick. Rails documents that WEBrick buffers responses, so Live streaming won’t behave correctly. Use our normal Puma server instead. (Ruby on Rails Guides)

1. First verify the route

Run:

bin/rails routes | grep stream

You should see our route, something like:

conversation_messages_stream
GET /conversations/:conversation_id/messages/stream

Then get a conversation ID:

bin/rails c
Conversation.last.id

For example:

1

Exit:

exit

2. Test with curl first

This is the easiest way to prove that the Rails endpoint is actually streaming.

Start Rails with Puma:

bin/rails server

Then in another terminal:

curl -N \
  -H "Accept: text/event-stream" \
  http://localhost:3000/conversations/1/messages/stream

Replace 1 with your real conversation ID.

Why -N?

curl -N means:

Don’t buffer the response.

Without it, you may receive everything at once and incorrectly conclude that streaming isn’t working.

3. What you should see

Because we’re using ActionController::Live::SSE, our response should look roughly like:

event: message
data: {"content":"Ruby"}
event: message
data: {"content":" is"}
event: message
data: {"content":" a"}
event: message
data: {"content":" programming"}
event: done
data: {"done":true}

The exact chunks will vary.

The important thing is that the output arrives progressively, not as one giant response at the end.

Rails’ SSE helper formats events and data for the text/event-stream response. (Ruby on Rails API)

4. Very important: our current stream action has a logical problem

Our current endpoint is probably something like:

def stream
  conversation = Conversation.find(params[:conversation_id])

  response.headers["Content-Type"] = "text/event-stream"

  sse = ActionController::Live::SSE.new(response.stream)

  messages = Ai::PromptBuilder
    .new(conversation: conversation)
    .build

  Ai::Client.new.stream_chat(messages: messages) do |delta|
    sse.write({ content: delta }, event: "message")
  end

  sse.write({ done: true }, event: "done")
ensure
  sse.close
end

This can stream the existing conversation, but it doesn’t receive a new user question.

Eventually our endpoint needs something like:

POST /conversations/:id/messages/stream

with:

content=Explain Ruby blocks

Otherwise we’re streaming whatever messages already exist in the conversation.

So for this first test, we’re only proving:

Rails
  ↓
ActionController::Live
  ↓
SSE
  ↓
Browser/curl

We’ll fix the request lifecycle immediately afterward.

5. Test directly from the browser

You can also open the endpoint in Chrome:

http://localhost:3000/conversations/1/messages/stream

But I prefer curl -N for the first test because the browser doesn’t give you a very useful raw view of SSE events.

Rails’ documentation uses essentially this same pattern – writing to response.stream periodically and closing the stream in ensure. (Ruby on Rails Guides)

One architectural correction before we commit

Don’t commit our current streaming implementation yet.

we’ll change the flow to the proper one:

Browser
   │
   │ POST message
   ▼
MessagesController
   │
   ├── save user message
   │
   ▼
LLM streaming
   │
   ├── SSE chunk → Browser
   ├── SSE chunk → Browser
   ├── SSE chunk → Browser
   │
   ▼
Complete response
   │
   ▼
Save assistant message

That is the version worth keeping in our portfolio and discussing in an int. scenario. Rails requires the response headers to be set before the first stream write and requires the stream to be closed when finished. (Ruby on Rails API)

the next step will be to connect the actual user message → streaming endpoint → browser UI rather than having a standalone stream endpoint.

Debug:ActionController::Live::ClientDisconnected – 500 Internal Server Error

Yes – very likely from our rescue behavior, but the deeper issue is that ActionController::Live::ClientDisconnected is not the same exception as IOError in Rails 8.1.

Rails 8.1 explicitly has:

ActionController::Live::ClientDisconnected

as its own exception class. (Ruby on Rails API)

So this:

rescue IOError
  # Browser disconnected

does not necessarily catch the exception you’re seeing.

Why the 500 appears

Our stream is working, then eventually the client closes the connection – or example:

  • browser finishes and closes the SSE connection
  • EventSource.close() is called
  • browser navigates/reloads
  • user closes the tab
  • network connection disappears

Rails detects that the client is gone while processing the Live response and raises:

ActionController::Live::ClientDisconnected

Rails’ Live processing happens in a separate thread, and once the response has been committed Rails handles exceptions differently from a normal controller request. (Ruby on Rails API)

Fix our rescue

add:

rescue ActionController::Live::ClientDisconnected
  Rails.logger.info("SSE client disconnected")

You can optionally also handle IOError:

rescue ActionController::Live::ClientDisconnected, IOError
  Rails.logger.info("SSE client disconnected")

And keep:

ensure
  sse.close
end

So our action should have roughly:

begin
  # streaming logic
rescue IOError
  Rails.logger.debug(">>>>>>>>>>>>>> Error Occured: IOError")
rescue ActionController::Live::ClientDisconnected
  Rails.logger.debug(">>>>>>>>>>>>>> SSE client disconnected")
ensure
  sse.close
end

But there is an important point

Don’t interpret ClientDisconnected as an application failure.

It’s closer to:

Rails: "I'm streaming."
Browser: "I'm no longer listening."
Rails: "Okay."

For SSE, that’s a normal lifecycle event.

Why you’re seeing Completed 500

This is the part that initially looks strange.

With ActionController::Live, Rails starts processing the action in a separate thread. When an exception occurs after the response has already been committed/started streaming, Rails can’t behave like a normal controller request with a fresh 500 response. Rails’ Live implementation has special handling for committed responses and closes the stream after logging the exception.

So our log can be misleading:

Completed 500 Internal Server Error
ActionController::Live::ClientDisconnected

even though:

  • the user received the streamed answer successfully,
  • the connection closed normally afterward.

That’s exactly why I would treat ClientDisconnected separately from genuine application errors.

Unknown's avatar

Author: Abhilash

Hi, I’m Abhilash! A seasoned web developer with 15 years of experience specializing in Ruby and Ruby on Rails. Since 2010, I’ve built scalable, robust web applications and worked with frameworks like Angular, Sinatra, Laravel, Node.js, Vue and React. Passionate about clean, maintainable code and continuous learning, I share insights, tutorials, and experiences here. Let’s explore the ever-evolving world of web development together!

Leave a comment