Integrate AI with Rails: AI bootcamp for Developers – Day 7 – AI Response Streaming

Now let’s implement streaming. OpenRouter supports Server-Sent Events (SSE) when stream: true, and the current Ruby SDK exposes Chat Completions streaming through stream_raw; its higher-level stream helper is not available in every released SDK version. (OpenRouter)

We’ll keep the implementation practical and compatible with the SDK behavior you’re using.

Step 9 – Stream the AI response

What changes?

Currently:

Browser
  ↓
POST
  ↓
Rails waits for entire LLM response
  ↓
redirect

We want:

Browser
  ↓
POST
  ↓
Rails
  ↓
OpenRouter SSE stream
  ↓
token
token
token
token
  ↓
Browser

SSE is a long-lived HTTP response where the server sends incremental events. OpenRouter explicitly supports this with stream: true.

9.1 First, prove streaming works from Ruby

Before involving Rails, modify Ai::Client temporarily with a method:

def stream_chat(messages:, &on_delta)
  stream = @client.chat.completions.stream_raw(
    model: MODEL,
    messages: messages
  )

  stream.each do |chunk|
    delta = chunk.choices.first&.delta&.content
    on_delta.call(delta) if delta.present?
  end
end

The current SDK’s stream_raw returns an enumerable stream of chat completion chunks. (RubyDoc)

Now from Rails console:

conversation = Conversation.first

messages = Ai::PromptBuilder
  .new(conversation: conversation)
  .build

Then:

Ai::Client.new.stream_chat(messages: messages) do |delta|
  print delta
  $stdout.flush
end

You should see the answer appearing progressively:

Ruby is a programming language...

instead of getting the entire answer at once.

Why `$stdout.flush`?

Ruby can buffer stdout. Flushing makes each chunk visible immediately in the console.

9.2 Now expose streaming from Rails

Instead of making MessagesController#create wait for the completed response, we’ll create a streaming endpoint.

Open:

config/routes.rb

Add:

resources :conversations, only: [:create, :show] do
  resources :messages, only: [:create]
end

get "/conversations/:conversation_id/messages/stream",
    to: "messages#stream",
    as: :conversation_messages_stream

9.3 Add the streaming controller action

Open:

app/controllers/messages_controller.rb

Add:

include ActionController::Live

and:

def stream
  conversation = Conversation.find(params[:conversation_id])

  response.headers["Content-Type"] = "text/event-stream"
  response.headers["Cache-Control"] = "no-cache"
  response.headers["X-Accel-Buffering"] = "no"

  sse = SSE.new(response.stream)

  messages = Ai::PromptBuilder
    .new(conversation: conversation)
    .build

  content = +""

  begin
    Ai::Client.new.stream_chat(messages: messages) do |delta|
      next if delta.blank?

      content << delta

      sse.write(
        { content: delta },
        event: "message"
      )
    end

    sse.write(
      { done: true },
      event: "done"
    )
  ensure
    sse.close
    response.stream.close
  end
end

But Rails doesn’t provide SSE automatically.

Add:

include ActionController::Live

and use Rails’ ActionController::Live::SSE if available in our Rails 8.1 setup, or otherwise we can use the standard SSE format directly. Rails 8.1’s Live controller infrastructure is the relevant mechanism here.

To avoid another dependency, let’s actually use the raw SSE format ourselves.

Replace the sse.write(...) parts with:

response.stream.write(
  "event: message\n" \
  "data: #{JSON.generate(content: delta)}\n\n"
)

and completion:

response.stream.write(
  "event: done\n" \
  "data: #{JSON.generate(done: true)}\n\n"
)

So the complete action becomes:

include ActionController::Live

def stream
  conversation = Conversation.find(params[:conversation_id])

  response.headers["Content-Type"] = "text/event-stream"
  response.headers["Cache-Control"] = "no-cache"
  response.headers["X-Accel-Buffering"] = "no"

  messages = Ai::PromptBuilder
    .new(conversation: conversation)
    .build

  begin
    Ai::Client.new.stream_chat(messages: messages) do |delta|
      next if delta.blank?

      response.stream.write(
        "event: message\n" \
        "data: #{JSON.generate(content: delta)}\n\n"
      )
    end

    response.stream.write(
      "event: done\n" \
      "data: #{JSON.generate(done: true)}\n\n"
    )
  rescue IOError
    # Browser disconnected.
  ensure
    response.stream.close
  end
end

9.4 What’s happening?

The server sends chunks like:

event: message
data: {"content":"Ruby"}

event: message
data: {"content":" is"}

event: message
data: {"content":" a"}

event: message
data: {"content":" programming"}

That’s SSE.

The browser doesn’t need to wait for the entire LLM response.

9.5 Important limitation

Our current stream action is only streaming the display.

We are not yet persisting the final assistant message.

That’s deliberate.

The next iteration will accumulate:

content << delta

and after the stream finishes:

			
conversation.messages.create!(
  role: :assistant,
  content: content,
  model: ...,
  input_tokens: ...,
  output_tokens: ...
)

		

So we ultimately want:

			
LLM
 ↓
stream chunks
 ↓
Browser
 ↓
accumulate full response
 ↓
PostgreSQL

		

9.6 Browser side

We can consume SSE with JavaScript:

const source = new EventSource(
  `/conversations/${conversationId}/messages/stream`
);

let content = "";

source.addEventListener("message", (event) => {
  const data = JSON.parse(event.data);

  content += data.content;

  document.querySelector("#assistant-response").innerHTML =
    content;
});

source.addEventListener("done", () => {
  source.close();
});

For our application, we’ll eventually use a Stimulus controller rather than inline JavaScript.

9.7 Don’t spend time styling this

Our immediate objective is proving:

LLM → SSE → Browser

Once you can see the response arriving incrementally, we’ve achieved the important part.

9.8 Commit

Once Ruby streaming works:

			
git add app/services/ai/client.rb
git commit -m "feat: stream LLM responses"
git push

Then we’ll wire the browser properly.

Int. knowledge from this step

You should now be able to explain:

What is SSE?

A persistent HTTP connection where the server pushes events to the client.

Why use it for AI?

Because LLM output naturally arrives incrementally, and streaming improves perceived latency.

Why not Action Cable?

WebSockets are bidirectional; SSE is simpler when the server primarily needs to push generated output to the browser.

Where does the LLM stream end?

At the Rails server, which consumes the provider’s SSE stream and forwards its own stream to the browser.

OpenRouter documents its AI streaming as SSE, while the Ruby SDK provides streaming chat-completion chunks through stream_raw.

Next step

Since ActionController::Live::SSE exists in Rails 8.1, let’s test the controller before committing.

One important point first: don’t test this through bin/rails server with WEBrick. Rails documents that WEBrick buffers responses, so Live streaming won’t behave correctly. Use our normal Puma server instead. (Ruby on Rails Guides)

1. First verify the route

Run:

bin/rails routes | grep stream

You should see our route, something like:

			
conversation_messages_stream
GET /conversations/:conversation_id/messages/stream

Then get a conversation ID:

bin/rails c

Conversation.last.id

For example:

Exit:

exit

2. Test with `curl` first

This is the easiest way to prove that the Rails endpoint is actually streaming.

Start Rails with Puma:

bin/rails server

Then in another terminal:

curl -N \
  -H "Accept: text/event-stream" \
  http://localhost:3000/conversations/1/messages/stream

Replace 1 with your real conversation ID.

Why `-N`?

curl -N means:

Don’t buffer the response.

Without it, you may receive everything at once and incorrectly conclude that streaming isn’t working.

3. What you should see

Because we’re using ActionController::Live::SSE, our response should look roughly like:

			
event: message
data: {"content":"Ruby"}
event: message
data: {"content":" is"}
event: message
data: {"content":" a"}
event: message
data: {"content":" programming"}
event: done
data: {"done":true}

		

The exact chunks will vary.

The important thing is that the output arrives progressively, not as one giant response at the end.

Rails’ SSE helper formats events and data for the text/event-stream response. (Ruby on Rails API)

4. Very important: our current stream action has a logical problem

Our current endpoint is probably something like:

def stream
  conversation = Conversation.find(params[:conversation_id])

  response.headers["Content-Type"] = "text/event-stream"

  sse = ActionController::Live::SSE.new(response.stream)

  messages = Ai::PromptBuilder
    .new(conversation: conversation)
    .build

  Ai::Client.new.stream_chat(messages: messages) do |delta|
    sse.write({ content: delta }, event: "message")
  end

  sse.write({ done: true }, event: "done")
ensure
  sse.close
end

This can stream the existing conversation, but it doesn’t receive a new user question.

Eventually our endpoint needs something like:

POST /conversations/:id/messages/stream

with:

content=Explain Ruby blocks

Otherwise we’re streaming whatever messages already exist in the conversation.

So for this first test, we’re only proving:

			
Rails
  ↓
ActionController::Live
  ↓
SSE
  ↓
Browser/curl

		

We’ll fix the request lifecycle immediately afterward.

5. Test directly from the browser

You can also open the endpoint in Chrome:

http://localhost:3000/conversations/1/messages/stream

But I prefer curl -N for the first test because the browser doesn’t give you a very useful raw view of SSE events.

Rails’ documentation uses essentially this same pattern – writing to response.stream periodically and closing the stream in ensure. (Ruby on Rails Guides)

One architectural correction before we commit

Don’t commit our current streaming implementation yet.

we’ll change the flow to the proper one:

Browser
   │
   │ POST message
   ▼
MessagesController
   │
   ├── save user message
   │
   ▼
LLM streaming
   │
   ├── SSE chunk → Browser
   ├── SSE chunk → Browser
   ├── SSE chunk → Browser
   │
   ▼
Complete response
   │
   ▼
Save assistant message

That is the version worth keeping in our portfolio and discussing in an int. scenario. Rails requires the response headers to be set before the first stream write and requires the stream to be closed when finished. (Ruby on Rails API)

the next step will be to connect the actual user message → streaming endpoint → browser UI rather than having a standalone stream endpoint.

Debug:ActionController::Live::ClientDisconnected – 500 Internal Server Error

Yes – very likely from our rescue behavior, but the deeper issue is that ActionController::Live::ClientDisconnected is not the same exception as IOError in Rails 8.1.

Rails 8.1 explicitly has:

ActionController::Live::ClientDisconnected

as its own exception class. (Ruby on Rails API)

So this:

			
rescue IOError
  # Browser disconnected

does not necessarily catch the exception you’re seeing.

Why the 500 appears

Our stream is working, then eventually the client closes the connection – or example:

browser finishes and closes the SSE connection
EventSource.close() is called
browser navigates/reloads
user closes the tab
network connection disappears

Rails detects that the client is gone while processing the Live response and raises:

ActionController::Live::ClientDisconnected

Rails’ Live processing happens in a separate thread, and once the response has been committed Rails handles exceptions differently from a normal controller request. (Ruby on Rails API)

Fix our rescue

add:

			
rescue ActionController::Live::ClientDisconnected
  Rails.logger.info("SSE client disconnected")

You can optionally also handle IOError:

			
rescue ActionController::Live::ClientDisconnected, IOError
  Rails.logger.info("SSE client disconnected")

And keep:

			
ensure
  sse.close
end

So our action should have roughly:

			
begin
  # streaming logic
rescue IOError
  Rails.logger.debug(">>>>>>>>>>>>>> Error Occured: IOError")
rescue ActionController::Live::ClientDisconnected
  Rails.logger.debug(">>>>>>>>>>>>>> SSE client disconnected")
ensure
  sse.close
end

		

But there is an important point

Don’t interpret ClientDisconnected as an application failure.

It’s closer to:

			
Rails: "I'm streaming."
Browser: "I'm no longer listening."
Rails: "Okay."

For SSE, that’s a normal lifecycle event.

Why you’re seeing `Completed 500`

This is the part that initially looks strange.

With ActionController::Live, Rails starts processing the action in a separate thread. When an exception occurs after the response has already been committed/started streaming, Rails can’t behave like a normal controller request with a fresh 500 response. Rails’ Live implementation has special handling for committed responses and closes the stream after logging the exception.

So our log can be misleading:

			
Completed 500 Internal Server Error
ActionController::Live::ClientDisconnected

even though:

the user received the streamed answer successfully,
the connection closed normally afterward.

That’s exactly why I would treat ClientDisconnected separately from genuine application errors.

Fix Streaming issue: After entering the question the page get stuck

Our server logs are as follows:

Started POST "/conversations"
Processing by ConversationsController#create as TURBO_STREAM
Parameters: {"...., "conversation" => {"title" => "Tell me about Node js"}, "commit" => "Start conversation"}

Then goes to
Started GET "/conversations/13"
Processing by ConversationsController#show as TURBO_STREAM
Parameters: {"id" => "13"}
Rendering conversations/show.html.erb
200 OK

Then the page stuck on the Form: form_with in the conversations/show.html.erb.
the div streaming-response is blank and it is not going to messages controller#stream or I think it is not executing the chat_controller.js also.

Yes – the server logs strongly suggest the problem is before MessagesController#stream. The conversation page loads correctly, then the form submission isn’t triggering our Stimulus chat#submit.

The likely culprit is Turbo + Stimulus registration, not SSE.

Let’s debug it quickly in the right order.

1. First confirm Stimulus is actually loading

Open your browser DevTools → Console.

Put this temporarily at the top of:

app/javascript/controllers/chat_controller.js

import { Controller } from "@hotwired/stimulus"

console.log("chat_controller.js loaded")

export default class extends Controller {
  connect() {
    console.log("Chat controller connected")
  }

  submit(event) {
    console.log("Chat submit triggered")
    // existing code...
  }
}

Reload:

http://localhost:3000/conversations/13

You should see:

			
chat_controller.js loaded
Chat controller connected

If you don’t see these

Then the problem is Stimulus registration/import, not the form.

2. Check your Stimulus setup

Because you’re using Rails 8.1, check:

app/javascript/controllers/index.js

It should contain something similar to:

			
import { application } from "controllers/application"
import ChatController from "controllers/chat_controller"
application.register("chat", ChatController)

Depending on the Rails 8 application template/setup, your index.js may use automatic controller loading instead. The important thing is that chat_controller.js is being registered under:

chat

3. Check `application.js`

Open:

app/javascript/application.js

You should have the normal Rails setup, typically something along the lines of:

			
import "@hotwired/turbo-rails"
import "controllers"

The important line is:

import "controllers"

Without it, your Stimulus controllers won’t be registered.

4. Check the actual HTML generated by Rails

This is very important.

Inspect the form in Chrome DevTools.

You should see:

<form
  data-controller="chat"
  data-action="submit->chat#submit"
  ...
>

If you don’t see:

data-controller="chat"

then our view isn’t producing the attributes we expect.

Our form should look approximately like:

<%= form_with(
  url: conversation_messages_path(@conversation),
  method: :post,
  data: {
    controller: "chat",
    action: "submit->chat#submit"
  }
) do |form| %>

5. There is another issue in our previous implementation

This is important.

We currently have:

<%= form_with(... method: :post) %>

but the Stimulus controller is trying to create:

GET /conversations/:id/messages/stream

That means the normal Turbo form submission and our SSE request are two different mechanisms.

We don’t actually want Turbo submitting the form at all.

Let Stimulus own the submission.

Change the form to:

<%= form_with(
  url: conversation_messages_path(@conversation),
  method: :post,
  data: {
    controller: "chat",
    action: "submit->chat#submit",
    turbo: false
  }
) do |form| %>

  <%= form.text_area :content,
      rows: 4,
      placeholder: "Ask something..." %>

  <%= form.submit "Send" %>
<% end %>

<div id="streaming-response"></div>

The important addition is:

turbo: false

This prevents Turbo from hijacking the form submission.

6. But there’s a second problem: `EventSource`

Our previous controller used:

const source = new EventSource(streamUrl)

That means the browser makes:

GET /conversations/13/messages/stream

and that endpoint expects:

params[:content]

So the URL must contain:

?content=Tell+me+about+Node+js

Let’s make the controller simpler and more reliable.

Use:

app/javascripts/controllers/chat_controller.js

import { Controller } from "@hotwired/stimulus"

console.log("chat_controller.js loaded")

export default class extends Controller {
  connect() {
    console.log("Chat controller connected")
  }

  submit(event) {
    console.log("Chat submit triggered")

    event.preventDefault()

    const form = event.currentTarget
    const url = form.action
    const formData = new FormData(form)

    const responseElement = document.querySelector("#streaming-response")
    responseElement.textContent = ""

    const conversationId = url.match(/conversations\/(\d+)\/messages/)[1]
    const streamUrl = 
      `/conversations/${conversationId}/messages/stream?${new URLSearchParams(
          formData
        )}`

    const source = new EventSource(streamUrl)
    source.addEventListener("message", (event) => {
      const data = JSON.parse(event.data)

      responseElement.textContent += data.content
    })

    source.addEventListener("done", () => {
      source.close()

      // Reload for now so the persisted assistant message appears.
      window.location.reload()
    })

    source.onerror = () => {
      source.close()
      responseElement.textContent += "\n\n[AI stream disconnected]"
    }
  }
}

7. Verify the route

Run:

bin/rails routes | grep messages

You should have something like:

			
conversation_messages
conversation_messages_stream

Specifically:

GET /conversations/:conversation_id/messages/stream

If you don’t have that route, the browser will never reach:

MessagesController#stream

8. Test the route independently

Before testing Stimulus, prove the endpoint itself still works:

			
curl -N \
  -H "Accept: text/event-stream" \
  "http://localhost:3000/conversations/13/messages/stream?content=Explain%20Node.js"

Since you already confirmed SSE works, this should still stream.

9. Then test the browser

Reload the conversation page and open DevTools → Console.

When the page loads:

Chat controller connected

When you submit:

Chat submit triggered

Then open DevTools → Network.

You should see:

GET /conversations/13/messages/stream?content=...

and its response should remain open while the AI streams.

If you see that request, the pipeline is working:

			
Form
 ↓
Stimulus
 ↓
EventSource
 ↓
GET /messages/stream
 ↓
MessagesController#stream
 ↓
LLM
 ↓
SSE
 ↓
Browser

		

10. Why our current log looks the way it does

You currently see:

			
POST /conversations
        ↓
GET /conversations/13
        ↓
Rendering conversations/show
        ↓
200 OK

		

and nothing afterward.

That means:

			
Rails page rendering ✅
Stimulus submit ❌

The fact that we don’t see:

Started GET "/conversations/13/messages/stream"

is the strongest clue.

So don’t debug ActionController::Live or OpenRouter yet.

The problem is almost certainly:

			
Stimulus controller registration
        OR
Turbo intercepting the form
        OR
data-controller/data-action not present

		

Cheers! We solved it.

Recommended architectural change (Optional)

One option is we can stop using EventSource with a GET query parameter for the actual chat request.

It’s fine for learning SSE, but for a real AI application, sending the user prompt as:

GET /messages/stream?content=...

isn’t ideal.

We’ll eventually use:

POST /conversations/:id/messages

with fetch() and consume the streaming response body:

			
POST
 ↓
Rails
 ↓
LLM stream
 ↓
ReadableStream
 ↓
Browser

		

We have to keep the user’s message in the POST body and have a single request lifecycle.

But this make our current app/javascript/controllers/chat_controller.js to re-write completely. So let’s move on to the next step ASAP.

to be continued…

Author: Abhilash

Hi, I’m Abhilash! A seasoned web developer with 15 years of experience specializing in Ruby and Ruby on Rails. Since 2010, I’ve built scalable, robust web applications and worked with frameworks like Angular, Sinatra, Laravel, Node.js, Vue and React. Passionate about clean, maintainable code and continuous learning, I share insights, tutorials, and experiences here. Let’s explore the ever-evolving world of web development together! View all posts by Abhilash

M	T	W	T	F	S	S
		1	2	3	4	5
6	7	8	9	10	11	12
13	14	15	16	17	18	19
20	21	22	23	24	25	26
27	28	29	30

Step 9 – Stream the AI response

What changes?

9.1 First, prove streaming works from Ruby

Why $stdout.flush?

9.2 Now expose streaming from Rails

9.3 Add the streaming controller action

9.4 What’s happening?

9.5 Important limitation

9.6 Browser side

9.7 Don’t spend time styling this

9.8 Commit

Int. knowledge from this step

Next step

1. First verify the route

2. Test with curl first

Why -N?

3. What you should see

4. Very important: our current stream action has a logical problem

5. Test directly from the browser

One architectural correction before we commit

Debug:ActionController::Live::ClientDisconnected – 500 Internal Server Error

Why the 500 appears

Fix our rescue

But there is an important point

Why you’re seeing Completed 500

Fix Streaming issue: After entering the question the page get stuck

1. First confirm Stimulus is actually loading

If you don’t see these

2. Check your Stimulus setup

3. Check application.js

4. Check the actual HTML generated by Rails

5. There is another issue in our previous implementation

6. But there’s a second problem: EventSource

7. Verify the route

8. Test the route independently

9. Then test the browser

10. Why our current log looks the way it does

Recommended architectural change (Optional)

Share this:

Related

Author: Abhilash

Leave a comment Cancel reply

Why `$stdout.flush`?

2. Test with `curl` first

Why `-N`?

Why you’re seeing `Completed 500`

3. Check `application.js`

6. But there’s a second problem: `EventSource`