Day: October 2, 2025

Sidekiq Testing Gotchas: When Your Tests Pass Locally But Fail in CI

A deep dive into race conditions, testing modes, and the mysterious world of background job testing

The Mystery: “But It Works On My Machine!” 🤔

Picture this: You’ve just refactored some code to improve performance by moving slow operations to background workers. Your tests pass locally with flying colors. You push to CI, feeling confident… and then:

X expected: 3, got: 2
X expected: 4, got: 0

Welcome to the wonderful world of Sidekiq testing race conditions – one of the most frustrating debugging experiences in Rails development.

The Setup: A Real-World Example

Let’s examine a real scenario that recently bit us. We had a OrdersWorker that creates orders for new customers:

# app/workers/signup_create_upcoming_orders_worker.rb
class OrdersWorker
  include Sidekiq::Worker

  def perform(client_id, reason)
    client = Client.find(client_id)
    # Create orders - this is slow!
    client.orders.create
    # ... more setup logic
  end
end

The worker gets triggered during customer activation:

# lib/settings/update_status.rb
def setup(prev)
  # NEW: Move slow operation to background
  OrdersWorker.perform_async(@user.client.id, @reason)
  # ... other logic
end

And our test helper innocently calls this during setup:

# spec/helper.rb
def init_client(tags = [], sub_menus = nil)
  client = FactoryBot.create(:client, ...)
  # This triggers the worker! 
  Settings::Status.new(client, { status: 'active', reason: 'test'}).save
  client
end

Understanding Sidekiq Testing Modes

Sidekiq provides three testing modes that behave very differently:

1. Default Mode (Production-like)

# Workers run asynchronously in separate processes
OrdersWorker.perform_async(client.id, 'signup')
# Test continues immediately - worker runs "sometime later"

2. Fake Mode

Sidekiq::Testing.fake!
# Jobs are queued but NOT executed
expect(OrdersWorker.jobs.size).to eq(1)

3. Inline Mode

Sidekiq::Testing.inline!
# Jobs execute immediately and synchronously
OrdersWorker.perform_async(client.id, 'signup')
# ^ This blocks until the job completes

The Environment Plot Twist

Here’s where it gets interesting. The rspec-sidekiq gem can completely override these modes:

Local Development

# Your test output
[rspec-sidekiq] WARNING! Sidekiq will *NOT* process jobs in this environment.

Translation: “I don’t care what Sidekiq::Testing mode you set – workers aren’t running, period.”

CI/Staging

# No warning - workers run normally
Sidekiq 7.3.5 connecting to Redis with options {:url=>"redis://redis:6379/0"}

Translation: “Sidekiq testing modes work as expected.”

The Race Condition Emerges

Now we can see the perfect storm:

RSpec.describe 'OrderBuilder' do
  it "calculates order quantities correctly" do
    client = init_client([],[])  # * Triggers worker async in CI
    client.update!(order_count: 5)  # * Sets expected value

    order = OrderBuilder.new(client).create(week)  # * Reads client state

    expect(order.products.first.quantity).to eq(3)  # >> Fails in CI
  end
end

What happens in CI:

  1. init_client triggers OrdersWorker.perform_async
  2. Test sets order_count = 5
  3. Worker runs asynchronously, potentially resetting client state
  4. OrderBuilder reads modified/stale client data
  5. Calculations use wrong values → test fails

What happens locally:

  1. init_client triggers worker (but rspec-sidekiq blocks it)
  2. Test sets order_count = 5
  3. No worker interference
  4. OrderBuilder reads correct client data
  5. Test passes ✅

Debugging Strategies

1. Look for the Warning

# Local: Workers disabled
[rspec-sidekiq] WARNING! Sidekiq will *NOT* process jobs in this environment.

# CI: Workers enabled (no warning)

2. Trace Worker Triggers

Look for these patterns in your test setup:

# Direct calls
SomeWorker.perform_async(...)

# Indirect calls through model callbacks, service objects
client.setup!  # May trigger workers internally
Settings::Status.new(...).save  # May trigger workers

3. Check for State Mutations

Workers that modify the same data your tests depend on:

# Test expects this value
client.update!(important_field: 'expected_value')

# But worker might reset it
class ProblematicWorker
  def perform(client_id)
    client = Client.find(client_id)
    client.update!(important_field: 'default_value')  # 💥 Race condition
  end
end

Solutions & Best Practices

Solution 1: File-Level Inline Mode

For specs heavily dependent on worker behavior:

RSpec.describe 'OrderBuilder' do
  before(:each) do
    # Force all workers to run synchronously
    Sidekiq::Testing.inline!
    # ... other setup
  end

  # All tests now have consistent worker behavior
end

Solution 2: Context-Specific Inline Mode

For isolated problematic tests:

context "with background jobs" do
  before { Sidekiq::Testing.inline! }

  it "works with synchronous workers" do
    # Test that needs worker execution
  end
end

Solution 3: Stub the Workers

When you don’t need the worker logic:

before do
  allow(ProblematicWorker).to receive(:perform_async)
end

Solution 4: Test the Worker Separately

Isolate worker testing from business logic testing:

# Test the worker in isolation
RSpec.describe OrdersWorker do
  it "creates orders correctly" do
    Sidekiq::Testing.inline!
    worker.perform(client.id, 'signup')
    expect(client.orders.count).to eq(4)
  end
end

# Test business logic without worker interference
RSpec.describe OrderBuilder do
  before { allow(OrdersWorker).to receive(:perform_async) }

  it "calculates quantities correctly" do
    # Pure business logic test
  end
end

The Golden Rules

1. Be Explicit About Worker Behavior

Don’t rely on global configuration – be explicit in your tests:

# ✅ Good: Clear intent
context "with synchronous jobs" do
  before { Sidekiq::Testing.inline! }
  # ...
end

# ❌ Bad: Relies on global config
context "testing orders" do
  # Assumes some global Sidekiq setting
end

2. Understand Your Test Environment

Know how rspec-sidekiq is configured in each environment:

# config/environments/test.rb
if ENV['CI']
  # Allow workers in CI for realistic testing
  Sidekiq::Testing.fake!
else
  # Disable workers locally for speed
  require 'rspec-sidekiq'
end

3. Separate Concerns

  • Test business logic without worker dependencies
  • Test worker behavior in isolation
  • Test integration with controlled worker execution

Real-World Fix

Here’s how we actually solved our issue:

RSpec.describe 'OrderBuilder' do
  before(:each) do |example|
    # CRITICAL: Ensure Sidekiq workers run synchronously to prevent race conditions
    # The init_client helper triggers OrdersWorker via Settings::Status,
    # which can modify client state (rte_meal_count) asynchronously in CI, causing test failures.
    Sidekiq::Testing.inline!

    unless example.metadata[:skip_before]
      create_diet_restrictions
      create_recipes
      assign_recipe_tags
    end
  end

  # All tests now pass consistently in both local and CI! ✅
end

Takeaways

  1. Environment Parity Matters: Your local and CI environments may handle Sidekiq differently
  2. Workers Create Race Conditions: Background jobs can interfere with test state
  3. Be Explicit: Don’t rely on global Sidekiq test configuration
  4. Debug Systematically: Look for worker triggers in your test setup
  5. Choose the Right Solution: Inline, fake, or stubbing – pick what fits your test needs

The next time you see tests passing locally but failing in CI, ask yourself: “Are there any background jobs involved?” You might just save yourself hours of debugging! 🎯

Have you encountered similar Sidekiq testing issues? Share your war stories and solutions in the comments below!