Sidekiq Testing Gotchas: When Your Tests Pass Locally But Fail in CI

A deep dive into race conditions, testing modes, and the mysterious world of background job testing

The Mystery: “But It Works On My Machine!” 🤔

Picture this: You’ve just refactored some code to improve performance by moving slow operations to background workers. Your tests pass locally with flying colors. You push to CI, feeling confident… and then:

X expected: 3, got: 2
X expected: 4, got: 0

Welcome to the wonderful world of Sidekiq testing race conditions – one of the most frustrating debugging experiences in Rails development.

The Setup: A Real-World Example

Let’s examine a real scenario that recently bit us. We had a OrdersWorker that creates orders for new customers:

# app/workers/signup_create_upcoming_orders_worker.rb
class OrdersWorker
  include Sidekiq::Worker

  def perform(client_id, reason)
    client = Client.find(client_id)
    # Create orders - this is slow!
    client.orders.create
    # ... more setup logic
  end
end

The worker gets triggered during customer activation:

# lib/settings/update_status.rb
def setup(prev)
  # NEW: Move slow operation to background
  OrdersWorker.perform_async(@user.client.id, @reason)
  # ... other logic
end

And our test helper innocently calls this during setup:

# spec/helper.rb
def init_client(tags = [], sub_menus = nil)
  client = FactoryBot.create(:client, ...)
  # This triggers the worker! 
  Settings::Status.new(client, { status: 'active', reason: 'test'}).save
  client
end

Understanding Sidekiq Testing Modes

Sidekiq provides three testing modes that behave very differently:

1. Default Mode (Production-like)

# Workers run asynchronously in separate processes
OrdersWorker.perform_async(client.id, 'signup')
# Test continues immediately - worker runs "sometime later"

2. Fake Mode

Sidekiq::Testing.fake!
# Jobs are queued but NOT executed
expect(OrdersWorker.jobs.size).to eq(1)

3. Inline Mode

Sidekiq::Testing.inline!
# Jobs execute immediately and synchronously
OrdersWorker.perform_async(client.id, 'signup')
# ^ This blocks until the job completes

The Environment Plot Twist

Here’s where it gets interesting. The rspec-sidekiq gem can completely override these modes:

Local Development

# Your test output
[rspec-sidekiq] WARNING! Sidekiq will *NOT* process jobs in this environment.

Translation: “I don’t care what Sidekiq::Testing mode you set – workers aren’t running, period.”

CI/Staging

# No warning - workers run normally
Sidekiq 7.3.5 connecting to Redis with options {:url=>"redis://redis:6379/0"}

Translation: “Sidekiq testing modes work as expected.”

The Race Condition Emerges

Now we can see the perfect storm:

RSpec.describe 'OrderBuilder' do
  it "calculates order quantities correctly" do
    client = init_client([],[])  # * Triggers worker async in CI
    client.update!(order_count: 5)  # * Sets expected value

    order = OrderBuilder.new(client).create(week)  # * Reads client state

    expect(order.products.first.quantity).to eq(3)  # >> Fails in CI
  end
end

What happens in CI:

  1. init_client triggers OrdersWorker.perform_async
  2. Test sets order_count = 5
  3. Worker runs asynchronously, potentially resetting client state
  4. OrderBuilder reads modified/stale client data
  5. Calculations use wrong values → test fails

What happens locally:

  1. init_client triggers worker (but rspec-sidekiq blocks it)
  2. Test sets order_count = 5
  3. No worker interference
  4. OrderBuilder reads correct client data
  5. Test passes ✅

Debugging Strategies

1. Look for the Warning

# Local: Workers disabled
[rspec-sidekiq] WARNING! Sidekiq will *NOT* process jobs in this environment.

# CI: Workers enabled (no warning)

2. Trace Worker Triggers

Look for these patterns in your test setup:

# Direct calls
SomeWorker.perform_async(...)

# Indirect calls through model callbacks, service objects
client.setup!  # May trigger workers internally
Settings::Status.new(...).save  # May trigger workers

3. Check for State Mutations

Workers that modify the same data your tests depend on:

# Test expects this value
client.update!(important_field: 'expected_value')

# But worker might reset it
class ProblematicWorker
  def perform(client_id)
    client = Client.find(client_id)
    client.update!(important_field: 'default_value')  # 💥 Race condition
  end
end

Solutions & Best Practices

Solution 1: File-Level Inline Mode

For specs heavily dependent on worker behavior:

RSpec.describe 'OrderBuilder' do
  before(:each) do
    # Force all workers to run synchronously
    Sidekiq::Testing.inline!
    # ... other setup
  end

  # All tests now have consistent worker behavior
end

Solution 2: Context-Specific Inline Mode

For isolated problematic tests:

context "with background jobs" do
  before { Sidekiq::Testing.inline! }

  it "works with synchronous workers" do
    # Test that needs worker execution
  end
end

Solution 3: Stub the Workers

When you don’t need the worker logic:

before do
  allow(ProblematicWorker).to receive(:perform_async)
end

Solution 4: Test the Worker Separately

Isolate worker testing from business logic testing:

# Test the worker in isolation
RSpec.describe OrdersWorker do
  it "creates orders correctly" do
    Sidekiq::Testing.inline!
    worker.perform(client.id, 'signup')
    expect(client.orders.count).to eq(4)
  end
end

# Test business logic without worker interference
RSpec.describe OrderBuilder do
  before { allow(OrdersWorker).to receive(:perform_async) }

  it "calculates quantities correctly" do
    # Pure business logic test
  end
end

The Golden Rules

1. Be Explicit About Worker Behavior

Don’t rely on global configuration – be explicit in your tests:

# ✅ Good: Clear intent
context "with synchronous jobs" do
  before { Sidekiq::Testing.inline! }
  # ...
end

# ❌ Bad: Relies on global config
context "testing orders" do
  # Assumes some global Sidekiq setting
end

2. Understand Your Test Environment

Know how rspec-sidekiq is configured in each environment:

# config/environments/test.rb
if ENV['CI']
  # Allow workers in CI for realistic testing
  Sidekiq::Testing.fake!
else
  # Disable workers locally for speed
  require 'rspec-sidekiq'
end

3. Separate Concerns

  • Test business logic without worker dependencies
  • Test worker behavior in isolation
  • Test integration with controlled worker execution

Real-World Fix

Here’s how we actually solved our issue:

RSpec.describe 'OrderBuilder' do
  before(:each) do |example|
    # CRITICAL: Ensure Sidekiq workers run synchronously to prevent race conditions
    # The init_client helper triggers OrdersWorker via Settings::Status,
    # which can modify client state (rte_meal_count) asynchronously in CI, causing test failures.
    Sidekiq::Testing.inline!

    unless example.metadata[:skip_before]
      create_diet_restrictions
      create_recipes
      assign_recipe_tags
    end
  end

  # All tests now pass consistently in both local and CI! ✅
end

Takeaways

  1. Environment Parity Matters: Your local and CI environments may handle Sidekiq differently
  2. Workers Create Race Conditions: Background jobs can interfere with test state
  3. Be Explicit: Don’t rely on global Sidekiq test configuration
  4. Debug Systematically: Look for worker triggers in your test setup
  5. Choose the Right Solution: Inline, fake, or stubbing – pick what fits your test needs

The next time you see tests passing locally but failing in CI, ask yourself: “Are there any background jobs involved?” You might just save yourself hours of debugging! 🎯

Have you encountered similar Sidekiq testing issues? Share your war stories and solutions in the comments below!

Unknown's avatar

Author: Abhilash

Hi, I’m Abhilash! A seasoned web developer with 15 years of experience specializing in Ruby and Ruby on Rails. Since 2010, I’ve built scalable, robust web applications and worked with frameworks like Angular, Sinatra, Laravel, Node.js, Vue and React. Passionate about clean, maintainable code and continuous learning, I share insights, tutorials, and experiences here. Let’s explore the ever-evolving world of web development together!

Leave a comment