A deep dive into race conditions, testing modes, and the mysterious world of background job testing
The Mystery: “But It Works On My Machine!” 🤔
Picture this: You’ve just refactored some code to improve performance by moving slow operations to background workers. Your tests pass locally with flying colors. You push to CI, feeling confident… and then:
X expected: 3, got: 2
X expected: 4, got: 0
Welcome to the wonderful world of Sidekiq testing race conditions – one of the most frustrating debugging experiences in Rails development.
The Setup: A Real-World Example
Let’s examine a real scenario that recently bit us. We had a OrdersWorker that creates orders for new customers:
# app/workers/signup_create_upcoming_orders_worker.rb
class OrdersWorker
include Sidekiq::Worker
def perform(client_id, reason)
client = Client.find(client_id)
# Create orders - this is slow!
client.orders.create
# ... more setup logic
end
end
The worker gets triggered during customer activation:
# lib/settings/update_status.rb
def setup(prev)
# NEW: Move slow operation to background
OrdersWorker.perform_async(@user.client.id, @reason)
# ... other logic
end
And our test helper innocently calls this during setup:
# spec/helper.rb
def init_client(tags = [], sub_menus = nil)
client = FactoryBot.create(:client, ...)
# This triggers the worker!
Settings::Status.new(client, { status: 'active', reason: 'test'}).save
client
end
Understanding Sidekiq Testing Modes
Sidekiq provides three testing modes that behave very differently:
1. Default Mode (Production-like)
# Workers run asynchronously in separate processes
OrdersWorker.perform_async(client.id, 'signup')
# Test continues immediately - worker runs "sometime later"
2. Fake Mode
Sidekiq::Testing.fake!
# Jobs are queued but NOT executed
expect(OrdersWorker.jobs.size).to eq(1)
3. Inline Mode
Sidekiq::Testing.inline!
# Jobs execute immediately and synchronously
OrdersWorker.perform_async(client.id, 'signup')
# ^ This blocks until the job completes
The Environment Plot Twist
Here’s where it gets interesting. The rspec-sidekiq gem can completely override these modes:
Local Development
# Your test output
[rspec-sidekiq] WARNING! Sidekiq will *NOT* process jobs in this environment.
Translation: “I don’t care what Sidekiq::Testing mode you set – workers aren’t running, period.”
CI/Staging
# No warning - workers run normally
Sidekiq 7.3.5 connecting to Redis with options {:url=>"redis://redis:6379/0"}
Translation: “Sidekiq testing modes work as expected.”
The Race Condition Emerges
Now we can see the perfect storm:
RSpec.describe 'OrderBuilder' do
it "calculates order quantities correctly" do
client = init_client([],[]) # * Triggers worker async in CI
client.update!(order_count: 5) # * Sets expected value
order = OrderBuilder.new(client).create(week) # * Reads client state
expect(order.products.first.quantity).to eq(3) # >> Fails in CI
end
end
What happens in CI:
init_clienttriggersOrdersWorker.perform_async- Test sets
order_count = 5 - Worker runs asynchronously, potentially resetting client state
OrderBuilderreads modified/stale client data- Calculations use wrong values → test fails
What happens locally:
init_clienttriggers worker (butrspec-sidekiqblocks it)- Test sets
order_count = 5 - No worker interference
OrderBuilderreads correct client data- Test passes ✅
Debugging Strategies
1. Look for the Warning
# Local: Workers disabled
[rspec-sidekiq] WARNING! Sidekiq will *NOT* process jobs in this environment.
# CI: Workers enabled (no warning)
2. Trace Worker Triggers
Look for these patterns in your test setup:
# Direct calls
SomeWorker.perform_async(...)
# Indirect calls through model callbacks, service objects
client.setup! # May trigger workers internally
Settings::Status.new(...).save # May trigger workers
3. Check for State Mutations
Workers that modify the same data your tests depend on:
# Test expects this value
client.update!(important_field: 'expected_value')
# But worker might reset it
class ProblematicWorker
def perform(client_id)
client = Client.find(client_id)
client.update!(important_field: 'default_value') # 💥 Race condition
end
end
Solutions & Best Practices
Solution 1: File-Level Inline Mode
For specs heavily dependent on worker behavior:
RSpec.describe 'OrderBuilder' do
before(:each) do
# Force all workers to run synchronously
Sidekiq::Testing.inline!
# ... other setup
end
# All tests now have consistent worker behavior
end
Solution 2: Context-Specific Inline Mode
For isolated problematic tests:
context "with background jobs" do
before { Sidekiq::Testing.inline! }
it "works with synchronous workers" do
# Test that needs worker execution
end
end
Solution 3: Stub the Workers
When you don’t need the worker logic:
before do
allow(ProblematicWorker).to receive(:perform_async)
end
Solution 4: Test the Worker Separately
Isolate worker testing from business logic testing:
# Test the worker in isolation
RSpec.describe OrdersWorker do
it "creates orders correctly" do
Sidekiq::Testing.inline!
worker.perform(client.id, 'signup')
expect(client.orders.count).to eq(4)
end
end
# Test business logic without worker interference
RSpec.describe OrderBuilder do
before { allow(OrdersWorker).to receive(:perform_async) }
it "calculates quantities correctly" do
# Pure business logic test
end
end
The Golden Rules
1. Be Explicit About Worker Behavior
Don’t rely on global configuration – be explicit in your tests:
# ✅ Good: Clear intent
context "with synchronous jobs" do
before { Sidekiq::Testing.inline! }
# ...
end
# ❌ Bad: Relies on global config
context "testing orders" do
# Assumes some global Sidekiq setting
end
2. Understand Your Test Environment
Know how rspec-sidekiq is configured in each environment:
# config/environments/test.rb
if ENV['CI']
# Allow workers in CI for realistic testing
Sidekiq::Testing.fake!
else
# Disable workers locally for speed
require 'rspec-sidekiq'
end
3. Separate Concerns
- Test business logic without worker dependencies
- Test worker behavior in isolation
- Test integration with controlled worker execution
Real-World Fix
Here’s how we actually solved our issue:
RSpec.describe 'OrderBuilder' do
before(:each) do |example|
# CRITICAL: Ensure Sidekiq workers run synchronously to prevent race conditions
# The init_client helper triggers OrdersWorker via Settings::Status,
# which can modify client state (rte_meal_count) asynchronously in CI, causing test failures.
Sidekiq::Testing.inline!
unless example.metadata[:skip_before]
create_diet_restrictions
create_recipes
assign_recipe_tags
end
end
# All tests now pass consistently in both local and CI! ✅
end
Takeaways
- Environment Parity Matters: Your local and CI environments may handle Sidekiq differently
- Workers Create Race Conditions: Background jobs can interfere with test state
- Be Explicit: Don’t rely on global Sidekiq test configuration
- Debug Systematically: Look for worker triggers in your test setup
- Choose the Right Solution: Inline, fake, or stubbing – pick what fits your test needs
The next time you see tests passing locally but failing in CI, ask yourself: “Are there any background jobs involved?” You might just save yourself hours of debugging! 🎯
Have you encountered similar Sidekiq testing issues? Share your war stories and solutions in the comments below!
The Rails Drop 