Understanding Core Computer Language Concepts: Design Patterns, Polymorphism, and Object Relationships

In this comprehensive guide, we’ll explore four fundamental concepts in computer science and object-oriented programming: the Template Method pattern, Strategy patterns, parameterized types, and object relationships through aggregation and acquaintance. These concepts form the backbone of modern software design and appear across virtually every programming language.


1. Template Method Pattern: Defining the Skeleton of an Algorithm

What is Template Method?

The Template Method is a behavioral design pattern that defines the skeleton of an algorithm in a base class but lets subclasses override specific steps without changing the algorithm’s structure. Think of it as a recipe where the overall cooking process is fixed, but individual chefs can customize certain steps.

The Core Idea

Instead of having multiple classes each implement the complete algorithm, you create:

  • A base/parent class that outlines the overall process
  • Subclasses that override specific “hook” methods to customize behavior

This follows the “Hollywood Principle”: “Don’t call us, we’ll call you.” The parent class controls the flow and calls the methods that subclasses provide.

Ruby Example

Let’s create a beverage brewing system:

# Base class defining the template method
class BeverageMaker
  def brew
    gather_ingredients
    heat_water
    add_ingredients
    steep
    serve
  end

  def gather_ingredients
    puts "Gathering ingredients..."
  end

  def heat_water
    puts "Heating water to appropriate temperature..."
  end

  # These are hook methods that subclasses will override
  def add_ingredients
    raise NotImplementedError, "Subclasses must implement add_ingredients"
  end

  def steep
    raise NotImplementedError, "Subclasses must implement steep"
  end

  def serve
    puts "Pouring into a cup..."
  end
end

# Tea subclass
class TeaMaker < BeverageMaker
  def add_ingredients
    puts "Adding tea leaves to the infuser..."
  end

  def steep
    puts "Steeping for 3-5 minutes..."
  end
end

# Coffee subclass
class CoffeeMaker < BeverageMaker
  def add_ingredients
    puts "Adding ground coffee to the filter..."
  end

  def steep
    puts "Brewing for 4-6 minutes..."
  end

  def serve
    puts "Adding milk and sugar as desired, then pouring..."
  end
end

# Usage
puts "=== Making Tea ==="
tea = TeaMaker.new
tea.brew

puts "\n=== Making Coffee ==="
coffee = CoffeeMaker.new
coffee.brew

Output:

=== Making Tea ===
Gathering ingredients...
Heating water to appropriate temperature...
Adding tea leaves to the infuser...
Steeping for 3-5 minutes...
Pouring into a cup...
=== Making Coffee ===
Gathering ingredients...
Heating water to appropriate temperature...
Adding ground coffee to the filter...
Brewing for 4-6 minutes...
Adding milk and sugar as desired, then pouring...

Real-World Application: Data Processing

class DataProcessor
  def process(file_path)
    data = read_file(file_path)
    data = validate(data)
    data = transform(data)
    data = enrich(data)
    save_output(data)
  end

  def read_file(file_path)
    raise NotImplementedError
  end

  def validate(data)
    puts "Validating data..."
    data
  end

  def transform(data)
    raise NotImplementedError
  end

  def enrich(data)
    puts "Enriching data with metadata..."
    data
  end

  def save_output(data)
    raise NotImplementedError
  end
end

class CSVProcessor < DataProcessor
  def read_file(file_path)
    puts "Reading CSV file: #{file_path}"
    [["Name", "Age"], ["Alice", 30], ["Bob", 25]]
  end

  def transform(data)
    puts "Transforming CSV data to hash format..."
    data
  end

  def save_output(data)
    puts "Saving processed data to database..."
  end
end

class JSONProcessor < DataProcessor
  def read_file(file_path)
    puts "Reading JSON file: #{file_path}"
    {"users" => [{"name" => "Alice", "age" => 30}]}
  end

  def transform(data)
    puts "Transforming JSON data to standardized format..."
    data
  end

  def save_output(data)
    puts "Saving to API endpoint..."
  end
end

Benefits

  • Code Reuse: Common logic is written once in the parent class
  • Consistency: Ensures all subclasses follow the same algorithm structure
  • Flexibility: Subclasses can customize only what they need
  • Maintainability: Changes to the overall algorithm are made in one place

2. Strategy Pattern: Encapsulating Interchangeable Algorithms

What is Strategy Pattern?

The Strategy pattern defines a family of algorithms, encapsulates each one, and makes them interchangeable. It lets the algorithm vary independently from clients that use it. Unlike Template Method, where variations happen through inheritance, Strategy uses composition to swap algorithms at runtime.

The Core Idea

You create:

  • A Strategy interface that defines the algorithm contract
  • Concrete strategy classes that implement different variants
  • A context class that uses a strategy object

This allows you to change the algorithm used without modifying the client code.

Ruby Example

Let’s create a payment processing system:

# Strategy interface (in Ruby, we use duck typing or modules)
module PaymentStrategy
  def pay(amount)
    raise NotImplementedError
  end
end

# Concrete strategies
class CreditCardPayment
  include PaymentStrategy

  def initialize(card_number, cvv)
    @card_number = card_number
    @cvv = cvv
  end

  def pay(amount)
    puts "Processing credit card payment of $#{amount}"
    puts "Card: #{@card_number[-4..-1]}"
    validate_cvv
    puts "Payment approved!"
  end

  private

  def validate_cvv
    puts "Validating CVV..."
  end
end

class PayPalPayment
  include PaymentStrategy

  def initialize(email)
    @email = email
  end

  def pay(amount)
    puts "Sending $#{amount} via PayPal to #{@email}"
    authenticate
    puts "PayPal payment processed!"
  end

  private

  def authenticate
    puts "Authenticating with PayPal..."
  end
end

class CryptocurrencyPayment
  include PaymentStrategy

  def initialize(wallet_address, crypto_type = "Bitcoin")
    @wallet_address = wallet_address
    @crypto_type = crypto_type
  end

  def pay(amount)
    puts "Sending #{amount} satoshis to wallet #{@wallet_address}"
    puts "Cryptocurrency: #{@crypto_type}"
    confirm_blockchain
    puts "Transaction confirmed on blockchain!"
  end

  private

  def confirm_blockchain
    puts "Confirming on blockchain..."
  end
end

# Context class
class ShoppingCart
  def initialize(payment_strategy)
    @payment_strategy = payment_strategy
    @total = 0
  end

  def add_item(price)
    @total += price
  end

  def checkout
    @payment_strategy.pay(@total)
  end

  # Strategy can be changed at runtime
  def change_payment_method(new_strategy)
    @payment_strategy = new_strategy
  end
end

# Usage
puts "=== Customer 1: Credit Card Payment ==="
cart1 = ShoppingCart.new(CreditCardPayment.new("4532-1234-5678-9010", "123"))
cart1.add_item(50)
cart1.add_item(30)
cart1.checkout

puts "\n=== Customer 2: PayPal Payment ==="
cart2 = ShoppingCart.new(PayPalPayment.new("user@example.com"))
cart2.add_item(100)
cart2.checkout

puts "\n=== Customer 3: Changes mind about payment ==="
cart3 = ShoppingCart.new(CreditCardPayment.new("5412-9876-5432-1098", "456"))
cart3.add_item(75)
puts "Initial strategy: Credit Card"
cart3.change_payment_method(CryptocurrencyPayment.new("1A1z7agoat4WYvtQy06YnYs73m7nEChoCM", "Bitcoin"))
puts "Changed strategy: Cryptocurrency"
cart3.checkout

Real-World Application: Sorting Algorithms

module SortStrategy
  def sort(array)
    raise NotImplementedError
  end
end

class BubbleSort
  include SortStrategy

  def sort(array)
    puts "Sorting using Bubble Sort..."
    n = array.length
    (0...n).each do |i|
      (0...n - i - 1).each do |j|
        array[j], array[j + 1] = array[j + 1], array[j] if array[j] > array[j + 1]
      end
    end
    array
  end
end

class QuickSort
  include SortStrategy

  def sort(array)
    puts "Sorting using Quick Sort..."
    return array if array.length <= 1
    pivot = array[0]
    left = array[1..-1].select { |x| x < pivot }
    right = array[1..-1].select { |x| x >= pivot }
    sort(left) + [pivot] + sort(right)
  end
end

class DataSorter
  def initialize(strategy)
    @strategy = strategy
  end

  def execute(data)
    @strategy.sort(data)
  end

  def change_strategy(strategy)
    @strategy = strategy
  end
end

# Usage
data = [64, 34, 25, 12, 22, 11, 90]
sorter = DataSorter.new(BubbleSort.new)
puts sorter.execute(data.dup).inspect

sorter.change_strategy(QuickSort.new)
puts sorter.execute(data.dup).inspect

Benefits

  • Runtime Flexibility: Algorithms can be selected at runtime
  • Code Isolation: Each algorithm is encapsulated in its own class
  • Easy to Extend: New strategies can be added without modifying existing code
  • Testability: Each strategy can be tested independently

Template Method vs. Strategy

AspectTemplate MethodStrategy
MechanismInheritanceComposition
When to useRelated algorithms sharing common structureInterchangeable algorithms
ImplementationSubclasses override methodsDifferent classes implement interface
Change timingCompile-time (class selection)Runtime (object swap)

3. Parameterized Types: Generic Programming

What are Parameterized Types?

Parameterized types (also called generics) allow you to write code that works with different data types while maintaining type safety. They enable you to create classes and functions that operate on various types specified as parameters.

C++ Templates

C++ uses templates to implement generics at compile-time:

#include <iostream>
#include <vector>

// Generic function template
template <typename T>
T max_value(T a, T b) {
    return (a > b) ? a : b;
}

// Generic class template
template <typename T>
class Stack {
private:
    std::vector<T> elements;

public:
    void push(T value) {
        elements.push_back(value);
    }

    T pop() {
        T value = elements.back();
        elements.pop_back();
        return value;
    }

    bool is_empty() const {
        return elements.empty();
    }
};

int main() {
    // Using template functions with different types
    std::cout << "Max of 5 and 10: " << max_value(5, 10) << std::endl;
    std::cout << "Max of 3.5 and 2.1: " << max_value(3.5, 2.1) << std::endl;

    // Using template classes
    Stack<int> intStack;
    intStack.push(10);
    intStack.push(20);
    std::cout << "Popped: " << intStack.pop() << std::endl;

    Stack<std::string> stringStack;
    stringStack.push("Hello");
    stringStack.push("World");
    std::cout << "Popped: " << stringStack.pop() << std::endl;

    return 0;
}

Key Features:

  • Compile-time code generation: Compiler generates specific code for each type used
  • Type safety: Type checking happens at compile time
  • Zero runtime overhead: Generic code is instantiated for each type
  • Template specialization: Can provide specific implementations for certain types

Ada Generics

Ada’s generics provide a similar mechanism but with a different syntax:

generic
    type Item_Type is private;
    Max_Length : Integer;
package Stacks is
    type Stack_Type is limited private;

    procedure Push(S : in out Stack_Type; Item : Item_Type);
    procedure Pop(S : in out Stack_Type; Item : out Item_Type);
    function Is_Empty(S : Stack_Type) return Boolean;

private
    type Item_Array is array (1..Max_Length) of Item_Type;
    type Stack_Type is record
        Items : Item_Array;
        Top : Integer := 0;
    end record;
end Stacks;

Usage:

with Stacks;
procedure Use_Integer_Stack is
package Int_Stacks is new Stacks(Item_Type => Integer, Max_Length => 100);
My_Stack : Int_Stacks.Stack_Type;
begin
Int_Stacks.Push(My_Stack, 42);
-- ...
end Use_Integer_Stack;

Ruby Generics (Runtime Polymorphism)

Ruby doesn’t have compile-time generics, but uses duck typing and metaprogramming:

# Ruby approach: Using blocks and duck typing
class Container
def initialize
@items = []
end
def add(item)
@items << item
end
def process(&block)
@items.each { |item| block.call(item) }
end
def map(&block)
@items.map { |item| block.call(item) }
end
def select(&block)
@items.select { |item| block.call(item) }
end
end
# Using with different types
int_container = Container.new
int_container.add(1)
int_container.add(2)
int_container.add(3)
puts "Original integers:"
int_container.process { |x| puts x }
puts "\nDoubled integers:"
doubled = int_container.map { |x| x * 2 }
puts doubled.inspect
string_container = Container.new
string_container.add("Hello")
string_container.add("World")
string_container.add("Ruby")
puts "\nOriginal strings:"
string_container.process { |s| puts s }
puts "\nUppercased strings:"
uppercased = string_container.map { |s| s.upcase }
puts uppercased.inspect

Using Generic Patterns

# A more sophisticated generic-like pattern using modules
module Enumerable
def filter_map(&block)
map(&block).select { |item| !item.nil? }
end
def partition_by(&block)
Hash.new { |h, k| h[k] = [] }.tap do |hash|
each { |item| hash[block.call(item)] << item }
end
end
end
class MyList
include Enumerable
def initialize(items)
@items = items
end
def each(&block)
@items.each(&block)
end
def map(&block)
@items.map(&block)
end
def select(&block)
@items.select(&block)
end
end
# Usage
numbers = MyList.new([1, 2, 3, 4, 5, 6])
evens = numbers.partition_by { |n| n.even? ? :even : :odd }
puts evens.inspect

Benefits

  • Type Safety: Errors caught at compile-time (in typed languages)
  • Code Reuse: Write once for multiple types
  • Performance: No runtime type checking overhead in compiled languages
  • Expressiveness: Can write sophisticated data structures and algorithms

4. Object Aggregation and Acquaintance: Structuring Relationships

Understanding the Difference

Aggregation and acquaintance are two ways objects relate to each other in object-oriented design:

  • Aggregation (has-a relationship): An object contains another object as a part of its structure. The contained object is a permanent part of the container.
  • Acquaintance (uses-a relationship): An object temporarily knows about another object, typically passed as a parameter or obtained through a method call. The relationship is less permanent.

Aggregation Examples

Aggregation represents a “part-of” relationship where an object owns or contains other objects:

# Strong aggregation: Car owns its parts
class Engine
def initialize(horsepower)
@horsepower = horsepower
end
def start
puts "Engine with #{@horsepower}hp starting..."
end
def stop
puts "Engine stopping..."
end
end
class Wheel
def initialize(size)
@size = size
end
def rotate
puts "#{@size}\" wheel rotating..."
end
end
class Car
def initialize(make, model)
@make = make
@model = model
# Aggregation: Car contains Engine and Wheels
@engine = Engine.new(200)
@wheels = [
Wheel.new(18),
Wheel.new(18),
Wheel.new(18),
Wheel.new(18)
]
end
def start
puts "Starting #{@make} #{@model}..."
@engine.start
end
def drive
@wheels.each(&:rotate)
puts "Car is moving!"
end
def stop
@engine.stop
puts "Car stopped."
end
end
# Usage
car = Car.new("Toyota", "Camry")
car.start
car.drive
car.stop

Key characteristics of aggregation:

  • The container creates/owns the contained objects
  • The contained objects are part of the container’s structure
  • Destroying the container may destroy the contained objects
  • The relationship is relatively permanent

Acquaintance Examples

Acquaintance represents a “knows-about” relationship where objects interact but don’t own each other:

# Acquaintance: Order knows about Customer and Product
class Customer
def initialize(name, email)
@name = name
@email = email
end
def name
@name
end
def email
@email
end
end
class Product
def initialize(name, price)
@name = name
@price = price
end
def name
@name
end
def price
@price
end
end
class Order
def initialize(order_id)
@order_id = order_id
@customer = nil # Acquaintance: will know about a customer
@products = [] # Acquaintance: will know about products
@total = 0
end
# Receives a customer as a parameter
def assign_customer(customer)
@customer = customer
puts "Order #{@order_id} assigned to #{customer.name}"
end
# Receives products as parameters
def add_product(product, quantity = 1)
@products << { product: product, quantity: quantity }
@total += product.price * quantity
end
def display_summary
puts "\n=== Order Summary ==="
puts "Order ID: #{@order_id}"
puts "Customer: #{@customer.name}"
puts "Items:"
@products.each do |item|
puts " - #{item[:product].name}: $#{item[:product].price} x #{item[:quantity]}"
end
puts "Total: $#{@total}"
end
end
# Usage
customer = Customer.new("Alice Johnson", "alice@example.com")
product1 = Product.new("Laptop", 999)
product2 = Product.new("Mouse", 25)
order = Order.new("ORD-001")
order.assign_customer(customer)
order.add_product(product1)
order.add_product(product2, 2)
order.display_summary

Key characteristics of acquaintance:

  • Objects are passed as parameters or obtained through method calls
  • The relationship is temporary and context-dependent
  • Objects don’t create or own each other
  • Objects can exist independently

Real-World Comparison: Restaurant System

# AGGREGATION: Restaurant owns its Menu and Tables
class MenuItem
def initialize(name, price)
@name = name
@price = price
end
def description
"#{@name}: $#{@price}"
end
end
class Table
def initialize(table_number, capacity)
@table_number = table_number
@capacity = capacity
@is_occupied = false
end
def occupy
@is_occupied = true
end
def free
@is_occupied = false
end
def available?
!@is_occupied
end
end
class Menu
def initialize(cuisine_type)
@cuisine_type = cuisine_type
@items = []
end
def add_item(item)
@items << item
end
def list_items
@items.map(&:description)
end
end
class Restaurant
def initialize(name)
@name = name
# AGGREGATION: Restaurant owns these objects
@menu = Menu.new("Italian")
@tables = [
Table.new(1, 4),
Table.new(2, 6),
Table.new(3, 2)
]
end
def setup_menu
@menu.add_item(MenuItem.new("Pasta Carbonara", 15))
@menu.add_item(MenuItem.new("Lasagna", 18))
@menu.add_item(MenuItem.new("Tiramisu", 8))
end
def show_menu
puts "=== #{@name} Menu ==="
@menu.list_items.each { |item| puts item }
end
def reserve_table(party_size)
available_table = @tables.find { |t| t.available? && t.capacity >= party_size }
if available_table
available_table.occupy
"Table reserved!"
else
"No suitable tables available"
end
end
end
# ACQUAINTANCE: Reservation knows about Customer and Restaurant
class Reservation
def initialize(reservation_id)
@reservation_id = reservation_id
@customer = nil
@restaurant = nil
@time = nil
@party_size = nil
end
def make_reservation(customer, restaurant, time, party_size)
@customer = customer
@restaurant = restaurant
@time = time
@party_size = party_size
puts "Reservation #{@reservation_id} made for #{customer.name} at #{time} for #{party_size} people"
end
def confirm
puts "Confirming reservation for #{@customer.name}..."
result = @restaurant.reserve_table(@party_size)
puts result
end
end
# Usage
restaurant = Restaurant.new("Luigi's Italian Kitchen")
restaurant.setup_menu
restaurant.show_menu
customer = Customer.new("Bob Smith", "bob@example.com")
reservation = Reservation.new("RES-001")
reservation.make_reservation(customer, restaurant, "7:00 PM", 4)
reservation.confirm

When to Use Each

AspectAggregationAcquaintance
RelationshipPart-of, ownsUses, knows-about
LifetimeContainer controlsIndependent
CreationContainer createsExternal creation
Use CaseCar-Engine, House-RoomsCustomer-Order, Client-Service
DependencyStrong couplingLoose coupling

Benefits

Aggregation:

  • Clear ownership and lifecycle management
  • Encapsulation of related components
  • Simplified understanding of object structure

Acquaintance:

  • Loose coupling between objects
  • Better testability and modularity
  • More flexible object interactions
  • Easier to extend and modify

Putting It All Together: A Complete Example

Let’s create a library system that demonstrates all four concepts:

# TEMPLATE METHOD: Base class for different user types
class LibraryUser
def initialize(name)
@name = name
@borrowed_books = []
end
def process_checkout(book)
check_eligibility
check_availability(book)
checkout_book(book)
send_confirmation(book)
end
protected
def check_eligibility
raise NotImplementedError
end
def check_availability(book)
puts "Checking if #{book.title} is available..."
end
def checkout_book(book)
@borrowed_books << book
puts "Book checked out successfully"
end
def send_confirmation(book)
puts "Sending confirmation to #{@name}"
end
end
class Student < LibraryUser
def check_eligibility
puts "Checking student ID and membership status..."
end
def send_confirmation(book)
puts "Emailing confirmation to student: #{@name}"
end
end
class Faculty < LibraryUser
def check_eligibility
puts "Checking faculty status..."
end
def checkout_book(book)
@borrowed_books << book
puts "Faculty member can borrow up to 20 items"
end
end
# AGGREGATION: Library owns Books and has Shelves
class Book
attr_reader :title, :author
def initialize(title, author, isbn)
@title = title
@author = author
@isbn = isbn
@is_available = true
end
def available?
@is_available
end
def checkout
@is_available = false
end
def return_book
@is_available = true
end
end
class Shelf
def initialize(section, capacity)
@section = section
@capacity = capacity
@books = []
end
def add_book(book)
@books << book if @books.length < @capacity
end
def list_books
@books.map(&:title)
end
end
class Library
def initialize(name)
@name = name
# AGGREGATION: Library owns shelves and manages books
@shelves = {
fiction: Shelf.new("Fiction", 100),
science: Shelf.new("Science", 100),
history: Shelf.new("History", 100)
}
@all_books = []
end
def add_book(book, section)
@all_books << book
@shelves[section].add_book(book)
end
def find_book(title)
@all_books.find { |book| book.title == title }
end
end
# STRATEGY: Different checkout strategies
module CheckoutStrategy
def apply_fee(days_borrowed)
raise NotImplementedError
end
end
class StudentCheckoutStrategy
include CheckoutStrategy
def apply_fee(days_borrowed)
days_borrowed > 14 ? days_borrowed - 14 * 0.25 : 0
end
end
class FacultyCheckoutStrategy
include CheckoutStrategy
def apply_fee(days_borrowed)
days_borrowed > 30 ? (days_borrowed - 30) * 0.10 : 0
end
end
class LateFeesCalculator
def initialize(strategy)
@strategy = strategy
end
def calculate(days_borrowed)
@strategy.apply_fee(days_borrowed)
end
def change_strategy(strategy)
@strategy = strategy
end
end
# ACQUAINTANCE: Loan connects User and Book temporarily
class Loan
def initialize(loan_id)
@loan_id = loan_id
@user = nil
@book = nil
@checkout_date = nil
end
def create_loan(user, book)
@user = user
@book = book
@checkout_date = Date.today
puts "Loan #{@loan_id}: #{user.class} borrowed '#{book.title}'"
end
def return_book
days = (Date.today - @checkout_date).to_i
fee_calculator = LateFeesCalculator.new(StudentCheckoutStrategy.new)
fee = fee_calculator.calculate(days)
puts "Book returned. Days borrowed: #{days}, Late fee: $#{fee}"
end
end
# Usage demonstration
puts "=== Library Management System ==="
# Setup library (aggregation)
library = Library.new("City Public Library")
book1 = Book.new("The Ruby Way", "Hal Fulton", "ISBN001")
book2 = Book.new("Design Patterns", "Gang of Four", "ISBN002")
library.add_book(book1, :science)
library.add_book(book2, :fiction)
# User checkout with template method
student = Student.new("John Doe")
student.process_checkout(book1)
faculty = Faculty.new("Dr. Smith")
faculty.process_checkout(book2)
# Loan with acquaintance
loan1 = Loan.new("LOAN001")
loan1.create_loan(student, book1)
loan1.return_book

Conclusion

These four concepts represent essential tools in the software architect’s toolkit:

  1. Template Method – Use inheritance to define algorithm structure
  2. Strategy – Use composition to swap algorithms at runtime
  3. Parameterized Types – Write generic code for multiple data types
  4. Aggregation/Acquaintance – Structure object relationships appropriately

Understanding when and how to apply each concept leads to more flexible, maintainable, and scalable software. Ruby’s flexibility makes these patterns particularly elegant to implement, though the principles apply across all modern programming languages.

The key is choosing the right tool for the right problem: use Template Method when you have variations of a fixed process, use Strategy for interchangeable algorithms, use generics for type-flexible code, and use appropriate aggregation/acquaintance patterns to structure your object relationships cleanly.

Happy Coding! 🚀

Sidekiq & Redis Optimization: Reducing Overhead and Scaling Worker Jobs

When you run thousands of background jobs through Sidekiq, Redis becomes the bottleneck. Every job enqueue adds Redis writes, network round-trips, and memory pressure. This post covers a real-world optimization we applied and a broader toolkit for keeping Sidekiq lean.


The Problem: One Job Per Item

Imagine sending weekly emails to 10,000 users. The naive approach:

# ❌ Bad: 10,000 Redis writes, 10,000 scheduled entries
user_ids.each do |id|
WeeklyEmailWorker.perform_async(id)
end

Each perform_async does:

  • A Redis LPUSH (or ZADD for scheduled jobs)
  • Serialization of job payload
  • Network round-trip

At 10,000 users, that’s 10,000 Redis operations and 10,000 scheduled entries. At 1M users, that’s 1M scheduled jobs in Redis. That’s expensive and slow.


The Fix: Batch + Staggered Scheduling

Instead of one job per user, we batch users and schedule each batch with a small delay:

# ✅ Good: 100 Redis writes, 100 scheduled entries
BATCH_SIZE = 100
BATCH_DELAY = 0.2 # seconds
pending_user_ids.each_slice(BATCH_SIZE).with_index do |batch_ids, batch_index|
delay_seconds = batch_index * BATCH_DELAY
WeeklyEmailByWorker.perform_in(delay_seconds, batch_ids)
end

What this achieves:

MetricBefore (1 per user)After (batched)
Redis ops10,000100
Scheduled jobs10,000100
Scheduled jobs at 1M users1,000,00010,000

Each worker still processes one user at a time internally, but we only enqueue one job per batch. Redis overhead drops by roughly 100x.

Why perform_in instead of chaining?

  • perform_in(delay, batch_ids) — all jobs are scheduled immediately with their future timestamps. Sidekiq moves them into the ready queue at the right time regardless of other queue traffic.
  • Chaining (each job enqueues the next) — the next batch only enters the queue after the current one finishes. If other jobs are busy, your email chain sits behind them and can be delayed significantly.

For time-sensitive jobs like “send at 8:46 AM local time,” upfront scheduling is the right choice.


Other Sidekiq Optimization Strategies

1. Bulk Enqueue (Sidekiq Pro/Enterprise)

Sidekiq::Client.push_bulk pushes many jobs in one Redis call:

# Single Redis call instead of N
Sidekiq::Client.push_bulk(
'class' => WeeklyEmailWorker,
'args' => user_ids.map { |id| [id] }
)

Useful when you don’t need per-job delays and want to minimize Redis round-trips.

2. Adjust Concurrency

Default is 10 threads per process. More threads = more concurrency but more memory:

# config/sidekiq.yml
:concurrency: 25 # Tune based on CPU/memory

Higher concurrency helps if jobs are I/O-bound (HTTP, DB, email). For CPU-bound jobs, lower concurrency is usually better.

3. Use Dedicated Queues

Separate heavy jobs from light ones:

# config/sidekiq.yml
:queues:
- [critical, 3] # 3x weight
- [default, 2]
- [low, 1]

Critical jobs get more CPU time. Low-priority jobs don’t block the rest.

4. Rate Limiting (Sidekiq Enterprise)

Throttle jobs that hit external APIs:

class EmailWorker
include Sidekiq::Worker
sidekiq_options throttle: { threshold: 100, period: 1.minute }
end

Prevents hitting rate limits and keeps Redis usage predictable.

5. Unique Jobs (sidekiq-unique-jobs)

Avoid duplicate jobs for the same work:

sidekiq_options lock: :until_executed, on_conflict: :log

Reduces redundant work and Redis load when jobs are retried or triggered multiple times.

6. Dead Job Cleanup

Dead jobs accumulate in Redis. Set retention and cleanup:

# config/initializers/sidekiq.rb
Sidekiq.configure_server do |config|
config.death_handlers << ->(job, ex) {
# Log, alert, or move to DLQ
}
end

Use dead_max_jobs and periodic cleanup so Redis doesn’t grow unbounded.

7. Job Size Limits

Large payloads increase Redis memory and serialization cost:

# Keep payloads small; pass IDs, not full objects
WeeklyEmailWorker.perform_async(user_id) # ✅
WeeklyEmailWorker.perform_async(user.to_json) # ❌

8. Connection Pooling

Ensure each worker process has a bounded Redis connection pool:

# config/initializers/sidekiq.rb
Sidekiq.configure_server do |config|
config.redis = { url: ENV['REDIS_URL'], size: 25 }
end

Prevents connection exhaustion under load.

9. Scheduled Job Limits

Scheduled jobs live in Redis. If you schedule millions of jobs, you may need to cap or paginate:

# Avoid scheduling 1M jobs at once
# Use batch + perform_in with reasonable batch sizes

10. Redis Memory and Eviction

Configure Redis for Sidekiq:

maxmemory 2gb
maxmemory-policy noeviction # or volatile-lru for cache-only keys

Monitor memory and eviction to avoid unexpected data loss.


Summary

StrategyWhen to Use
Batch + perform_inMany similar jobs at a specific time; reduces Redis ops by ~100x
push_bulkLarge batches of jobs without per-job delays
Dedicated queuesDifferent priority levels for job types
Rate limitingExternal APIs or rate-limited services
Unique jobsIdempotent or duplicate-prone jobs
Small payloadsAlways; pass IDs instead of full objects
Connection poolingHigh concurrency or many processes

The batch + perform_in pattern is especially effective for time-sensitive jobs that must run in a narrow window while keeping Redis overhead low.

Happy Coding with Sidekiq!


Fixing Let’s Encrypt SSL Certificate Renewal on a Server: A Step-by-Step Guide

Background

We recently provisioned two new staging environments (staging1.mydomain.com and staging2.mydomain.com) mirroring our production Rails infrastructure. Production uses a Cloudflare load balancer fronting multiple origin servers running nginx, with a cron-driven script to renew Let’s Encrypt certificates across all origins.

When we added the staging environments, the existing renew_certificate.sh cron script wasn’t set up on them yet — so the certificates expired. This post documents everything we encountered trying to fix it, every error we hit, and how we resolved each one.


The Renewal Architecture

Before diving in, it’s worth understanding how SSL renewal works in this setup:

Cloudflare (DNS + Load Balancer)
nginx (origin server)
/apps/mydomain/current ← Rails app lives here
/etc/letsencrypt/live/ ← Certs live here

The renewal script (scripts/cron/renew_certificate.sh) does the following:

  1. Fetches the load balancer pool config from the Cloudflare API
  2. Disables all origin servers in the pool except the GCP instance (takes servers out of rotation)
  3. Turns off Cloudflare’s “Always Use HTTPS” setting (allows HTTP for the ACME challenge)
  4. Runs sudo certbot renew locally
  5. Copies the new cert to all other origin servers via SCP and SSH
  6. Re-enables all origin servers in the Cloudflare pool
  7. Re-enables “Always Use HTTPS”

The problem: this script was never added to the cron on staging1/staging2, so the certs expired.


First Attempt: Running the Renewal Script Manually

SSH’d into staging2 and ran:

bash /apps/mydomain/current/scripts/cron/renew_certificate.sh

Error #1: RSpec / webmock LoadError

An error occurred while loading spec_helper. - Did you mean?
rspec ./spec/helper.rb
Failure/Error: require 'webmock/rspec'
LoadError:
cannot load such file -- webmock/rspec

What happened: The script calls bundle exec rake google_chat:send_message[...] to send failure notifications to Google Chat. On staging, test gems like webmock aren’t installed in the bundle, so the rake task blew up loading the Rails environment.

Lesson: This is a notification side-effect, not the core renewal logic. But it masked the real error.

Error #2: certbot failing because port 80 was in use

After isolating the issue, running sudo certbot renew directly gave:

Renewing an existing certificate for staging2.mydomain.ca and www.staging2.mydomain.ca
Failed to renew certificate staging2.mydomain.ca with error: Could not bind TCP port 80
because it is already in use by another process on this system (such as a web server).
Please stop the program in question and then try again.

What happened: The original certificate was issued using certbot’s standalone authenticator, which spins up its own HTTP server on port 80 to answer the ACME challenge. Since nginx was already running on port 80, the renewal failed.

Meanwhile there was a second certificate (staging2.mydomain.ca-0001) that had been created earlier with sudo certbot --nginx -d staging2.mydomain.ca. This cert was valid — but it created a mess.


Inspecting the Damage

sudo certbot certificates

Output:

Renewal configuration file /etc/letsencrypt/renewal/staging2.mydomain.ca.conf produced
an unexpected error: expected /etc/letsencrypt/live/staging2.mydomain.ca-0001/cert.pem
to be a symlink. Skipping.
The following renewal configurations were invalid:
/etc/letsencrypt/renewal/staging2.mydomain.ca.conf

The nginx config at /etc/nginx/sites-enabled/mydomain was also a mess — certbot had injected its own server block for the HTTP→HTTPS redirect, and the two 443 server blocks were pointing to different cert paths:

# Certbot-injected block (unwanted)
server {
if ($host = staging2.mydomain.ca) {
return 301 https://$host$request_uri;
} # managed by Certbot
...
}
# Redirect server pointing to -0001 certs (also unwanted)
server {
server_name staging2.mydomain.ca;
listen 443 ssl http2;
ssl_certificate /etc/letsencrypt/live/staging2.mydomain.ca-0001/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/staging2.mydomain.ca-0001/privkey.pem; # managed by Certbot
...
}
# Main www server pointing to original path
server {
server_name www.staging2.mydomain.ca;
listen 443 ssl http2;
ssl_certificate /etc/letsencrypt/live/staging2.mydomain.ca/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/staging2.mydomain.ca/privkey.pem;
...
}

The Fix

Step 1: Remove all broken certbot state

sudo rm -f /etc/letsencrypt/renewal/staging2.mydomain.ca.conf
sudo rm -f /etc/letsencrypt/renewal/staging2.mydomain.ca-0001.conf
sudo rm -rf /etc/letsencrypt/live/staging2.mydomain.ca
sudo rm -rf /etc/letsencrypt/live/staging2.mydomain.ca-0001
sudo rm -rf /etc/letsencrypt/archive/staging2.mydomain.ca
sudo rm -rf /etc/letsencrypt/archive/staging2.mydomain.ca-0001

Step 2: Stop nginx and get a fresh cert with standalone authenticator

sudo service nginx stop
sudo certbot certonly --standalone -d staging2.mydomain.ca -d www.staging2.mydomain.ca
sudo service nginx start

This gave us a clean, single certificate at /etc/letsencrypt/live/staging2.mydomain.ca/.

Step 3: Clean up the nginx config

Removed the certbot-injected if ($host = ...) server block, and updated both 443 server blocks to point to the same cert path:

ssl_certificate /etc/letsencrypt/live/staging2.mydomain.ca/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/staging2.mydomain.ca/privkey.pem;

Reloaded nginx:

sudo service nginx reload

The site was live again with a valid cert.


Making Future Renewals Work Without Stopping nginx

The next problem: the cert renewal config was still using standalone authenticator. Future automated renewals would fail again the moment nginx was running.

The fix is to switch to the webroot authenticator. Our nginx config already had an ACME challenge location block:

location ^~ /.well-known/acme-challenge/ {
root /apps/certbot;
default_type "text/plain";
allow all;
}

This means certbot can write a challenge file to /apps/certbot and nginx will serve it over HTTP — no need to stop nginx.

Attempt 1: Manually edit the renewal config

Edited /etc/letsencrypt/renewal/staging2.mydomain.ca.conf:

[renewalparams]
authenticator = webroot
server = https://acme-v02.api.letsencrypt.org/directory
key_type = ecdsa
[[webroot]]
staging2.mydomain.ca = /apps/certbot
www.staging2.mydomain.ca = /apps/certbot

Dry run:

sudo certbot renew --dry-run

Error #3: webroot mapping not found

Failed to renew certificate staging2.mydomain.ca with error: Missing command line flag or
config entry for this setting:
Input the webroot for staging2.mydomain.ca:

The config looked correct but certbot was still asking interactively. This is a known certbot quirk — manually converting a standalone config to webroot doesn’t always work reliably because of how certbot parses its internal config format.

Attempt 2: Delete and re-issue with webroot from the start (this worked)

sudo certbot delete --cert-name staging2.mydomain.ca
sudo mkdir -p /apps/certbot
sudo certbot certonly --webroot -w /apps/certbot \
-d staging2.mydomain.ca \
-d www.staging2.mydomain.ca

This time certbot generated the renewal config correctly itself. Dry run:

sudo certbot renew --dry-run
Simulating renewal of an existing certificate for staging2.mydomain.ca and www.staging2.mydomain.ca
Congratulations, all simulated renewals succeeded:
/etc/letsencrypt/live/staging2.mydomain.ca/fullchain.pem (success)

Key Lessons

  1. Never run certbot --nginx on a server where you manage the nginx config manually. It injects its own server blocks and creates confusing duplicate certs with -0001 suffixes.
  2. Standalone vs webroot authenticator: Standalone is simpler to set up initially but requires stopping nginx. Webroot is the right choice for servers where nginx runs continuously — as long as you have the ACME challenge location block configured.
  3. Manually editing certbot renewal configs is fragile. Let certbot generate the renewal config by passing the correct authenticator flags at issuance time.
  4. certbot renew --dry-run is your best friend. Always confirm future renewals will work before leaving the server. Discovering a broken renewal config 2 days before expiry is stressful.
  5. Let’s Encrypt ACME server outages are real but brief. If dry-run fails with “The service is down for maintenance”, check https://letsencrypt.status.io/ and retry in a few hours.

A Clean Auto-Renewal Script for nginx + webroot

Here’s a standalone script you can drop into any server using this stack. It handles renewal, nginx reload, and sends a notification if anything fails. It assumes the webroot authenticator is already configured in the certbot renewal config.

#!/bin/bash
# /etc/cron.d/certbot-renew or called via crontab
# Requires: certbot, nginx, curl (for Slack/Google Chat webhook)
set -euo pipefail
CERT_NAME="${CERT_NAME:-}" # e.g. staging2.mydomain.ca
NOTIFY_WEBHOOK="${NOTIFY_WEBHOOK:-}" # Slack or Google Chat webhook URL
ACME_WEBROOT="${ACME_WEBROOT:-/apps/certbot}"
LOG_FILE="/var/log/certbot-renew.log"
TIMESTAMP=$(date '+%Y-%m-%d %H:%M:%S')
log() {
echo "[$TIMESTAMP] $*" | tee -a "$LOG_FILE"
}
notify() {
local message="$1"
log "NOTIFY: $message"
if [[ -n "$NOTIFY_WEBHOOK" ]]; then
curl -s -X POST "$NOTIFY_WEBHOOK" \
-H "Content-Type: application/json" \
-d "{\"text\": \"$message\"}" \
>> "$LOG_FILE" 2>&1 || true
fi
}
# Ensure webroot directory exists
mkdir -p "$ACME_WEBROOT"
log "Starting certificate renewal..."
# Attempt renewal
RENEW_OUTPUT=$(sudo certbot renew \
--quiet \
--non-interactive \
${CERT_NAME:+--cert-name "$CERT_NAME"} \
2>&1) || {
notify "SSL RENEWAL FAILED on $(hostname): $RENEW_OUTPUT"
log "ERROR: $RENEW_OUTPUT"
exit 1
}
# Check if any cert was actually renewed (certbot exits 0 even if nothing renewed)
if echo "$RENEW_OUTPUT" | grep -q "Congratulations"; then
log "Certificate renewed. Reloading nginx..."
sudo service nginx reload || {
notify "SSL RENEWAL WARNING on $(hostname): cert renewed but nginx reload failed!"
exit 1
}
notify "SSL cert successfully renewed on $(hostname)"
log "Done."
else
log "No certificates due for renewal. Nothing to do."
fi

Usage

# Set executable
chmod +x /usr/local/bin/certbot-renew.sh
# Set environment variables and run
CERT_NAME=staging2.mydomain.ca \
NOTIFY_WEBHOOK=https://chat.googleapis.com/v1/spaces/.../messages?key=... \
/usr/local/bin/certbot-renew.sh

Crontab entry (runs twice daily — Let’s Encrypt recommendation)

0 3,15 * * * deployer CERT_NAME=staging2.mydomain.ca NOTIFY_WEBHOOK=https://... /usr/local/bin/certbot-renew.sh

Running twice daily ensures that if one attempt fails due to a transient ACME server issue, the next attempt 12 hours later will succeed — giving you plenty of time before expiry.


Summary

ProblemRoot CauseFix
certbot failed to bind port 80standalone authenticator conflicted with nginxSwitch to webroot authenticator
Duplicate -0001 cert createdRan certbot --nginx after standalone cert existedDelete all cert state, re-issue cleanly
nginx serving expired certMixed cert paths after certbot injected its own configManually fix nginx config to consistent paths
Manual webroot config edit didn’t workCertbot’s conf format is fragile when converted manuallyDelete and re-issue with --webroot flag from scratch

Happy Debugging!