Two Sidekiq race conditions hid behind one pause button

A pause/resume feature on a Ruby on Rails campaign sender kept killing itself in production. The cause was two independent race conditions stacked behind the same button.

Campaigns paused, had their throttle changed, and resumed would go quiet. The status would be stuck at running, thousands of scheduled records would still be in the database, and zero Sidekiq jobs would be left to send them.

No exception, no retries, no alert. It was just a campaign that had stopped existing as far as the workers were concerned.

Recovery meant a human noticing and manually re-kicking the scheduler.

SWARECO traced it to two races that produce the identical symptom by completely different mechanisms. Fixing either one alone would have left the bug intermittently alive. This is exactly why a "we fixed the pause bug" that closes after one repro is not a fix.

Here is both.

The setup: two Sidekiq queues moving at very different speeds

The mechanics matter. Sidekiq executes background jobs asynchronously: an enqueued job is pushed to Redis and waits in its queue until a Sidekiq process with a free worker thread picks it up, so how quickly a job actually runs depends entirely on the queue it lands in. Pausing a campaign enqueues an async cleanup job that sweeps the campaign's not-yet-sent jobs out of the queue. Resuming re-enqueues the campaign's scheduled records for sending.

Those two operations run on two different Sidekiq queues with very different capacity. The cleanup sweep runs on a low-priority queue served by a single worker and is frequently backlogged. The send-scheduler runs on a high-throughput queue and gets picked up almost instantly.

That asymmetry is the whole story.

When the fast queue and the slow queue disagree about ordering, the results are these two races.

Race one: the cleanup sweep deletes the resume's fresh work

The first race is a cleanup that runs too late and matches too broadly.

Pause enqueues the sweep on the slow, backlogged queue, so it sits. The user changes the throttle and hits resume. The scheduler runs on the fast queue and re-enqueues all the campaign's records straight away, so sending starts.

Then the delayed sweep finally runs. It deletes the jobs it finds, matching only on job class plus campaign id.

It has no notion of "which enqueue am I supposed to be cleaning up." So it deletes the jobs the resume just created, and the campaign is left running with nothing queued.

The fix is to give the sweep a sense of time. The cleanup now carries a cutoff timestamp (Time.current.to_f, captured at pause) and deletes only jobs whose Sidekiq payload created_at predates that cutoff.

# only sweep jobs that existed at pause time
cancel_scheduled_job(cutoff: Time.current.to_f)

Jobs re-enqueued by a later resume are newer than the cutoff, so the sweep physically cannot touch them, no matter how far behind the slow queue has fallen. A nil cutoff, for the cancel and destroy paths and for any legacy payloads already sitting in Redis at deploy time, preserves the old delete-everything behaviour. This is what makes the change safe to ship without draining the queue first.

Race two: the scheduler reads the status before the transaction commits

The second race produces the same dead campaign, and no cutoff timestamp touches it. This one happens before the sweep is even relevant.

Resume flips the campaign's state through an AASM state machine, and the scheduler was enqueued from an AASM after callback. The subtlety is that an AASM after callback fires inside the state-change database transaction, before COMMIT.

The scheduler job was enqueued with perform_async, reached Redis immediately, and on a fast queue a worker could pick it up and query the campaign's status before the resuming transaction had committed. The worker read the still-paused status, failed its own running? guard, and, because it runs with retry: 0, exited silently.

Nothing threw. The campaign never started.

This is the canonical enqueue-inside-a-transaction bug, and the canonical fix applies.

# was: SendMessageSchedulerJob.perform_async(id)
SendMessageSchedulerJob.perform_in(5.seconds, id)

perform_in(5.seconds) delays pickup long enough that the status COMMIT is guaranteed visible before the scheduler's guard runs. It is a deliberately boring fix.

The interesting part is why the bug was invisible. retry: 0 plus a guard clause that returns quietly means the failure mode is silence, not an error. A job that raised would have retried and probably self-healed, or at least paged someone. A job that reads stale state and politely gives up leaves no trace at all.

Why both had to ship together

The two races are independent, and each is individually sufficient to strand a campaign.

• Fix the cutoff timestamp but keep perform_async, and a fast resume still loses to its own uncommitted transaction.
• Fix the enqueue timing but keep the class-plus-id match, and a backlogged sweep still deletes a resume's fresh jobs.

Either fix alone would have made the bug rarer, which on a race condition is the worst possible outcome. That is the trap with intermittent concurrency bugs, a single repro that goes green is not evidence the class of failure is gone. It becomes rare enough to look fixed, frequent enough to keep stranding real campaigns, and now with a closed ticket claiming victory.

Neither fix needed a schema change, a new queue, or an infrastructure change. Pause stays instant for the user, as sending stops immediately via the send job's own paused check. The Redis sweep is only queue hygiene that is now allowed to lag harmlessly. The whole correction is a cutoff argument and a five-second delay.

How to avoid Sidekiq race conditions like these two

The standard best-practices toolbox for a Sidekiq race condition is well known: make every job idempotent so a retry is harmless, use unique-jobs locks to keep duplicate jobs from being enqueued, and protect shared data with optimistic locking or a unique index in Postgres. That toolbox is aimed at the common case, two threads executing the same job concurrently.

Neither race here was concurrent execution of the same job. One was a delete racing a re-enqueue across two queues; the other was a worker reading state a database transaction had not yet committed. Idempotency and uniqueness would not have touched either one.

Two patterns are worth carrying instead, to any Rails app that pauses and resumes background work.

1. A cleanup job that matches on identity but not on time will eventually delete work it did not create. If a sweep can be outrun by a re-enqueue on a faster queue, it needs a cutoff, not just a class + id filter. "Delete this campaign's jobs" is under-specified. "Delete this campaign's jobs that existed as of this moment" is the actual intent.
2. Enqueuing from inside a transaction, including from an AASM after callback, is a visibility race, not a timing quirk. The job can win the trip to Redis. Use perform_in with a small delay, or an after_commit hook, so the worker cannot observe state the database has not yet committed. A retry: 0 job behind a silent guard clause will hide the race from you completely. Treat "the job just didn't run and nothing errored" as a signature of exactly this bug.

SWARECO found both of these behind one button because the symptom was identical from the outside, a campaign that quietly stopped.

The lesson is not the two fixes, which are standard.

It is that "same symptom" is not "same bug," and a concurrency failure that a single green repro declares fixed is usually still there.