Solid Queue Went Silent at 3am. This Is What Held the Lock.

The limits_concurrency semaphore interaction that silently starves your Solid Queue and shows no error in the logs

A terminal reveals a 97-minute advisory lock blocking jobs — this is how Solid Queue silently starves your queue.

At 3:17am our job queue stopped. Not crashed. Not errored. Stopped. Workers were running. The Solid Queue tables were there. Jobs were enqueued. Nothing was processing them.

No alert fired. No exception hit Honeybadger. The queue looked alive from the outside.

We had set **limits_concurrency** on our ReportGenerationJob. Capped at one concurrent execution with a 3-minute duration. The job typically ran in 22 minutes. On the night this happened, an underlying dataset had grown and the job ran for 91 minutes.

Every subsequent report job enqueued during those 91 minutes ended up blocked, waiting for the semaphore to open.

Solid Queue does not use advisory locks for job claiming. It uses **SELECT ... FOR UPDATE SKIP LOCKED**, a PostgreSQL row-level mechanism that lets workers skip rows already claimed by another worker and move on to the next available job. Workers never block each other during claiming.

The starvation we saw came from the limits_concurrency semaphore system entirely.

When limits_concurrency is set on a job class, Solid Queue checks a semaphore before allowing a job into solid_queue_ready_executions. If all concurrency slots are taken, the job goes into solid_queue_blocked_executions instead. Workers never see it. The semaphore releases when the running job finishes and signals it. If that signal fails, or if the job dies mid-execution, the semaphore stays closed.

Whether the signal failed or was never sent, the next job was 91 minutes late. The documented maintenance cycle should have forced the expired semaphore open well before that. Something in our setup prevented it. The dispatcher was running. I still do not know what it missed.

• **limits_concurrency** controls concurrent execution through a semaphore, not a database row lock.
• A long-running job that holds all concurrency slots blocks every subsequent job sharing its concurrency_key.
• Blocked jobs do not sit in solid_queue_ready_executions. They sit in solid_queue_blocked_executions until the semaphore opens.

What the Logs Showed

The Rails logs were clean. The Solid Queue worker logs showed normal polling messages. No lock wait errors. No retries. No timeouts.

Honeybadger had nothing.

I checked solid_queue_processes first. Six workers, all with a recent last_heartbeat_at. The workers were not dead.

I had not run this query yet. I should have run it first.

SELECT id, job_id, queue_name, scheduled_at
FROM solid_queue_ready_executions
ORDER BY scheduled_at ASC
LIMIT 20;

Jobs from other queues — mailers, notifications, default — sitting unclaimed since before 3am. The report jobs were not here. Jobs blocked by a semaphore never reach solid_queue_ready_executions at all. They go straight to solid_queue_blocked_executions and stay there.

What the ready table told me was that workers were not claiming anything, not just report jobs. That pointed away from a semaphore problem and toward a worker problem. Wrong read. The workers were fine. The claiming failure for those jobs had a different cause I chased first and lost 20 minutes on.

The distinction matters. FOR UPDATE SKIP LOCKED means workers do not block each other during claiming. If report jobs are not reaching solid_queue_ready_executions, the problem is upstream of claiming entirely.

• Workers showing current heartbeats while ready jobs pile up points at a claiming problem, not a worker crash.
• Jobs blocked by a semaphore never appear in solid_queue_ready_executions — they are in solid_queue_blocked_executions, invisible to workers.
• The two failure modes look identical from the outside. Both produce queue silence. But they live in different tables.

How to Find What Is Holding the Semaphore

The first thing I should have checked was solid_queue_semaphores. Solid Queue stores one semaphore row per concurrency key, tracking the current value (how many slots remain) and an expiry timestamp.

SELECT key, value, expires_at
FROM solid_queue_semaphores
ORDER BY expires_at ASC;

One row. Our ReportGenerationJob concurrency key. Value of zero, meaning no slots available. Expiry set to the job's original enqueue time plus the configured duration of 3 minutes. That expiry had passed hours ago.

Value zero, expiry in the past. That is the starvation signature.

When a job finishes and the semaphore release fails, or when the job dies without completing cleanly, the semaphore stays at zero. No new jobs with that key enter ready state.

The dispatcher’s concurrency_maintenance_interval is supposed to catch this and unblock eligible jobs. The default is 10 minutes. With a 3-minute duration and a 10-minute maintenance cycle, the semaphore should have been force-released within roughly 13 minutes of the job start. At 4:47am it had not been. The maintenance task had run multiple times and left it in place. I still do not know exactly why. The dispatcher was up. The heartbeats were current. Something in that maintenance pass did not fire the release.

I had never looked at solid_queue_blocked_executions before that night. I probably should have had this query in the runbook already.

SELECT sbe.id, sj.class_name, sbe.concurrency_key, sbe.created_at
FROM solid_queue_blocked_executions sbe
JOIN solid_queue_jobs sj ON sbe.job_id = sj.id
WHERE sbe.concurrency_key LIKE '%ReportGeneration%'
ORDER BY sbe.created_at ASC;

Forty-seven rows. All ReportGenerationJob. The oldest blocked at 3:04am.

I had configured limits_concurrency once and assumed it handled itself. It does not always.

• Query solid_queue_semaphores first when jobs stop entering ready state. A value of zero with a past expiry is the starvation signature.
• solid_queue_blocked_executions holds the jobs waiting behind a closed semaphore.
• The dispatcher maintenance interval defaults to 10 minutes. A brief semaphore release failure can sit unresolved for that entire window before auto-recovery kicks in.

What Happens to Jobs Behind a Closed Semaphore

Jobs blocked in solid_queue_blocked_executions do not get claimed by workers. Workers only see solid_queue_ready_executions. A job blocked at 3:04am is not visible to workers at all until the semaphore opens and the dispatcher moves it.

When the semaphore finally releases, the dispatcher moves one job at a time from blocked to ready. Workers then claim it on their next poll cycle.

Recovery is sequential, not parallel. With 47 blocked jobs and a single concurrency slot, they process one by one. We had two time-sensitive notification jobs in that queue. They went out an hour and a half late. The client noticed.

The part I had misread was duration. The README describes it as a failsafe for when something can happen that prevents the first job from releasing the semaphore. I had treated it as a job execution timeout. It is not.

duration controls how long a stuck semaphore survives before the maintenance task is allowed to force-release it.

A 3-minute duration with a 10-minute maintenance interval means you could wait 10 minutes after a semaphore failure, even if the duration expired 7 minutes ago. The maintenance task does not run continuously.

• Blocked jobs are invisible to workers until the dispatcher explicitly moves them to ready state.
• duration is a failsafe for semaphore release failures, not a job execution timeout.
• Short duration combined with long concurrency_maintenance_interval creates a recovery gap where jobs stay blocked longer than the duration value implies.

How to Set Limits That Do Not Starve the Queue

The fix had two parts.

First, we changed the duration on ReportGenerationJob to match its worst-case runtime plus a buffer.

class ReportGenerationJob < ApplicationJob
  queue_as :reports
  limits_concurrency to: 1, duration: 45.minutes, key: -> (report_id) { "report_generation" }
  def perform(report_id)
    # job body
  end
end

We had set 3 minutes because the documentation example used a short duration.

The problem is that duration controls how long a stuck semaphore survives before the maintenance task forces release. If your job legitimately runs for 22 minutes, a 3-minute duration means the semaphore expires before the job finishes. The maintenance task can try to release it while the job is still running.

We set 45 minutes because the job had never exceeded 35 minutes before that night. 45 gave us headroom.

Second, we isolated the report queue with its own worker pool and tightened the maintenance interval.

# config/solid_queue.yml
dispatchers:
  - polling_interval: 1
    batch_size: 500
    concurrency_maintenance_interval: 30
workers:
  - queues: [default, mailers, notifications]
    threads: 4
    processes: 2
  - queues: [reports]
    threads: 1
    processes: 1

We set concurrency_maintenance_interval to 30 seconds. The Solid Queue README's own production example uses 300 seconds. Going lower than 30 is possible but it runs a maintenance query against solid_queue_blocked_executions on every cycle. At 2 seconds that is continuous database load. On any app with meaningful queue volume, it adds up. Start at 30, monitor your database, and go lower only if you have measured the impact. The default 10 minutes was fine for most queues. Not for jobs where a delay gets noticed by a client.

• Set duration to cover the job's worst-case runtime plus 30-50% buffer, not a short failsafe value.
• concurrency_maintenance_interval controls how quickly Solid Queue recovers from a stuck semaphore. 30 seconds is a reasonable starting point for latency-sensitive queues. Below that you are running a maintenance query on every cycle, which adds continuous database load — measure before going lower.
• Isolating long-running jobs to a named queue limits blast radius. If the report semaphore gets stuck again, notifications are not blocked.

What Monitoring Actually Catches This

We had alerting on job failure rates and queue depth. Neither fired. The jobs were not failing. They were blocked. Queue depth was not growing in a visible way because we were not enqueuing many jobs at 3am.

After the incident we added two monitors.

The first tracks oldest scheduled_at in solid_queue_ready_executions. Worker crash, semaphore block, misconfigured queue name. All produce the same symptom and all fire this alert.

oldest_waiting = SolidQueue::ReadyExecution.minimum(:scheduled_at)
if oldest_waiting && oldest_waiting < 15.minutes.ago
  StatsD.gauge(
    "solid_queue.oldest_ready_job_age_seconds",
    (Time.current - oldest_waiting).to_i
  )
  PagerDutyAlert.trigger("Solid Queue stall detected")
end

The second tracks stuck semaphores directly.

stuck = SolidQueue::Semaphore.where(value: 0).where("expires_at < ?", Time.current)
if stuck.any?
  StatsD.gauge("solid_queue.stuck_semaphores", stuck.count)
end

A semaphore with value zero and a past expiry is always wrong. That monitor has fired twice since we added it. Once for a job that died mid-execution without releasing. Once for a deploy that killed a worker mid-job. Both times the alert fired before any user impact.

The pg_locks query is irrelevant here. Solid Queue does not hold advisory locks. If you are looking for advisory locks to diagnose a Solid Queue stall, you will find nothing useful. The state you need is in solid_queue_semaphores and solid_queue_blocked_executions.

• A semaphore stuck at zero with a past expiry is a direct signal, not an inference. Monitor it explicitly.
• Job age monitoring catches starvation that failure-rate and queue-depth monitors miss.
• pg_locks is the wrong diagnostic for Solid Queue. The tables are solid_queue_semaphores and solid_queue_blocked_executions.

What the Default Rails 8 Config Does Not Tell You

The default Rails 8 Solid Queue config has no worker timeouts and routes all queues through the same worker pool. It also has a 10-minute concurrency_maintenance_interval in the dispatcher.

That is fine for apps that do not use limits_concurrency. The moment you add it, the maintenance interval becomes part of your recovery SLA. A stuck semaphore that takes 10 minutes to release on its own is a 10-minute minimum delay for every blocked job.

Three things worth configuring before you go to production with limits_concurrency. Set duration to match worst-case job runtime, not a short failsafe value. Lower concurrency_maintenance_interval for queues where latency matters. Add explicit monitoring on solid_queue_semaphores for zero-value expired rows.

Solid Queue is still the right default for most Rails 8 apps. Not having an external broker is a real operational win. The failure modes just live in Postgres tables now, which means they are actually inspectable.

The night I ran that semaphore query and saw one row with value zero and an expiry from 91 minutes earlier, I knew immediately what had happened. That kind of clarity at 4am is not something I take for granted.

If your queue is silent and workers are alive, check solid_queue_semaphores before anything else.

Related reads

• Your Rails 8 App Is Running Solid Queue. It Has a 1.2-Second Job Pickup Delay. Here Is How to Know If That Is Killing You.
• Rails 8.1 ActiveJob Changes Will Break Your Jobs. Here Is How to Migrate Safely.