Background Jobs for Async Python, With a Dashboard You’ll Actually Want to Open (toro + matador)

Photo by Stephane YAICH on Unsplash

We’ve all been there. Your FastAPI app needs to send an email after a signup. Easy, you think. Then you need to retry it when the SMTP server has a bad day. Then a nightly report. Then a rate limit so the invoice API stops yelling at you. And somewhere around the third asyncio.create_task() that silently swallowed an exception, you admit it: you need a real job queue.

So you go shopping, and the shopping trip is weirdly disappointing. The big established option is built for a synchronous world, and your whole app is async def. The lightweight options feel great for about a week, until you ask “so… how do I see what failed last night?” and the answer is redis-cli and good intentions.

I wanted the boring middle: an async-first queue that takes reliability seriously, plus a dashboard that ships in the box. I couldn’t find it, so I built it. Two small packages:

toro, an async-first, Redis-backed job queue for Python

matador, a live dashboard for it (yes, the names are a bullfighting joke, the matador watches the toro)

Here’s the pitch, and why I think it’s worth twenty minutes of your time.

The queue: async from the first line

toro is not a sync library with async bolted on. Everything is await, end to end, on top of redis.asyncio.

from toro import Queue

queue = Queue("emails")

await queue.add("send-welcome", {"to": "ada@example.com"})
await queue.add("send-digest", {"user": 42}, delay=60_000, attempts=3,
                backoff={"type": "exponential", "delay": 1000})

The worker side is just as small. You write one async function, it gets jobs:

from toro import Worker

async def process(job):
    if job.name == "send-welcome":
        await send_welcome(job.data["to"])
    return {"sent": True}

worker = Worker("emails", process, concurrency=10)
await worker.run()

That’s the whole mental model. One Redis, one queue object, one worker function. No broker daemon, no result-backend matrix, no separate scheduler process to forget about.

The part I refuse to compromise on: not losing jobs

A job queue has exactly one sacred duty. If you give it a job, it must not lose it. Everything else is decoration.

Under the hood, every state transition in toro is a Lua script, so it’s atomic on the Redis server. There is no client-side window where a job is “between” states. Claiming a job, completing it, retrying it, recovering it: one script each.

If a worker dies mid-job, a mark-and-sweep pass notices the orphaned job and puts it back in line. And every worker holds a per-job lock with a random token, so even if a job somehow runs twice during a recovery, only one of them gets to commit the result. At-least-once execution, exactly-once results.

You get the production toolbox on top: priorities, delayed jobs, automatic retries with backoff, deduplication windows, cron and interval schedulers, queue-wide rate limiting, graceful worker draining. I went down the rabbit hole of what the big ecosystems charge for in their paid tiers, and it’s mostly this exact list. Here it’s just the library.

The dashboard: server-rendered, lives with your app

matador is one pip install and one mount:

from fastapi import Depends
from matador import create_app

app.mount("/jobs", create_app(
    ["emails", "billing"],
    connection=redis,                       # share your existing pool
    dependencies=[Depends(require_admin)],  # your auth, not mine
))

It’s FastAPI + Jinja + HTMX. No SPA, no build step, no node_modules the size of a small moon. Every tab and page is a real URL, live updates stream over SSE, and the whole thing reads straight from Redis through toro’s own API.

You can watch queues fill and drain, expand a job to see its payload and stack trace, retry or delete things (one at a time or in bulk), pause a queue, manage schedulers, and see every worker’s heartbeat, including the ones that died and what they were running when it happened.

And because it ships no auth of its own, it inherits yours. Mount it behind your admin guard and you’re done. If you mount it with no auth at all, it warns you loudly at startup, because someone had to be the adult here.

The metrics are where it got fun

For the latest release I went deep on one question: what should a queue dashboard actually show you? I read everything I could find: SRE writeups, monitoring vendor docs, postmortems of queue meltdowns. Two findings changed the design.

First: queue depth is a lie. Ten thousand 1ms jobs and ten 1s jobs are the same backlog. The number that tells the truth is latency, the age of the next job in line. If it reads 0ms, your workers are keeping up. If it reads 31 minutes, nobody is consuming the queue, no matter how innocent the depth looks. So latency is the first chip on every queue page.

Second: averages hide exactly what hurts you. One job class quietly degrading from 50ms to 5s disappears inside a healthy-looking mean. So toro records durations into small log-scale histogram buckets (about 52 bytes per job name per minute, in plain Redis hashes that expire on their own), and matador shows real p50/p95 percentiles, per queue and per job name.

There’s a table on every queue page that answers the actual 3am question, “which job is responsible,” with the worst offender sorted to row one. And the percentiles try hard not to lie to you: values are computed the statistically honest way (merge histogram buckets first, never average percentiles), and if a job only ran 7 times in the last hour, its p95 is dimmed with a tooltip telling you it’s just the slowest run wearing a costume.

pip install toro-queue matador-dashboard

Both are MIT, fully typed, tested against real Redis on every commit:

toro: https://github.com/ilovepixelart/toro

matador: https://github.com/ilovepixelart/matador