Which worker runs what, in which order, and when.
tarsk worker --queues high,low
The order is a priority, not a preference: nothing from low is claimed while high has work.
Postgres sorts by it in the same statement that claims the row, so it costs nothing there.
Redis needs a read per queue — the highest with anything waiting wins, and only when they are
all empty is there anything to wait on, which is one blocking read across all of them. The
price is one extra round trip per claim while a high queue is empty and a low one is not, paid
by workers that asked for more than one queue.
Strict at the moment of claiming, which is the only moment it can be. A batch already claimed is already leased to this worker and still runs; priority decides what is read next, not what is already held. On Redis that is a prefetch batch of slack. Postgres takes only as many rows as the worker has free slots to start them in, highest queue first, so nothing is held back behind them.
This page used to say --queues high,low merely preferred the first within one read, which was
true and is what a single XREADGROUP over two streams does — Redis answers it with up to
COUNT from each, so a batch held low work that ran ahead of high work arriving while it
drained.
--hard-max-rss is the exception, and it is off by default. Set it and a child that reaches it
mid-task is killed rather than allowed to keep growing — the task is retried, and if it cannot
fit it is dead-lettered instead of retried forever. It cannot make an oversized task succeed;
it decides who loses when one task threatens the box. It must sit above --max-rss, or the
graceful ceiling could never fire.
Redis Streams, Postgres, and RabbitMQ. The first two are the full-featured pair: at-least-once, leasing per task rather than per worker, neither needing lease renewal — a task’s timeout is capped, so a lease cannot outlive a known ceiling.
RabbitMQ, or anything speaking AMQP 0.9.1, is the basic backend: amqp:// and amqps://.
It delivers jobs and delivers them well, and that is the whole of what it does, because AMQP
is a message transport with no shared store. Choose it when RabbitMQ is what you already run
and jobs are what you need moved. If you want any of the things in the right-hand column
below, choose Redis or Postgres; they are not on a roadmap for AMQP, because there is nowhere
in the protocol to put them.
| Redis, Postgres | RabbitMQ | |
|---|---|---|
| delivery, retries, delays, priority, chains, groups, results, dead letters | yes | yes |
rate limits, max_concurrency |
one limit across every worker | one limit per worker |
| cron | elected once, missed runs caught up | fires once per worker, not caught up |
| cancel, pause, send dedup, chords | yes | refused, with an error that says why |
result.get() |
told when the result lands | asks twenty times a second |
tarsk status |
ready, running, delayed, dead | ready and dead |
That cuts both ways.
What it does better than a lease can: an unacked delivery is redelivered the instant the worker’s connection dies — the connection is the lease, and there is no timeout to wait out. Delays ride per-value TTL queues that dead-letter back when they fire, and clean themselves up when idle. The dead-letter queue, results, progress, chains, groups, per-send expiry and strict queue priority all work as they do elsewhere.
What degrades to per-worker, which is the shape Celery has always had on RabbitMQ: rate
limits, max_concurrency, and the cron election. Each is enforced correctly inside one
worker; N workers each enforce their own copy, so a 5/s limit across four workers is 20/s.
Two things are refused rather than faked, with an error that says why: cancelling queued work
and send deduplication — both need state every worker can read. And retries counts handler
failures, not crashes: a crash redelivery reuses the server’s copy of the message, whose
attempt counter the dead worker never got to bump. tarsk status shows ready and dead counts;
in-flight and delayed sit in places AMQP proper cannot count. If any of these are the feature
you came for, Redis and Postgres hold them broker-side.
There is a fourth, memory://, and it is deliberately not a production option: an in-process
broker that lives and dies with the batch run the test suite and the benchmarks use. Inside
one process it carries the full job record — chains, groups fan out, max_concurrency holds,
expires drops stale work, rate limits meter, results and progress read back — so every
behaviour that is about a job can be tested without a server. What it cannot express is
anything that is about more than one process: a second worker, the CLI inspecting a running
worker’s queues or dead letters, cancellation from outside, send deduplication between
producers. Those need a broker that outlives a process, which is the entire difference between
it and the other two.
On Postgres a claim and an ack do not wait for the disk. Those two statements run with
synchronous_commit off, for themselves alone: the server answers once the change is in its
memory and writes it out within a fraction of a second. A server that crashes inside that
window forgets it, and the job is delivered again — which is the at-least-once you already
have to write handlers for. A job sent, a result, and a dead letter still wait for the disk,
because those are the ones that would be lost and not repeated. It is half of what a job
costs on Postgres.
A Postgres queue is a table that is nearly all churn, and two things keep that from
showing. A claim does not start from the bottom of the table: it starts just below the
highest id the worker has already taken, so the dead rows finished jobs leave behind are not
walked on every claim, and once a second it looks from the bottom anyway for anything handed
back with no delay. And tarsk_jobs asks for a vacuum after 2,000 dead rows instead of the
default fifth of the table, set as a storage parameter on that one table when the schema is
created. If your Postgres runs with autovacuum off, vacuum tarsk_jobs yourself; the queue
keeps its speed either way, and the table keeps its size only if something does.
Both speak TLS, which every managed instance requires. rediss:// for Redis, and for Postgres
whatever sslmode says — require, verify-ca and verify-full all verify the chain and the
hostname, since rustls does that on every handshake. The last two are spellings tokio-postgres
does not know and are mapped to require rather than refused, because a connection string
copied from a provider usually says one of them. sslmode=disable, or no sslmode against a
server without TLS, still connects in the clear.
Point SSL_CERT_FILE at your provider’s CA bundle if it is not one the system already trusts.
rediss:// connects over TLS, which every managed Redis requires — verified against the
system trust store, so an untrusted certificate is refused rather than accepted. Append
#insecure to skip that check for a self-signed certificate you control; nothing else turns
it off.
Failures follow the task’s own retries and backoff, and neither backend needed a delay
queue for it: Postgres already has a visibility timer, and Redis has the idle clock its pending
sweep reads. What runs out of retries lands in tarsk:{queue}:dead or the tarsk_dead table,
with the error and traceback.
Nothing here grows without a bound. Results and progress carry the TTL the task asked for
and expire on their own; the live stream loses an entry when it is acked; reservations, buckets
and cancellations run on their own clocks. Dead letters were the exception — one payload and
one full traceback per row, kept forever — so --max-dead caps them at 1,000 a queue and
drops the oldest past that.
That is a count, and a count is the proxy the tables above spend their time arguing against: it
bounds bytes only if you know what a row weighs. Redis Streams offer MAXLEN and MINID and
nothing by size, so a count is what the datastore gives. Measured, a failure costs about 2 KB
plus its payload — mostly traceback — so a thousand is a couple of megabytes a queue with
small payloads, and ten thousand of a one-megabyte payload would be ten gigabytes. Raise it
knowing what yours weigh.
The default is small because of which way it fails. Too small loses old failures, which is
visible and documented. Too large runs the broker out of memory and takes the healthy work down
with it, which is the failure this whole project exists to prevent. --max-dead 0 keeps every
failure, and is right only if something else is emptying them.
On Redis the trim is approximate — it drops whole nodes and charges almost nothing for it — so the cap is a bound rather than an exact count.
job = await send_email.send_async(user.id)
answer = await app.result(job).get_async(timeout=30)
send() and get() are blocking, and in a web handler that matters more than the microseconds
suggest. Measured on 500 sequential enqueues against a local Redis: the blocking pair held the
event loop for 45ms of the 46ms it took, while the async pair kept it responsive at 103µs
median lag. get() is worse than either — it polls, so awaiting a result on the loop holds it
for up to the timeout, thirty seconds by default.
The Rust producer releases the GIL for the round trip, so the async versions hand it to a thread and the loop is free for the whole wait rather than the parts Python was not holding. That is a thread hop rather than a native async client: 500 concurrent enqueues take 57ms, and the ceiling is asyncio’s default executor rather than the network.
from tarsk import chain, group
pipeline = chain(fetch.s(url), parse.s(), store.si("bucket"))
result_id = pipeline.send() # only the first step is queued now
answer = app.result(result_id).get() # the id of the last step, known up front
ids = group(resize.s(f) for f in files).send()
t.s(...) is fed the previous step’s result as its first argument; t.si(...) is not, for
steps that take only what they were given. Every step’s id is minted by the client before
anything is sent, which is why a handle to the end of a chain exists while its first step is
still being written.
The chain travels as data on the job record, a list of [id, name, payload, timeout, queue,
feed]. The supervisor splices the result into the next step’s arguments and queues it; it never
learns what any of those names mean, which is the same rule that keeps it from importing your
app (§4.1). The next step is queued before the current one is acked, so a crash in between
redelivers the step rather than dropping the rest of the chain — at-least-once, as everywhere
else here.
A chain that fails partway stops there: the remaining steps were never queued, so there is
nothing to cancel. A group is deliberately thin — the queue already runs what it can in
parallel, and what group adds is handles that exist before the sending does.
from tarsk import chord, group
work = chord(group(resize.si(f) for f in files), publish.s(album_id))
answer = app.result(work.send()).get()
Members run side by side; the callback runs once, after all of them, and is handed a list of
what they returned — in the order they were given, whichever order they finished in — ahead of
its own arguments. publish.si(...) for a callback that does not want the list.
Each member carries its place and the callback where a chain step would be, so this too is data on the job record. What it adds is a count: finished members are recorded in the broker, and the member whose arrival completes the set queues the callback. Exactly one member is told so, however they race, and the same one is told again if it is redelivered, so a callback that could not be queued the first time is queued the second.
If a member fails for good, the callback never runs — there is no list to hand it. The failed member is in the dead letters like any other failure, and replaying it there completes the chord. What has been recorded is kept for a day.
Not on RabbitMQ: counting needs a store every worker can write to, and send() says so before
anything is queued rather than letting the members run and the callback go missing.
ids = app.send_many(resize.si(f) for f in files)
One round trip instead of one a job, and on Redis and Postgres one unit: they are all queued
or none are, where a loop of send() leaves half a batch behind when the connection drops in
the middle. A batch holding one job the broker cannot take is refused whole, on every backend. A group and a chord are sent this way. Jobs that have to be decided one at a time
still are — anything delayed, and anything with unique= or a dedup key — and RabbitMQ
confirms each in turn, having no way to take a batch as a unit.
send_in(60, …) runs a task once, later; send_at(when) takes a timezone-aware datetime.
task.options(queue=…, timeout=…, task_id=…) overrides any of it for one send — keyword-only
and on its own object, so nothing can collide with an argument the task itself takes. A
caller-supplied id doubles as an idempotency key.
@app.task(cron="*/5 * * * *") runs it on a
schedule — five fields, evaluated in UTC.
UTC is the whole timezone story, deliberately. A local schedule has to answer what “02:30 daily” means on the night a clock jumps, and every answer is someone’s outage.
AsyncResult.get() does not poll. Redis publishes and Postgres notifies when a result is
stored, so the answer arrives when it exists — about 2ms after the task finishes on a local
Redis and 9ms on Postgres, where asking every 50ms made it 53 and 55 — and waiting on a slow
task costs the broker nothing in the meantime. A Postgres worker is told about new work the
same way, rather than asking forty times a second while there is none.
The schedule lives with the supervisor, because that is the only process holding the registry and no user code. Each run is elected through the broker, so running ten workers still runs the task once.
A run that came due while no worker was up is caught up. When a worker returns it fires the
most recent run that was missed — once, however many were: an hourly job that was down for six
hours gets one run, not six. It looks back a day and no further, so a worker switched on after
a month does not start by running last month’s reports. A schedule seen for the first time is
not owed anything: deploying a nightly job at noon does not run it at noon. If a late run is
worse than none, give the task an expires.
Catching up needs a record of the last run that outlives the worker, which Redis and Postgres
keep and RabbitMQ has nowhere to put: on amqp:// a schedule fires when a worker is up for it
and is not caught up after a restart.
7 is Sunday as well as 0. An expression that cannot be read is reported once on the
worker’s stderr, and that task then never fires — it is worth looking for the line after
adding a schedule.