tarsk

Routing and scheduling

Which worker runs what, in which order, and when.

← Back to the index

Priority

tarsk worker --queues high,low

The order is a priority, not a preference: nothing from low is claimed while high has work. Postgres sorts by it in the same statement that claims the row, so it costs nothing there. Redis needs a read per queue — the highest with anything waiting wins, and only when they are all empty is there anything to wait on, which is one blocking read across all of them. The price is one extra round trip per claim while a high queue is empty and a low one is not, paid by workers that asked for more than one queue.

Strict at the moment of claiming, which is the only moment it can be. A batch already claimed is already leased to this worker and still runs; priority decides what is read next, not what is already held. On Redis that is a prefetch batch of slack. Postgres takes only as many rows as the worker has free slots to start them in, highest queue first, so nothing is held back behind them.

This page used to say --queues high,low merely preferred the first within one read, which was true and is what a single XREADGROUP over two streams does — Redis answers it with up to COUNT from each, so a batch held low work that ran ahead of high work arriving while it drained.

--hard-max-rss is the exception, and it is off by default. Set it and a child that reaches it mid-task is killed rather than allowed to keep growing — the task is retried, and if it cannot fit it is dead-lettered instead of retried forever. It cannot make an oversized task succeed; it decides who loses when one task threatens the box. It must sit above --max-rss, or the graceful ceiling could never fire.

Brokers

Redis Streams, Postgres, and RabbitMQ. The first two are the full-featured pair: at-least-once, leasing per task rather than per worker, neither needing lease renewal — a task’s timeout is capped, so a lease cannot outlive a known ceiling.

RabbitMQ, or anything speaking AMQP 0.9.1, is the basic backend: amqp:// and amqps://. It delivers jobs and delivers them well, and that is the whole of what it does, because AMQP is a message transport with no shared store. Choose it when RabbitMQ is what you already run and jobs are what you need moved. If you want any of the things in the right-hand column below, choose Redis or Postgres; they are not on a roadmap for AMQP, because there is nowhere in the protocol to put them.

  Redis, Postgres RabbitMQ
delivery, retries, delays, priority, chains, groups, results, dead letters yes yes
rate limits, max_concurrency one limit across every worker one limit per worker
cron elected once, missed runs caught up fires once per worker, not caught up
cancel, pause, send dedup, chords yes refused, with an error that says why
result.get() told when the result lands asks twenty times a second
tarsk status ready, running, delayed, dead ready and dead

That cuts both ways.

What it does better than a lease can: an unacked delivery is redelivered the instant the worker’s connection dies — the connection is the lease, and there is no timeout to wait out. Delays ride per-value TTL queues that dead-letter back when they fire, and clean themselves up when idle. The dead-letter queue, results, progress, chains, groups, per-send expiry and strict queue priority all work as they do elsewhere.

What degrades to per-worker, which is the shape Celery has always had on RabbitMQ: rate limits, max_concurrency, and the cron election. Each is enforced correctly inside one worker; N workers each enforce their own copy, so a 5/s limit across four workers is 20/s. Two things are refused rather than faked, with an error that says why: cancelling queued work and send deduplication — both need state every worker can read. And retries counts handler failures, not crashes: a crash redelivery reuses the server’s copy of the message, whose attempt counter the dead worker never got to bump. tarsk status shows ready and dead counts; in-flight and delayed sit in places AMQP proper cannot count. If any of these are the feature you came for, Redis and Postgres hold them broker-side.

There is a fourth, memory://, and it is deliberately not a production option: an in-process broker that lives and dies with the batch run the test suite and the benchmarks use. Inside one process it carries the full job record — chains, groups fan out, max_concurrency holds, expires drops stale work, rate limits meter, results and progress read back — so every behaviour that is about a job can be tested without a server. What it cannot express is anything that is about more than one process: a second worker, the CLI inspecting a running worker’s queues or dead letters, cancellation from outside, send deduplication between producers. Those need a broker that outlives a process, which is the entire difference between it and the other two.

On Postgres a claim and an ack do not wait for the disk. Those two statements run with synchronous_commit off, for themselves alone: the server answers once the change is in its memory and writes it out within a fraction of a second. A server that crashes inside that window forgets it, and the job is delivered again — which is the at-least-once you already have to write handlers for. A job sent, a result, and a dead letter still wait for the disk, because those are the ones that would be lost and not repeated. It is half of what a job costs on Postgres.

A Postgres queue is a table that is nearly all churn, and two things keep that from showing. A claim does not start from the bottom of the table: it starts just below the highest id the worker has already taken, so the dead rows finished jobs leave behind are not walked on every claim, and once a second it looks from the bottom anyway for anything handed back with no delay. And tarsk_jobs asks for a vacuum after 2,000 dead rows instead of the default fifth of the table, set as a storage parameter on that one table when the schema is created. If your Postgres runs with autovacuum off, vacuum tarsk_jobs yourself; the queue keeps its speed either way, and the table keeps its size only if something does.

Both speak TLS, which every managed instance requires. rediss:// for Redis, and for Postgres whatever sslmode says — require, verify-ca and verify-full all verify the chain and the hostname, since rustls does that on every handshake. The last two are spellings tokio-postgres does not know and are mapped to require rather than refused, because a connection string copied from a provider usually says one of them. sslmode=disable, or no sslmode against a server without TLS, still connects in the clear.

Point SSL_CERT_FILE at your provider’s CA bundle if it is not one the system already trusts.

rediss:// connects over TLS, which every managed Redis requires — verified against the system trust store, so an untrusted certificate is refused rather than accepted. Append #insecure to skip that check for a self-signed certificate you control; nothing else turns it off.

Failures follow the task’s own retries and backoff, and neither backend needed a delay queue for it: Postgres already has a visibility timer, and Redis has the idle clock its pending sweep reads. What runs out of retries lands in tarsk:{queue}:dead or the tarsk_dead table, with the error and traceback.

Nothing here grows without a bound. Results and progress carry the TTL the task asked for and expire on their own; the live stream loses an entry when it is acked; reservations, buckets and cancellations run on their own clocks. Dead letters were the exception — one payload and one full traceback per row, kept forever — so --max-dead caps them at 1,000 a queue and drops the oldest past that.

That is a count, and a count is the proxy the tables above spend their time arguing against: it bounds bytes only if you know what a row weighs. Redis Streams offer MAXLEN and MINID and nothing by size, so a count is what the datastore gives. Measured, a failure costs about 2 KB plus its payload — mostly traceback — so a thousand is a couple of megabytes a queue with small payloads, and ten thousand of a one-megabyte payload would be ten gigabytes. Raise it knowing what yours weigh.

The default is small because of which way it fails. Too small loses old failures, which is visible and documented. Too large runs the broker out of memory and takes the healthy work down with it, which is the failure this whole project exists to prevent. --max-dead 0 keeps every failure, and is right only if something else is emptying them.

On Redis the trim is approximate — it drops whole nodes and charges almost nothing for it — so the cap is a bound rather than an exact count.

Enqueueing from async code

job = await send_email.send_async(user.id)
answer = await app.result(job).get_async(timeout=30)

send() and get() are blocking, and in a web handler that matters more than the microseconds suggest. Measured on 500 sequential enqueues against a local Redis: the blocking pair held the event loop for 45ms of the 46ms it took, while the async pair kept it responsive at 103µs median lag. get() is worse than either — it polls, so awaiting a result on the loop holds it for up to the timeout, thirty seconds by default.

The Rust producer releases the GIL for the round trip, so the async versions hand it to a thread and the loop is free for the whole wait rather than the parts Python was not holding. That is a thread hop rather than a native async client: 500 concurrent enqueues take 57ms, and the ceiling is asyncio’s default executor rather than the network.

Chains and groups

from tarsk import chain, group

pipeline = chain(fetch.s(url), parse.s(), store.si("bucket"))
result_id = pipeline.send()          # only the first step is queued now
answer = app.result(result_id).get() # the id of the last step, known up front

ids = group(resize.s(f) for f in files).send()

t.s(...) is fed the previous step’s result as its first argument; t.si(...) is not, for steps that take only what they were given. Every step’s id is minted by the client before anything is sent, which is why a handle to the end of a chain exists while its first step is still being written.

The chain travels as data on the job record, a list of [id, name, payload, timeout, queue, feed]. The supervisor splices the result into the next step’s arguments and queues it; it never learns what any of those names mean, which is the same rule that keeps it from importing your app (§4.1). The next step is queued before the current one is acked, so a crash in between redelivers the step rather than dropping the rest of the chain — at-least-once, as everywhere else here.

A chain that fails partway stops there: the remaining steps were never queued, so there is nothing to cancel. A group is deliberately thin — the queue already runs what it can in parallel, and what group adds is handles that exist before the sending does.

Chords

from tarsk import chord, group

work = chord(group(resize.si(f) for f in files), publish.s(album_id))
answer = app.result(work.send()).get()

Members run side by side; the callback runs once, after all of them, and is handed a list of what they returned — in the order they were given, whichever order they finished in — ahead of its own arguments. publish.si(...) for a callback that does not want the list.

Each member carries its place and the callback where a chain step would be, so this too is data on the job record. What it adds is a count: finished members are recorded in the broker, and the member whose arrival completes the set queues the callback. Exactly one member is told so, however they race, and the same one is told again if it is redelivered, so a callback that could not be queued the first time is queued the second.

If a member fails for good, the callback never runs — there is no list to hand it. The failed member is in the dead letters like any other failure, and replaying it there completes the chord. What has been recorded is kept for a day.

Not on RabbitMQ: counting needs a store every worker can write to, and send() says so before anything is queued rather than letting the members run and the callback go missing.

Sending many at once

ids = app.send_many(resize.si(f) for f in files)

One round trip instead of one a job, and on Redis and Postgres one unit: they are all queued or none are, where a loop of send() leaves half a batch behind when the connection drops in the middle. A batch holding one job the broker cannot take is refused whole, on every backend. A group and a chord are sent this way. Jobs that have to be decided one at a time still are — anything delayed, and anything with unique= or a dedup key — and RabbitMQ confirms each in turn, having no way to take a batch as a unit.

Scheduling

send_in(60, …) runs a task once, later; send_at(when) takes a timezone-aware datetime. task.options(queue=…, timeout=…, task_id=…) overrides any of it for one send — keyword-only and on its own object, so nothing can collide with an argument the task itself takes. A caller-supplied id doubles as an idempotency key.

@app.task(cron="*/5 * * * *") runs it on a schedule — five fields, evaluated in UTC.

UTC is the whole timezone story, deliberately. A local schedule has to answer what “02:30 daily” means on the night a clock jumps, and every answer is someone’s outage.

AsyncResult.get() does not poll. Redis publishes and Postgres notifies when a result is stored, so the answer arrives when it exists — about 2ms after the task finishes on a local Redis and 9ms on Postgres, where asking every 50ms made it 53 and 55 — and waiting on a slow task costs the broker nothing in the meantime. A Postgres worker is told about new work the same way, rather than asking forty times a second while there is none.

The schedule lives with the supervisor, because that is the only process holding the registry and no user code. Each run is elected through the broker, so running ten workers still runs the task once.

A run that came due while no worker was up is caught up. When a worker returns it fires the most recent run that was missed — once, however many were: an hourly job that was down for six hours gets one run, not six. It looks back a day and no further, so a worker switched on after a month does not start by running last month’s reports. A schedule seen for the first time is not owed anything: deploying a nightly job at noon does not run it at noon. If a late run is worse than none, give the task an expires.

Catching up needs a record of the last run that outlives the worker, which Redis and Postgres keep and RabbitMQ has nowhere to put: on amqp:// a schedule fires when a worker is up for it and is not caught up after a restart.

7 is Sunday as well as 0. An expression that cannot be read is reported once on the worker’s stderr, and that task then never fires — it is worth looking for the line after adding a schedule.


← Back to the index