The contract

A backend is a Jobs.Backend: a record of functions, since Agentica has no traits. Jobs.enqueue and the others call its fields, as Db.query calls a connection, and a backend package fills them. A wrapper that replaces one field (backend.with(fetch: ...)) can log, count or inject faults.

struct Backend
  name: String
  caps: Caps
  enqueue: Fn[List[NewJob], Result[List[Enqueued]]]
  fetch: Fn[Fetch, Result[List[Lease]]]
  ack: Fn[Lease, Result[Void]]
  nack: Fn[Lease, Nack, Result[Void]]
  extend: Fn[Lease, Int, Result[Lease]]
  counts: Fn[String, Result[Counts]]
  close: Fn[Int, Void]
end

What every backend guarantees #

  1. At least once. After enqueue gives ok, the job is delivered until it is acked or dead. A consumer that crashes or hangs loses its lease when it runs out, and the job comes back.
  2. One live lease. While a lease is live, no fetch gets that job. A new delivery has a new token, and ack, nack and extend with the old token fail with jobs.lease_lost. A lease is over at its expires_at_ms: a call at exactly that time finds it expired, and one a millisecond before finds it live.
  3. Not early. No job is delivered before its run_at_ms.
  4. Attempts count at the fetch. attempt is 1 on the first delivery and counts every delivery, also one whose lease ran out. A delivery past max_attempts does not happen: a lease that runs out on the last attempt makes the job dead, and so does a Retry on the last attempt. A job that crashes its consumer cannot loop forever. Release does not count.
  5. Unique keys. While a job with the same queue and unique_key is ready, scheduled or leased, enqueue gives its id with duplicate: true. The key is free when the job is acked or dead.
  6. enqueue_all is all or nothing: a job that is not valid, or that does not fit, fails the call and adds none. A backend writes the batch in one transaction, or in one step of its owner, so no caller sees part of it.
  7. last_error. A Retry that carries an error makes it the last_error of the next delivery, a Retry with no error keeps the previous one, and a dead job keeps its error (the one of the Dead, of the last Retry, or why it ran out of attempts or lease). Version 1 has no call that reads a dead job, so a caller cannot observe that last part: only test/store_test.agc checks it, for the memory store, and the conformance suite does not.
  8. close refuses new calls with jobs.closed and waits up to drain_ms for the backend to stop. Closing twice does nothing. A fetch that waits when close runs ends with jobs.closed.
  9. Groups. A job may have a group, a name in its queue. A fetch with max_per_group above 0 never takes a job of a group that already has that many live leases in the queue: it leaves the job where it is and takes the next ready one, so a full group does not hold back the jobs behind it, and the leases of the fetch itself count. max_per_group: 0 is no limit, and a job with no group is never limited. A lease is live from the fetch until an ack, a nack of any kind or its expiry (a Release and a Retry too), and a fetch that waits for a full group is woken when one ends. max_per_group below 0 and an empty group fail with argument.

There is no order guarantee. Ready jobs come roughly by run_at_ms, then by insertion. Acked jobs are deleted, and dead jobs are kept (the memory backend keeps max_dead of them). The payload is bytes: every store holds bytes, and Jobs.json is the typed form on top.

Times are Unix milliseconds. A backend takes a clock option (Fn[Int]) and defaults to the wall clock, which a database also stores. The wall clock can jump: a jump forward ends leases early, and at-least-once still holds.

The memory backend #

Jobs.memory(Jobs.MemoryOptions(...)) keeps the jobs in one actor. max_jobs (10 000) is how many live jobs it holds: past it enqueue fails with jobs.full, and a producer decides to log, drop or answer 503. max_dead (1 000) is how many dead jobs it keeps; the oldest is dropped. clock is the time source. Two behaviors are the memory backend's own, not the contract's, and the conformance suite does not check them: a job that a caller that gave up would have got goes back to the queue without counting an attempt, and a fetch that waits also ends at wait_ms of real time. specs/memory.feature tests both ("A job that a caller that gave up would have got stays in the queue" and "A fetch that waits ends at its wait_ms of real time"). Every call applies the clock first, so a lease that ran out is back before the call is served.

A fetch with a limit per group passes over a full group in one step, whatever its backlog, so it costs the number of full groups, not of their jobs; a fetch that waits for a full group is asked again only when a job comes or comes due, or a lease ends. A fetch with wait_ms parks until a job comes or is due, or its time is up: wait_ms on the clock of the backend, and also wait_ms of real time, so on a manual clock that nobody moves a fetch still ends with ok([]). fetch_timeout_ms is the longest a caller waits for a fetch, its wait included (the default is wait_ms and 10 s more); a caller that gives up with jobs.timeout leaves its fetch parked for one more second at most, and a job that the fetch gets goes back to the queue. fetch_timeout_ms must be at least 1. A caller that gives up while it waits does not lose the jobs the actor had for it: they go back to the queue, and the delivery does not count.

The conformance suite #

import Conformance from "pkg/jobs/conformance" runs the cases that every backend must pass, so a new backend keeps the contract. Each case gets a fresh backend from your Maker, a function from a clock to a backend, and moves a manual clock, so no case sleeps.

type Maker = Fn[Fn[Int], Result[Jobs.Backend]]
Conformance.names() -> List[String]
Conformance.run(name: String, make: Maker) -> Result[Void]
Conformance.run_all(make: Maker) -> List[Outcome]
Conformance.failed(outcomes: List[Outcome]) -> List[String]
Conformance.failures(outcomes: List[Outcome]) -> String

A backend package has one scenario, "the backend passes every contract case", that runs run_all and fails with Conformance.failures. A backend that cannot follow an injected clock (Caps.has_clock false, as a database that uses its own time) gets the real clock: its time cases sleep, with a unit of one second, and check what holds a little before and a little after a time, not to the millisecond. has_clock is the only capability: every guarantee above, unique keys and extend included, is required of every backend, so a store that has no unique index or no way to move a lease keeps them in its own code, or does not fill the contract.

import Conformance from "pkg/jobs/conformance"
import Jobs from "pkg/jobs/jobs"

def make(clock: Fn[Int]) -> Result[Jobs.Backend] = Jobs.memory(Jobs.MemoryOptions(clock: some(clock)))

def main() -> Void
  outcomes = Conformance.run_all(&make)
  puts("cases: #{outcomes.size}")
  puts("failed: #{Conformance.failed(outcomes)}")
end
Output
cases: 39
failed: []

The cases cover: the last_error of a retry, the job as it was enqueued, byte for byte; empty queues and queues that do not see each other; a fetch that waits, for a job that comes and for a job that is due; leases that run out, a late ack, nack or extend with an old token, and the same calls after the lease ran out but before any other fetch; ack, retry, release and dead; the last attempt; extend, and the job that comes back at the new end; a job that is not delivered early; unique keys, held through a retry, a release and an expired lease, free again when the job dies on its last attempt by expiry or by a retry, and two jobs of one batch with one key; a lease that is over at exactly its end and not a millisecond before; the groups: a full group skipped while the jobs of the others are served, kept apart by queue, freed by an ack, a retry, a release, a dead job and an expiry, woken from a fetch that waits, and a limit of 0 or a job with no group that is never held back; enqueue_all with a job that is not valid; eight tasks that fetch 200 jobs at once, each delivered once; the calls after close, and a second close; and a fetch that waits when the backend is closed, which ends with jobs.closed.

For the author of a backend #

A backend package tests itself with the suite, in BDD first: one scenario runs every case, and a new case of a newer jobs tag needs no change in the package.

Scenario: The backend passes every contract case
  When the suite runs on the backend
  Then no case failed
import BDD from "std/bdd"
import Conformance from "pkg/jobs/conformance"
import Jobs from "pkg/jobs/jobs"

private

FAILED = "failed"

def make(clock: Fn[Int]) -> Result[Jobs.Backend] = Jobs.memory(Jobs.MemoryOptions(clock: some(clock)))

def main() -> Void
  BDD.run([
    BDD.step("the suite runs on the backend") do |ctx, _args|
      failed = Conformance.failed(Conformance.run_all(&make))
      BDD.set(ctx, FAILED, JSON.from(failed))
    end,
    BDD.step("no case failed") do |ctx, _args|
      failed = failed_cases(ctx)
      BDD.assert(ctx, failed.empty?, "cases failed: #{failed.join("; ")}")
    end,
  ])
end

def failed_cases(ctx: BDD.Context) -> List[String]
  JSON.decode(BDD.get(ctx, FAILED).unwrap!, List[String]).unwrap_or([])
end
  • Each case needs a fresh, empty backend: the maker is called once for each case, and a backend that shares a database between calls must empty its tables, or use a schema of its own.
  • The cases use the queue names conformance and other, and the keys and payloads that they make up. A table of the backend can hold nothing else that those names would meet.
  • The maker runs in the task that calls run_all, so a backend that starts tasks (an actor) is a child of it and stays up while the case runs. The suite closes the backend after each case, so the maker must not close it.
  • On a backend with has_clock: false the cases on time run on the real clock and sleep. The whole suite takes about 31 s then, and under a second on a manual clock. To time the kit alone, run time agentica FILE on a file whose main calls Conformance.run_all with the maker and prints Conformance.failed of its outcomes.