jobs says what happens to a job with the telemetry package, as std/logger says it in text for a person. The calls of Jobs (enqueue, enqueue_all, fetch, ack, nack) emit an event after the backend answered, so every backend gets them, and Jobs.drain also runs each job in a span. Nothing is emitted for a call that failed: an event says what happened. With no handler attached a call costs a few dozen steps more (one table read, and no map is built), so a program that does not use telemetry pays nothing it can see. Once any handler is attached, even one for other events, each call costs a little more, about 100 steps for each check of whether a handler hears it.

An app attaches the handlers, after Telemetry.start() in its main, with the names it wants (["jobs"] for all of them). Use the same telemetry tag as jobs does, or the manifest has two sources for one name (PKG-SOURCE-CONFLICT).

EventMeasurementsMetadata
jobs.enqueuecount (jobs added), duplicate_count (jobs not added)queue, backend
jobs.fetchcount (jobs leased)queue, backend
jobs.ackcount (1)queue, id, attempt, backend
jobs.nackcount (1)queue, id, attempt, backend, how (retry, release or dead)
jobs.job.startnonequeue, id, attempt, backend, span_id
jobs.job.stopduration_nsas start
jobs.job.errorduration_nsas start, and error_kind
jobs.job.settlecount (1)queue, id, attempt, backend, verdict (ack, retry or dead)
  • jobs.enqueue is one event for each queue of a batch. count is the jobs that were added, and duplicate_count the ones that were a duplicate of a unique key and so were not added: together they are the jobs that the caller offered.
  • jobs.job.start|stop|error and jobs.job.settle come only from Jobs.drain: a consumer that calls fetch, ack and nack itself gets the four events of the calls, and no span of its own.
  • A fetch that leased nothing emits nothing: a consumer that polls would fill a handler with empty events. The event of a fetch that did lease has the number of jobs.
  • jobs.job.start|stop|error are the span of the handler in Jobs.drain. A handler that returns an error ends in jobs.job.error, with the error_kind of that error (jobs.discard for Jobs.discard, and error for a plain one). The time is the time of the handler, not of the ack.
  • jobs.job.settle is what the drain decided after the handler: ack, retry (the job runs again) or dead (a discard, or a failure on the last attempt). It comes after the jobs.ack or jobs.nack that carried it out.
  • The metadata has ids and names, never the payload or the text of an error. The id is a string that the backend makes up, a good key for a trace and a bad label for a metric: keep it out of the labels of a reporter. The logger lines of a drain stay as they were, for people.
  • Events of one task arrive in order, and events of different tasks have no order between them. A handler is slow or fails without hurting the consumer (see the telemetry docs); the events it cannot take are counted by Telemetry.dropped.

Read them in a test #

Testing.session of telemetry captures the events of a block and leaves no task running, so it fits in a test of an app. This one drains a queue and reads what jobs said.

import Jobs from "pkg/jobs/jobs"
import Telemetry from "pkg/telemetry/telemetry"
import Testing from "pkg/telemetry/testing"

def handle(job: Jobs.Job) -> Result[Void]
  return Jobs.discard("not an address") if job.attempt == 1

  ok()
end

def line(event: Telemetry.Event) -> String
  extra = event.metadata.get("error_kind", event.metadata.get("verdict", event.metadata.get("how", "")))
  return event.name if extra == ""

  "#{event.name} #{extra}"
end

def drain_one(backend: Jobs.Backend, capture: Testing.Capture) -> Result[List[String]]
  _ = Jobs.enqueue(backend, Jobs.NewJob(queue: "mail", payload: "nobody".bytes)).unwrap!
  _ = Jobs.drain(backend, Jobs.Drain(queue: "mail"), &handle)
  found = try Testing.events(capture)
  ok(found.map(&line))
end

def main() -> Void
  backend = Jobs.memory().unwrap!
  for text in Testing.session(["jobs"], ->(capture) { drain_one(backend, capture) }).unwrap!
    puts(text)
  end

  Jobs.close(backend, 1_000)
end
Output
jobs.enqueue
jobs.fetch
jobs.job.start
jobs.job.error jobs.discard
jobs.nack dead
jobs.job.settle dead