telemetry
telemetry is for the numbers of a program: how long a job took, how many were retried, how many connections a pool has. A library calls Telemetry.execute to say that something happened, and does not decide where the number goes. The app attaches a handler that counts it, logs it or sends it to a monitor.
std/logger is for text that a person reads. An event of telemetry is data for a program: a name, numbers and a few strings.
Add it in a project with an agentica.toml, with its prefixed tag and its folder as --subdir:
agentica add telemetry https://github.com/agentica-lang/packages --tag telemetry-0.1.0 --subdir telemetryThat writes this line of agentica.toml, and the entry of agentica.lock:
[dependencies]
telemetry = { git = "https://github.com/agentica-lang/packages", tag = "telemetry-0.1.0", subdir = "telemetry" }An example #
Start the hub once, in main. A handler is a function from an event to a Result[Void], attached under an id to the events it wants.
import Telemetry from "pkg/telemetry/telemetry"
def show(event: Telemetry.Event) -> Result[Void]
puts("#{event.name} #{event.ints} #{event.metadata}")
ok()
end
def main() -> Void
Telemetry.start().unwrap!
Telemetry.attach("show", ["jobs"], &show).unwrap!
Telemetry.execute("jobs.job.stop", ints: {"duration_ns": 1_200_000}, metadata: {"queue": "mail"})
Telemetry.execute("pool.checkout.stop", ints: {"duration_ns": 40})
Telemetry.flush().unwrap!
Telemetry.stop().unwrap!
endjobs.job.stop {"duration_ns": 1200000} {"queue": "mail"}The handler heard jobs.job.stop and not pool.checkout.stop. flush waits until the handlers have handled what was emitted before it: delivery is async, so a program or a test calls it before it reads what a handler did, or before it ends. The testing page shows how to assert on events in a test.
An event #
An event has:
name: a dotted name such asjobs.job.stop(the conventions say how to choose one);ints: aMap[String, Int]for the counts, the durations (duration_ns) and the sizes (size_bytes);floats: aMap[String, Float]for the fractions and ratios;metadata: aMap[String, String]that says which one it was (queue,pool);time_ms: the wall clock whenexecuteran, andtask: the id of the task that called it.
There are two maps of measurements because the type tells a handler what it can do with a number. Integers are exact and ATS.add sums them atomically, so a counter that many tasks update needs no lock; a float is for a number that is not whole. The names say the type (ints and floats) and not the meaning, since a count, a duration and a size are all integers: the meaning is in the key, with its unit as a suffix. A key is in one map only.
Choosing the events #
attach(id, events, handler) takes a list of patterns. A pattern is an event name, or the start of one at a dot: "jobs" hears jobs, jobs.job and jobs.job.stop, and not jobs_extra.stop or job. "*" hears every event. A handler with several patterns that match one event hears it once. A pattern is one to four words of a to z, digits and _ joined by dots, or *; anything else is an argument error.
execute never fails, never waits and never checks its name. An event whose name is not valid is heard by the handlers whose patterns it happens to match, like any other ("*", or a pattern that is its first words), so a library must use good names: the conventions say which.
import Telemetry from "pkg/telemetry/telemetry"
def count(event: Telemetry.Event) -> Result[Void] = ok()
def main() -> Void
Telemetry.start().unwrap!
puts(Telemetry.enabled?("jobs.job.stop"))
Telemetry.attach("count", ["jobs.job"], &count).unwrap!
puts(Telemetry.enabled?("jobs.job.stop"))
puts(Telemetry.enabled?("jobs.enqueue"))
puts(Telemetry.handlers())
Telemetry.detach("count").unwrap!
puts(Telemetry.handlers())
Telemetry.stop().unwrap!
endfalse
true
false
["count"]
[]enabled? says whether an execute of a name reaches a handler. With no handler attached, execute costs one table read, so a library can emit everywhere; guard with enabled? only an event whose arguments cost something to build.
Spans #
Telemetry.span(name, metadata) do ... end runs a block and says how it went. It emits NAME.start before the block, and then NAME.stop with duration_ns in ints when the block gave ok, or NAME.error with duration_ns and the metadata error_kind when it gave an error. The three events have your metadata and the same span_id, so a reporter can match them. The span gives back the result of the block, whatever it was.
import Telemetry from "pkg/telemetry/telemetry"
def show(event: Telemetry.Event) -> Result[Void]
puts("#{event.name} span_id=#{event.metadata.get("span_id", "")} duration=#{event.ints.has_key?("duration_ns")}")
ok()
end
def send(to: String) -> Result[Int]
return error("no address", kind: "mail.address") if to == ""
ok(to.size)
end
def main() -> Void
Telemetry.start().unwrap!
Telemetry.attach("show", ["mail"], &show).unwrap!
sent = Telemetry.span("mail.send", {"queue": "welcome"}) do send("ana@example.com") end
Telemetry.flush().unwrap!
puts(sent)
refused = Telemetry.span("mail.send", {"queue": "welcome"}) do send("") end
Telemetry.flush().unwrap!
puts(refused)
Telemetry.stop().unwrap!
endmail.send.start span_id=1 duration=false
mail.send.stop span_id=1 duration=true
ok(15)
mail.send.start span_id=2 duration=false
mail.send.error span_id=2 duration=true
error("no address", kind: "mail.address")When no handler hears any of the three events, a span only calls the block: it reads no clock and makes no id, and costs a few steps more than the call. A handler that is attached while a span runs may get only its end event. A block that crashes (a runtime error, not an error result) ends its task, so its span emits NAME.start and no NAME.stop or NAME.error: do not rely on the end event to close a span. Use Telemetry.enabled?(name) before building metadata that costs something.
Slow and broken handlers #
Each handler runs in an actor of its own, with a queue of Options.capacity events (1024 by default). execute puts the event in the queue of each handler that hears it and goes on, so a slow handler delays only itself. When a queue is full the newest event is dropped, and Telemetry.dropped(id) counts it (the count stays after the handler is detached, until its id is attached again). Events of one emitter reach a handler in order; events of different tasks have no order between them.
import Telemetry from "pkg/telemetry/telemetry"
def stuck(event: Telemetry.Event) -> Result[Void]
Task.sleep(300)
ok()
end
def main() -> Void
Telemetry.start().unwrap!
Telemetry.attach("slow", ["tick"], &stuck, Telemetry.Options(capacity: 1)).unwrap!
for i in 1..10
Telemetry.execute("tick.n", ints: {"i": i})
end
puts("the emitter did not wait")
puts(Telemetry.dropped("slow") >= 8)
Telemetry.stop().unwrap!
endthe emitter did not wait
trueA handler that returns an error, or crashes, is detached at once: its actor ends and its routes go. The hub writes one Logger.warning with the id and the kind of the error (or the RUNTIME- code of the crash), and nothing of the event. The other handlers and the emitter are not affected. After a flush, a handler that failed on an earlier event is already detached, and its id is free again.
import Telemetry from "pkg/telemetry/telemetry"
def fussy(event: Telemetry.Event) -> Result[Void] = error("no thanks", kind: "fussy.no")
def main() -> Void
Telemetry.start().unwrap!
Telemetry.attach("fussy", ["tick"], &fussy).unwrap!
Telemetry.execute("tick.a")
Telemetry.flush().unwrap!
puts(Telemetry.handlers())
Telemetry.stop().unwrap!
end[]A crash prints its diagnostic on stderr, as any crash of a task does, and does not fail the run: the hub watches the handler, so it is observed.
The API #
Telemetry.start() -> Result[Void]
Telemetry.stop(timeout_ms: Int = 5000) -> Result[Void]
Telemetry.execute(name: String, ints: Map[String, Int] = {}, floats: Map[String, Float] = {},
metadata: Map[String, String] = {}) -> Void
Telemetry.span(name: String, metadata: Map[String, String], body: Fn[Result[T]]) -> Result[T]
Telemetry.enabled?(name: String) -> Bool
Telemetry.attach(id: String, events: List[String], handler: Fn[Telemetry.Event, Result[Void]],
options: Telemetry.Options = Telemetry.Options()) -> Result[Void]
Telemetry.detach(id: String) -> Result[Void]
Telemetry.handlers() -> List[String]
Telemetry.dropped(id: String) -> Int
Telemetry.flush(timeout_ms: Int = 5000) -> Result[Void]startstarts the hub, an actor that is the only task that changes the handlers; it is a child of the task that calls it, so call it inmain. Emitters read the routes without asking the hub. Beforestart,executedoes nothing andattachfails. Starts take turns: astartthat finds another running waits for it (up to 2 seconds) and givestelemetry.started, so after anystartreturns, the hub is registered.stopdrains the handlers (asflush), detaches them all and ends the hub. A test that shares a process with others callsstopat its end andstartat its beginning.detachremoves a handler at once: the events in its queue are not delivered, and none is routed to it oncedetachhas returned.flushandstopwait for the handlers, so a handler that calls them would wait for itself: they givetelemetry.in_handler. A hub that ended with the task that started it is replaced by the nextstart; until thenflushgivesok, andstop,attachanddetachgivetelemetry.not_started.- Errors have a kind:
telemetry.not_started,telemetry.started,telemetry.duplicate(the id is in use),telemetry.unknown(detachof an id that is not attached),telemetry.timeout(flushorstopran out of time, or the hub did not answer anattachordetachin 5 seconds: the call may still take effect later),telemetry.in_handler, andargumentfor a bad id, pattern or capacity (1 to 1_048_576).