agentica test runs Gherkin .feature files. Each scenario starts your program as a separate process, sends it input and checks its output.

A first feature #

greet.agc greets the name it reads:

def main() -> Void
  name = gets!.strip
  puts("Hello, #{name}!")
end
Input
Ana
Output
Hello, Ana!

specs/greet.feature:

Feature: Greeting

  Scenario: Greet by name
    Given the Agentica program "../greet.agc"
    When I send the line "Ana"
    Then I receive the line "Hello, Ana!" within 5000 milliseconds
    And the program exits with code 0 within 5000 milliseconds

Run it:

agentica test specs/

Exit code 0: every scenario passes or is skipped (also when all are skipped, so a CI job should check that the skipped count is the one it expects). 1: one fails. 2: the tests could not run (also a shared temporary directory without the sticky bit: set TMPDIR to a private folder).

  • --ai and --json change the report format. --only TEXT keeps scenarios and tests whose name contains TEXT.
  • --tags EXPR keeps the scenarios whose tags match: @a, not @a, joined with and and or (not binds tighter than and, and and tighter than or; no parentheses), as in --tags "@db and not @slow". A scenario has the tags of its Feature and itself, and a Scenario Outline row also those of its Examples block. Unit test files have no tags, so they are left out. A bad expression exits with 2.
  • --jobs N and --isolate (each scenario in a process of its own, see below): Test.
  • A failed step is reported at its line and column in the feature file.
  • Each row of a Scenario Outline is named with the outline name and the row’s values in the columns the name does not show, such as Name (sqlite, 42), so --only "(sqlite, 42)" selects one row. Identical rows keep one name; --json has each row’s line in example.
  • --seed N makes Random repeat in steps and programs. Without it, the runner draws one seed for every step program and program under test, and shows it when the run fails.
  • It also runs *_test.agc unit test files (Test) in the same report. With no paths it searches specs/ and test/.

Built-in steps #

No code needed. {string} is text in double quotes, {int} a whole number.

StepDoes
the environment variable {string} is {string}sets a variable for the program, before it starts
the program argument {string}adds a command-line argument for the program, before it starts
the Agentica program {string}starts the program, a path relative to the feature file
I send the line {string}sends the text and a new line
I send a line:sends the Doc String below and a new line
I send input:sends the Doc String below as it is
I close stdincloses the program’s input
I receive the line {string} within {int} millisecondschecks the next output line
I receive a line within {int} milliseconds:checks the next line against the Doc String
I receive a JSON line within {int} milliseconds:compares the next line as JSON, in any key order
stdin remains openchecks that the program still runs
the program exits with code {int} within {int} millisecondswaits for the exit code
stdout is: / stderr is:after the program ends, compares the stdout that no line step has read yet, or all of stderr
stdout includes: / stderr includes:after the program ends, checks that the same text contains the Doc String as a substring

Given, When, Then, And and But match the same steps. Background, Scenario Outline with Examples, Doc Strings (""") and Data Tables work. A <...> in an outline is replaced when it names an Examples column; any other <...>, in a step or a Doc String, stays as written, as in Cucumber (#include <stdio.h> or HTML is fine). A misspelled column is not an error: the step text keeps its <name>, and the step fails as undefined if no pattern matches it.

Environment and arguments #

The two steps before the Agentica program give the program its environment and its System.args:

Scenario Outline: Greet in a language
  Given the environment variable "GREETING" is "<greeting>"
  And the program argument "--name"
  And the program argument "<name>"
  And the Agentica program "../greet.agc"
  Then I receive the line "<line>" within 5000 milliseconds

  Examples:
    | greeting | name | line      |
    | Hola     | Ana  | Hola, Ana |
  • A variable is added to the environment agentica test runs in; the program sees the others as they are. Setting a name again replaces its value. The variable reaches only the program of that scenario: not the runner, and no other scenario.
  • Each the program argument adds one argument, in order, with no splitting on spaces: "a b" is one argument. The program reads them with System.args.
  • Both come before the Agentica program; after it they fail with RUNTIME-BDD-CONTEXT, since the program has already started. A variable name must be nonempty and cannot contain =, a name or value cannot contain a NUL byte, and AGENTICA_CHANNEL, AGENTICA_VM_METRICS, AGENTICA_VM_METRICS_FILE and UV_USE_IO_URING, which the VM owns, cannot be set: each is RUNTIME-BDD-CONTEXT. On Windows names ignore case, so Path and then PATH set one variable, to the second value; two names that differ only in the case of a letter outside ASCII, such as Ä and ä, make the Agentica program fail with RUNTIME-BDD-IO.
  • A scenario starts one program. To run it again with other values, write another scenario (an outline row is one). For a peer that runs beside the program, see A peer program.

Your own steps #

A step program builds a list with BDD.step in main and passes it to BDD.run. Most BDD.* functions take the context first and return it. BDD.Context can be named but not built: a context exists only inside a step, so use the context that a call returns. The modules under std/bdd/ and std/test/ are internal to agentica test and unsupported. Load the steps with --steps:

agentica test specs/ --steps steps.agc

Without --steps, agentica test uses the nearest steps.agc found in the folder of each path that holds a .feature file (the path itself if it is a folder) or above it, up to the working directory, so specs/steps.agc needs no flag, also for specs/auth/ or specs/a.feature. It applies only to a run with .feature files. If the paths lead to two different steps.agc files, the run fails with RUNTIME-BDD-USAGE; pass --steps. --steps overrides it.

import BDD from "std/bdd"

def remember(ctx: BDD.Context, args: List[String]) -> BDD.Context = BDD.set(ctx, "count", JSON.from(args[0].to_i!))

def check_count(ctx: BDD.Context, args: List[String]) -> BDD.Context
  count = JSON.int(BDD.get(ctx, "count").unwrap!).unwrap!
  BDD.assert(ctx, count == args[0].to_i!, "wrong count")
end

def main() -> Void
  BDD.run([BDD.step("I remember {int}", &remember), BDD.step("the count is {int}", &check_count)])
end
Feature: Memory

  Scenario: Remember a number
    When I remember 3
    Then the count is 3
  • A step gets the scenario’s context ctx and returns it; the next step gets what the previous one returned. Most BDD calls take and return the context: keep the one returned. BDD.doc_string, BDD.table, BDD.get, BDD.call and BDD.read_line return their value instead. Every Then step must make at least one BDD check.
  • Each --jobs slot runs its scenarios one after another in one worker process, each scenario in a task of its own: its values (BDD.set, BDD.keep), its program under test, the handles and tasks it opened, its seed and Task.pending_count are its own. A crash, System.exit or an endless loop fails only its scenario, and the next scenario gets a new worker; so does any scenario that fails or is skipped. A step must finish the tasks it starts (RUNTIME-BDD-PENDING-TASKS); an unawaited crash fails it with RUNTIME-TASK-CRASHED.
  • A worker runs main once, so state that main makes before BDD.run and the steps capture (an ATS table, a Channel) is shared by the scenarios that worker runs, and so is process state that a step sets, such as Logger.set_level. Make such state in a step (a Background) to have it fresh in each scenario, or run with --isolate, which gives each scenario a process of its own and runs main again for each, as before 0.32. main should only build the list and call BDD.run. Run alone, a step program prints how to run it and exits with code 2.
  • A step may import files and use Process.spawn. Its standard input is empty (IO.gets gives the end-of-input error). It must not call Process.parent_read_line or Process.parent_send: they belong to the runner.

BDD.step and BDD.run #

BDD.step(pattern: String, handler: Fn[BDD.Context, List[String], BDD.Context]) -> BDD.Step
BDD.run(steps: List[BDD.Step]) -> Void

The handler gets the context and the {string} and {int} captures as strings, in order. A block works too and sees the variables around it:

import BDD from "std/bdd"

def main() -> Void
  answer = "42"
  BDD.run([BDD.step("the answer is {int}") do |ctx, args| BDD.assert(ctx, args[0] == answer, "wrong answer") end])
end

BDD.run puts the built-in steps in front of the list, runs what agentica test asks for, and never returns.

BDD.assert #

BDD.assert(ctx: BDD.Context, condition: Bool, message: String) -> BDD.Context

Fails the scenario with the message when the condition is false.

BDD.skip #

BDD.skip(ctx: BDD.Context, reason: String) -> BDD.Context

Ends the step and its scenario as skipped, with the reason in the report: the later steps do not run, and the run still passes. Use it when a scenario needs something the machine lacks.

import BDD from "std/bdd"

def need_postgres(ctx: BDD.Context, args: List[String]) -> BDD.Context
  return ctx if System.env("POSTGRES_URL").ok?

  BDD.skip(ctx, "POSTGRES_URL is not set")
end

def main() -> Void
  BDD.run([BDD.step("a PostgreSQL server", &need_postgres)])
end

The report shows SKIP, the summary counts skipped, and --json gives "status": "skipped" and the reason. Combine it with a tag, such as @postgres, and --tags "not @postgres" leaves those scenarios out altogether.

BDD.set and BDD.get #

BDD.set(ctx: BDD.Context, name: String, value: JSON.Value) -> BDD.Context
BDD.get(ctx: BDD.Context, name: String) -> Result[JSON.Value]

Keep JSON values between the steps of one scenario (each starts empty). BDD.get gives an error for a missing name.

BDD.keep and BDD.call #

BDD.keep(ctx: BDD.Context, name: String, service: Fn[JSON.Value, Result[JSON.Value]]) -> BDD.Context
BDD.call(ctx: BDD.Context, name: String, request: JSON.Value) -> Result[JSON.Value]

Keep a value that is not JSON between the steps of one scenario, such as an open connection. The service is a function from a request to a result, both JSON, and it captures the value. BDD.call runs the service kept under name, or gives an error when there is none. Each scenario starts with no services. Keeping a name again replaces the service, and the old handle is not closed until the scenario’s process ends (use BDD.after to close it sooner). A step cannot leave a task running, so the service does its work in the calling step.

import BDD from "std/bdd"

def open_pair(ctx: BDD.Context, args: List[String]) -> BDD.Context
  listener = TCP.listen("127.0.0.1", 0).unwrap!
  client = TCP.connect("127.0.0.1", TCP.local_port(listener), 5000).unwrap!
  server = TCP.accept(listener, 5000).unwrap!
  BDD.keep(ctx, "pair") do |request|
    try TCP.send(client, JSON.encode(request) + "\n", 5000)
    JSON.parse(try TCP.read_line(server, 5000))
  end
end

def check_echo(ctx: BDD.Context, args: List[String]) -> BDD.Context
  reply = BDD.call(ctx, "pair", JSON.from(args[0])).unwrap!
  BDD.assert(ctx, JSON.string(reply).unwrap! == args[0], "wrong reply")
end

def main() -> Void
  BDD.run([BDD.step("a connected pair", &open_pair), BDD.step("the pair echoes {string}", &check_echo)])
end

A peer program #

BDD.given_program starts one program, and every step that sends, reads or checks talks to it. A second named program would need a name in each of those steps, and a peer needs its own steps anyway (to learn its port, or to read what it received). So a step starts the peer itself with Process.spawn_captured and keeps the handle with BDD.keep. A program started by a step belongs to the scenario and stops when it ends, so no cleanup is needed. Start the peer first, then pass what the program needs from it with BDD.given_env or BDD.given_argument:

import BDD from "std/bdd"

# ./peer prints the port it listens on, then one line for each message it receives.
def start_peer(ctx: BDD.Context, args: List[String]) -> BDD.Context
  peer = Process.spawn_captured("./peer", [], Process.Options(timeout_ms: 5000)).unwrap!
  port = Process.read_line(peer, 5000).unwrap!
  kept = BDD.keep(ctx, "peer") do |_request| ok(JSON.from(Process.read_line(peer, 5000).unwrap_or(""))) end
  BDD.given_env(kept, "PEER_PORT", port)
end

def check_received(ctx: BDD.Context, args: List[String]) -> BDD.Context
  line = JSON.string(BDD.call(ctx, "peer", JSON.null()).unwrap!).unwrap!
  BDD.assert(ctx, line == args[0], "the peer received #{line.inspect}")
end

def main() -> Void
  BDD.run([BDD.step("a peer", &start_peer), BDD.step("the peer received {string}", &check_received)])
end
Scenario: The program talks to the peer
  Given a peer
  And the Agentica program "../app.agc"
  When I send the line "hello"
  Then the peer received "hello"

The peer’s handle lives in the closure of the service, so a later step reads from it through BDD.call, and the read has a timeout so a silent peer fails the step. A read that times out kills the peer, as it does a program under test (see Process), so make it the last use. Start the peer in the step handler itself: a program belongs to the task that started it, so one started in a Task.spawn stops when that task ends, and a step may not leave a task running.

BDD.after #

BDD.after(ctx: BDD.Context, hook: Fn[BDD.Context, Result[Void]]) -> BDD.Context

Registers a cleanup that runs when the scenario ends, whether it passed, failed or was skipped. A step registers it right after it makes what needs cleaning, so the hook exists only when there is something to clean.

  • Hooks run in the reverse order of registration, in the process of the scenario. A hook gets the context the scenario ended with, and returns ok() or error(message). Use that context for any BDD call in the hook.
  • A hook that returns an error, crashes or fails a BDD check is a failed hook, and the other hooks still run. That holds for a check on any context, also one the hook captured when it was registered. A failed hook fails a scenario that passed, with the code RUNTIME-BDD-AFTER at the scenario line. For a scenario that failed, the failure stays the one reported and the message of the hook is added to it.
  • Hooks do not run when the process of the scenario itself ends: a step that crashes (an .unwrap! on an error, a division by zero), a step or hook that calls System.exit, or the 30 second limit. A scenario that fails by a crash still leaves what it made, so keep a cleanup of stale folders for that case, or make steps return error and check it with BDD.assert instead of .unwrap!.
  • Hooks run before the program under test is stopped, and no BDD call stops it first. On Windows a folder the program still holds open may not be removable by a hook.
  • A hook is kept in the context that BDD.after returns. A step that registers a hook and then fails on an older context (BDD.assert(ctx, ...) with the ctx from before) does not run it: pass on the context that BDD.after returned.
  • A hook cannot call BDD.skip. A hook that fails after BDD.skip turns the skipped scenario into a failed one (RUNTIME-BDD-AFTER). A hook that never ends is reported when the 30 second limit of the scenario passes.
import BDD from "std/bdd"

def remove(path: String) -> Result[Void] = Path.remove(path, 5000)

def make_folder(ctx: BDD.Context, args: List[String]) -> BDD.Context
  path = args[0]
  Dir.mkdir(path, 5000).unwrap!
  BDD.after(ctx) do |_| remove(path) end
end

def main() -> Void
  BDD.run([BDD.step("a folder {string} for this scenario", &make_folder)])
end

BDD.doc_string and BDD.table #

BDD.doc_string(ctx: BDD.Context) -> String
BDD.table(ctx: BDD.Context) -> List[List[String]]

Read the Doc String or the Data Table under the current step.

import BDD from "std/bdd"

def check_rows(ctx: BDD.Context, args: List[String]) -> BDD.Context
  rows = BDD.table(ctx)
  BDD.assert(ctx, rows.size == 3, "expected a header and two rows")
end

def send_body(ctx: BDD.Context, args: List[String]) -> BDD.Context = BDD.when_line(ctx, BDD.doc_string(ctx))

def main() -> Void
  BDD.run([BDD.step("the table has two rows:", &check_rows), BDD.step("I send the record:", &send_body)])
end

Driving the program #

BDD.given_env(ctx: BDD.Context, name: String, value: String) -> BDD.Context
BDD.given_argument(ctx: BDD.Context, value: String) -> BDD.Context
BDD.given_program(ctx: BDD.Context, path: String) -> BDD.Context
BDD.when_line(ctx: BDD.Context, text: String) -> BDD.Context
BDD.when_input(ctx: BDD.Context, text: String) -> BDD.Context
BDD.when_eof(ctx: BDD.Context) -> BDD.Context
BDD.read_line(ctx: BDD.Context, timeout_ms: Int) -> Result[String]
BDD.then_line(ctx: BDD.Context, expected: String, timeout_ms: Int) -> BDD.Context
BDD.then_json_line(ctx: BDD.Context, expected: String, timeout_ms: Int) -> BDD.Context
BDD.then_stdin_open(ctx: BDD.Context) -> BDD.Context
BDD.then_exit(ctx: BDD.Context, code: Int, timeout_ms: Int) -> BDD.Context
BDD.then_stdout(ctx: BDD.Context, expected: String) -> BDD.Context
BDD.then_stderr(ctx: BDD.Context, expected: String) -> BDD.Context
BDD.then_stdout_includes(ctx: BDD.Context, expected: String) -> BDD.Context
BDD.then_stderr_includes(ctx: BDD.Context, expected: String) -> BDD.Context

The built-in steps as functions, to combine in your own steps. Pass on the context BDD.given_program returns: it holds the program. BDD.given_program compiles a program once per run, the first time a scenario starts it, and later scenarios start that bytecode while the file’s text and the variables that place packages and the C compiler (AGENTICA_PATH, HOME, CC and the like, from BDD.given_env) are the same: a change during the run to a file it imports is not seen. A program that does not compile starts through agentica FILE.agc, which prints its diagnostic. BDD.given_env and BDD.given_argument come before BDD.given_program and return the context to pass to it.

BDD.read_line gives the next output line instead of comparing it, so a step can use it. It is an error when the output ended or no line came within timeout_ms; a timeout stops the program under test, as for BDD.then_line: every later read, send or exit check fails with RUNTIME-BDD-IO, so make it the last use of the program.

stdout is: compares the output that no line step has read yet. stdout includes: and stderr includes: check the same text for a substring, as s.include? does (an empty text is RUNTIME-BDD-CONTEXT), so a log scenario can assert one line of a noisy stderr; a failure shows the whole stream as actual. All four need the program to have exited. Output is kept up to 1 MiB per stream.

A failed then_* check reports expected and actual in the inspect form, as Test.eq does: a line or output in quotes, an exit code as a number, a JSON line as its sorted-key text. An undefined or ambiguous step’s actual is the step text in quotes. Text over 64 Ki characters is cut and ends with ... and the count left out. A front-end error of the step program keeps the raw type names (Int, String), so its --json form depends on the code.

Running tests from a program #

No function runs features from a program. Start agentica test --json with Process.spawn and read its report.