native-gen
native-gen reads the extern module block of an .agc file and writes a C header for it. The C author writes plain functions with C types; the header has their prototypes, and the wrappers that check the arguments and the function and type tables of the module. The .agc is the one source: every signature and shape comes from std/analysis, the same text that the VM compares with the tables when it loads the module.
Install #
Needs Agentica 0.31.0 or later. Add it with the prefixed tag of the release, in a project with an agentica.toml:
agentica add native-gen https://github.com/agentica-lang/packages --tag native-gen-0.4.0 --subdir native-genThat writes this line of agentica.toml, and the entry of agentica.lock:
[dependencies]
native-gen = { git = "https://github.com/agentica-lang/packages", tag = "native-gen-0.4.0", subdir = "native-gen" }The releases and their notes are in the changelog.
Example #
A block of stats.agc:
struct Summary
count: Int
min: Float
max: Float
mean: Float
end
enum Cell
Empty
Number(Float)
Text(String)
end
extern module "stats"
## The count, smallest, largest and mean of the values; none for no values.
def summary(values: List[Float]) -> Optional[Summary]
def rows(text: String) -> Result[List[Map[String, Cell]]]
endMake the header, and make it again after each change to the block:
agentica deps/native-gen/cli.agc stats.agc --output native/stats_native.hThe C file defines STATS_NATIVE_IMPLEMENTATION before it includes the header, writes one function per prototype, and ends with STATS_NATIVE_MODULE(NULL):
#define STATS_NATIVE_IMPLEMENTATION
#include "stats_native.h"
an_term stats_summary(an_env *env, stats_list_Float values) {
double x;
int more;
if (values.len == 0) return an_none(env);
while ((more = stats_list_Float_next(env, &values, &x)) == 1) {
/* ... */
}
if (more < 0) return an_void(env); /* a bad item: the call has failed */
stats_Summary summary = {count, min, max, total / count};
return stats_optional_Summary_some(env, summary);
}
an_term stats_rows(an_env *env, ng_str text) {
/* text.data, text.len; a cell is stats_Cell_Empty(env), stats_Cell_Number(env, x) or stats_Cell_Text(env, s);
the rows end with stats_result_list_map_String_Cell_ok(env, stats_list_map_String_Cell_end(env, &b)), b being
the builder of the rows, or with an_error(env, message, len) */
}
STATS_NATIVE_MODULE(NULL)A function you did not write is a link error, not a failure when the module loads. Other .c files of the module include the header without the define and get the prototypes and constants only; the implementation part has its own guard, so the one file may include the header before the define too.
The header #
- The prefix of the C names is the library name of the block up to its first
.(stats), or--prefix NAME. Functions use it in lower case, macros and constants in capitals.an,ng,agenticaand a prefix that starts withan_orng_belong to the C API or to native-gen and are refused. - One prototype per function,
an_term PREFIX_name(an_env *env, ...), with the##doc above it.Intisint64_t,Floatdouble,Boolbool,Stringng_strandBytesng_buf(dataandlen, borrowed for the call: copy them to keep them). A struct or enum of the program is its C value, aListorMapa cursor and anOptionalorResulta struct (below). A resource is its state,struct PREFIX_Name_state *(below). The result is always anan_term. - Each wrapper checks the arguments in order and fails the call at the first one that does not decode: a scalar or a resource with
an_badarg(env, i), a struct, enum,OptionalorResultwith the path of its bad part (argument 1: Shape.Circle.0.x: expected Int). AListorMapargument is only opened (one that is not fails asargument 1: List[Float]: expected List[Float]): its items are read by_next. defisAN_SHORTandcompute defisAN_DIRTY.PREFIX_TYPE_Nameis the row of a struct or enum, theTYPEofan_structandan_enum;PREFIX_Name_memberis a field or variant, in declared order.PREFIX_RESOURCE_Nameis the row of a resource.- A struct is
typedef struct {...} PREFIX_Name, a field by its name (a C keyword gets a trailing_). An enum is atagof typePREFIX_Name_tag(its values arePREFIX_Name_Variant, so aswitchon it is checked by-Wswitch) and a unionaswith one struct per variant that has a payload,_0,_1, ... in order:shape.as.Circle._1. A field or payload of another struct or enum of the block is embedded by value, unless that type leads back to its owner (a recursive enum), where it is anan_term. A variant with a payload cannot be named after a C macro (NULL,EOF, ...). - For a type that a parameter reaches,
bool PREFIX_Name_get(env, term, &out)reads a term into its C value. On false the call has already failed, with the path of the bad part; a function whose signature does not reach the type reads none (Point: expected Point). For a type that a result reaches,PREFIX_Name_make(env, &value)makes the term, and an enum has one constructor per variant,PREFIX_Shape_Circle(env, center, radius). A String or Bytes in a C value is borrowed and a term lasts for the call: keep C values in locals. - A
ListorMapthat a parameter reaches is a cursor,PREFIX_list_FloatorPREFIX_map_String_Point(Map[String, List[Int]]isPREFIX_map_String_list_Int), withlen.PREFIX_list_Float_next(env, &values, &x)reads the next item (_next(env, &map, &key, &value)for a Map) and gives 1 for an item, 0 at the end, and -1 when the item does not decode: the call has then failed with the path of the item (argument 1: List[Int].1: expected Int,argument 1: Map[String, Point].0.value.x: expected Int), and every later_nextgives -1 too. Nothing is copied to an array. An item that is aListorMapis its own cursor, and its failures name the path from it (argument 1: List[Float].1); one that is anOptionalorResultis its struct (_nextfills aPREFIX_optional_String). Theargument N:comes from the argument the wrapper opened: the cursor of an argument, and every cursor under it (an item, or thevalueof anOptionalorResult), keep that N in their fieldargument.PREFIX_list_Float_open(env, term, &cursor)opens one held as a term, such as a field of a struct; its failures have noargument N:(List[Int].1: expected Int). Open cursors with_openonly, never by hand. - A
ListorMapthat a result reaches has a builder:PREFIX_list_Float_builder b,_begin(env, &b, hint)(hintthe expected item count, 0 when unknown),_push(env, &b, x)and_end(env, &b), which gives the term; a Map has_begin(env, &b),_put(env, &b, key, value)and_end. An item is the C value of its type, and aList,Map,OptionalorResultitem is the term its own builder or constructor made. - An
Optionalthat a parameter reaches isPREFIX_optional_Point(some,value) and aResultisPREFIX_result_Int(ok,value, anderror, its message as anng_strborrowed for the call).valueis zero when there is none.PREFIX_optional_Point_get(env, term, &out)reads one held as a term. One that a result reaches has a constructor,PREFIX_optional_Point_some(env, point)orPREFIX_result_Int_ok(env, n);an_none(env)andan_error(env, message, len)make the others. - Cursors, builders and the structs of
OptionalandResulthold terms: keep them in C locals, for the call only. - A resource needs
struct PREFIX_Name_stateandPREFIX_Name_FREE(avoid (*)(void *state), orNULL), both defined beforePREFIX_NATIVE_MODULE. C11 has no empty struct: a resource with no state still needs one member.struct PREFIX_Name_state *PREFIX_Name_new(env, &out)makes one, setsoutto its term (to return, or to push in a builder) and gives its zeroed state; in a function whose signature does not reach the resource it gives NULL, setsoutto Void and fails the call. The VM callsPREFIX_Name_FREEonce, when the resource is closed, its task ends or the run ends; it gets only the state and may run on another VM thread while other calls run, so what it shares with them must be thread-safe. The state holds C data only, never a term.PREFIX_Name_get(env, term)gives the state of a resource that is an argument of the call, and NULL for any other term, one made by_newin the same call included. A state pointer is valid until the call returns: do not keep it. A wrong or closed resource never reaches the wrapper: the checker (CHECK-NATIVE-TYPE) and the VM (RUNTIME-NATIVE-CLOSED) stop it first, so itsan_badargis a guard only. - Names:
empty?isPREFIX_empty_pandsave!isPREFIX_save_bang. A parameter that is a C or C++ keyword,env, a type or macro a prototype may use (int64_t,size_t,uint8_t,uint32_t,errno), or that starts withan_orng_, gets a trailing_(int_). The header compiles as C11 and as C++. - The header starts with a check of the version of
agentica_native.h: it stops the build when the API is older than the block needs.
A new module #
new starts a module in a package (the command is cli.agc new, native-gen new in its errors):
agentica deps/native-gen/cli.agc new demo
agentica deps/native-gen/cli.agc new path/to/pkg/demoIt writes, in the folder of the path (the current one for a bare name):
demo.agc, with anextern module "demo"block of one example function,add(a: Int, b: Int) -> Int, and amainthat calls it;native/demo.c, with one body per function that fails the call withnot written yetuntil you write it, andDEMO_NATIVE_MODULE(NULL)at its end;native/demo_native.h, the header made from the new.agc;- the
[native.demo]entry ofagentica.toml(build = { sources = ["native/demo.c"] }), added at the end of the file, or a newagentica.tomlwithname = "demo". A comment above the entry gives the two commands that make and check the header, run from that folder.
The package builds and loads as it is: running demo.agc fails at the call with add: not written yet. Replace the example with your functions, then make the header again.
The name is the [native.NAME] of agentica.toml, the library of the block and the start of the C names: a lowercase letter, then lowercase letters, digits and _, at most 64 characters, not a device name of Windows (con, nul, com1, ...), and not a prefix of the C API (an, ng, agentica, or one that starts with an_ or ng_). Pass NAME or FOLDER/NAME, without .agc.
With --lang rust, new starts a Rust module instead:
agentica deps/native-gen/cli.agc new --lang rust demo
agentica deps/native-gen/cli.agc build demo.agcIt writes demo.agc (the same block and main) and the crate rust/, laid out as the example below: Cargo.toml, .gitignore (target/ and Cargo.lock), src/lib.rs with one method per function that returns Err(Raise::new("not written yet")), and src/demo_native.rs, the Rust file made from the new .agc. The crate goes to the package root, the nearest folder up with an agentica.toml, as for build, so new --lang rust src/demo in a package writes src/demo.agc and the root's rust/. With no agentica.toml up the tree, the folder of the path gets one with name = "demo"; one that is there is never changed. A package has one Rust module, since its crate builds one library: a second new --lang rust in it is NATIVE-GEN-EXISTS. It adds no [native.demo] entry: build adds the one of the system, with the library it built. The name also cannot be a Rust keyword or the name of a Rust crate (fn, test, std, ...), end with _ or hold __. After build, running demo.agc fails at the call with add: not written yet, as for C.
new writes only new files. It writes nothing when demo.agc, native/demo.c or the header (for Rust, a file of rust/ that it writes) is there, or when agentica.toml has native.demo already (NATIVE-GEN-EXISTS), or when agentica.toml with the entry added is not TOML (NATIVE-GEN-MANIFEST). Every text is made before the first write; only a write that fails midway (NATIVE-GEN-IO) leaves the files written before it.
A Rust module #
A module can be written in Rust instead, on the agentica-native crate that native-gen carries in deps/native-gen/native/agentica-native. The crate of the module is the folder rust/ at the package root (the nearest folder up from the .agc with an agentica.toml), and depends on that crate by path. The package examples/rust_math is a whole one; CI builds it and runs it, and checks that the files below are its files.
It needs cargo, and the install above, which puts the crate in deps/native-gen/.
agentica.toml, before the build:
# A native module in Rust (the native-gen docs show it). `agentica deps/native-gen/cli.agc build math.agc` builds it
# and adds the [native.math] entry of the system below; rust/build-check.sh does that on a copy in CI.
name = "native-gen-rust-math"math.agc:
## A native module written in Rust: `rust/src/lib.rs` implements the block on the file that native-gen makes from it,
## and `cli.agc build math.agc` builds it into native/ and adds its entry to agentica.toml.
struct Point
x: Float
y: Float
end
extern module "math"
## The sum of a and b.
def add(a: Int, b: Int) -> Int
## The mean of the values; none for no values.
def mean(values: List[Float]) -> Optional[Float]
## x kept between low and high.
def clamp(x: Float, low: Float, high: Float) -> Float
## The point halfway between a and b.
def midpoint(a: Point, b: Point) -> Point
end
def main() -> Void
puts(add(2, 3))
puts(mean([1.0, 2.0, 6.0]))
puts(mean([]))
puts(clamp(12.5, 0.0, 10.0))
puts(midpoint(Point(x: 0.0, y: 0.0), Point(x: 4.0, y: 2.0)))
endrust/Cargo.toml:
[package]
name = "math"
version = "0.1.0"
edition = "2024"
rust-version = "1.90"
publish = false
[lib]
crate-type = ["cdylib"]
[dependencies]
agentica-native = { path = "../deps/native-gen/native/agentica-native" }rust/src/lib.rs:
//! The Rust side of math.agc, on the file `src/math_native.rs` that native-gen makes from its block.
#[rustfmt::skip]
mod math_native;
use agentica_native::{Env, Raise};
use crate::math_native::{MathNative, Point, module};
struct Math;
impl MathNative for Math {
fn add(_env: Env<'_>, a: i64, b: i64) -> Result<i64, Raise> {
a.checked_add(b).ok_or_else(|| Raise::new("the sum is out of range"))
}
fn mean(_env: Env<'_>, values: Vec<f64>) -> Result<Option<f64>, Raise> {
if values.is_empty() {
return Ok(None);
}
Ok(Some(values.iter().sum::<f64>() / values.len() as f64))
}
fn clamp(_env: Env<'_>, x: f64, low: f64, high: f64) -> Result<f64, Raise> {
if low.is_nan() || high.is_nan() || low > high {
return Err(Raise::new("low is above high, or not a number"));
}
Ok(x.clamp(low, high))
}
fn midpoint(_env: Env<'_>, a: Point, b: Point) -> Result<Point, Raise> {
Ok(Point { x: (a.x + b.x) / 2.0, y: (a.y + b.y) / 2.0 })
}
}
agentica_native::export!(module::<Math>());rust/src/math_native.rs is the file that build writes from the block: the struct Point, the trait MathNative with one method per function, and module. Build the module, then run the program, from the package root:
agentica deps/native-gen/cli.agc build math.agc
agentica math.agcThe output of build on Linux (cargo prints its progress on stderr):
wrote rust/src/math_native.rs
copied rust/target/release/libmath.so to native/libmath.so
added native.math.linux to agentica.tomlWith a cargo workspace or CARGO_TARGET_DIR, the library that cargo built is at another path, which the copied line names.
The entry that build adds to agentica.toml:
[native.math]
linux = { file = "native/libmath.so" }The output of agentica math.agc:
5
some(3.0)
none
10.0
Point(x: 2.0, y: 1.0)A native module cannot call back into Agentica: its functions get values and return values, and never call back into Agentica; a def must not wait (use a compute def). Return the data, or a resource, and loop in Agentica. A plain extern block can pass a function of the program to C as a function pointer, with a callback line.
A def that calls a library that switches to a stack of its own (a coroutine, as the non-blocking calls of MariaDB Connector/C run), or that blocks for a while inside one, runs that part in env.blocking(|| ...), which gives the closure's value: the collector leaves the thread alone meanwhile. The closure may use Rust memory and read the &str and &[u8] borrowed from the call's arguments, never env or a term (its Send bound keeps them out). It runs on the calling thread, so it still stops that scheduler: keep it short. A plain wait goes in a compute def, which runs on the compute pool and needs no blocking. It is an_blocking of Native API 1.3, so a module on the crate needs Agentica 0.30.0 or later (0.29.0 has API 1.3, but its an_blocking could lose a value on macOS x86_64).
What build does, with paths from the package root:
- It writes
rust/src/NAME_native.rs(NAME is the library of the block,math) when it changed: the types of the block with their codecs, the traitMathNativethatsrc/lib.rsimplements, andmodule::<Impl>()foragentica_native::export!. Declare it insrc/lib.rswith#[rustfmt::skip] mod math_native;. - It runs
cargo build --releaseon the crate (from the current folder, so a.cargo/config.tomlis found from there, not fromrust/), and copies the library tonative/libmath.soon Linux,native/libmath.dylibon macOS ornative/math.dllon Windows. - It adds the entry of the system to
agentica.toml,linux = { file = "native/libmath.so" }under[native.math](or a new[native.math]at the end). An entry that is there already is kept, and the library goes to itsfile, which must be innative/.agentica.tomlis read again after cargo, so an edit made meanwhile is kept. - It prints one line per step: what it wrote, or that the file is current. A second run with nothing changed writes nothing.
A package that others install does not build its Rust module: commit the library in native/ for each system, and the entry of each, as you build them on that system. build does not cross-build: the library cargo makes is copied as the entry of the system it runs on. Commit rust/src/NAME_native.rs too, and check in CI that it is current; build --check runs no cargo:
agentica deps/native-gen/cli.agc build math.agc --check--check covers only the generated Rust file, not the library or the manifest entry.
build needs cargo on PATH and an agentica.toml at the package root, and the library of the block must be a module name (the rule of new). The block is read without the [native.NAME] entries of the manifest, so a library that is not built yet does not stop it.
Errors #
An error is one line on stderr, native-gen: [CODE] message, and the exit code is 1. Bad usage prints the usage error and a Try 'native-gen --help' line (Try 'native-gen new --help' for new, which also takes a missing or an extra argument as bad usage), and the exit code is 2.
| Code | When |
|---|---|
NATIVE-GEN-NAME | two names map to one C identifier (empty? and empty_p), or a variant with a payload is named after a C macro |
NATIVE-GEN-BLOCK | the file has no extern module block, or more than one; for build, its library is not a module name |
NATIVE-GEN-PREFIX | the prefix is not a C identifier, or is the C API's; give another with --prefix. For new, the name breaks the rule above, or is the C API's (an, ng, agentica, or starts with an_ or ng_); for new --lang rust, also a Rust keyword, the name of a Rust crate (fn, test, std, agentica_native), or a name that ends with _ or holds __ |
NATIVE-GEN-UNKNOWN | std/analysis gives a function mode or a type kind that this version does not know |
NATIVE-GEN-STALE | --check: the file at --output is not the header that the .agc gives; build --check: rust/src/NAME_native.rs is not the one it gives |
NATIVE-GEN-EXISTS | new: a file it would write is there, or agentica.toml has native.NAME already |
NATIVE-GEN-MANIFEST | new: agentica.toml, with the entry added, is not TOML. build: [native.NAME] has a build table, its entry for the system has no file or one outside native/, or the entry cannot be added (no [native.NAME] line to add it under, a native table written inline) |
NATIVE-GEN-OS | build: a system other than linux, macos and windows |
NATIVE-GEN-CARGO | build: cargo is not on PATH, fails, or builds no single cdylib |
NATIVE-GEN-IO | a file cannot be read or written; for build, also no rust/Cargo.toml, or no agentica.toml up from the file |
| any other | the code of the diagnostic when the .agc does not check |
Check in CI #
Commit the header, and fail the build when it is stale:
agentica deps/native-gen/cli.agc stats.agc --output native/stats_native.h --checkFrom a program, NativeGen.header(path, prefix = "") from import NativeGen from "pkg/native-gen/native_gen" returns the text of the header.