Code analysis (std)
std/analysis answers, inside a program, the questions that the command line answers: import Analysis from "std/analysis". The answers are plain structs. The *_json functions give the same bytes as the --json output of agentica outline, type-at, symbol and refs (Command line), and Analysis.diagnostic_json gives what --json check prints on stderr.
Analysis.workspace() -> Analysis.Workspace
Analysis.open(files: Analysis.Workspace) -> Analysis.Session
Analysis.update(session: Analysis.Session, path: String, text: String) -> Analysis.Session
Analysis.close(session: Analysis.Session, path: String) -> Analysis.Session
Analysis.refresh(session: Analysis.Session, path: String) -> Analysis.Session
Analysis.entry(session: Analysis.Session, path: String) -> Result[Analysis.Entry]
Analysis.adopt(session: Analysis.Session, entry: Analysis.Entry) -> Analysis.Session
Analysis.check(session: Analysis.Session, path: String) -> Analysis.Checked
Analysis.outline(session: Analysis.Session, path: String) -> Result[Analysis.Outline]
Analysis.type_at(session: Analysis.Session, path: String, line: Int, column: Int) -> Result[Analysis.TypeAt]
Analysis.type_at_partial(session: Analysis.Session, path: String, line: Int, column: Int) -> Result[Analysis.TypeAt]
Analysis.symbol(session: Analysis.Session, name: String, paths: List[String]) -> Result[Analysis.Symbols]
Analysis.refs(session: Analysis.Session, path: String, line: Int, column: Int) -> Result[Analysis.Refs]
Analysis.refs_partial(session: Analysis.Session, path: String, line: Int, column: Int) -> Result[Analysis.Refs]
Analysis.native_modules(session: Analysis.Session, path: String) -> Result[Analysis.NativeModules]
Analysis.bodies(session: Analysis.Session, path: String) -> Result[Analysis.Bodies]
Analysis.diagnostic(message: String) -> Result[Analysis.Diagnostic]
Analysis.outline_json(found: Analysis.Outline) -> JSON.Value
Analysis.type_at_json(found: Analysis.TypeAt) -> JSON.Value
Analysis.symbols_json(found: Analysis.Symbols) -> JSON.Value
Analysis.refs_json(found: Analysis.Refs) -> JSON.Value
Analysis.native_modules_json(found: Analysis.NativeModules) -> JSON.Value
Analysis.bodies_json(found: Analysis.Bodies) -> JSON.Value
Analysis.diagnostic_json(found: Analysis.Diagnostic) -> JSON.ValueSessions and buffers #
A Session reads files from the disk under the working folder, except files that have a buffer: text an editor has open and has not saved. Analysis.update(session, path, text) returns a session where path has text, whether or not the disk has the file. Analysis.close removes the buffer. Two spellings of one file are one buffer. Create the folder of a new file before update, and close it with the spelling you gave. A buffer answers for its own file and for every file that imports it.
Analysis.open takes a Workspace: Analysis.workspace() gives an empty one, or build one whose buffers map holds paths and texts. Its version must be 1. When two paths name one file, the later one wins.
A session is a value. Tasks may share one and ask it at once; an update in one task leaves the copies of the others unchanged. Its fields are not API. A new session has no cache: each question runs the stages from the start, until refresh fills it (next section).
Refresh and the cache #
Analysis.refresh(session, path) returns a session whose cache holds the linked and checked program of path, and the parsed modules it reads. A loop that keeps one session runs update, then refresh, then the questions, which then skip parsing and checking about path.
check, type_at and refs use the checked program in the cache; symbol links each file it searches and uses only its parsed modules. Each question reads the files again, starting at its own file, and uses the cache only where the modules and their texts are the ones the cache was made from. Buffer edits, close, changes on disk, new and removed files, and a change of agentica.toml that changes what an import means all count. So the answers are the same with or without refresh: a stale cache costs time, never a wrong answer. A refresh checks again only when a module changed, and parses only the modules whose text changed. A module whose text is the same, with every module it imports, is not checked again: the check takes what the last program of path that checked left for it, so an edit of path costs about the file itself, and an edit of an imported module checks that module and those that import it. The answers are those of a check from the start. A file that cannot be read or linked leaves the cache as it was, and the questions say why.
The cache keeps at most 16 checked files, the least recently refreshed leaving first, with the parsed modules they reach. It lives in the value: there is no task, channel or lock, so the caller owns the loop. Run one refresh at a time per session and keep the session it returns. Two refreshes from one session make two caches, and keeping one drops the other’s work, which costs time only. A program that refreshes in a task and calls update on its own copy meanwhile adopts the refreshed session only if no update or close came after the refresh started; otherwise it keeps its own session and refreshes again later.
Sharing the cache between tasks #
A Session holds functions, so an ATS table cannot hold one. An Analysis.Entry is what refresh kept for one file, as plain data. Analysis.entry(session, path) takes it from a refreshed session; it is an error when the cache holds none for path (not refreshed, a file that does not link, or an entry that left the cache). Give entry the same spelling of the path as refresh: a file that only has a buffer is not canonicalized, so another spelling finds nothing. Analysis.adopt(session, entry) returns the session with the cache of that entry alone, and the same buffers. A refresh or a question about that file then takes the program from the entry when the files and buffers are as they were, and else checks again only the modules that changed or import one that did. A stale entry costs time, never a wrong answer. The fields of an Entry are not API.
So tasks that refresh different files lose nothing. Each task refreshes its own session, puts Analysis.entry(refreshed, path) in an ATS[String, Analysis.Entry] under the path, and a task that asks about a file adopts the entry of that file first. Entries of two files are separate caches: the table shares the checked program of each file, not the modules that two files both import. The program decides when an entry leaves the table, for instance when the file is closed.
def refresh_into(entries: ATS[String, Analysis.Entry], session: Analysis.Session, path: String) -> Void
found = Analysis.entry(Analysis.refresh(session, path), path)
if found.ok?
entries.put(path, found.unwrap!)
end
end
def check_from(entries: ATS[String, Analysis.Entry], session: Analysis.Session, path: String) -> Analysis.Checked
found = entries.get(path)
return Analysis.check(Analysis.adopt(session, found.unwrap!), path) if found.ok?
return Analysis.check(session, path)
endAnswers #
The answers are structs of strings, ints, bools and lists, with no enum. Their fields are the keys of the --json forms, with the same names, except that range is a Region with start and stop (JSON end). An absent value is "", or an empty list for a struct: Outline.doc, Item.doc and Import.alias are "" where the JSON has null. TypeAt.definition is empty or holds one Definition. A kind is a string, an open set: a program accepts a kind it does not know.
The types are aliases of the library’s own. A program reads their fields but cannot build them by name: Analysis.Position(line: 3, column: 9) is CHECK-UNKNOWN-TYPE. To get a Diagnostic, for a timeout say, read one from JSON with Analysis.diagnostic.
A file is named as on the command line: by its canonical path, relative to the working folder when it is below it. A diagnostic keeps the root file as the path you gave, as check does, so Checked.file and the file of its diagnostic may be spelled differently.
Analysis.checklinks and checks the file.Checked.passedis true when there is no problem;diagnosticsholds the first one, if any.Analysis.outlinelists what the file declares. The file is only parsed. A module block is anItemof kind"module"whoseitemsare its declarations;itemsis empty for every other kind.Analysis.type_atgives the type and the definition at a position. Line and column count from 1.Analysis.symbolfinds wherenameis declared in the files and folders ofpaths(.when the list is empty). A folder means every*.agcfile below it. The files are linked together, and those that do not link are listed inSymbols.skipped.Symbol.moduleis the file (""for a builtin), andhelpis the object ofagentica help NAME --jsonfor a builtin, JSON null for the others. Every file with a buffer in the session is searched too, first, as--stdin FILEdoes, whether or notpathsnames it, so a query cannot be scoped to a folder while other buffers are open. Buffers come in the bytewise order of their canonical paths, whatever order they were put in, somatchesdoes not depend on the history ofupdatecalls. Folders are listed on the disk: a folder that exists only as buffers (buffernewdir/new.agcwithpaths["newdir"]) is aQUERY-USAGE“cannot read”. Name the buffer’s file, or leavepathsempty.Analysis.refsgives the declaration of the name at the position and every reference to it, asagentica refsdoes. It links and checks the file astype_atdoes, and sees the file and what it imports; a file that only imports it is not seen.Refs.definitionis empty for a builtin, whichRefs.builtinnames (""for anything else). AReferenceis therangeof the name in itsfile.Analysis.type_at_partialandAnalysis.refs_partialaretype_atandrefswith--partial: when some functions do not check, they answer from the others, withpartialtrue and the failures indiagnostics; a declaration,mainor constant that does not check, or a field or builtin forrefs_partial, is still the first problem as an error.TypeAt.typeis""for a name inside a function that did not check, and in such a function a position the link cannot name gives the first problem as the error.
Native modules #
Analysis.native_modules(session, path) describes the extern module "NAME" blocks of a file for a tool that generates code for a native module, such as a C header, without its own parser or type rules. The file is linked and checked as for type_at, and needs no main. It has no command line form; native_modules_json gives the JSON of the answer, with command native-modules.
A NativeModules has the blocks of the file, in source order, two blocks of one library being one. A NativeBlock has:
library, theNAMEas written, and therangeof its declarations, from the first to the last, without theextern moduleline and theend(the syntax tree keeps no place for them). Blocks of one library are one block, so its range may cover the declarations of another block, and two ranges may overlap;functions, each with itsnameas written (empty?), itsmode(shortfor adef,dirtyfor acompute def),parameters(nameand canonicaltype),result,signature,exported,docandrange, andtypes_inandtypes_out, the names of the structs, enums and resources that the parameters and that the result reach, through the fields and payloads too, each once, each list in the order of its own walk, which is not a subsequence of the order of the block’stypes: for a parameterA{b: B},B{c: C}and a resultCthe block lists A, C, B buttypes_inis A, B, C, so do not merge them by position (a type intypes_outonly is made by the module and never read, and one intypes_inonly is read and never made);resources, theresource Namelines, withexported,docandrange;types, the structs and enums that cross, each with itskind(structorenum), its barename, itsmembers(fields or variants, with their canonicaltypes) and itsshape;api_majorandapi_minor, the least version of the C API (AN_API_MAJORandAN_API_MINORofagentica_native.h) that has what the block uses: 1.0 for scalars and collections, 1.1 when a struct or enum crosses, 1.2 for a resource.
A doc is "" in the record when there is none, and null in the JSON. A signature is the text the module’s function table must have, (List[Float]) -> Optional[Summary], and a shape the text its type table must have, count: Int, mean: Float or Empty | Number(Float) | Text(String). They are written by the code that the compiler binds a call with, so they are the bytes the loader compares. A shape is the text after = in the {S<index>=...} or {E<index>=...} that the compiler writes, whose <index> is the type’s place in the program and is not part of the answer.
Function bodies #
Analysis.bodies(session, path) gives the functions of a file with their bodies as a typed tree, for a tool that translates them, such as a prover. It is experimental: until a first tool works with it, its shape may change between releases without a breaking mark, though the answer has a version. The file is linked and checked as for type_at, and needs no main. A std/ name without a buffer, such as std/binary, is that module as the binary carries it, and the answer names it std/binary.agc. It has no command line form; bodies_json gives the JSON of the answer, with command bodies.
A Bodies has the functions of the file that have a body, private ones too, in source order, the prelude, and the modules of the linked program, each a BodyModule with its prefix (the @N:: that the linked names of its declarations start with, "" for the file itself) and its file. A Body has its name as written, its linked name (the one a call targets), its file, exported, its range and name_range, its generics, its params, its result and its body, a list of Node. A parameter has a name, type, range and default: its default value as one node, or an empty list. A parameter that a call fills no argument for takes its default: a function of the file has it in its params, and a builtin, which has no Body, in the parameters of agentica help NAME --json, with their names in order. A type is a TypeShape: kind is builtin (name is Int, Bytes, List, Result, Fn, …), struct, enum or opaque (name is the linked name), var (a type parameter) or none (a node with no value), and args are its type arguments.
A Node has a kind, its type, its range, a text, a target, new, its children (in source order), type_args and prelude:
int,float,string,bool,nil: a literal, with its value intext.local: a read of a name, intext.let:text = value, withnewtrue when it makes a new local and false when it changes one.discardis_ = value.call: thetargetis what the checker resolved: a function by its linked name, or a builtin, such asInt.shift_left,Bytes.lengthfordata.size,Bytes.atfordata[i],okorsome. The children are the arguments, the receiver of a method first, in the order they run, which is the order written: the positional ones fill the parameters from the left, eachname: valueis anamednode (textthe parameter, the value its child), and in a call with named arguments a trailing block is anamednode too (xs.reduce(initial: 0) do ... endgivesxs,namedinitial,namedblock).type_argsare the types a call of a generic function or builtin binds: one per type parameter of the function, in order, or one per type variable of the builtin, in the order they first appear in its parameters, then its result (xs.map do |x| x.to_s endon aList[Int]hasIntandString); a type the call leaves open has the kindnone, and on every other nodetype_argsis an empty list. A builtin written in Agentica (List.sum,Optional.unwrap_or,File.foreach) has inpreludethe linked name of the function that runs it for these types (@prelude::list::sum_int), and""otherwise.indexistarget[index]on aListorMap.op: a binary operator, written intextas in the source (+,<=,==,!=, …); its two children have their types.and,or,not(text!) andneg(unary minus,text-) are kinds of their own.-9223372036854775808, which only fits under its minus, is oneintnode.try,return, andinterpolation, whose children arestringparts and typed expressions.if: abranch(the condition, then ablock) for eachifandelsif, then ablockforelse.case: the value, then anarmeach: its pattern, aguardwhen written, and ablock. Acasethat gives a value has its type, and the last node of each arm’s block is that arm’s value. A pattern is awildcard, aliteralor avariant(targetShape.Square,ok,error,someornone) whose children are its slots:bind,skipor a nested pattern. An arm of several patterns ("ex", "px" ->) has analternativesnode whose children are those patterns.while: the condition, then ablock.for:textis the variable, with its type, then the iterable and ablock.nextandbreakhave no children.list: its elements.map: anentryeach, its key then its value.range:text..or..., then its two ends.struct:targetis the linked struct name, then anamednode for eachname: valueas written (textthe name, the value its child).with: the struct, then itsnamednodes.variantas a value:targetas for a pattern, then its arguments, and anamednode for each named one.field:textis the field, the child its struct.method: a method that every type has and that is not a builtin (to_s,inspectof a struct,T.members),textits name, the receiver then the arguments.function:&name, with the function or builtin intargetandpreludeas for acall.invoke: a call of a function value (f.call(x),textcall, the value then the arguments) or of anFnfield (h.run(x),textthe field, the struct then the arguments).lambda: a block (textblock) or a lambda (textlambda): aparamnode for each parameter (textits name, with its type), then ablock.type: a type used as a value (Point.members),textits name.constant: a read of a module constant,textits linked name.regex:text/body/flags, its child the call that builds it.template: its child the call that builds its text.
A Body is pure when no call it makes, through every function it calls, has an effect or reads state that another task or the system may change. A builtin has an effect when agentica help NAME --json lists one in its effects (stdout, filesystem, network, …), and the reads of shared state have one too: concurrency for a read of an ATS or a Channel and for ch.lazy, filesystem for File.foreach and File.chunks, clock for System.Profile.steps. So +, b[i] and unwrap! are pure: they may halt, but do nothing else. A call of a function of an extern module has the effect native. A call that may run a function value has the effect function-value: f.call(x), a call of an Fn field (h.run(x)), a builtin given a function value that is not a block or &name (xs.map(f)), and a terminal of a Lazy (to_a, first, …), which runs the source and the steps that made it. A method call the checker did not resolve has the effect unresolved. A reference &name counts as a call of name. impure gives each reason once, as a Reason: the effect, the call (a builtin, a linked function name, or call), its range, and through, the linked names of the functions from the first one called to the one that makes the call (empty when the function makes it itself). The call is in the file of the last name’s prefix. Recursion and cycles are followed once.
The prelude of a Bodies has those functions, each once, in the order the functions of the file first call them, with their bodies, pure and impure as for any function. A reference to a builtin (&Int.abs) runs a function that the prelude makes for it, listed too, from the file <prelude>/builtin_refs.agc, which the front end writes. The others have the file <prelude>/NAME.agc, the file std/prelude/NAME.agc that the binary carries, and their generic types are the letters A, B, K and V.
A block runs from its first statement to its last. Some ranges are coarse: a string part of an interpolation has the range of the interpolation, a skip slot that of its variant, and an if or case statement runs to the start of the next line. The type arguments of a call of a generic function are not in the answer yet.
Errors #
outline, type_at, type_at_partial, symbol, refs, refs_partial, native_modules and bodies return a Result whose error message is the diagnostic as one JSON object, the --json form, so try works. Analysis.diagnostic reads it back into an Analysis.Diagnostic. The codes are those of the command line: LEX-, PARSE-, CHECK- and PKG-; QUERY-NO-EXPRESSION for a position on no expression; QUERY-NOT-FOUND for a symbol name that nothing declares; and QUERY-USAGE for a file that cannot be read, a position below line 1 or column 1, or a symbol with an empty name or a path that names no *.agc file. For outline, type_at and refs the diagnostic names the file; for symbol its file is <arguments>, as on the command line. Unlike the command line, QUERY-USAGE has no usage help line; QUERY-NOT-FOUND keeps its help (the files not searched).
import Analysis from "std/analysis"
def main() -> Void
session = Analysis.open(Analysis.workspace())
session = Analysis.update(session, "app.agc", "def twice(n: Int) -> Int\n return n * 2\nend\n")
outline = Analysis.outline(session, "app.agc")
if outline.ok?
for item in outline.unwrap!.items
puts("#{item.kind} #{item.signature}")
end
else
puts(Analysis.diagnostic(outline.message).unwrap!.code)
end
endCost #
The compiler is part of the program that imports std/analysis: its pure core is carried by the binary as the private std/internal/compiler/, and only std/ modules may import that. A program that imports std/analysis therefore checks and lowers the files of stdlib/CORE too when it runs from source, and the checker and the catalog of builtins it asks are the ones of the binary that compiled it, frozen in the compiled program. The binary refuses a .agcb file of another Agentica version, so recompile a tool after upgrading agentica. A program that runs often can pay the compile time once: agentica compile tool.agc -o tool.agcb, then agentica run -- tool.agcb.