Lexical conventions

  • The file is UTF-8; core identifiers use ASCII characters. A source file has no size limit. Bytes that are not valid UTF-8 (a stray continuation byte, an overlong form, a surrogate, a code point above U+10FFFF, a sequence cut short, also at the end of the file) are LEX-INVALID-UTF8, in the file that has them, whether it is the root file or an imported one. The span is one character long and starts at the first byte of the first sequence that does not decode, at the line and column of the characters before it. The text is never repaired with replacement characters.
  • A comment starts with # and ends at the end of the line. The newline after it is kept.
  • A line whose first non-blank characters are ## is a doc line (see “Doc comments” below), not an ordinary comment.
  • Names of variables, functions and fields start with a lowercase letter or _. Type names start with an uppercase letter. Identifiers are case-sensitive.
  • Spaces, tabs and indentation have no structural meaning. CRLF is read as LF.
  • A newline separates statements, except inside (...), [...] and {...}, and before a line that starts with .name (see below).
  • The lexer emits one NL token for each newline outside those delimiters, including blank and comment lines. Inside a block that is itself within (...), [...] or {...}, newlines are emitted too, so a multi-line block can be an argument, and so they are inside the {...} body of a lambda. The parser ignores extra NL tokens between declarations and statements.
  • A -> after a token that can end an expression (a name, a literal, a closing bracket, end) is the arrow of a def result or a case arm; anywhere else it starts a lambda (->(x) { ... }), and the { after it, or after its (...), opens the lambda’s body.
  • If the file does not end with a newline, the lexer adds one before EOF.
  • A line whose first token is a . followed by a lowercase name (.map, .with(; not .Type or a number) continues the expression of the line before: the lexer drops the NL that ends that line, so a method chain can span lines without parentheses. The line before must end in a token that can end an expression (a name, a literal, a closing bracket or end; not a type annotation such as -> Void), and it must not be the head of an if, elsif, while, case or for: a dot on the first line of a body never changes the condition. Comment-only lines (#) between the two lines are skipped, and a trailing comment on the line before is fine. A blank line ends the statement, and so does a ## doc line. .. and ... at the start of a line do not continue (ranges stay as they are), and neither does |>. A keyword after the dot (.end) is rejected as it is on one line. A leading dot that continues nothing (the first line of a block or of an if, while, case or for body, after a blank line, after do |x| or after an operator) is PARSE-UNEXPECTED-TOKEN, at the dot, except after an operator, where it is at the line break.
  • ;, backslash continuation and implicit continuation after an operator are not supported. Wrap a long expression in parentheses. Inside lists and calls, use several lines and a trailing comma. The parameters of a def header, the entries of a map, the names of an import list and the fields of a struct literal take a trailing comma too; the payload types of an enum variant do not.
total = (
  base +
  extra
)

lines = File.foreach("ledger.csv", 65536, TIMEOUT_MS)
  .drop(1)                                     # header
  .map { |line| parse(line) }
  .filter(&big_expense?)

Reserved words of the core, which cannot be used as identifiers:

def struct enum case do import export end if elsif else while for in return try
true false nil and or not

A reserved word where a name goes (a parameter, a function, a field, a local before =, : or +=, a loop variable, a block parameter or a pattern binding) is PARSE-RESERVED-WORD at the word, with another name as the help. After a dot or & (JSON.true, xs.end) it is the same code, and the help says that no function, field or method has that name.

break and next are not reserved words: they can name variables, and as statements they are loop control. with is not reserved either: it means a struct update only after a . and before ( (§5), and it can still name variables, functions and struct fields.

Reserved builtin names cannot be declared as functions, types, variables, import aliases or module names (RESERVED_TYPE_NAMES in compiler/catalog/builtins_data.agc): ATS, Await, Base64, Bool, Bytes, Channel, Codec, Dir, Error, File, Float, Fn, IO, Int, JSON, Lazy, List, Map, Math, Nil, OpenSSL, Optional, Path, Process, Random, Range, Result, Set, String, StructMember, System, TCP, TLS, Task, TaskDown, TaskExit, TaskRef, Test, Time, Void and Zlib. A builtin family reserves only its own name: its members are written Family.Member, such as OpenSSL.Cipher, and its functions Family.Member.fn, such as OpenSSL.Cipher.aes_256_gcm_encrypt (§6, “Minimal builtin functions”), so Cipher or Digest inside a family reserve nothing. LogRegistry and HttpWire, the std-only families of std/logger and std/http, are not reserved, but an enum of either name is CHECK-REDEFINED-NAME: its variants would clash with the family’s records (LogRegistry.Event, HttpWire.Response). A struct or a type alias may take the name. HTTP is not reserved: it is the usual alias of std/http. ok, error, some and none are names, but cannot be declared as functions, variables, parameters or loop variables (CHECK-REDEFINED-NAME); a struct field may have them, since it is read through a value (reply.error). The same holds for puts, print, gets and gets! as variables, parameters, loop variables and pattern bindings.

A trailing ? or ! belongs to a lowercase name, whatever follows it: include? (s, part) is valid. ! is not a suffix before =, so a!=b is a != b. By convention ? marks a predicate that returns Bool, and ! marks the form that halts the run with an error instead of returning a Result. Variables, fields and parameters cannot have the suffix.

The pipe operator |> is one token (the longest match next to | and ||), so block parameters do |x| and || are unchanged.

Doc comments #

A doc line is a line whose first non-blank characters are ##. Its text is the rest of the line after ## and one optional space. ## alone is an empty doc line, which separates paragraphs. Only the first two #s are the marker: ### Heading is a doc line whose text is # Heading. A ## after code on the same line (x = 1 ## note) is an ordinary comment, and ## inside a string literal or an interpolation is text.

A doc block is one or more consecutive doc lines. A blank line or an ordinary # comment line ends the block. The lexer emits one DOC token for each block, with its lines joined by newlines as the text and the span of the whole block. The newline after the block is an NL, as after any other line. Inside (...), [...] and {...}, where no NL is emitted, the DOC token is still emitted, and the parser rejects it (PARSE-DETACHED-DOC).

DOC_LINE   = { " " | TAB }, "##", [ " " ], { ANY_BUT_NEWLINE } ;
DOC        = DOC_LINE, { NL, DOC_LINE } ;

TAB is the tab character, and ANY_BUT_NEWLINE is any character except a line feed or carriage return.

The text is CommonMark. Section 5 says where a doc block may stand and what it documents.

Literals #

NAME       = LOWER_OR_UNDERSCORE, { LETTER | DIGIT | "_" }, [ "?" | "!" ] ;
TYPE_NAME  = UPPER, { LETTER | DIGIT | "_" } ;
INTEGER    = DIGIT, { [ "_" ], DIGIT }
           | "0", ( "x" | "X" ), HEX_DIGIT, { [ "_" ], HEX_DIGIT }
           | "0", ( "b" | "B" ), BIN_DIGIT, { [ "_" ], BIN_DIGIT }
           | "0", ( "o" | "O" ), OCT_DIGIT, { [ "_" ], OCT_DIGIT } ;
FLOAT      = DIGIT, { [ "_" ], DIGIT }, [ ".", DIGIT, { [ "_" ], DIGIT } ], EXPONENT
           | DIGIT, { [ "_" ], DIGIT }, ".", DIGIT, { [ "_" ], DIGIT } ;
EXPONENT   = ( "e" | "E" ), [ "+" | "-" ], DIGIT, { [ "_" ], DIGIT } ;
STRING     = '"', { STRING_CHAR | ESCAPE | INTERPOLATION }, '"' ;
ESCAPE     = "\\", ( '"' | "\\" | "#" | "n" | "r" | "t" ) ;
INTERPOLATION = "#{", expression, "}" ;

LETTER is A–Z or a–z; DIGIT is 0–9; HEX_DIGIT is a DIGIT or a–f in either case; OCT_DIGIT is 0–7; BIN_DIGIT is 0 or 1; LOWER_OR_UNDERSCORE is a–z or _; UPPER is A–Z. STRING_CHAR is any Unicode character except double quotes, backslash, newlines, and a # followed by {. NL, DOC and EOF are lexer tokens.

The minus sign is an operator, not part of a literal. An exponent follows the digits or the fraction, as in Ruby: 1.0e300, 1.5e-3, 2.0E+10 and 1e5 are Floats, and Float#to_s writes 1e+300, which reads back. The e needs a digit before it and a digit after the optional sign, so 1.e5 is the call 1.e5 and 1e is the Int 1 next to the name e. An exponent too large gives infinity (1e999 is +inf.0). A string literal cannot span lines; text of two or more lines is a heredoc (below).

A _ may separate digits in INTEGER and in both parts of FLOAT: 1_000, 1_000.000_5, 1e1_0. It is only visual: 1_000 is the same Int as 1000, and the lexer drops the _s before it reads the value. _ is allowed only between two digits. A trailing _ (1_, 1_000_, 1_a), a double __ (1__0) and a _ next to the dot or the e (1_.5, 1._5, 1_e5) are LEX-INVALID-NUMBER, with the literal as written as the span: the run of digits and _, plus a . followed by a digit or a _ and the run after it, plus an exponent. A . followed by a _ belongs to the number, so 1._5 is an error; a . followed by a letter or by another . ends the number (1_000.times, 1_000..2_000). _1 is still a NAME. The text form of a number has no separators: puts(1_000) prints 1000, and s.to_i and JSON still reject _.

0x, 0b and 0o, in either case, start a hex, binary or octal INTEGER, as in Ruby: 0xFF, 0b1010, 0o755. It is the same Int as the decimal one (0xFF == 255), with the same range: 0x8000_0000_0000_0000 is CHECK-INT-RANGE and -0x8000_0000_0000_0000 is the least Int. A _ sits only between two digits. Every letter, digit and _ after the prefix belongs to the literal, so a prefix with no digit (0x), a digit outside the base (0b102, 0o8, 0xFG) and a misplaced _ (0x_FF, 0xFF_, 0b1__0) are LEX-INVALID-NUMBER, with the literal as the span. Unlike Ruby, a leading zero alone is decimal, not octal (0755 is 755), and a Float has no other base.

"text #{expression} text" interpolates; the result is a String. Each value is inserted as its to_s (see “Methods on builtin types”). The expression must end on the same line and cannot contain comments; strings, maps and nested interpolation may appear inside it. A Void call is CHECK-VOID-VALUE. A function, task or resource handle has no to_s and is CHECK-TYPE-MISMATCH. \# writes a literal #, so "\#{x}" is the text #{x}. case patterns and import paths take plain string literals only, so a heredoc there is PARSE-UNEXPECTED-TOKEN.

Heredocs #

A heredoc is a string of several lines, in the squiggly form of Ruby: <<~NAME, then the text on the lines below, then a line that holds only NAME. The common indentation of the text is removed.

HEREDOC      = "<<~", HEREDOC_NAME ;
HEREDOC_NAME = UPPER, { UPPER | DIGIT | "_" } ;
def help_text(name: String) -> String
  text = <<~TEXT
    Usage: #{name} [options]

      --verbose   print more
    Exit code 0 on success.
  TEXT
  return text
end

For name = "tool" the value is "Usage: tool [options]\n\n --verbose print more\nExit code 0 on success.\n".

The lexer removes the smallest number of leading spaces among the body lines that are not blank from every line. A line of only spaces becomes an empty line. Each body line ends with \n, the last one too. The indentation is removed from the source text before escapes and interpolation, so an interpolated value is never re-indented.

The text is read like a string literal: the escapes are the same (\n, \r, \t, \", \\, \#, and a bad one is LEX-INVALID-ESCAPE), #{...} interpolates and must end on its line, and a " needs no escape.

The rest of the line that holds <<~NAME is lexed as usual, so puts(<<~TEXT) and <<~TEXT.length work, and the body starts on the next line. Two heredocs on one line take their bodies one after the other, in order. Inside parentheses the call goes on after the end line: f(<<~A, with the next argument on the line after the end line. The body lines and the end line give no NL token; the line of the opener gives its NL where it would give one without a heredoc.

The end line has only optional spaces and the name; it may be the last line of the file, with no newline after it. A tab in the leading whitespace of a body line or of the end line is LEX-HEREDOC-INDENT, at the tab, because the width of a tab would make the removed indentation unclear. A file that ends before the end line is LEX-UNTERMINATED-HEREDOC, at <<~NAME.

A heredoc is the form for text of two or more lines: a heredoc with one body line, or none, is LEX-HEREDOC-ONE-LINE, at <<~NAME, with a help that shows the string literal ("text\n"); <<~EAG is the exception (below). A heredoc is never required: "a\nb" stays valid.

Only the squiggly form exists. <<~text, <<~'NAME', <<NAME and <<-NAME are LEX-UNEXPECTED-CHAR, at the << and the ~ or - of the opener, with a help that shows <<~NAME. Every other <<, such as xs << x, is two < tokens, which the parser rejects (PARSE-UNEXPECTED-TOKEN, with a help that shows xs = xs.push(x)). A heredoc inside the #{...} of another string or heredoc cannot end on its line and is LEX-UNTERMINATED-STRING.

The lexer gives a heredoc the tokens of a string literal (string, or interp-start, the tokens of each expression, interp-middle and interp-end), and every one of those string pieces has the span of <<~NAME, so the nodes built from them stay on the opener line. The tokens inside #{...} keep their line and column in the body, and a diagnostic about them prints that line. A span inside a heredoc can therefore lie after the span of the expression that holds it.

EAG templates #

EAG (“Embedded AGentica”) is Agentica’s ERB. A heredoc named EAG, <<~EAG, is a template: a String built from its text and its tags.

import EAG from "std/eag"

def page(names: List[String]) -> String
  return <<~EAG
    <ul>
    <% for  name in names %>
      <li><%= EAG.html_escape(name) -%>
      </li>
    <% end  %>
    </ul>
  EAG
end

The lexer keeps the body lines as written in one eag token (the rules of a heredoc apply: the indentation, the end line, LEX-HEREDOC-INDENT, LEX-UNTERMINATED-HEREDOC; a body may be one line or empty). The text is raw, with no escape and no #{}. The tags (<%= %>, <%== %>, <% %>, <%# %>, <%%, -%>, <%-, the <%@ escape %> directive), the trimming rules and their codes (PARSE-EAG-BLOCK, PARSE-EAG-UNTERMINATED-TAG, PARSE-EAG-DIRECTIVE) are in the std/eag page.

The parser makes the template an expression that builds the text, from tokens that keep their places in the file, so every diagnostic about a tag has its place in the template; the checker checks it as ordinary code, so the literal can stand anywhere an expression can. The names the expansion uses have a #, which no identifier has. The file must import std/eag in any form, or the literal is CHECK-EAG-IMPORT.

Regex literals #

/pattern/flags is a regular expression of std/regex: the value of Regex.compile of the body, with the type Regex.Regex (semantics in §6). The body has no string escapes. A backslash is read with the character after it, so \/ is a slash in the pattern and \d needs no second backslash. The body ends at the first unescaped /; # in the body is pattern text, never a comment.

  • A line break or the end of the file before the closing / is LEX-UNTERMINATED-REGEX, at the literal.
  • A / inside a class [...] ends the literal and leaves the class open, which is also LEX-UNTERMINATED-REGEX, with a help: write the slash as \/.
  • #{ in the body is LEX-REGEX-INTERPOLATION; \#{ matches the text #{. A pattern built at run time is Regex.compile(text).
  • After the closing / come the flags i (ignore case, (?i)) and m (. matches a newline, (?s)), each at most once, in any order. Any other letter or digit, or a repeated flag, is LEX-REGEX-FLAG, at that character.

A / starts a literal when the token before it cannot end an operand. The operand ends are number and string literals, names, type names, true, false, nil, end, ), ], } and a regex literal; after one of them / is division and /= is the compound operator. So a / b / c and x /= 2 keep their meaning, while f(/=x/), [/a/, /b/] and return /a/ hold literals. A line that starts with / 2 is LEX-UNTERMINATED-REGEX, with a help: put the operator at the end of the line before, or wrap the expression in parentheses. //, /* and */ stay two operators, which the parser explains as comment habits. puts /x/ is division after a name, a PARSE-UNEXPECTED-TOKEN whose help shows puts(/x/). A regex is not a case pattern (PARSE-UNEXPECTED-TOKEN).