Login
Track P: The grammar and its matcher
Login

Everything a person types to Glint is Crystal syntax. The shell, and later any configuration Glint reads, accepts only lines that also parse as Crystal and mean the same thing. The shell and human-facing syntax in PLAN.md gives the reasons. Track P builds the machinery for that: one grammar table, written as Crystal data in the form used by cleave's Cleave::GlyphParser, the progressive multipass parser that began in crystal-robots (docs/PARSER.md there). That parser reduces a string of token glyphs one rule at a time, each rule being a regular expression over glyphs, and can print every pass as a derivation.

On Linux the rules run on Crystal's Regex, which uses PCRE2. Inside Glint, and on a 16 KiB microcontroller, there is no PCRE2 and no dynamic memory to spare. Track P keeps PCRE2 for the hard part, which is compiling and validating each pattern with all its corner cases, and does it at build time. A Crystal run macro compiles each rule with the host's PCRE2 (10.44, the version BQ2 pins), reads the compiled bytecode, and emits Crystal that needs no PCRE2 at run time. The long-term aim is to regenerate PCRE2 itself with the glyph parser. Until then, PCRE2's own compiler is the specification, and the target binary links none of it.

The track runs alongside Phase 1, and Phase 1's shell (1.8) waits on P.4. Slices cite PCRE2's source and documentation, cleave's GlyphParser and crystal-robots' docs/PARSER.md, in place of the development text.

Provisional slices.

Slices

P.1 The shell grammar as data. This slice writes the first grammar table in Cleave::GlyphParser's form: a lexer and one rule that reduces a call with literal arguments (ls, cat "hello.txt", echo "booted"). The derivation of each example is checked in specs on Linux using Regex. It comes first because every later slice compiles or matches this table. Using cleave's parser rather than a second one means the grammar engine exists once, which is Parsimony.

P.2 Crystal is the oracle for syntax. This slice adds a spec that runs every example line in the grammar table through Crystal's own parser (Crystal::Parser, from the compiler BQ2 ships) and fails if a line does not parse as Crystal, or parses as something other than the call the shell means. It comes right after P.1 because the rule that the shell accepts only Crystal must hold from the first rule onward. Letting the real compiler judge the syntax is Demonstrability. P.3 builds on this check, since only checked rules are worth compiling.

P.3 Reading PCRE2's bytecode. This slice adds tools/pcre2_extract.cr, run at build time by a run macro. It compiles one rule with the host's PCRE2 10.44 and locates the bytecode the way pcre2_match.c does: after the pcre2_real_code header and its name table (name_count entries of name_entry_size). It emits the bytecode as a constant. The opcode numbers are read from the pinned pcre2_internal.h rather than written by hand, following the plan's rule of working from the data. Any opcode outside the supported set stops the build with its name. The slice comes after P.2 because only checked rules are worth compiling. Keeping PCRE2's version pinned and its opcodes read, not copied, is Clarity about a dependency on internals that PCRE2 does not promise to keep stable.

P.4 A small matcher. This slice adds one Crystal interpreter for the opcodes the grammar uses. That covers brackets and alternation (OP_BRA, OP_ALT, OP_KET), literal and UTF-8 characters (OP_CHAR), character classes over glyphs (OP_CLASS, OP_XCLASS), repeats, and the look-around assertions the parser's precedence guards rely on (OP_ASSERT_NOT, OP_ASSERTBACK_NOT). It runs over the constants from P.3 with no allocation, so it works in Glint's kernel and on the Zepto. A differential spec on Linux matches every rule against PCRE2 itself on the grammar's examples and on generated glyph strings, and fails on any difference. It comes after P.3 because it consumes P.3's constants. One interpreter for all rules is Parsimony, and PCRE2 as the oracle is Demonstrability.

P.4b The matcher's bytecode without start-up. Merged as check-in e9b57a84. A repair slice, found when 2.6 first typed into the shell on firmware and every line, help included, gave syntax error. The image takes its bytecode from ImageBytecode::CALL_STATIC_BYTECODE, a StaticArray constant. Crystal initialises such a constant through a const_init function at start-up, which firmware never runs. So in every real image the bytecode was zero from 1.8 on, and no boot test typed a command to notice. The bytecode becomes a value built at call time. A spec reads the linked image's symbols and fails on any start-up initialiser, whatever source it came from, and a boot test types help into the shell. A check on the built image, rather than on chosen source directories, is Demonstrability.

P.5 Faster matching, when needed. Once a profile shows matching matters, this slice emits straight-line Crystal per rule from the same bytecode, so LLVM can merge character comparisons, and keeps P.4's differential spec as the check. It comes last because P.4 is enough until measurements say otherwise, which is Parsimony: no mechanism before the need for it.

Track demo: the shell's grammar runs in Glint's EFI images with no PCRE2 symbols (nm shows none), matching exactly what PCRE2 matches on Linux.

References