Documentation

Testing Guide

This guide covers how to run tests, what the different test types are, and how to interpret their output. For the full technical reference (.ci format spec, .et directives, fuzz target list), see docs/functional-tests.md.

First-time setup

./le setup install
./le setup check
./le test unit pkg path ./internal/core/textbuf/

./le setup check reports every tool setup installs, and the scoped package test proves the checkout builds and its tests run under the feature tags (../guide/developer-setup.md, "After Setup").

./le verify current is the full verification, Thomas's release check, which GitHub CI also runs on a pushed commit. It runs only when stdin and stdout are both terminals, so Thomas starts it by hand at one. No agent runs any ./le verify command, ./le verify changed included: the evidence for a change is the scoped test runs of what it reaches. The main thread runs them in the main checkout, from the owed commands a subagent lists in its report (ai/rules/commands.md).

./le is the compiled repository development entry point. Each area exposes its own verbs, for example ./le test functional plugin, ./le test qemu installer-iso, and ./le test unit bgp. Legacy command names have no compatibility aliases.

Action words describe the contract: check is a read-only verdict, verify is a composite policy gate, report is advisory output, and write or update changes tracked state.

The escalation ladder

Use the narrowest test that covers your change. Escalate only when needed.

Step What you run When Time
1 ./le test unit pkg path ./internal/core/textbuf/ run TestName Iterating on one test seconds
2 ./le test unit bgp Checking one component group 10s to 1:30
3 ./le test functional plugin Checking the user-visible path varies
4 GitHub CI on the pushed commit Judging the pushed change set: no agent runs it locally minutes
5 ./le verify current The release check, which only Thomas starts, at a terminal the longest

Steps 1 to 3 are development: run the ones that cover what you changed, and ./le repo changed scope print both names them (running-commands.md, "The changed-set selector"). Run them in the main checkout, never in a worktree or a copy of the tree. Step 4 is CI's: it runs once the work is pushed, and ./le commit push asks for no local verify record. A red it finds gets a fix commit; it does not hold the commit it judged. ./le verify changed still exists for Thomas, and no agent runs it.

Step 5 is the full verification, and it is not an agent's to run. The dispatcher refuses ./le verify current, ./le verify current fast and every form of ./le verify worktree unless stdin and stdout are both terminals, because a full run loads the whole shared machine and only Thomas knows when that is safe. The refusal prints the exact command line for Thomas, and the scoped tests that are the change's evidence, which the main thread runs. No keyword or environment variable skips it. When Thomas starts ./le verify current, it runs every stage in a fresh detached worktree at HEAD, through the same lifecycle as ./le verify worktree, so an edit made while it runs cannot move the tree it judges; ./le verify current fast runs the changed-files stages in place, over the working tree. The narrower commands are development tools. ./le test unit pkg admits its run through internal/le/job, and ./le job run admits any other heavy command the same way, so concurrent sessions do not oversubscribe the machine.

One package: ./le test unit pkg

./le test unit pkg path <dir>/ [run <regex>] [vet] tests the packages it names under the feature tags from feature-gates.txt, with the race detector and the toolchain's timeout, admitted under the job label unit-pkg. Repeat path for each package. run hands its regex to go test -run. vet runs go vet on the same packages and their test files instead, under vet-pkg. changed in place of path takes the packages the change reaches.

The output goes to this session's scratch log. A green run answers one line, test unit pkg: exit 0, <n> packages ok, ..., log <path>, which counts only the packages that ran a test. With changed, a left out <package> line comes before it for each reached package that changed set aside behind a group's own tags, naming the group's command. A red run prints each failed test's --- FAIL: line with the lines beneath it, then that summary. A run in which no package ran a test, a run that matched nothing included, answers exit 1.

It refuses with exit 2, before it takes a slot, a line that would test more than the packages meant: path ./... (the whole tree, ./le test unit all, is the owner's run), a group's whole pattern (run that group), a value that spans two or more groups, and a package under the installer group, whose ze_installer tag the feature set lacks (run ./le test unit installer). A changed answer the selector widened to every package is refused the same way. The race detector is always on, and the action takes no other go test flag: a run that needs -v, -count, -bench or a cover profile stays on ./le job run label unit-pkg quiet command go test -race -tags "$tags" ... with the tags written out (running-commands.md, "Testing one package while you develop it").

Component groups for step 3

Command Scope Time
./le test unit bgp BGP engine, wire, reactor about 1:30
./le test unit core Core libraries about 30 seconds
./le test unit plugins All plugins about 40 seconds
./le test unit config Config parsing and YANG about 20 seconds
./le test unit cli CLI component about 10 seconds
./le test unit installer Installer initrd logic behind the ze_installer tag about 10 seconds
./le test unit all The whole checkout, then the groups their build tags hide from it minutes: it is every package under -race

Pick the group matching your change. Each group is a subset of all, so a green group says nothing about the rest of the tree.

The installer group runs the tests on Linux. On another host go test cross-compiles a Linux binary it cannot start, so the group runs go vet and type-checks the same files instead. ./le test qemu all runs them for real inside the Alpine virtual machine.

Test types

Ze has four levels of testing, each covering a different concern.

Unit tests (*_test.go)

Standard Go tests. They validate logic in isolation: does the parser produce the right output, does the state machine transition correctly, does the encoder round-trip.

./le test unit pkg path ./internal/component/bgp/message/...
./le test unit pkg path ./internal/component/bgp/route/ run TestParseOrigin

Dynamic-group coverage exercises each plugin's real configuration boundary: capability declarations, policy lookups, validation, and command output. Opposite group and peer values expose lost inheritance; same-name peers and groups expose namespace collisions. A list of caller filenames or helper names proves neither. For example, the redirect tests pass plugin JSON through the RIB parser before querying the session's effective setting.

Kernel tests in a user namespace

A unit test that needs kernel privilege (links, routes, XFRM, nftables, conntrack, raw or AF_PACKET sockets, a low port, a named network namespace) runs unprivileged through userns.Enter (internal/test/userns). Enter re-runs the one test as root over a user, network and mount namespace of its own, with lo up and a private /run/netns, so nothing it creates reaches the host. Write such a test this way:

  • if !userns.Enter(t) { return } is the first statement of each top-level test that reaches a privileged operation. Never call it from a helper or a subtest: the child selects the test by its top-level name. A helper that needs privilege is reached through the Enter of every top-level test that calls it.
  • The test never calls t.Parallel, because the child re-runs that one named test in a fresh process.
  • A refused operation inside the namespace fails the test with the operation and the errno, and never skips: the namespace grants the privilege, so a refusal is a defect or an unmet need. A skip that names an absent kernel feature or module (for example CONFIG_MPLS_ROUTING) stays a skip.
  • Every child startup error fails the parent, namespace denial included. The message carries the error and names ./le setup install and its userns-unrestricted check, which lifts the AppArmor restriction (developer setup). Only a child that ran and reported its test skipped skips the parent.
  • The file's build tag is linux, so every Linux unit run and the hosted CI unit job execute it. Keep integration && linux only when the file runs a binary ./le setup does not install (nft), needs a host grant (/dev/ppp, through ppp-access), takes an explicitly supplied ze executable, or needs a privilege only the initial user namespace holds. Say which in the file's header comment.

Run the evidence as your own user through the launcher, for example ./le job run label unit-pkg quiet command go test -v -count=1 -race -tags "$tags" -run '^TestName$' ./internal/core/routewatch, and read --- PASS for each test in the quiet log. It keeps the long form because the short action takes no -v, so $tags comes from the recipe in running-commands.md, "A bare go test is not ./le test unit". Off Linux, Enter returns true at once and the test runs in place, which serves a portable test such as the traceroute command tests. The namespace mechanics are in QEMU integration testing, "Build Tags".

Fixture environment isolation

Native Go test commands remove the launching le's ze.repo.root and ze.le.build.name values under every case/dot/underscore spelling. The test process discovers its checkout from its working directory; a fixture can then set its own root without an inherited canonical spelling winning after a cache reset. Toolchain caches, tags and unrelated environment settings remain intact. Go test callers using gotoolchain.Toolchain.Environment MUST set EnvOptions.Test, including fuzz and benchmark runs. Ordinary builds and production commands retain operator checkout overrides and named-build checks.

A go test that ./le job run starts as its direct child, and the one ./le test unit pkg starts through it, use the same isolation at the process boundary, including Go's optional leading -C directory option. The admission process retains its root and job identity; only the test child's checkout/build identity is removed. The test child also gets GOTMPDIR set to the test temp dir on the home device, as every le-started test does (running-commands.md, "Where a Go test keeps its temp dirs").

Tests that invoke fixture binaries remove inherited launcher identity before adding their explicit child environment. The production named-build refusal is not disabled: a child given an explicit mismatching identity still fails. Tests that run git-changing actions MUST verify the resolved root is their own temporary fixture before invoking those actions. They MUST NOT use the developer's checkout or linked worktrees as mutation fixtures.

Verifier lifecycle fixtures need a real Git repository with HEAD, even when they inject the lifecycle's Git calls. Admission measures the actual checkout before the worktree operation: an empty directory fails that measurement instead of exercising the intended branch, timeout, or add refusal.

CLI personality parity runs both real binaries against one isolated Git repository with a committed original, a tracked modification, and an untracked file. The fixture copies the checkout's real .gitignore before committing, so runtime scratch files follow the same ignore policy. Both binaries receive the same explicit fixture root. Stdout equality remains exact; stderr keeps only the existing program-name normalization. Concurrent developer edits are not fixture inputs.

Functional tests (.ci files)

These spin up real Ze processes and test behavior end-to-end: does the config parse, does the BGP session establish, does the plugin produce the right output.

Each test/ subdirectory has its own runner and format:

Directory What it tests Runner
test/encode/ BGP wire encoding le test bgp encode
test/decode/ Wire decoding le test bgp decode
test/parse/ Config parsing (valid/invalid) le test bgp parse
test/plugin/ Plugin behavior le test bgp plugin
test/reload/ Config reload le test bgp reload
test/ui/ CLI completion le test ui
test/editor/ TUI editor (.et files) le test editor
test/managed/ Managed config le test managed
test/web/ Web UI le test web
test/l2tp/ L2TP daemon le test l2tp
test/firewall/ Firewall le test firewall
test/policy/ Policy routing le test policy
test/exabgp-compat/ ExaBGP compatibility le test exabgp

Run a single test by one-based ID or exact name, list the available IDs, or resume from the last printed ID after an interrupted run. Each of these runners admits itself through the same job admission as a full suite, so it needs no ./le job run around it (running-commands.md, "One verify at a time").

./le test bgp plugin 42              # test id 42
./le test bgp encode --list          # list N/TOTAL, id, and name
./le test bgp plugin --start 42      # id 42 and every later test
./le test editor 7                   # editor test id 7
./le test editor -p nav              # editor tests matching "nav"
./le test exabgp --start 20          # resume ExaBGP compatibility

Run a full suite:

./le test functional encode
./le test functional plugin
./le test functional gating
./le test functional exabgp

Mutation tests (gomu)

Coverage tells you which lines ran. Mutation testing tells you whether the tests would notice if those lines did something different. gomu rewrites the AST (arithmetic, conditional, logical, bitwise, branch, return value, and error handling operators), runs the test suite against each mutation, and reports which mutations survived.

gomu uses overlay-based execution, so it never modifies source files on disk. It is a Go tool recorded in tools.go. The native ./le test mutation combine command combines completed reports, and ./le test mutation record appends their per-package scores to the committed history.

Mutation testing is advisory. It never gates Thomas's release check, ./le verify current, or CI. A surviving mutant is a signal that a test could be stronger, not a blocking failure.

One consumer reads a gomu report for a narrower question. ./le rfc discriminate <selector> report <path> scopes the report to ONE RFC-tagged test: it keeps the KILLED mutants that lie in code that test's own coverage profile executes, and offers them as candidate breaks. ./le rfc record then applies one, runs that test alone, and records the red it observed. gomu's status stays unchanged, because nothing in ./le rfc check starts a mutation run: the gate reads a stored record. See docs/contributing/rfc-conformance-gates.md, "The discrimination record".

Files with custom build tags and cmd/ze/ are excluded via .gomuignore because gomu has no --tags support. Reports land in tmp/ (gitignored). Mutation score history is tracked in test/mutation/history.ndjson.

gomu is vendored and runs through go run, so it needs no install.

Command Purpose
go run github.com/sivchari/gomu/cmd/gomu run --output json --incremental=false --fail-on-gate=false Full advisory mutation run
go run github.com/sivchari/gomu/cmd/gomu run --output json --incremental --base-branch=main --fail-on-gate=false Changed-file advisory mutation run
./le test mutation combine Combine the per-package JSON reports

Tuning is by flag on gomu run, not by environment variable. gomu reads only GITHUB_TOKEN and GITHUB_REPOSITORY, both for its GitHub integration.

Flag Default Effect
--workers 4 Parallel workers
--timeout 30 Test timeout in seconds
--incremental true Analyse changed files only
--base-branch main Base branch for the incremental analysis
--fail-on-gate true Fail the build when the quality gate is not met
--output console console, json, html, or text

Interop and integration tests

These require external infrastructure (Docker, root/CAP_NET_ADMIN, QEMU, or internet). They are not part of the normal development cycle.

./le test integration interop
./le test integration iface
./le test qemu ci test test/<dir>/<name>.ci   # one needs-linux test in Ze's QEMU guest
./le test integration live-rpki
./le test integration docker-peers   # every package test tagged `docker`
./le test integration clean dry    # what a killed lab run left, and why the rest stays
./le test integration clean        # remove the lab resources whose owner is gone

A package test that starts a real peer container carries //go:build docker, never docker and integration together: the integration population runs in the QEMU guest, which has no Docker. docker-peers derives its packages from the tree and sets ZE_REQUIRE_DOCKER=1. Such a test starts its server with labtest.Start(t, spec) from internal/le/interoplab/labtest, never with its own exec.Command("docker"), which ./le arch tier check refuses outside internal/le/interoplab; the contract is docs/architecture/testing/interop.md, "A package test's own server container (labtest)".

When a test must be weakened

A red test means the CODE is wrong by default. Fix the code. When the coverage is genuinely gone, because the feature it proved was removed or another test now proves it, the removal is recorded rather than silent.

A test weakening is approved BEFORE the edit by an independent agent running the ze-test-judge skill (ai/skills/ze-test-judge.md), never by Thomas and never by the agent that wrote the change. The judge approves only when the new test proves at least what the old one did, or when the old expectation is wrong against its source; otherwise it refuses, and the refusal stands. The edit hook refuses the weakening, names the test, and prints the command that records the verdict.

The route, in order:

  1. Hand the test, the diff and your reason to a fresh agent running ze-test-judge.
  2. On APPROVE the judge records its verdict, for one test at a time: ./le test weakened approve unit <package>.<TestName> reason "<the judge's verdict>". The row goes to tmp/commit-weakening-approved-<session>.md, keyed on the main thread's commit namespace, so a subagent's edit sees it too. git never reads that file. <package> is the name of the directory holding the test, not its Go package clause; the command refuses a unit no directory carries and names the key to use. When two files of the directory declare the name, it refuses the bare unit and prints the form that names the file: ./le test weakened approve unit <package>.<Name> file <path> reason "<the judge's verdict>". file is accepted only then: on a unit one file declares, the command refuses it and prints the form without it. The file lives only in the approval row; the commit's trailer names the unit alone. The row records the unit's content as the judge read it, so it approves that change only: a different later change of the same test needs a new verdict (docs/architecture/testing/test-health.md, "One approval names one test"). On REFUSE, fix the code instead.
  3. Make the edit.
  4. Commit with ./le commit create as usual. The message gains one Weakening-approved: <package>.<TestName>: <the judge's verdict> line per test, and the script drops each used row once the commit lands. A row a landed commit already used approves nothing further.

The unit is the name of the directory holding the file, a dot, then the enclosing top-level func TestXxx, or the file stem for a .ci or a .et. A bare name, a path or a directory is refused: one approval names one test. An RFC-approved: row for a unit (./le rfc approve) also accepts that unit's weakening. Because the unit keeps only the directory's base name, a commit that changes cmd.TestX in both a/cmd and b/cmd is refused: split it so each commit carries one of them.

A helper is any function of a _test.go file that is not a Test, Fuzz or Benchmark function (TestMain counts as a helper). A weakening inside a helper (an assertion it no longer makes) is the helper's own finding: the row names the helper, <package>.<helperName>, because a calling test's verdict judged the test's text, not the helper's. A changed helper also changes every existing test that calls it in a file the commit carries, so that test's live row is spent and a stale one refuses. A test calls a helper when it names the helper as a call or a function value (never after a selector's dot), directly or through other helpers, in the _test.go files of the same directory. The rule matches names, so a local of the same name also counts as a call. A helper no test calls keeps its finding all the same.

A weakening is the loss of proof the committed tree holds. The edit hook judges only the units HEAD records for the file. It follows a committed test file that the working tree lacks into the other test files of the same directory: an Edit of a file moved there (by git mv or by mv) is judged against the committed units. A Go test file at a path HEAD does not hold is judged, on the Write that creates it and on every later whole-file Write or Edit, against the committed files missing from its directory, for each unit it declares and the edit changes under the name of one the missing file held, so a stub written first cannot carry a weaker copy in later. A renamed file whose units are renamed too is caught only at the commit gate. A Write that recreates a deleted committed file is judged against HEAD's text of it. An alias of a committed test (a symlink, a hard link in its directory, or a case variant of its name on a case-insensitive filesystem) is judged as the committed test. A test or helper that only the working tree holds, in a new untracked file or added to a tracked one, is yours to rename, replace or remove without a verdict, unless it shares a name with a unit of a committed file missing from its directory. While a committed .ci is missing from a directory, every untracked .ci there is guarded against losing what its own text proves, which can refuse an edit to an unrelated new .ci, never admit a loss. When a refusal comes from a committed file missing from the directory, it names that file: have an independent judge approve the change, or, if the two files are unrelated, stop and report it, so that whoever deleted the file commits the deletion first. A unit the edit leaves as it was is not judged again; it was judged only if a Write or Edit the hook sees wrote it after the committed file went missing, or through a link to the committed file. The hook does not see these, and the commit gate judges them as an addition beside a deletion, or as a deletion it cannot pair:

  • a move to another directory, and a new .ci written under another name
  • an edit through a hard link made in another directory
  • a unit written by a shell command (a redirect, sed -i, cp, or a draft moved into place with mv): the Bash hook guards shell writes only under plan/ and ai/rules/
  • a unit written while its committed namesake still stood, that file deleted afterwards
  • a weaker copy of a moved unit added to a file HEAD already tracks, while a committed sibling is missing: a tracked file is judged against its own HEAD text only

The rules are in docs/architecture/testing/test-health.md, "One approval names one test".

A test edited again after its verdict is refused at commit, even when the new edit weakens nothing: the row is stale, because the judge saw other text. Run the judge again on the change as it stands and record its fresh verdict, or remove the stale row when the change needs none.

The writer resolves the directory key against current and HEAD carriers, and refuses an unknown or ambiguous key before it changes an approval file. The registered action checks the unit and the reason in a read-only preparation before startup, so a refused request writes nothing. Its admitted answer creates the commit namespace and writes the row.

Known limit: two kinds of unit cannot be approved at all, because the unit must match <package>.<TestName> with one dot and no empty side. A .ci or .et file whose stem holds a dot (x.y.ci is suite.x.y), and a test file at the repository root (its package side is empty), are refused by the approve writer and by every reader.

An edit that only lowers a COUNT lands with a notice and no approval: consolidating three cases into one table lowers a count exactly as deleting a check does. The COMMIT still needs an approval for it, and the judge's verdict is where the commit says which of the two happened.

Git history is where past approvals are read: git log --grep '^Weakening-approved:' lists each one beside the change it accepted, and ./le test weakened check reads the same lines when it reviews a commit. The rule is ai/rules/testing.md, and the design is docs/architecture/testing/test-health.md.

How Thomas's release check works

When Thomas starts ./le verify current, it runs the full population in a fresh detached worktree at HEAD; ./le verify current fast runs the changed-files population in place. Both are release checks that start only at a terminal (see the ladder above). ./le verify changed runs the changed population in a throwaway worktree; Thomas may start it, and no agent does. ./le verify list full and ./le verify list fast print each population. The unit stages use a two-pass strategy to stay fast:

  1. Lint (27 linters via golangci-lint)
  2. Vet evidence (cross-compile the Go evidence packages for Linux)
  3. Cached full pass (go test without -race): Go caches by source hash, so when nothing changed this is instant. Catches logic regressions everywhere.
  4. Changed-group pass: uses the test-only CGO_ENABLED=1 go test -race path on Linux and Darwin. Its test binaries are never release/build evidence.
  5. Functional tests from internal/le/test/functional/catalog.go
  6. ExaBGP compatibility

Common case (one group changed): ~2 min total instead of 6+.

The builds the linter reads

golangci-lint analyzes ONE build for each run: one GOOS, one GOARCH, one tag set. ./le go lint run therefore runs more than one.

Pass Build What only it reads
1 the host GOOS, .golangci.yml tags the shipped daemon
2 GOOS=linux, plus integration every kernel-facing //go:build integration test
3..N one for each row of FLAVORS. internal/le/go/lint.Answer ze_installer, ze_distro, ze_appliance, ze_setup, tinygo, and the capability tags (debug, race, live, ...). Also the GOOS and GOARCH targets no other pass compiles: darwin, freebsd, openbsd, dragonfly, wasip1, linux/arm64 and linux/riscv64. Also the compile-out build, which drops every feature gate and keeps ze_core alone

Waiting for the linter's own lock

golangci-lint takes one lock for the whole machine, at $TMPDIR/golangci-lint.lock. Several sessions share this checkout, so a second linter is normal here. Every command line Ze builds therefore carries --allow-serial-runners, which makes the child WAIT for that lock. Without it the child gives up after five seconds and exits parallel golangci-lint is running, and the stage turns that into a red that names no file.

A pass that waits prints nothing while it waits. A whole-tree pass holds the lock for about ten minutes on this workstation, so a pass behind one can stay silent for that long. That is the linter in a queue, not a hang.

The post-edit lint takes no index lock

After each Go edit the post-write hook lints the edited package with --new-from-rev=HEAD --timeout=30s. For --new-from-rev, golangci-lint runs git status and git diff itself, and its timeout SIGKILLs them. A git killed while it refreshed the index left an empty .git/index.lock that no process held, and every session's next git write failed until someone removed it. The hook therefore runs the linter with GIT_OPTIONAL_LOCKS=0, which stops git status refreshing the index, and diff.autoRefreshIndex=false, which stops git diff writing the refresh back. The patch the linter reads is the same either way. le's other git status queries under a deadline carry the first setting through gitenv.ReadOnly. The weakening check's queries, which the edit hook runs on every test edit, carry gitenv.ReadOnlyQuery: the first setting plus maintenance.auto=false and gc.auto=0, so none of them starts a detached maintenance run under .git. The post-edit hook starts the real linter, not the golangci-lint shim, so the kill at the hook's 60 second budget ends the linter itself and no le exec waits for a job slot on an edit (postLinter).

Post-tool hooks block only on the author's own files

A post-tool hook's blocking text enters the agent's context, and every later turn re-reads it. Several sessions share this checkout, so a check that judged every uncommitted file blocked each session on the others' work. Each check therefore blocks only on what the tool call itself touched:

Check Blocks on Said at code 0, to the person only
Post-edit lint An issue in the edited file, or a package that does not build (* typecheck: N in the summary) The count of issues in other changed files of the package
Post-command gofmt An unformatted Go file the command names, by path or by a directory holding it The count of unformatted files it does not name

When any check blocks, the hook drops the code-0 hints from what it prints, because a hint carries nothing for the agent to act on.

The gofmt check reads what a command names from its parse as bash, never from its text. A file is named when a whole word of the line is its path or a directory above it:

  • ./, a trailing slash and an absolute path under the checkout are spelling.
  • Go's dir/... names dir.
  • A word that is one unquoted glob (pkg/*.go) names what it matches, and * does not cross a directory.
  • Any word that starts with - and holds = names what follows the first = (--file=pkg/x.go).
  • The checkout itself (., ./..., its absolute path) and a path outside it name nothing.

A read-only line names nothing, because it cannot have changed a file. A line is read-only when every command on it is a reader (grep, cat, git log, the set in readerCommands), a command inside $(...) included, and no redirection may write a Go file of the checkout, which is the only way a line of readers changes one:

Redirection target Ends the exemption
Outside the checkout (2>/dev/null), or a descriptor (2>&1, >&-) ✕
A known file that does not end in .go (> tmp/out.txt, 2> tmp/err.log) ✕
A file that ends in .go, or a glob ✓
Not known: an expansion (> "$out"), or a relative path that climbs with .. (> ../x.go) ✓

The exemption is all or nothing: cd pkg && cat > x.go <<EOF is not read-only, so the word of cd names the file, as it does in cd pkg && sed -i ... x.go.

A command that does not parse names nothing, and the code-0 line says the command could not be read and how to check what it changed. So does a line whose tree names fewer commands than bash runs (treeMisreadsLine, read by commandUnread in internal/le/hookruntime/postbash.go): a comment that ends in a backslash, which the parser joins to the next line while bash runs that line, and a $ followed by a backslash-newline. Such a tree would read echo a # x \ then gofmt -w x.go as one read-only echo, so the check does not read it at all.

The limits that remain. In each case the file is counted at code 0 and does not block:

  • A word bash expands other than as a plain glob names nothing: a parameter (pkg/$name/x.go), a command substitution, arithmetic, a tilde, braces, or a glob that also carries a quote or a backslash. The check cannot know what bash passes for it.
  • A path inside a longer word is not seen: dd of=x.go, an attached short option value (-ox.go), the script of bash -c '...'.
  • A path that arrives through a variable or xargs is not seen.
  • A relative word is read against the checkout root, not against the shell's directory after an earlier cd on the line. Only the word of cd names, and a relative word that climbs with .. names nothing.
  • An absolute path is compared as it is spelled, so one that reaches the checkout through a symlink is read as outside it.
  • A redirection target that starts with - and holds no = (cat > -x.go) is read as an option word and names nothing, so such a line stays read-only.
  • A glob that path.Match refuses (an unmatched [) is compared as it is spelled, so a file it would reach by matching is not named.

Some limits go the other way, and block for a file the command did not change:

  • A command that is not a reader blocks for another session's file under a directory it names (gofmt -l internal/le).
  • A reader under an assignment is not taken for a reader (LC_ALL=C grep -rn x internal/le, GIT_PAGER=cat git log -- internal/le). This is deliberate: the check cannot know what the variable changes.
  • On a line that is not read-only, a word inside ${x:-word}, (( )), [[ ]] or a case pattern is a candidate like any other.

The slot a lint holds

./le go lint run claims the lint label in the shared job registry (internal/le/job) before it plans anything. A lint uses cores allocated for the whole machine, so admission decides how many run at once, and it prints [lint] waiting: <holder> running (pid N, Ns elapsed) while it queues. The banner also names the entry and log, the log's silence, and its last nonempty line. A quiet-window diagnostic does not establish that a job is stuck: the holder remains occupied and only its owner can stop it. An unreadable log has no stage or measured silence to report. An unreadable entry or an unsuccessful process query keeps capacity occupied.

A second session asking for the SAME work over the SAME inputs does not queue. It attaches: it replays the running lint's output, takes that run's verdict, and the tree is linted once for both. [lint] attaching to the lint already running over these inputs is that answer. A full run and a scoped run are different work, so they never share a verdict.

The inputs are the ones a lint READS, and not the whole checkout. Nine sessions write this checkout. A journal row or a spec edit that one of them made while your lint runs used to void the match, and the tree was linted a second time. Go source, .golangci.yml, go.mod, go.sum and vendor/modules.txt each void a share. A file under .claude/, ai/, backups/, docs/, plan/, rfc/ or website/ does not.

That list is declared once, as lintIgnores beside job.LintLabel (internal/le/job/treehash.go), and it EXCLUDES rather than includes. An input nobody listed still voids the share, so a missing entry costs a duplicate run and never a stale verdict. Before admission, each build's single go list -deps -test query supplies selected JSON metadata for both the authored direct lint scope and the ordinary/test dependency compiler population. A real source in an excluded tree, an excluded or out-of-checkout symlink target, or a source-resolution error refuses admission rather than sharing a verdict that did not measure its inputs. Generated test-main source is a separate compiler role: the planner matches its main package to the selected tested package by import path and directory, not by cache location. It neither joins an absolute generated file beneath that directory nor promotes the harness or its dependencies into direct lint scope. Authored internal and external test files remain direct lint inputs.

The findings a shared verdict names are read back out of the replay, so an attached red still says which files it was about. A red that names no file is charged to every commit in the checkout.

Each flavor pass lints only the packages holding a file the two passes above do not load. That package set is DERIVED from the tree with go list on every run. A hand-written list drifts the moment somebody adds a //go:build debug file in a new package, and the drift is silent.

The consequence is that a flavor pass is a lint pass, not a compile check. A package whose files every target shares, such as internal/test/runner or internal/core/diskspace, is linted once under the host GOOS and never type-checked under FreeBSD or OpenBSD, where syscall.Rlimit is signed and Statfs_t names its fields F_bavail and F_bsize. ./le go vet-platforms is the check that does compile it: it vets ./cmd/... ./internal/... ./pkg/... with the shipped personality tags under darwin, freebsd and openbsd, and Thomas's release check, ./le verify worktree, runs all three. The split is deliberate: linting every package again under each BSD would repeat the host pass's findings to type-check what one go vet per target already type-checks.

A pass that stops before it lints is not a pass with findings. golangci-lint exits 1 when it reports findings; any other non-zero exit (a package load that failed because go was missing from PATH, a timeout, a configuration it could not read) or a child that never started means that pass checked nothing. The driver names those passes on their own line, lint: N pass(es) did not complete and linted nothing, and records the cause in the pass's error field, so a stopped pass never reads as one more red with findings.

The driver then asserts coverage. Every tracked Go file must be loaded by some pass. The exceptions are vendor/, gokrazy/modcache/, and the //go:build ignore files that belong to no build. ./le go lint run executes the native plan and returns its flavor rows as structured output; ./le go lint run retains the established target interface.

Two files are still outside it, and the driver names both on every run. examples/plugin/go/main.go, which is a separate Go module. And tools.go, whose imports are programs rather than packages.

The //go:build !ze_<feature> compile-out stubs -- the code an operator reaches when a feature is OFF -- were a third population until 2026-08-24. No pass could select one, because --build-tags only ADDS to the config's list. The compile-out row reaches every stub: it runs against a derived copy of .golangci.yml carrying no build tags at all (tagless_config), so the command line is the whole tag set and that set is ze_core. One row is enough because the gates are independent, so a build with none of them on satisfies every negated term at once. A feature-only helper must therefore carry its consumer's build constraint. Without it, the bare-core build reports the helper as unused, which is what it is in a binary that compiles its only caller out.

The accounting every gate owes its population

The lint driver above is the shape every gate is held to, and internal/le/population is that shape as a callable type. A gate states what it governs, what it walked, and a reason for each member it deliberately skipped. Claim.Assess then answers a Coverage. It goes red on two things: a member neither walked nor excused, and an excuse that has stopped being needed.

Both halves matter. The first is the gate covering less than it says. The second is a stated exception nobody rechecked, which hides the next member to land on that same path. An empty population is an error rather than a clean report, because a walk that found nothing prints what a healthy tree prints.

A count floor -- refuse below N members -- is NOT this. It catches only the empty case, and comparing sizes where the question is about sets is the same defect one level up. plan/journal/gate-excludes-part-of-its-population.md records both.

The exemptions are a population too

A gate that either scans a member or exempts it balances by construction. Its own accounting proves nothing, because every member it read fell into one of the two buckets by definition. population.Exemptions asks the other question: of the rules the gate declares, which ones still do work?

A rule earns its place only where it SUPPRESSED something. The weaker reading, that a file exists under it, stays true for a rule whose file stopped doing the thing it was excused for. So the walk scans the exempt file and then drops the result, rather than skipping it, and credits the rule only when it dropped something.

iface-resolution, plugin-boundary and fs-persistence each split Check from CheckCheckout for this. Over a fixture, a rule that suppresses nothing means the tree does not hold that code. Over the real checkout it means an exemption nobody rechecked. Only the caller knows which tree it passed.

An empty rule set is a clean answer, not the refusal an empty population gets. The two empties are different facts. A walked population that came back empty is a walk that found nothing, which looks like a healthy tree. A rule set was written empty, in source a reader can see.

Feature-tag structural type check

./le go staticcheck check type-checks the working tree in N+2 configurations derived from the N unique features in feature-gates.txt: one distro all-on row, one bare-core row, and one row that omits each feature. Staticcheck includes selected _test.go files. This stage type-checks those files without running their tests.

The matrix guarantees these direct single-feature omissions. It makes no guarantee for arbitrary combinations with two or more omitted features.

Inside a verify run the stage judges only the rows the change set can move: the distro all-on and bare-core rows, plus one row per feature tag the change reached. Typing the target yourself judges every row, because only a verify run publishes the feature-tag answer it scopes by. What widens the scope back to every row is ../architecture/testing/verify-freshness-scope.md.

A verify run also CUTS those rows. It runs six stages, check part 1 of 6 through check part 6 of 6, and each one judges the rows dealt to its piece. CI deals the stage list to its shards round robin, so the six pieces run on six shards rather than on one job's clock. Each piece names the rows it judged in its own log. Rerun one piece, or the whole matrix:

./le go staticcheck check part 3 of 6
./le go staticcheck check

The matrix checks package and test variants in the working tree. ./le repo compiles check remains the committed-tree final-link check for shipped build flavors.

The one stage that does not read your working tree

Every stage above compiles and runs the files on your disk, uncommitted ones included. ./le repo compiles check (internal/le/repo/compiles.Answer) is the exception: it extracts the commit with git archive and compiles the extracted tree, so it sees only what git holds.

That is the population that breaks when a commit takes a consumer and leaves its producer uncommitted. The build is green on your disk and red for everybody who clones. Run it after the commit script when the commit carried Go:

./le repo compiles check
REV=7abe8a07e ./le repo compiles check

The action builds every flavor in internal/le/repo/compiles/matrix.go over ./.... Each row pins its tags, operating system where required, and a tag-gated anchor file that proves the flavor selected code. Naming the package alone is insufficient because go build ./... can skip every constrained file and still exit zero. About 45 seconds warm. It does not compile _test.go files, because go build never does.

Output is captured to tmp/ze-verify.log. On failure:

grep -E "^--- FAIL|^FAIL|TEST FAILURE" tmp/ze-verify.log

Interpreting output

Unit test failures

Standard Go test output. Look for --- FAIL: TestName lines:

--- FAIL: TestParseOrigin (0.00s)
    origin_test.go:42: expected 0, got 1
FAIL	github.com/ze-software/ze/internal/core/bgp/attribute	0.003s

Functional test failures

le test prints a summary at the end of each suite. Look for the test name and the expectation that failed:

FAIL  encode/addpath.ci
  expect=bgp:conn=1:seq=1:hex=...
  got: FFFF...0038...

Fuzz failures

Go places the failing input in testdata/fuzz/<TestName>/ under the package. The file contains the input that triggered the crash. Fix the code, then the fuzz corpus entry becomes a regression test automatically.

A test that exists is not a test that gates

A .ci or .et can pass whether or not the feature works when the observed effect reaches the assertion by a path OTHER than the one under test. That is a FALSE PASS. Three redistribute-late-join*.ci tests kept passing with the late-join replay (handleReplayBatch) disabled: the route reached the peer by some other path, so they guarded nothing and shipped green.

Mutation testing through the Go gomu binary runs unit tests only; it never executes .ci or .et. Proving that a functional test gates is therefore the discipline this page has always prescribed: disable the producing function, confirm the test flips to red, and revert.

For an RFC-tagged carrier that walk now has a runner and a record. ./le rfc record ... route revert producer <path>::<Func> disables the named function, runs ONE .ci or ONE interop scenario, requires the red, and puts the producer back byte for byte. What it writes is a rfc/discrimination/<stem>/<requirement-id>.json record that ./le rfc check re-verifies on every run, so the red is remembered rather than done once and forgotten. An untagged .ci still owes the walk by hand.

When a behavior genuinely cannot be made to fail under mutation because it is not observable end to end (the reactor suppresses a duplicate announce, so per-peer targeting is wire-indistinguishable), guard it with a unit test that inspects the producing value directly and say so in the test comment. Do not keep a .ci that passes with the feature disabled.

Boundary values for common ranges

Every numeric range is tested at three points: the last valid value, the first invalid value below, and the first invalid value above.

Range Last valid Invalid below Invalid above
Port 1-65535 65535 0 65536
Hold time 0, 3+ 0, 3 1, 2 none
Prefix IPv4 0-32 32 none 33
Message length 19-4096 4096 18 4097

Schema-driven web tests

Finder tests check rendered navigation against the live schema and peer rows against the configured tree, rather than pinning the whole page's HTML. They exercise authentication, edit permissions, full-page and HTMX responses, links, CLI context and escaped field values. A newly registered schema container must not require a snapshot rewrite; omitting a configured peer must still fail.

Flake shapes seen in this repository

Check each of these against the test before investigating a new race or isolation flake: locked-write with unlocked-read, subscribe-before-broadcast, gate-handler queue state, barrier FIFO order, cleanup-drains-work, a fixed port behind an SO_REUSEPORT gate, and colliding test-fake pool IDs.

Symptom Root cause Fix
Port reuse race in reactor tests Stop() not waiting for cleanup Ensure cleanup goroutines complete before returning
Completion test fails intermittently A real bug, not a flake Check completeShowPath includes YANG schema children
Inter-message timing in plugin tests Sleep too tight under load Increase the inter-message delay, or synchronize

Keep portable protocol assertions running on every platform. An unsupported platform service is not a reason to skip authentication or lifecycle checks. The PPP NCP lifetime fixtures require the authenticated principal, negotiated address and address-before-session-up ordering everywhere. They require no IPv6 backend call from the non-Linux stub; Linux additionally requires the negotiated link-local address pair.

Reproducing a load-dependent flake is ../architecture/testing/runner-architecture.md.

Cheat sheet

I want to... Run
Check my setup ./le setup check, then ./le test unit pkg path ./internal/core/textbuf/
Test one package ./le test unit pkg path ./internal/core/textbuf/; add vet to vet it instead
Run one Go test ./le test unit pkg path ./internal/core/textbuf/ run TestName
Test the packages my change reaches ./le test unit pkg changed
Run one functional test ./le test bgp plugin 42
Run a component group ./le test unit bgp
Judge my committed change set before a push the scoped package tests and functional suites ./le repo changed scope print both names; GitHub CI verifies the pushed commit, and no agent runs ./le verify
Have the release check run: GitHub CI runs it on a pushed commit, and Thomas runs it at his terminal ./le verify current
Type-check every supported feature combination ./le go staticcheck check
List native test actions ./le help
List functional tests ./le test bgp encode --list
Run one fuzz target FUZZ=FuzzName PKG=./path/... TIME=30s ./le test fuzz run
Run all fuzz targets ./le test fuzz run
Check the commit compiles ./le repo compiles check
Check web behavior ./le test functional web
Run every package test that starts a real server container (docker tag) ./le test integration docker-peers
List, then remove, Docker resources a killed lab run left behind ./le test integration clean dry, then ./le test integration clean
Check that every *_templ.go matches its .templ source ./le doc check templ-output. No action writes the output back (./le repo update carries no templ stage), so bring it back in step with the checker's own invocation minus -check: ./le job run label templ-gen command go run -mod=vendor github.com/a-h/templ/cmd/templ generate -keep-orphaned-files -path "$PWD/internal". Both walk internal/ only. Never run a bare templ generate, and switch off an editor's on-save templ integration: a bare run walks from the repo root, writes that root into every generated file, and reds the gate. ./le