Testing Guide
This guide covers how to run tests, what the different test types are, and how to
interpret their output. For the full technical reference (.ci format spec, .et
directives, fuzz target list), see docs/functional-tests.md.
First-time setup
./le setup install
./le setup check
./le test unit pkg path ./internal/core/textbuf/
./le setup check reports every tool setup installs, and the scoped package
test proves the checkout builds and its tests run under the feature tags
(../guide/developer-setup.md, "After Setup").
./le verify current is the full verification, Thomas's release check, which
GitHub CI also runs on a pushed commit. It runs only when stdin and stdout are
both terminals, so Thomas starts it by hand at one. No agent runs any
./le verify command, ./le verify changed included: the evidence for a
change is the scoped test runs of what it reaches. The main thread runs them in
the main checkout, from the owed commands a subagent lists in its report
(ai/rules/commands.md).
./le is the compiled repository development entry point. Each area exposes
its own verbs, for example ./le test functional plugin,
./le test qemu installer-iso, and ./le test unit bgp. Legacy command names
have no compatibility aliases.
Action words describe the contract: check is a read-only verdict, verify is
a composite policy gate, report is advisory output, and write or update
changes tracked state.
The escalation ladder
Use the narrowest test that covers your change. Escalate only when needed.
| Step | What you run | When | Time |
|---|---|---|---|
| 1 | ./le test unit pkg path ./internal/core/textbuf/ run TestName |
Iterating on one test | seconds |
| 2 | ./le test unit bgp |
Checking one component group | 10s to 1:30 |
| 3 | ./le test functional plugin |
Checking the user-visible path | varies |
| 4 | GitHub CI on the pushed commit | Judging the pushed change set: no agent runs it locally | minutes |
| 5 | ./le verify current |
The release check, which only Thomas starts, at a terminal | the longest |
Steps 1 to 3 are development: run the ones that cover what you changed, and
./le repo changed scope print both names them
(running-commands.md, "The changed-set selector"). Run them in the main
checkout, never in a worktree or a copy of the tree. Step 4 is CI's: it runs
once the work is pushed, and ./le commit push asks for no local verify
record. A red it finds gets a fix commit; it does not hold the commit it
judged. ./le verify changed still exists for Thomas, and no agent runs it.
Step 5 is the full verification, and it is not an agent's to run. The dispatcher refuses ./le verify current, ./le verify current fast and every form of ./le verify worktree
unless stdin and stdout are both terminals, because a full run
loads the whole shared machine and only Thomas knows when that is safe. The
refusal prints the exact command line for Thomas, and the scoped tests that
are the change's evidence, which the main thread runs. No keyword or environment variable
skips it. When Thomas starts ./le verify current, it runs every stage in a fresh detached worktree
at HEAD, through the same lifecycle as ./le verify worktree, so an edit made
while it runs cannot move the tree it judges; ./le verify current fast runs
the changed-files stages in place, over the working tree.
The narrower commands are development tools. ./le test unit pkg admits its
run through internal/le/job, and ./le job run admits any other heavy
command the same way, so concurrent sessions do not oversubscribe the machine.
One package: ./le test unit pkg
./le test unit pkg path <dir>/ [run <regex>] [vet] tests the packages it
names under the feature tags from feature-gates.txt, with the race detector
and the toolchain's timeout, admitted under the job label unit-pkg. Repeat
path for each package. run hands its regex to go test -run. vet runs
go vet on the same packages and their test files instead, under vet-pkg.
changed in place of path takes the packages the change reaches.
The output goes to this session's scratch log. A green run answers one line,
test unit pkg: exit 0, <n> packages ok, ..., log <path>, which counts only
the packages that ran a test. With changed, a left out <package> line
comes before it for each reached package that changed set aside behind a
group's own tags, naming the group's command. A red run prints each failed test's
--- FAIL: line with the lines beneath it, then that summary. A run in which
no package ran a test, a run that matched nothing included, answers exit 1.
It refuses with exit 2, before it takes a slot, a line that would test more
than the packages meant: path ./... (the whole tree, ./le test unit all,
is the owner's run), a group's whole pattern (run that group), a value that
spans two or more groups, and a package under the installer group, whose
ze_installer tag the feature set lacks (run ./le test unit installer). A
changed answer the selector widened to every package is refused the same
way. The race detector is always on, and the action takes no other go test
flag: a run that needs -v, -count, -bench or a cover profile stays on
./le job run label unit-pkg quiet command go test -race -tags "$tags" ...
with the tags written out (running-commands.md, "Testing one package while
you develop it").
Component groups for step 3
| Command | Scope | Time |
|---|---|---|
./le test unit bgp |
BGP engine, wire, reactor | about 1:30 |
./le test unit core |
Core libraries | about 30 seconds |
./le test unit plugins |
All plugins | about 40 seconds |
./le test unit config |
Config parsing and YANG | about 20 seconds |
./le test unit cli |
CLI component | about 10 seconds |
./le test unit installer |
Installer initrd logic behind the ze_installer tag |
about 10 seconds |
./le test unit all |
The whole checkout, then the groups their build tags hide from it | minutes: it is every package under -race |
Pick the group matching your change. Each group is a subset of all, so a
green group says nothing about the rest of the tree.
The installer group runs the tests on Linux. On another host go test
cross-compiles a Linux binary it cannot start, so the group runs go vet
and type-checks the same files instead. ./le test qemu all runs them for
real inside the Alpine virtual machine.
Test types
Ze has four levels of testing, each covering a different concern.
Unit tests (*_test.go)
Standard Go tests. They validate logic in isolation: does the parser produce the right output, does the state machine transition correctly, does the encoder round-trip.
./le test unit pkg path ./internal/component/bgp/message/...
./le test unit pkg path ./internal/component/bgp/route/ run TestParseOrigin
Dynamic-group coverage exercises each plugin's real configuration boundary: capability declarations, policy lookups, validation, and command output. Opposite group and peer values expose lost inheritance; same-name peers and groups expose namespace collisions. A list of caller filenames or helper names proves neither. For example, the redirect tests pass plugin JSON through the RIB parser before querying the session's effective setting.
Kernel tests in a user namespace
A unit test that needs kernel privilege (links, routes, XFRM, nftables,
conntrack, raw or AF_PACKET sockets, a low port, a named network namespace)
runs unprivileged through userns.Enter (internal/test/userns). Enter
re-runs the one test as root over a user, network and mount namespace of its
own, with lo up and a private /run/netns, so nothing it creates reaches
the host. Write such a test this way:
if !userns.Enter(t) { return }is the first statement of each top-level test that reaches a privileged operation. Never call it from a helper or a subtest: the child selects the test by its top-level name. A helper that needs privilege is reached through theEnterof every top-level test that calls it.- The test never calls
t.Parallel, because the child re-runs that one named test in a fresh process. - A refused operation inside the namespace fails the test with the operation
and the errno, and never skips: the namespace grants the privilege, so a
refusal is a defect or an unmet need. A skip that names an absent kernel
feature or module (for example
CONFIG_MPLS_ROUTING) stays a skip. - Every child startup error fails the parent, namespace denial included. The
message carries the error and names
./le setup installand itsuserns-unrestrictedcheck, which lifts the AppArmor restriction (developer setup). Only a child that ran and reported its test skipped skips the parent. - The file's build tag is
linux, so every Linux unit run and the hosted CI unit job execute it. Keepintegration && linuxonly when the file runs a binary./le setupdoes not install (nft), needs a host grant (/dev/ppp, throughppp-access), takes an explicitly suppliedzeexecutable, or needs a privilege only the initial user namespace holds. Say which in the file's header comment.
Run the evidence as your own user through the launcher, for example
./le job run label unit-pkg quiet command go test -v -count=1 -race -tags "$tags" -run '^TestName$' ./internal/core/routewatch,
and read --- PASS for each test in the quiet log. It keeps the long form
because the short action takes no -v, so $tags comes from the recipe in
running-commands.md, "A bare go test is not ./le test unit". Off Linux, Enter returns true at once and
the test runs in place, which serves a portable test such as the traceroute
command tests. The namespace mechanics are in
QEMU integration testing,
"Build Tags".
Fixture environment isolation
Native Go test commands remove the launching le's ze.repo.root and
ze.le.build.name values under every case/dot/underscore spelling. The test
process discovers its checkout from its working directory; a fixture can then
set its own root without an inherited canonical spelling winning after a cache
reset. Toolchain caches, tags and unrelated environment settings remain intact.
Go test callers using gotoolchain.Toolchain.Environment MUST set
EnvOptions.Test, including fuzz and benchmark runs. Ordinary builds and
production commands retain operator checkout overrides and named-build checks.
A go test that ./le job run starts as its direct child, and the one
./le test unit pkg starts through it, use the same isolation
at the process boundary, including Go's optional leading -C directory option.
The admission process retains its root and job identity; only the test child's
checkout/build identity is removed. The test child also gets GOTMPDIR set to
the test temp dir on the home device, as every le-started test does
(running-commands.md, "Where a Go test keeps its temp
dirs").
Tests that invoke fixture binaries remove inherited launcher identity before adding their explicit child environment. The production named-build refusal is not disabled: a child given an explicit mismatching identity still fails. Tests that run git-changing actions MUST verify the resolved root is their own temporary fixture before invoking those actions. They MUST NOT use the developer's checkout or linked worktrees as mutation fixtures.
Verifier lifecycle fixtures need a real Git repository with HEAD, even when
they inject the lifecycle's Git calls. Admission measures the actual checkout
before the worktree operation: an empty directory fails that measurement
instead of exercising the intended branch, timeout, or add refusal.
CLI personality parity runs both real binaries against one isolated Git
repository with a committed original, a tracked modification, and an untracked
file. The fixture copies the checkout's real .gitignore before committing, so
runtime scratch files follow the same ignore policy. Both binaries receive the
same explicit fixture root. Stdout equality remains exact; stderr keeps only
the existing program-name normalization. Concurrent developer edits are not
fixture inputs.
Functional tests (.ci files)
These spin up real Ze processes and test behavior end-to-end: does the config parse, does the BGP session establish, does the plugin produce the right output.
Each test/ subdirectory has its own runner and format:
| Directory | What it tests | Runner |
|---|---|---|
test/encode/ |
BGP wire encoding | le test bgp encode |
test/decode/ |
Wire decoding | le test bgp decode |
test/parse/ |
Config parsing (valid/invalid) | le test bgp parse |
test/plugin/ |
Plugin behavior | le test bgp plugin |
test/reload/ |
Config reload | le test bgp reload |
test/ui/ |
CLI completion | le test ui |
test/editor/ |
TUI editor (.et files) |
le test editor |
test/managed/ |
Managed config | le test managed |
test/web/ |
Web UI | le test web |
test/l2tp/ |
L2TP daemon | le test l2tp |
test/firewall/ |
Firewall | le test firewall |
test/policy/ |
Policy routing | le test policy |
test/exabgp-compat/ |
ExaBGP compatibility | le test exabgp |
Run a single test by one-based ID or exact name, list the available IDs, or
resume from the last printed ID after an interrupted run. Each of these
runners admits itself through the same job admission as a full suite, so it
needs no ./le job run around it (running-commands.md, "One verify at a
time").
./le test bgp plugin 42 # test id 42
./le test bgp encode --list # list N/TOTAL, id, and name
./le test bgp plugin --start 42 # id 42 and every later test
./le test editor 7 # editor test id 7
./le test editor -p nav # editor tests matching "nav"
./le test exabgp --start 20 # resume ExaBGP compatibility
Run a full suite:
./le test functional encode
./le test functional plugin
./le test functional gating
./le test functional exabgp
Mutation tests (gomu)
Coverage tells you which lines ran. Mutation testing tells you whether the tests would notice if those lines did something different. gomu rewrites the AST (arithmetic, conditional, logical, bitwise, branch, return value, and error handling operators), runs the test suite against each mutation, and reports which mutations survived.
gomu uses overlay-based execution, so it never modifies source files on disk.
It is a Go tool recorded in tools.go. The native ./le test mutation combine
command combines completed reports, and ./le test mutation record appends
their per-package scores to the committed history.
Mutation testing is advisory. It never gates Thomas's release check, ./le verify current, or CI. A surviving mutant
is a signal that a test could be stronger, not a blocking failure.
One consumer reads a gomu report for a narrower question. ./le rfc discriminate <selector> report <path> scopes the report to ONE RFC-tagged test: it keeps the
KILLED mutants that lie in code that test's own coverage profile executes, and
offers them as candidate breaks. ./le rfc record then applies one,
runs that test alone, and records the red it observed. gomu's status stays
unchanged, because nothing in ./le rfc check starts a mutation run: the gate
reads a stored record. See docs/contributing/rfc-conformance-gates.md, "The
discrimination record".
Files with custom build tags and cmd/ze/ are excluded via .gomuignore because
gomu has no --tags support. Reports land in tmp/ (gitignored). Mutation score
history is tracked in test/mutation/history.ndjson.
gomu is vendored and runs through go run, so it needs no install.
| Command | Purpose |
|---|---|
go run github.com/sivchari/gomu/cmd/gomu run --output json --incremental=false --fail-on-gate=false |
Full advisory mutation run |
go run github.com/sivchari/gomu/cmd/gomu run --output json --incremental --base-branch=main --fail-on-gate=false |
Changed-file advisory mutation run |
./le test mutation combine |
Combine the per-package JSON reports |
Tuning is by flag on gomu run, not by environment variable. gomu reads only
GITHUB_TOKEN and GITHUB_REPOSITORY, both for its GitHub integration.
| Flag | Default | Effect |
|---|---|---|
--workers |
4 | Parallel workers |
--timeout |
30 | Test timeout in seconds |
--incremental |
true | Analyse changed files only |
--base-branch |
main |
Base branch for the incremental analysis |
--fail-on-gate |
true | Fail the build when the quality gate is not met |
--output |
console |
console, json, html, or text |
Interop and integration tests
These require external infrastructure (Docker, root/CAP_NET_ADMIN, QEMU, or internet). They are not part of the normal development cycle.
./le test integration interop
./le test integration iface
./le test qemu ci test test/<dir>/<name>.ci # one needs-linux test in Ze's QEMU guest
./le test integration live-rpki
./le test integration docker-peers # every package test tagged `docker`
./le test integration clean dry # what a killed lab run left, and why the rest stays
./le test integration clean # remove the lab resources whose owner is gone
A package test that starts a real peer container carries //go:build docker,
never docker and integration together: the integration population runs in
the QEMU guest, which has no Docker. docker-peers derives its packages from
the tree and sets ZE_REQUIRE_DOCKER=1. Such a test starts its server with
labtest.Start(t, spec) from internal/le/interoplab/labtest, never with its
own exec.Command("docker"), which ./le arch tier check refuses outside
internal/le/interoplab; the contract is docs/architecture/testing/interop.md,
"A package test's own server container (labtest)".
When a test must be weakened
A red test means the CODE is wrong by default. Fix the code. When the coverage is genuinely gone, because the feature it proved was removed or another test now proves it, the removal is recorded rather than silent.
A test weakening is approved BEFORE the edit by an independent agent running
the ze-test-judge skill (ai/skills/ze-test-judge.md), never by Thomas and
never by the agent that wrote the change. The judge approves only when the new
test proves at least what the old one did, or when the old expectation is wrong
against its source; otherwise it refuses, and the refusal stands. The edit hook
refuses the weakening, names the test, and prints the command that records the
verdict.
The route, in order:
- Hand the test, the diff and your reason to a fresh agent running
ze-test-judge. - On APPROVE the judge records its verdict, for one test at a time:
./le test weakened approve unit <package>.<TestName> reason "<the judge's verdict>". The row goes totmp/commit-weakening-approved-<session>.md, keyed on the main thread's commit namespace, so a subagent's edit sees it too. git never reads that file.<package>is the name of the directory holding the test, not its Go package clause; the command refuses a unit no directory carries and names the key to use. When two files of the directory declare the name, it refuses the bare unit and prints the form that names the file:./le test weakened approve unit <package>.<Name> file <path> reason "<the judge's verdict>".fileis accepted only then: on a unit one file declares, the command refuses it and prints the form without it. The file lives only in the approval row; the commit's trailer names the unit alone. The row records the unit's content as the judge read it, so it approves that change only: a different later change of the same test needs a new verdict (docs/architecture/testing/test-health.md, "One approval names one test"). On REFUSE, fix the code instead. - Make the edit.
- Commit with
./le commit createas usual. The message gains oneWeakening-approved: <package>.<TestName>: <the judge's verdict>line per test, and the script drops each used row once the commit lands. A row a landed commit already used approves nothing further.
The unit is the name of the directory holding the file, a dot, then the
enclosing top-level func TestXxx, or the file stem for a .ci or a .et. A bare name, a path or a
directory is refused: one approval names one test. An RFC-approved: row for a
unit (./le rfc approve) also accepts that unit's weakening. Because the unit
keeps only the directory's base name, a commit that changes cmd.TestX in both
a/cmd and b/cmd is refused: split it so each commit carries one of them.
A helper is any function of a _test.go file that is not a Test, Fuzz or
Benchmark function (TestMain counts as a helper). A weakening inside a
helper (an assertion it no longer makes) is the helper's own finding: the row
names the helper, <package>.<helperName>, because a calling test's verdict
judged the test's text, not the helper's. A changed helper also changes every
existing test that calls it in a file the commit carries, so that test's live
row is spent and a stale one refuses. A test calls a helper when it names the
helper as a call or a function value (never after a selector's dot), directly
or through other helpers, in the _test.go files of the same directory. The
rule matches names, so a local of the same name also counts as a call. A
helper no test calls keeps its finding all the same.
A weakening is the loss of proof the committed tree holds. The edit hook
judges only the units HEAD records for the file. It follows a committed test
file that the working tree lacks into the other test files of the same
directory: an Edit of a file moved there (by git mv or by mv) is judged
against the committed units. A Go test file at a path HEAD does not hold is
judged, on the Write that creates it and on every later whole-file Write
or Edit, against the committed files missing from its directory, for each
unit it declares and the edit changes under the name of one the missing
file held, so a stub written first cannot carry a weaker copy in later. A
renamed file whose units are renamed too is caught only at the commit gate.
A Write that
recreates a deleted committed file is judged against HEAD's text of it. An
alias of a committed test (a symlink, a hard link in its directory, or a case
variant of its name on a case-insensitive filesystem) is judged as the
committed test. A test or helper that only the working tree holds, in a new
untracked file or added to a tracked one, is yours to rename, replace or
remove without a verdict, unless it shares a name with a unit of a committed
file missing from its directory. While a committed .ci is missing from a
directory, every untracked .ci there is guarded against losing what its own
text proves, which can refuse an edit to an unrelated new .ci, never admit
a loss. When a refusal comes from a committed file missing from the
directory, it names that file: have an independent judge approve the change,
or, if the two files are unrelated, stop and report it, so that whoever
deleted the file commits the deletion first. A unit the edit leaves as it was
is not judged again; it was judged only if a Write or Edit the hook sees
wrote it after the committed file went missing, or through a link to the
committed file. The hook does not see these, and the commit gate judges
them as an addition beside a deletion, or as a deletion it cannot pair:
- a move to another directory, and a new
.ciwritten under another name - an edit through a hard link made in another directory
- a unit written by a shell command (a redirect,
sed -i,cp, or a draft moved into place withmv): the Bash hook guards shell writes only underplan/andai/rules/ - a unit written while its committed namesake still stood, that file deleted afterwards
- a weaker copy of a moved unit added to a file HEAD already tracks, while a committed sibling is missing: a tracked file is judged against its own HEAD text only
The rules are in docs/architecture/testing/test-health.md, "One approval
names one test".
A test edited again after its verdict is refused at commit, even when the new edit weakens nothing: the row is stale, because the judge saw other text. Run the judge again on the change as it stands and record its fresh verdict, or remove the stale row when the change needs none.
The writer resolves the directory key against current and HEAD carriers, and refuses an unknown or ambiguous key before it changes an approval file. The registered action checks the unit and the reason in a read-only preparation before startup, so a refused request writes nothing. Its admitted answer creates the commit namespace and writes the row.
Known limit: two kinds of unit cannot be approved at all, because the unit must match
<package>.<TestName> with one dot and no empty side. A .ci or .et file
whose stem holds a dot (x.y.ci is suite.x.y), and a test file at the
repository root (its package side is empty), are refused by the approve writer
and by every reader.
An edit that only lowers a COUNT lands with a notice and no approval: consolidating three cases into one table lowers a count exactly as deleting a check does. The COMMIT still needs an approval for it, and the judge's verdict is where the commit says which of the two happened.
Git history is where past approvals are read: git log --grep '^Weakening-approved:' lists each one beside the change it accepted, and
./le test weakened check reads the same lines when it reviews a commit. The
rule is ai/rules/testing.md, and the design is
docs/architecture/testing/test-health.md.
How Thomas's release check works
When Thomas starts ./le verify current, it runs the full population in a fresh detached worktree at
HEAD; ./le verify current fast runs the changed-files population in place.
Both are release checks that start only at a terminal (see the ladder above).
./le verify changed runs the changed population in a throwaway worktree; Thomas
may start it, and no agent does.
./le verify list full and ./le verify list fast print each population. The
unit stages use a two-pass strategy to stay fast:
- Lint (27 linters via golangci-lint)
- Vet evidence (cross-compile the Go evidence packages for Linux)
- Cached full pass (
go testwithout-race): Go caches by source hash, so when nothing changed this is instant. Catches logic regressions everywhere. - Changed-group pass: uses the test-only
CGO_ENABLED=1 go test -racepath on Linux and Darwin. Its test binaries are never release/build evidence. - Functional tests from
internal/le/test/functional/catalog.go - ExaBGP compatibility
Common case (one group changed): ~2 min total instead of 6+.
The builds the linter reads
golangci-lint analyzes ONE build for each run: one GOOS, one GOARCH, one tag
set. ./le go lint run therefore runs more than one.
| Pass | Build | What only it reads |
|---|---|---|
| 1 | the host GOOS, .golangci.yml tags |
the shipped daemon |
| 2 | GOOS=linux, plus integration |
every kernel-facing //go:build integration test |
| 3..N | one for each row of FLAVORS. internal/le/go/lint.Answer |
ze_installer, ze_distro, ze_appliance, ze_setup, tinygo, and the capability tags (debug, race, live, ...). Also the GOOS and GOARCH targets no other pass compiles: darwin, freebsd, openbsd, dragonfly, wasip1, linux/arm64 and linux/riscv64. Also the compile-out build, which drops every feature gate and keeps ze_core alone |
Waiting for the linter's own lock
golangci-lint takes one lock for the whole machine, at
$TMPDIR/golangci-lint.lock. Several sessions share this checkout, so a second
linter is normal here. Every command line Ze builds therefore carries
--allow-serial-runners, which makes the child WAIT for that lock. Without it
the child gives up after five seconds and exits parallel golangci-lint is running, and the stage turns that into a red that names no file.
A pass that waits prints nothing while it waits. A whole-tree pass holds the lock for about ten minutes on this workstation, so a pass behind one can stay silent for that long. That is the linter in a queue, not a hang.
The post-edit lint takes no index lock
After each Go edit the post-write hook lints the edited package with
--new-from-rev=HEAD --timeout=30s. For --new-from-rev, golangci-lint runs
git status and git diff itself, and its timeout SIGKILLs them. A git killed
while it refreshed the index left an empty .git/index.lock that no process
held, and every session's next git write failed until someone removed it. The
hook therefore runs the linter with GIT_OPTIONAL_LOCKS=0, which stops
git status refreshing the index, and diff.autoRefreshIndex=false, which
stops git diff writing the refresh back. The patch the linter reads is the
same either way. le's other git status queries under a deadline carry the
first setting through gitenv.ReadOnly. The weakening check's queries, which
the edit hook runs on every test edit, carry gitenv.ReadOnlyQuery: the
first setting plus maintenance.auto=false and gc.auto=0, so none of them
starts a detached maintenance run under .git. The post-edit hook starts the real linter, not
the golangci-lint shim, so the kill at the hook's 60 second budget ends the linter
itself and no le exec waits for a job slot on an edit (postLinter).
Post-tool hooks block only on the author's own files
A post-tool hook's blocking text enters the agent's context, and every later turn re-reads it. Several sessions share this checkout, so a check that judged every uncommitted file blocked each session on the others' work. Each check therefore blocks only on what the tool call itself touched:
| Check | Blocks on | Said at code 0, to the person only |
|---|---|---|
| Post-edit lint | An issue in the edited file, or a package that does not build (* typecheck: N in the summary) |
The count of issues in other changed files of the package |
| Post-command gofmt | An unformatted Go file the command names, by path or by a directory holding it | The count of unformatted files it does not name |
When any check blocks, the hook drops the code-0 hints from what it prints, because a hint carries nothing for the agent to act on.
The gofmt check reads what a command names from its parse as bash, never from its text. A file is named when a whole word of the line is its path or a directory above it:
./, a trailing slash and an absolute path under the checkout are spelling.- Go's
dir/...namesdir. - A word that is one unquoted glob (
pkg/*.go) names what it matches, and*does not cross a directory. - Any word that starts with
-and holds=names what follows the first=(--file=pkg/x.go). - The checkout itself (
.,./..., its absolute path) and a path outside it name nothing.
A read-only line names nothing, because it cannot have changed a file. A line
is read-only when every command on it is a reader (grep, cat, git log,
the set in readerCommands), a command inside $(...) included, and no
redirection may write a Go file of the checkout, which is the only way a line
of readers changes one:
| Redirection target | Ends the exemption |
|---|---|
Outside the checkout (2>/dev/null), or a descriptor (2>&1, >&-) |
✕ |
A known file that does not end in .go (> tmp/out.txt, 2> tmp/err.log) |
✕ |
A file that ends in .go, or a glob |
✓ |
Not known: an expansion (> "$out"), or a relative path that climbs with .. (> ../x.go) |
✓ |
The exemption is all or nothing: cd pkg && cat > x.go <<EOF is not
read-only, so the word of cd names the file, as it does in cd pkg && sed -i ... x.go.
A command that does not parse names nothing, and the code-0 line says the
command could not be read and how to check what it changed. So does a line
whose tree names fewer commands than bash runs (treeMisreadsLine, read by
commandUnread in internal/le/hookruntime/postbash.go): a comment that ends
in a backslash, which the parser joins to the next line while bash runs that
line, and a $ followed by a backslash-newline. Such a tree would read
echo a # x \ then gofmt -w x.go as one read-only echo, so the check does
not read it at all.
The limits that remain. In each case the file is counted at code 0 and does not block:
- A word bash expands other than as a plain glob names nothing: a parameter
(
pkg/$name/x.go), a command substitution, arithmetic, a tilde, braces, or a glob that also carries a quote or a backslash. The check cannot know what bash passes for it. - A path inside a longer word is not seen:
dd of=x.go, an attached short option value (-ox.go), the script ofbash -c '...'. - A path that arrives through a variable or
xargsis not seen. - A relative word is read against the checkout root, not against the shell's
directory after an earlier
cdon the line. Only the word ofcdnames, and a relative word that climbs with..names nothing. - An absolute path is compared as it is spelled, so one that reaches the checkout through a symlink is read as outside it.
- A redirection target that starts with
-and holds no=(cat > -x.go) is read as an option word and names nothing, so such a line stays read-only. - A glob that
path.Matchrefuses (an unmatched[) is compared as it is spelled, so a file it would reach by matching is not named.
Some limits go the other way, and block for a file the command did not change:
- A command that is not a reader blocks for another session's file under a
directory it names (
gofmt -l internal/le). - A reader under an assignment is not taken for a reader (
LC_ALL=C grep -rn x internal/le,GIT_PAGER=cat git log -- internal/le). This is deliberate: the check cannot know what the variable changes. - On a line that is not read-only, a word inside
${x:-word},(( )),[[ ]]or a case pattern is a candidate like any other.
The slot a lint holds
./le go lint run claims the lint label in the shared job registry
(internal/le/job) before it plans anything. A lint uses cores allocated for
the whole machine, so admission decides how many run at once, and it prints
[lint] waiting: <holder> running (pid N, Ns elapsed) while it queues.
The banner also names the entry and log, the log's silence, and its last
nonempty line. A quiet-window diagnostic does not establish that a job is
stuck: the holder remains occupied and only its owner can stop it.
An unreadable log has no stage or measured silence to report.
An unreadable entry or an unsuccessful process query keeps capacity occupied.
A second session asking for the SAME work over the SAME inputs does not queue.
It attaches: it replays the running lint's output, takes that run's verdict, and
the tree is linted once for both. [lint] attaching to the lint already running over these inputs is that answer. A full run and a scoped run are different
work, so they never share a verdict.
The inputs are the ones a lint READS, and not the whole checkout. Nine sessions
write this checkout. A journal row or a spec edit that one of them made while
your lint runs used to void the match, and the tree was linted a second time. Go
source, .golangci.yml, go.mod, go.sum and vendor/modules.txt each void a
share. A file under .claude/, ai/, backups/, docs/, plan/, rfc/ or
website/ does not.
That list is declared once, as lintIgnores beside job.LintLabel
(internal/le/job/treehash.go), and it EXCLUDES rather than includes. An input
nobody listed still voids the share, so a missing entry costs a duplicate run
and never a stale verdict. Before admission, each build's single
go list -deps -test query supplies selected JSON metadata for both the authored
direct lint scope and the ordinary/test dependency compiler population. A real source in an excluded tree,
an excluded or out-of-checkout symlink target, or a source-resolution error
refuses admission rather than sharing a verdict that did not measure its inputs.
Generated test-main source is a separate compiler role: the planner matches its
main package to the selected tested package by import path and directory, not
by cache location. It neither joins an absolute generated file beneath that
directory nor promotes the harness or its dependencies into direct lint scope.
Authored internal and external test files remain direct lint inputs.
The findings a shared verdict names are read back out of the replay, so an attached red still says which files it was about. A red that names no file is charged to every commit in the checkout.
Each flavor pass lints only the packages holding a file the two passes above do
not load. That package set is DERIVED from the tree with go list on every run.
A hand-written list drifts the moment somebody adds a //go:build debug
file in a new package, and the drift is silent.
The consequence is that a flavor pass is a lint pass, not a compile check. A
package whose files every target shares, such as internal/test/runner or
internal/core/diskspace, is linted once under the host GOOS and never
type-checked under FreeBSD or OpenBSD, where syscall.Rlimit is signed and
Statfs_t names its fields F_bavail and F_bsize. ./le go vet-platforms
is the check that does compile it: it vets ./cmd/... ./internal/... ./pkg/...
with the shipped personality tags under darwin, freebsd and openbsd, and
Thomas's release check, ./le verify worktree, runs all three. The split is deliberate: linting every
package again under each BSD would repeat the host pass's findings to type-check
what one go vet per target already type-checks.
A pass that stops before it lints is not a pass with findings. golangci-lint
exits 1 when it reports findings; any other non-zero exit (a package load that
failed because go was missing from PATH, a timeout, a configuration it
could not read) or a child that never started means that pass checked nothing.
The driver names those passes on their own line, lint: N pass(es) did not complete and linted nothing, and records the cause in the pass's error
field, so a stopped pass never reads as one more red with findings.
The driver then asserts coverage. Every tracked Go file must be loaded by some
pass. The exceptions are vendor/, gokrazy/modcache/, and the //go:build ignore files that belong to no build. ./le go lint run executes the
native plan and returns its flavor rows as structured output; ./le go lint run
retains the established target interface.
Two files are still outside it, and the driver names both on every run.
examples/plugin/go/main.go, which is a separate Go module. And tools.go,
whose imports are programs rather than packages.
The //go:build !ze_<feature> compile-out stubs -- the code an operator reaches
when a feature is OFF -- were a third population until 2026-08-24. No pass could
select one, because --build-tags only ADDS to the config's list. The
compile-out row reaches every stub: it runs against a derived copy of
.golangci.yml carrying no build tags at all (tagless_config), so the command
line is the whole tag set and that set is ze_core. One row is enough because
the gates are independent, so a build with none of them on satisfies every
negated term at once. A feature-only helper must therefore carry its consumer's
build constraint. Without it, the bare-core build reports the helper as
unused, which is what it is in a binary that compiles its only caller out.
The accounting every gate owes its population
The lint driver above is the shape every gate is held to, and
internal/le/population is that shape as a callable type. A gate states what it
governs, what it walked, and a reason for each member it deliberately skipped.
Claim.Assess then answers a Coverage. It goes red on two things: a member
neither walked nor excused, and an excuse that has stopped being needed.
Both halves matter. The first is the gate covering less than it says. The second is a stated exception nobody rechecked, which hides the next member to land on that same path. An empty population is an error rather than a clean report, because a walk that found nothing prints what a healthy tree prints.
A count floor -- refuse below N members -- is NOT this. It catches only the empty
case, and comparing sizes where the question is about sets is the same defect one
level up. plan/journal/gate-excludes-part-of-its-population.md records both.
The exemptions are a population too
A gate that either scans a member or exempts it balances by construction. Its own
accounting proves nothing, because every member it read fell into one of the two
buckets by definition. population.Exemptions asks the other question: of the
rules the gate declares, which ones still do work?
A rule earns its place only where it SUPPRESSED something. The weaker reading, that a file exists under it, stays true for a rule whose file stopped doing the thing it was excused for. So the walk scans the exempt file and then drops the result, rather than skipping it, and credits the rule only when it dropped something.
iface-resolution, plugin-boundary and fs-persistence each split Check
from CheckCheckout for this. Over a fixture, a rule that suppresses nothing
means the tree does not hold that code. Over the real checkout it means an
exemption nobody rechecked. Only the caller knows which tree it passed.
An empty rule set is a clean answer, not the refusal an empty population gets. The two empties are different facts. A walked population that came back empty is a walk that found nothing, which looks like a healthy tree. A rule set was written empty, in source a reader can see.
Feature-tag structural type check
./le go staticcheck check type-checks the working tree in N+2
configurations derived from the N unique features in feature-gates.txt: one
distro all-on row, one bare-core row, and one row that omits each feature.
Staticcheck includes selected _test.go files. This stage type-checks those
files without running their tests.
The matrix guarantees these direct single-feature omissions. It makes no guarantee for arbitrary combinations with two or more omitted features.
Inside a verify run the stage judges only the rows the change set can move: the
distro all-on and bare-core rows, plus one row per feature tag the change
reached. Typing the target yourself judges every row, because only a verify run
publishes the feature-tag answer it scopes by. What widens the scope back to
every row is ../architecture/testing/verify-freshness-scope.md.
A verify run also CUTS those rows. It runs six stages,
check part 1 of 6 through check part 6 of 6, and each one judges the rows
dealt to its piece. CI deals the stage list to its shards round robin, so the
six pieces run on six shards rather than on one job's clock. Each piece names
the rows it judged in its own log. Rerun one piece, or the whole matrix:
./le go staticcheck check part 3 of 6
./le go staticcheck check
The matrix checks package and test variants in the working tree.
./le repo compiles check remains the committed-tree final-link check for shipped
build flavors.
The one stage that does not read your working tree
Every stage above compiles and runs the files on your disk, uncommitted ones
included. ./le repo compiles check (internal/le/repo/compiles.Answer) is the
exception: it extracts the commit with git archive and compiles the extracted
tree, so it sees only what git holds.
That is the population that breaks when a commit takes a consumer and leaves its producer uncommitted. The build is green on your disk and red for everybody who clones. Run it after the commit script when the commit carried Go:
./le repo compiles check
REV=7abe8a07e ./le repo compiles check
The action builds every flavor in internal/le/repo/compiles/matrix.go over
./.... Each row pins its tags, operating system where required, and a
tag-gated anchor file that proves the flavor selected code. Naming the package
alone is insufficient because go build ./... can skip every constrained file
and still exit zero.
About 45 seconds warm. It does not compile _test.go files, because go build
never does.
Output is captured to tmp/ze-verify.log. On failure:
grep -E "^--- FAIL|^FAIL|TEST FAILURE" tmp/ze-verify.log
Interpreting output
Unit test failures
Standard Go test output. Look for --- FAIL: TestName lines:
--- FAIL: TestParseOrigin (0.00s)
origin_test.go:42: expected 0, got 1
FAIL github.com/ze-software/ze/internal/core/bgp/attribute 0.003s
Functional test failures
le test prints a summary at the end of each suite. Look for the test name
and the expectation that failed:
FAIL encode/addpath.ci
expect=bgp:conn=1:seq=1:hex=...
got: FFFF...0038...
Fuzz failures
Go places the failing input in testdata/fuzz/<TestName>/ under the package.
The file contains the input that triggered the crash. Fix the code, then the
fuzz corpus entry becomes a regression test automatically.
A test that exists is not a test that gates
A .ci or .et can pass whether or not the feature works when the observed
effect reaches the assertion by a path OTHER than the one under test. That is a
FALSE PASS. Three redistribute-late-join*.ci tests kept passing with the
late-join replay (handleReplayBatch) disabled: the route reached the peer by
some other path, so they guarded nothing and shipped green.
Mutation testing through the Go gomu binary runs unit tests only; it never
executes .ci or .et. Proving that a functional test gates is therefore the
discipline this page has always prescribed: disable the producing function,
confirm the test flips to red, and revert.
For an RFC-tagged carrier that walk now has a runner and a record.
./le rfc record ... route revert producer <path>::<Func> disables
the named function, runs ONE .ci or ONE interop scenario, requires the red, and
puts the producer back byte for byte. What it writes is a
rfc/discrimination/<stem>/<requirement-id>.json record that ./le rfc check re-verifies on every
run, so the red is remembered rather than done once and forgotten. An untagged
.ci still owes the walk by hand.
When a behavior genuinely cannot be made to fail under mutation because it is
not observable end to end (the reactor suppresses a duplicate announce, so
per-peer targeting is wire-indistinguishable), guard it with a unit test that
inspects the producing value directly and say so in the test comment. Do not keep
a .ci that passes with the feature disabled.
Boundary values for common ranges
Every numeric range is tested at three points: the last valid value, the first invalid value below, and the first invalid value above.
| Range | Last valid | Invalid below | Invalid above |
|---|---|---|---|
| Port 1-65535 | 65535 | 0 | 65536 |
| Hold time 0, 3+ | 0, 3 | 1, 2 | none |
| Prefix IPv4 0-32 | 32 | none | 33 |
| Message length 19-4096 | 4096 | 18 | 4097 |
Schema-driven web tests
Finder tests check rendered navigation against the live schema and peer rows against the configured tree, rather than pinning the whole page's HTML. They exercise authentication, edit permissions, full-page and HTMX responses, links, CLI context and escaped field values. A newly registered schema container must not require a snapshot rewrite; omitting a configured peer must still fail.
Flake shapes seen in this repository
Check each of these against the test before investigating a new race or
isolation flake: locked-write with unlocked-read, subscribe-before-broadcast,
gate-handler queue state, barrier FIFO order, cleanup-drains-work, a fixed port
behind an SO_REUSEPORT gate, and colliding test-fake pool IDs.
| Symptom | Root cause | Fix |
|---|---|---|
| Port reuse race in reactor tests | Stop() not waiting for cleanup |
Ensure cleanup goroutines complete before returning |
| Completion test fails intermittently | A real bug, not a flake | Check completeShowPath includes YANG schema children |
| Inter-message timing in plugin tests | Sleep too tight under load | Increase the inter-message delay, or synchronize |
Keep portable protocol assertions running on every platform. An unsupported platform service is not a reason to skip authentication or lifecycle checks. The PPP NCP lifetime fixtures require the authenticated principal, negotiated address and address-before-session-up ordering everywhere. They require no IPv6 backend call from the non-Linux stub; Linux additionally requires the negotiated link-local address pair.
Reproducing a load-dependent flake is
../architecture/testing/runner-architecture.md.
Cheat sheet
| I want to... | Run |
|---|---|
| Check my setup | ./le setup check, then ./le test unit pkg path ./internal/core/textbuf/ |
| Test one package | ./le test unit pkg path ./internal/core/textbuf/; add vet to vet it instead |
| Run one Go test | ./le test unit pkg path ./internal/core/textbuf/ run TestName |
| Test the packages my change reaches | ./le test unit pkg changed |
| Run one functional test | ./le test bgp plugin 42 |
| Run a component group | ./le test unit bgp |
| Judge my committed change set before a push | the scoped package tests and functional suites ./le repo changed scope print both names; GitHub CI verifies the pushed commit, and no agent runs ./le verify |
| Have the release check run: GitHub CI runs it on a pushed commit, and Thomas runs it at his terminal | ./le verify current |
| Type-check every supported feature combination | ./le go staticcheck check |
| List native test actions | ./le help |
| List functional tests | ./le test bgp encode --list |
| Run one fuzz target | FUZZ=FuzzName PKG=./path/... TIME=30s ./le test fuzz run |
| Run all fuzz targets | ./le test fuzz run |
| Check the commit compiles | ./le repo compiles check |
| Check web behavior | ./le test functional web |
Run every package test that starts a real server container (docker tag) |
./le test integration docker-peers |
| List, then remove, Docker resources a killed lab run left behind | ./le test integration clean dry, then ./le test integration clean |
Check that every *_templ.go matches its .templ source |
./le doc check templ-output. No action writes the output back (./le repo update carries no templ stage), so bring it back in step with the checker's own invocation minus -check: ./le job run label templ-gen command go run -mod=vendor github.com/a-h/templ/cmd/templ generate -keep-orphaned-files -path "$PWD/internal". Both walk internal/ only. Never run a bare templ generate, and switch off an editor's on-save templ integration: a bare run walks from the repo root, writes that root into every generated file, and reds the gate. ./le |