Testing Guide

First-time setup

./le setup install
./le verify current mode full

./le is the compiled repository development entry point. Each area exposes its own verbs, for example ./le functional plugin, ./le qemu install-iso-test, and ./le test-unit bgp. Legacy command names have no compatibility aliases.

Action words describe the contract: check is a read-only verdict, verify is a composite policy gate, report is advisory output, and write or update changes tracked state.

The escalation ladder

Use the narrowest test that covers your change. Escalate only when needed.

./le verify current mode full is the pre-commit gate. The narrower commands are development tools. ./le job run admits an individual heavy command through internal/le/job, so concurrent sessions do not oversubscribe the machine.

Component groups for step 3

Pick the group matching your change. Each group is a subset of all, so a green group says nothing about the rest of the tree.

The installer group runs the tests on Linux. On another host go test cross-compiles a Linux binary it cannot start, so the group runs go vet and type-checks the same files instead. ./le qemu all-tests runs them for real inside the Alpine virtual machine.

Test types

Ze has four levels of testing, each covering a different concern.

Unit tests (*_test.go)

Standard Go tests. They validate logic in isolation: does the parser produce the right output, does the state machine transition correctly, does the encoder round-trip.

./le job run label bgp-message command go test ./internal/component/bgp/message/...
./le job run label parse-origin command go test ./internal/... -run TestParseOrigin
./le verify current mode full

Functional tests (.ci files)

These spin up real Ze processes and test behavior end-to-end: does the config parse, does the BGP session establish, does the plugin produce the right output.

Each test/ subdirectory has its own runner and format:

Run a single test by one-based ID or exact name, list the available IDs, or resume from the last printed ID after an interrupted run. Queue the runner through ./le job run label <label> command <argv...> to use the same admission as a full suite.

./le job run label plugin-42 command bin/ze-test bgp plugin 42          # test id 42
./le job run label encode-list command bin/ze-test bgp encode --list    # list N/TOTAL, id, and name
./le job run label plugin-from-42 command bin/ze-test bgp plugin --start 42  # id 42 and every later test
./le job run label editor-7 command bin/ze-test editor 7                # editor test id 7
./le job run label editor-nav command bin/ze-test editor -p nav         # editor tests matching "nav"
./le job run label exabgp-from-20 command bin/ze-test exabgp --start 20 # resume ExaBGP compatibility

Run a full suite:

./le functional encode
./le functional plugin
./le functional gating
./le functional exabgp-test

Mutation tests (gomu)

Coverage tells you which lines ran. Mutation testing tells you whether the tests would notice if those lines did something different. gomu rewrites the AST (arithmetic, conditional, logical, bitwise, branch, return value, and error handling operators), runs the test suite against each mutation, and reports which mutations survived.

gomu uses overlay-based execution, so it never modifies source files on disk. It is a Go tool recorded in tools.go. The native ./le mutation combine command combines completed reports, and ./le mutation record-history appends their per-package scores to the committed history.

Mutation testing is advisory. It never gates ./le verify current mode full or CI. A surviving mutant is a signal that a test could be stronger, not a blocking failure.

One consumer reads a gomu report for a narrower question. ./le rfc discriminate <selector> report <path> scopes the report to ONE RFC-tagged test: it keeps the KILLED mutants that lie in code that test's own coverage profile executes, and offers them as candidate breaks. ./le rfc discriminate-record then applies one, runs that test alone, and records the red it observed. gomu's status stays unchanged, because nothing in ./le rfc check starts a mutation run: the gate reads a stored record. See docs/contributing/rfc-conformance-gates.md, "The discrimination record".

Files with custom build tags and cmd/ze/ are excluded via .gomuignore because gomu has no --tags support. Reports land in tmp/ (gitignored). Mutation score history is tracked in test/mutation/history.ndjson.

gomu is vendored and runs through go run, so it needs no install.

Tuning is by flag on gomu run, not by environment variable. gomu reads only GITHUB_TOKEN and GITHUB_REPOSITORY, both for its GitHub integration.

Interop and integration tests

These require external infrastructure (Docker, root/CAP_NET_ADMIN, QEMU, or internet). They are not part of the normal development cycle.

./le integration interop
./le integration iface
./le qemu all-tests
./le integration live-rpki

When a test must be weakened

A red test means the CODE is wrong by default. Fix the code. When the coverage is genuinely gone, because the feature it proved was removed or another test now proves it, the removal is recorded rather than silent.

The record is test/weakened/<session>.md, the ledger shard your own commit session owns, where <session> is the eight hex characters ./le commit session prints. It holds one row per weakened test, | Test | Reason |. No other session reads your shard and none can write it, so nobody has to be careful about anybody else's rows.

Write the rows this commit owes and leave the rest to the gate: ./le commit create drops every row your shard holds whose text git already has at HEAD, because a row whose commit landed explains a diff history holds. Git history is where past rows are read, so git log -p -- test/weakened/ shows each one beside the change it accepted.

The route, in order:

1. Write the row first. The native write-edit hook reads the shard from disk, so a row added after the edit takes effect only after the edit is retried.
2. Make the edit.
3. Name test/weakened/<session>.md in the commit. internal/le/commit.Answer refuses a commit that weakens a test and leaves the row in the working tree.

./le test-weakened check prints every session's shard and the rows in it, and names yours. Read it when you want to know what the ledger holds without preparing a commit.

The test name is the enclosing top-level func TestXxx for Go, and the file stem for a .ci or a .et. Write package.TestName when the bare name matches two weakened tests in one commit.

An edit that only lowers a COUNT lands with a notice and no row: consolidating three cases into one table lowers a count exactly as deleting a check does. The COMMIT still needs a row for it, and that row is where you say which of the two happened.

./le test-weakened check runs the checker over the file and is a stage of ./le verify current mode full in both modes. The rule is ai/rules/testing.md, and the design is docs/architecture/testing/test-health.md.

How ./le verify current mode full works

./le verify current mode full is the pre-commit gate. It uses a two-pass strategy to stay fast:

1. Lint (27 linters via golangci-lint)
2. Vet evidence (cross-compile the Go evidence packages for Linux)
3. Cached full pass (go test without -race): Go caches by source hash, so when nothing changed this is instant. Catches logic regressions everywhere.
4. Changed-group pass: uses the test-only CGO_ENABLED=1 go test -race path on Linux and Darwin. Its test binaries are never release/build evidence.
5. Functional tests from internal/le/functional/catalog.go
6. ExaBGP compatibility

Common case (one group changed): ~2 min total instead of 6+.

The builds the linter reads

golangci-lint analyzes ONE build for each run: one GOOS, one GOARCH, one tag set. ./le verify lint run therefore runs more than one.

Waiting for the linter's own lock

golangci-lint takes one lock for the whole machine, at $TMPDIR/golangci-lint.lock. Several sessions share this checkout, so a second linter is normal here. Every command line Ze builds therefore carries --allow-serial-runners, which makes the child WAIT for that lock. Without it the child gives up after five seconds and exits parallel golangci-lint is running, and the stage turns that into a red that names no file.

A pass that waits prints nothing while it waits. A whole-tree pass holds the lock for about ten minutes on this workstation, so a pass behind one can stay silent for that long. That is the linter in a queue, not a hang.

The slot a lint holds

./le verify lint run claims the lint label in the shared job registry (internal/le/job) before it plans anything. A lint uses cores allocated for the whole machine, so admission decides how many run at once, and it prints [lint] waiting: <holder> running (pid N, Ns elapsed): <the holder's last log line> while it queues. That last line is the holder's current stage, read from the log the holder is writing (reportBusy, internal/le/job/registry.go), so a waiter can tell a run that is progressing from one that is stuck. A holder with no readable log gets the banner without it.

A second session asking for the SAME work over the SAME inputs does not queue. It attaches: it replays the running lint's output, takes that run's verdict, and the tree is linted once for both. [lint] attaching to the lint already running over these inputs is that answer. A full run and a scoped run are different work, so they never share a verdict.

The inputs are the ones a lint READS, and not the whole checkout. Nine sessions write this checkout. A journal row or a spec edit that one of them made while your lint runs used to void the match, and the tree was linted a second time. Go source, .golangci.yml, go.mod, go.sum and vendor/modules.txt each void a share. A file under .claude/, ai/, backups/, docs/, plan/, rfc/ or website/ does not.

That list is declared once, as lintIgnores beside job.LintLabel (internal/le/job/treehash.go), and it EXCLUDES rather than includes. An input nobody listed still voids the share, so a missing entry costs a duplicate run and never a stale verdict. A tree is on the list because it holds no Go file at all, and a //go:embed pattern cannot leave the directory of the package that writes it, so nothing under such a tree can reach a Go package.

The findings a shared verdict names are read back out of the replay, so an attached red still says which files it was about. A red that names no file is charged to every commit in the checkout.

Each flavor pass lints only the packages holding a file the two passes above do not load. That package set is DERIVED from the tree with go list on every run. A hand-written list drifts the moment somebody adds a //go:build debug file in a new package, and the drift is silent.

The driver then asserts coverage. Every tracked Go file must be loaded by some pass. The exceptions are vendor/, gokrazy/modcache/, and the //go:build ignore files that belong to no build. ./le verify lint run executes the native plan and returns its flavor rows as structured output; ./le verify lint run retains the established target interface.

Two files are still outside it, and the driver names both on every run. examples/plugin/go/main.go, which is a separate Go module. And tools.go, whose imports are programs rather than packages.

The //go:build !ze_<feature> compile-out stubs -- the code an operator reaches when a feature is OFF -- were a third population until 2026-08-24. No pass could select one, because --build-tags only ADDS to the config's list. The compile-out row reaches every stub: it runs against a derived copy of .golangci.yml carrying no build tags at all (tagless_config), so the command line is the whole tag set and that set is ze_core. One row is enough because the gates are independent, so a build with none of them on satisfies every negated term at once. A feature-only helper must therefore carry its consumer's build constraint. Without it, the bare-core build reports the helper as unused, which is what it is in a binary that compiles its only caller out.

The accounting every gate owes its population

The lint driver above is the shape every gate is held to, and internal/le/population is that shape as a callable type. A gate states what it governs, what it walked, and a reason for each member it deliberately skipped. Claim.Assess then answers a Coverage. It goes red on two things: a member neither walked nor excused, and an excuse that has stopped being needed.

Both halves matter. The first is the gate covering less than it says. The second is a stated exception nobody rechecked, which hides the next member to land on that same path. An empty population is an error rather than a clean report, because a walk that found nothing prints what a healthy tree prints.

A count floor -- refuse below N members -- is NOT this. It catches only the empty case, and comparing sizes where the question is about sets is the same defect one level up. plan/journal/gate-excludes-part-of-its-population.md records both.

The exemptions are a population too

A gate that either scans a member or exempts it balances by construction. Its own accounting proves nothing, because every member it read fell into one of the two buckets by definition. population.Exemptions asks the other question: of the rules the gate declares, which ones still do work?

A rule earns its place only where it SUPPRESSED something. The weaker reading, that a file exists under it, stays true for a rule whose file stopped doing the thing it was excused for. So the walk scans the exempt file and then drops the result, rather than skipping it, and credits the rule only when it dropped something.

iface-resolution, plugin-boundary and fs-persistence each split Check from CheckCheckout for this. Over a fixture, a rule that suppresses nothing means the tree does not hold that code. Over the real checkout it means an exemption nobody rechecked. Only the caller knows which tree it passed.

An empty rule set is a clean answer, not the refusal an empty population gets. The two empties are different facts. A walked population that came back empty is a walk that found nothing, which looks like a healthy tree. A rule set was written empty, in source a reader can see.

Feature-tag structural type check

./le staticcheck-feature-matrix check type-checks the working tree in N+2 configurations derived from the N unique features in feature-gates.txt: one distro all-on row, one bare-core row, and one row that omits each feature. Staticcheck includes selected _test.go files. This stage type-checks those files without running their tests.

The matrix guarantees these direct single-feature omissions. It makes no guarantee for arbitrary combinations with two or more omitted features.

Inside a verify run the stage judges only the rows the change set can move: the distro all-on and bare-core rows, plus one row per feature tag the change reached. Typing the target yourself judges every row, because only a verify run publishes the feature-tag answer it scopes by. What widens the scope back to every row is ../architecture/testing/verify-freshness-scope.md.

A verify run also CUTS those rows. It runs six stages, check part 1 of 6 through check part 6 of 6, and each one judges the rows dealt to its piece. CI deals the stage list to its shards round robin, so the six pieces run on six shards rather than on one job's clock. Each piece names the rows it judged in its own log. Rerun one piece, or the whole matrix:

./le staticcheck-feature-matrix check part 3 of 6
./le staticcheck-feature-matrix check

The matrix checks package and test variants in the working tree. ./le repository tracked-build check remains the committed-tree final-link check for shipped build flavors.

The one stage that does not read your working tree

Every stage above compiles and runs the files on your disk, uncommitted ones included. ./le repository tracked-build check (internal/le/repository/trackedbuild.Answer) is the exception: it extracts the commit with git archive and compiles the extracted tree, so it sees only what git holds.

That is the population that breaks when a commit takes a consumer and leaves its producer uncommitted. The build is green on your disk and red for everybody who clones. Run it after the commit script when the commit carried Go:

./le repository tracked-build check
REV=7abe8a07e ./le repository tracked-build check

The action builds every flavor in internal/le/repository/trackedbuild/matrix.go over ./.... Each row pins its tags, operating system where required, and a tag-gated anchor file that proves the flavor selected code. Naming the package alone is insufficient because go build ./... can skip every constrained file and still exit zero. About 45 seconds warm. It does not compile _test.go files, because go build never does.

Output is captured to tmp/ze-verify.log. On failure:

grep -E "^--- FAIL|^FAIL|TEST FAILURE" tmp/ze-verify.log

Interpreting output

Unit test failures

Standard Go test output. Look for --- FAIL: TestName lines:

--- FAIL: TestParseOrigin (0.00s)
    origin_test.go:42: expected 0, got 1
FAIL	github.com/ze-software/ze/internal/core/bgp/attribute	0.003s

Functional test failures

ze-test prints a summary at the end of each suite. Look for the test name and the expectation that failed:

FAIL  encode/addpath.ci
  expect=bgp:conn=1:seq=1:hex=...
  got: FFFF...0038...

Fuzz failures

Go places the failing input in testdata/fuzz/<TestName>/ under the package. The file contains the input that triggered the crash. Fix the code, then the fuzz corpus entry becomes a regression test automatically.

A test that exists is not a test that gates

A .ci or .et can pass whether or not the feature works when the observed effect reaches the assertion by a path OTHER than the one under test. That is a FALSE PASS. Three redistribute-late-join*.ci tests kept passing with the late-join replay (handleReplayBatch) disabled: the route reached the peer by some other path, so they guarded nothing and shipped green.

Mutation testing through the Go gomu binary runs unit tests only; it never executes .ci or .et. Proving that a functional test gates is therefore the discipline this page has always prescribed: disable the producing function, confirm the test flips to red, and revert.

For an RFC-tagged carrier that walk now has a runner and a record. ./le rfc discriminate-record ... route revert producer <path>::<Func> disables the named function, runs ONE .ci or ONE interop scenario, requires the red, and puts the producer back byte for byte. What it writes is a rfc/discrimination/<stem>.json record that ./le rfc check re-verifies on every run, so the red is remembered rather than done once and forgotten. An untagged .ci still owes the walk by hand.

When a behavior genuinely cannot be made to fail under mutation because it is not observable end to end (the reactor suppresses a duplicate announce, so per-peer targeting is wire-indistinguishable), guard it with a unit test that inspects the producing value directly and say so in the test comment. Do not keep a .ci that passes with the feature disabled.

Boundary values for common ranges

Every numeric range is tested at three points: the last valid value, the first invalid value below, and the first invalid value above.

Flake shapes seen in this repository

Check each of these against the test before investigating a new race or isolation flake: locked-write with unlocked-read, subscribe-before-broadcast, gate-handler queue state, barrier FIFO order, cleanup-drains-work, a fixed port behind an SO_REUSEPORT gate, and colliding test-fake pool IDs.

A darwin FAIL caused by a _other.go stub returning ErrUnsupported is a test-setup bug rather than a real failure. Keep the failure list meaningful.

Reproducing a load-dependent flake is ../architecture/testing/runner-architecture.md.

Cheat sheet