Architecture

.ci Test File Format

The .ci format is used by Ze's test runner to define functional tests. It supports embedded files (Tmpfs), test options, expectations, and commands.

For the execution architecture (how tests are scheduled and run concurrently) and the web .wb format, see runner-architecture.md.

Syntax Overview

All lines use key=value format with : separators:

action=type:key=value:key=value:...
Action Purpose
stdin= Embed stdin content for processes
tmpfs= Embed file content inline
option= Test configuration
cmd= Commands (API, shell, foreground/background)
expect= Expectations to validate
await= Block until the daemon's stderr carries every awaited line, then tear down (deterministic fence); then=stop makes the runner stop the daemon itself
reject= Negative expectations (fail if matched)
action= Actions (send notification, raw bytes)
http= HTTP endpoint checks and readiness polls

Key Concepts

An unparseable test file fails; it never hides or vanishes

A file that does not parse is recorded as a permanent failure and discovery continues. It is never dropped, and it never aborts the rest of the directory. All three discoverers behave identically.

Discoverer Format Marker
EncodingTests.Discover .ci, suites registered by harnesstool.SuiteAnswer and the bgp runner (encode, plugin, ui, ...) Record.ParseFailed + State=StateFail + FailureType=parse_error
ParsingTests.Discover .ci (parse suite) parsingTest.ParseError
ParsingTests.Discover legacy + with a companion .expect. valid/*.confinvalid/*.conf parsingTest.ParseError
DecodingTests.Discover .ci and .test (decode suite) decodingTest.ParseError

The legacy .conf layout has no instances in the tree today, and it is held to the same contract anyway: a missing, empty, or bad-regex .expect file records that one fixture as a failure rather than abandoning the directory. An unreachable abort becomes reachable the moment someone adds the directory, and the shape is the bug.

Both alternatives are silent-coverage-loss bugs this project has already paid for, which is why neither is permitted:

Anti-pattern What it costs
Return the parse error out of Discover The whole directory is abandoned. One bad .ci made the test/ui suite discover and run ZERO tests, and a suite that runs nothing reads as green.
continue past the bad file The file leaves the suite with no warning and no failure record. Its coverage disappears and nothing says so.

A guard that neither denies nor speaks does not exist (ai/rules/evidence.md). The runner short-circuits a parse-failed test before executing anything, so the reported error is the parse error rather than a confusing downstream symptom.

Unparseable outranks skipped. A file that both fails to parse and carries a skip marker (option=needs-linux, option=skip-os, option=needs-arch) is reported FAIL, not SKIP. Its skip marker was parsed from the same broken file, so it is not trustworthy evidence that the file need not run: a contradicting directive may sit past the break, or the marker itself may be what was mis-parsed. Honoring it would mean trusting a broken file's own claim that it can be ignored.

The consequences are asymmetric, which is what settles the ordering. 158 .ci files carry one of those markers (12 in test/ui), and on a non-Linux host they never run. A wrongly-SKIPPED malformed file is invisible indefinitely, which is exactly how test/ui rotted. A wrongly-FAILED one is loud and costs a single commit. The check therefore lives in parallel.go's per-test goroutine, ahead of the skip short-circuit: that is the real entry point, and both Runner.runTest and parsingRunner.runTest are reached through it.

Suite label, test id, and failure identity

The verify debugging protocol identifies a functional failure with:

Field Source Purpose
Suite label le test runner label such as plugin, ui, or managed First routing boundary inside ./le test functional
Test id One-based decimal id printed by --list and per-test result lines Exact single-test rerun scope
Run number N/TOTAL printed by --list and per-test result lines Human progress marker for long suites
CI file path Parsed .ci source path Full test definition and embedded fixtures
Failure kind Runner failure type, timeout state, or mismatch class Conservative grouping key
Expected / received evidence TEST FAILURE block detail Full debugging evidence in the stage log

In ZE_VERIFY_MODE=1, failed suites emit native failure-group metadata before the full failure blocks. The compact verify index uses that metadata for group routing and keeps the full TEST FAILURE blocks in the stage log.

Per-step trace output

All three runner families (.ci, .wb, .et) record per-step outcomes during execution and emit dual-format trace output:

  • Human: colored checkmark/cross glyph per step with kind, assert, and detail.
  • Machine: VERIFY STEP: {"file":"...","step":N,"kind":"...","status":"pass|fail",...} -- one JSON line per step, matching the VERIFY FAILURE GROUP: prefix convention.

Trace is emitted automatically for failed tests. Under -v, passing tests also show their trace. The .ci runner includes the trace in its TEST FAILURE report block when StepTrace is non-empty.

conn and seq

Most directives use conn=N and seq=N to identify message ordering:

  • conn (connection): 1-based TCP connection index. Each le test peer instance manages one TCP connection. Multi-peer tests use conn=1 for the first peer, conn=2 for the second, etc. The maximum is set by option=tcp_connections:value=N.
  • seq (sequence): 1-based message sequence within a connection. seq=1 is the first BGP message after OPEN/KEEPALIVE, seq=2 is the second, etc.

A test with two peers and one UPDATE each uses conn=1:seq=1 and conn=2:seq=1, not conn=1:seq=1 and conn=1:seq=2.

Port Substitution

Each test owns three consecutive ports, exposed as variables in commands, tmpfs= content, stdin blocks, option=env values, and URLs:

Variable Meaning
$PORT BGP peer port (leased by the runner, used by le test peer --port $PORT)
$PORT2 Secondary port, $PORT+1 (web UI, looking glass, plugin acceptor)
$PORT3 Third port, $PORT+2 (an RTR cache, a second listener, a session editor)

runner.ExpandPorts expands these placeholders in tmpfs= content, exec=, stdin blocks, option=env values, and HTTP check and wait URLs. Assertion values in expect= and reject= are not expanded. The config sweep in internal/component/config/cli calls the same function. Any other $PORT<n> fails the file at discovery, naming the placeholder: it would name a port the lease does not hold.

Never hardcode port numbers. Use $PORT in cmd= exec values and $PORT2 in http= URLs.

The span is LEASED when the test starts, not when the suite discovers it (runner.LeaseTestPorts, internal/test/runner/ports.go). Discovery numbers the Nth test of every suite from the same base, so the preference alone collides whenever two le test processes run at once. The lease takes per-port advisory locks in /tmp/le-port-locks-<effective-uid>, independent of TMPDIR, checkout, or session. The flat per-user directory avoids requiring access through another user's private lock directory. Directory and file creation modes are 0700 and 0600, respectively; cross-user coordination is unsupported. Lock files MUST NOT be unlinked: replacing a lock inode splits ownership between old and new users of the path.

All participating runners MUST be rebuilt for this namespace cutover. Old runners using le-port-locks under their temporary directory do not coordinate with this namespace; there is no fallback or hybrid locking mode.

While holding the locks, the runner probes 127.0.0.1 with TCP/IPv4 and ::1 with TCP/IPv6 for every port, keeping all probe listeners open until the whole span has been checked. IPv6 is omitted only when its family, protocol, or loopback address is unavailable, not when its port is occupied. Probe listeners close before children bind; the advisory locks remain until the test releases its lease. A preferred span that is locked or occupied moves to 25000-32759.

The locks coordinate participating same-user processes on the same host that share /tmp; they do not provide universal port exclusivity across users or isolated filesystems. An unrelated process can still bind after the probes close. A hardcoded number in a .ci file takes part in neither step, which is why the rule above is a rule.

There is no $PORT4 placeholder. A compiled fixture that starts more listeners than the three placeholders cover MUST reserve their ports with runner.ReservePorts or runner.LeaseTestPorts, and MUST retain the reservation until every child using those ports has stopped and been reaped. An offset from a placeholder is only a preferred allocation address, never a reservation.

The live managed-client restore fixture uses the runner's first two ports for the hub's SSH and plugin listeners and reserves a third port for its managed listener. Its hub and client configs declare no BGP peers or dynamic groups, so they bind no BGP listeners and need no BGP port offsets.

Stdin Blocks

A stdin block embeds content the runner pipes to a process's standard input. Two commands take it as a FILE instead, and the table under "Where the block goes" says which, why, and what the .ci writes to select each route.

Syntax

Multi-line (with terminator):

stdin=<name>:terminator=<TERM>
<content>
<TERM>

Single-line hex:

stdin=<name>:hex=<hex-value>

Single-line text:

stdin=<name>:text=<text-value>

Parameters

Parameter Description
name Identifier referenced by cmd=...:stdin=<name>
terminator End marker for multi-line content
hex Hex-encoded content (single-line)
text Plain text content (single-line, newline appended)
refused The diagnostic code ze config validate refuses this block with, when the test hands ze a config it must refuse (see refused under Tmpfs)

Examples

Multi-line (config):

stdin=ze-bgp:terminator=EOF_CONF
peer test-peer {
    remote {
        ip 127.0.0.1;
        as 65533;
    }
    local-as 65533;
}
EOF_CONF

cmd=foreground:seq=1:exec=ze -:stdin=ze-bgp

The block goes to a file and the daemon runs as ze start <file>. The example said exec=ze bgp server -:stdin=ze until 2026-09-07, naming a command the CLI does not have and a block the file does not declare.

Single-line hex (decode test):

stdin=payload:hex=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF003C020000001C...
cmd=foreground:seq=1:exec=ze bgp decode --json --family ipv4/unicast -:stdin=payload
expect=json:json={ "type": "update", ... }

The block is PIPED: the - belongs to decode, not to the daemon. The decode suite runs that exec= line as written, with ze replaced by the binary under test and the block piped as one line. Until 2026-10-14 it lifted the flags off the line and ran an argv of its own ending in the hex string, so the command text was never executed.

Single-line text:

stdin=cmd:text=update text next-hop 10.0.0.1 nlri add 10.0.0.0/24

Where the block goes

The runner pipes the block, with two exceptions. Each one exists because the child needs a path it can open, and each is selected by what the exec= line already writes: no new directive chooses between them.

The exec= line Where the block goes Why
ze -, and its flagged forms ze -d -, ze --plugin <p> -, ze --mcp <port> -, ze --web <port> --insecure-web - a FILE in a stable per-daemon directory, and argv becomes ze [flags] start <file> SIGHUP reads the source again and a restart reuses its tree
le test peer ... with NO - in argv a temporary FILE appended to argv le test peer takes its expect script as a path argument
le test peer ... - PIPED LoadExpectFile opens its argument through cliio, so - is standard input there
every other line, ze bgp decode -, ze config validate -, le test replay -, sh -c ... included PIPED - is the cliio stdin token, and the command reads standard input. ai/rules/cli.md

The daemon - is recognized by POSITION, not by a list of verbs: the runner asks DaemonConfigArg which argument is the config, and substitutes only when that argument is the -. A - belonging to a verb is left in argv and the block is piped, so ze config validate - and ze bgp decode pcap - test the form the operator types.

The route changes how the block travels, never what the runner reads from it. Whether a check peer lingers (option=linger:value=true, which the then=stop fence below waits on) is read from the .ci block itself with the peer's own parser, so a peer given the block as a file lingers exactly as one given it on standard input. TestPeerLingerSurvivesStdinRouting (internal/test/runner/one_off_proof_test.go) runs both routes through the fence.

The first daemon block uses ze-bgp.conf in the test work directory, so a fixture rewriting that bare name still edits the source the daemon reads. Further distinct blocks use daemon-2/ze-<block>.conf, daemon-3/…, each with its own sibling database/. Reusing a block reuses its directory. Numbered directories distinguish even block names that sanitise identically. Wrapped launches such as le test fail-syscall … -- ze - retain real stdin and receive an isolated ze.config.dir by the same allocation rule. Explicit file paths remain authoritative; the runner does not choose a backend.

Until 2026-09-07 the runner took the FIRST - in argv whatever it meant, so every verb form ran against a file path the author never wrote. A test that named a stdin block on a verb was testing the path form under a - that said otherwise, and test/ui/bgp-decode-stdin-hex.ci failed with invalid hex: encoding/hex: invalid byte: U+002F '/' for that reason.

What a ze-peer block may carry

A block handed to le test peer (named on a cmd=...:exec=le test peer ...:stdin=<name> line) is read first by ze-peer and then by the test runner. The block named peer is validated by the same rules even with no such line, because a .ci with no cmd= at all feeds its expect= lines to ze-peer by another route. A line neither of them acts on fails the file at parse time, naming the block, the line number and the directive.

When a file declares cmd= lines, every stdin=peer block must be named by a cmd=...:exec=le test peer ...:stdin=peer command. An orphan block fails parsing even without a rejection directive: its wire expectations would have no consumer. Files with no cmd= lines retain the legacy peer execution path. Peer command recognition uses the execution tokenizer: quoted command words such as "le" test 'peer' and quoted mode values retain their peer role.

Line Read by
expect=bgp, reject=bgp, action=*, and the peer's own option= (asn, bind, tcp_connections, conn_map, open, update, linger, remote_close, silent, stop-reading, await_eor, established-file, await-file) le test peer
expect=json, expect=stderr, expect=syslog, reject=stderr, reject=stdout, reject=syslog the test runner, where the line stands
cmd=api nobody. It documents the command that produced the expected bytes
option=timeout the test runner parses it, and adopts it only when the file declares none. Its scope is the whole test, not this peer, so a file-level value always wins. 450 tracked peer blocks carry one. Write the one that governs outside the block
option=env refused. It sets the environment of every process the test starts, not of this peer, so it must be written outside the block
a line neither parser accepts refused

Any OTHER directive the runner parses is accepted where it stands and applies to the whole test. option=file, expect=exit, await= and http= all work inside a peer block, and none of them is peer-scoped. Only option=env and option=timeout are singled out. Those two read as peer scope and are not.

A value the named option does not have is refused as well, not just an unknown option name. option=update:value=inspect-update-message and option=asn:value=abc each used to parse into nothing, which is the same silent drop one level down.

The accepted set is derived from the two parsers rather than written out a third time (peer.ClaimLine reports what ze-peer did with the line, and the runner's own line parser answers for the rest), so a directive added to either one cannot start being dropped here. Before the guard existed the block forwarded only expect= and action= lines and discarded everything else in silence: eleven reject=bgp directives across nine RFC-behaviour tests, and the reject=stderr of a tenth, asserted nothing while reading as the negative half of the proof.

Tmpfs (Virtual File System)

Tmpfs allows embedding multiple files within a single .ci file. Files are extracted to a temp directory at runtime.

File contents are arbitrary text or bytes, not implicitly Ze configuration. There is no special configuration meaning for a .src or .conf suffix. For example, a peer's action=rewrite may copy a human-readable acknowledgment into a completion marker without making either file a daemon configuration.

Syntax

tmpfs=<path>[:mode=<octal>][:encoding=<type>][:refused=<code>]:terminator=<TERM>
<content>
<TERM>

Parameters

Parameter Required Default Description
path ✓ - Relative path (no .., no absolute)
mode ✕ Auto File permissions (octal: 644, 755)
encoding ✕ text Content encoding: text or base64
terminator ✓ - End marker (alone on line)
refused ✕ - The diagnostic code ze config validate refuses this file with (config-parse, config-yang-reference, ...): the test feeds Ze a config it must refuse

refused= changes nothing at run time: the runner writes the file like any other. It is the test's declaration that the body is a Ze config Ze refuses on purpose. The repository config sweep (TestRepositoryConfigsValidate, internal/component/config/cli) validates every config a .ci carries, and for a declared file it asserts the opposite: the body must be refused with that code, and a declared body that validates, or is refused for another reason, is a finding.

The sweep discovers tracked files and untracked files that Git's ignore rules permit, using git ls-files --cached --others --exclude-standard. This keeps launcher exports and other ignored artifacts out while retaining tracked exceptions and new source files. The runner's draft tree remains outside the regression corpus. Discovery warnings and source read errors fail the sweep, including when a narrowed exclusion test would not select that file.

Mode Defaults

A file with no mode= takes its mode from its extension: 0755 for .py, .sh, .bash, .zsh, .pl, .rb and .run, 0644 for anything else. Use an explicit mode= when the test needs a different permission, such as an extensionless shim (wg, ip) put on PATH, which only mode=755 makes executable.

Both routes a .ci takes write the block's bytes unchanged and keep the mode the block declared: the parse suite writes the parsed tmpfs as it is, and every other suite rebuilds it per run (to expand $PORT) with the mode the record carries, never one derived again from the path. TestTmpfsContentAndModeReachBothRoutes (internal/test/runner/one_off_proof_test.go) runs an extensionless mode=755 shim and a control with no mode= through both routes and compares the bytes and the execute bit that reached disk.

Terminator Rules

  • Must be non-empty
  • Must be unique within file (no two Tmpfs blocks can share terminator)
  • Alphanumeric and underscore only: [A-Za-z0-9_]+
  • Matched exactly (no whitespace trimming)
  • Recommended: EOF_<PURPOSE> (for example, EOF_CONF)

Example

tmpfs=peer.conf:terminator=EOF_CONF
peer test-peer {
    remote {
        ip 127.0.0.1;
        as 65533;
    }
    local-as 65533;
}
EOF_CONF


option=file:path=peer.conf
option=asn:value=65533
expect=bgp:conn=1:seq=1:hex=FFFF...

Security Constraints

  1. No absolute paths - must be relative
  2. No parent traversal - no .. components
  3. No hidden files - no . prefix in path components
  4. Path length limit - max 256 characters
  5. Path depth limit - max 10 levels

Limits

Configurable via environment variables:

Limit Default Environment Variable
Max file size 1 MB ze.bgp.ci.max_file_size
Max total size 1 MB ze.bgp.ci.max_total_size
Max files 100 ze.bgp.ci.max_files
Max path length 256 ze.bgp.ci.max_path_length
Max path depth 10 ze.bgp.ci.max_path_depth

Options

A peer block may require option=remote_close:value=true when its final expectation must be followed by the remote ending the transport. The check peer waits for EOF or a connection reset before reporting success. Extra bytes, any other read error, or cancellation during runner teardown fail the check. The runner's command timeout bounds the peer if the remote never closes. It sends nothing and does not close its own connection to obtain that evidence. It requires check mode and cannot be combined with linger or conn_map. Use it after an exact shutdown NOTIFICATION expectation to prove both the protocol message and the remote close.

option=<type>:key=value[:key=value...]
Type Keys Description
file path=<name> Config file to use
asn value=<N>[:peer=<ip>] The AS ze-peer opens with. It reaches BOTH carriers RFC 6793 defines: the two-octet My Autonomous System field, narrowed to AS_TRANS (23456) above 65535, and the Capability Value of capability 65. The range is 1 to 4294967295 and a value outside it fails the file when it is read, naming the option and the value. peer=<ip> binds the declaration to one of ze's endpoint addresses, for a peer process that serves several of ze's peers at once (option=conn_map). With no option=asn line at all the runner derives one from the session { asn { remote N } } leaf of the ze configuration the .ci names. It reads a tmpfs= block, a stdin= block and the file an option=file:path= points at, and it follows the inheritance chain: the router's own bgp { session { asn {. } } }, then a group or template, then the peer, each level overriding the one above. Three things fail the file at read time rather than being guessed: two peers ze dials at ONE address expecting different ASNs, a declared AS the reader cannot read, and an eBGP peer the derivation reached with nothing. That last refusal knows only what the reader knows. A peer is judged eBGP by comparing the local and remote AS, so a peer for which NO local AS is declared at any level of the chain is not judged eBGP and is not refused; it inherits ze's own AS from the mirror, which is right for an iBGP session and wrong for an eBGP one. The derivation reaches no peer at all when a compiled fixture under internal/test/fixture writes the configuration and launches le test peer itself, because no .ci block is involved; such a fixture passes --asn (cliWirePeerAS). internal/test/fixture/ui_fixture_send_bgp.go
bind value=ipv6 Bind to IPv6
timeout value=<duration> Test timeout (e.g., 30s). Overrides auto-timeout.
tcp_connections value=<N> The number of TCP connections the peer serves. It is a BOUND, never a witness: the count says how many connections happened and not who caused them. A peer that closes when its expectations are met makes ze dial again on its retry timer, so value=2 is reached by a daemon that did nothing. To assert that the DAEMON dropped and restarted a session, add option=linger, which makes the peer incapable of causing the second connection.
linger value=true Peer-block only: the check peer never closes a connection itself. After ALL expectations complete it prints its success token and holds the session open (answering KEEPALIVEs) until test teardown. BETWEEN connections, where one connection's expectations are met and a later connection is still owed, it holds that connection until the REMOTE closes it, and fails the test if teardown comes first. Without it a completed peer closes, which ze correctly treats as session-down; that withdraws the peer's routes and races any forwarding still in flight toward other peers, and it also makes the daemon's OWN close unobservable, because ze dials again on its retry timer whoever closed (asserts a stop and a restart, and needs the peer to be incapable of causing one). A conn_map peer is excluded from the between-connections hold: it serves every connection of one batch in turn and then waits for the daemon to close them all, so holding the first would starve the batch. A reject=bgp that fires during either hold RETRACTS the success token already printed, so the peer still fails the test. test/reload/config-apply-ordering-address-swap.ci
silent value=true Peer-block only, check mode only: the peer stops sending the automatic KEEPALIVE reply it otherwise writes for every message it receives. It holds the TCP connection open and keeps reading and matching expectations. Needed to reach ze's receive hold timer: ze sends its own KEEPALIVE every hold/3 seconds, each automatic reply resets ze's hold timer, and "the peer went quiet" is otherwise unexpressible. A closed connection is a different event on a different code path, so action=close does not substitute. Explicit writes still happen: action=send, action=notification, the OPEN handshake itself, and option=linger's post-completion KEEPALIVE loop are unaffected, so silent with linger is not silent. Sink and echo modes ignore it. See. test/plugin/deadpeer-holddown.ci
stop-reading value=true[:conn=<N>] Peer-block only, check mode only: the mirror of silent. On connection <N> (counted from 1 like the conn= of an expect=bgp line, default 1) the peer reads the daemon's first UPDATE, judges it like any other frame, and then reads NOTHING more from that connection. It keeps the TCP connection open and writes a KEEPALIVE every second, so ze's receive hold timer never fires, while everything ze writes piles up unread until ze's own write blocks. Needed to reach a session whose peer stopped accepting data: a closed connection and a quiet peer are both different events. The fill does not depend on the host's default buffer sizes on the peer's side: the peer shrinks its receive buffer on that connection before the handshake, which also switches the kernel's receive autotuning off. The daemon's send buffer is still the kernel's, so the test must give ze more to send than a send buffer can autotune to (4 MiB with Linux and macOS defaults). The peer never closes that connection itself. It reads nothing, so it sees the daemon's close only on the write side: a KEEPALIVE written to a closed connection is answered with a reset and the write after it fails, within two seconds. It then accepts the next connection, which therefore exists only because the DAEMON dropped the session. If teardown comes first, the peer fails with the daemon never closed the connection the peer stopped reading. Every other connection is read normally. The first UPDATE MUST meet the last expectation connection <N> holds, because no later frame is read: an expectation left behind fails the peer by name. value=false leaves the option off; any other value, a conn= below 1 or above tcp_connections, and a combination with conn_map, silent or remote_close are refused. See. test/plugin/session-peer-stops-reading.ci
established-file path=<name> Peer-block only: ze-peer creates an empty marker file on each session's first UPDATE: <name> for the first session of the run, <name>-2 for the second, <name>-N for the Nth, so a fixture waits for the session a reload restarted (await-file established-2) the way it waits for the first. A relative name resolves in the test's work directory, where ze-peer and every compiled fixture run. The UPDATE, not ze's KEEPALIVE, is the event: RFC 4271 Section 9 permits an UPDATE only in Established, so the file proves the DAEMON is Established, while its KEEPALIVE arrives when it can still be in OpenConfirm. It lets a fixture wait on that state instead of a fixed delay: the reload trigger's awaitEstablished holds the SIGHUP until the file named established exists. A session that sends no UPDATE never writes the file. See. internal/test/fixture/misc_fixture_shellports.gotest/reload/reload-dynamic-peer-survives.ci
await-file path=<name>[:timeout=<duration>] Peer-block only: ze-peer sends nothing after the OPEN handshake, not an action=send, not an option=update route, until the file <name> exists. It resolves like established-file, so one peer block can hold its first UPDATE until another block's established-file proves the daemon has that session Established. A relay test needs it when its sender would otherwise race the receiver: ze forwards nothing to a peer it does not yet hold Established, and a route that arrives first reaches the receiver later through a peer-up replay, whose order is not the send order. The wait is bounded at 15 s, or at the timeout= the option declares (a positive Go duration such as 240s; anything else is refused), and fails naming the file and the bound. While it waits, ze-peer keeps the session alive: it answers and consumes each KEEPALIVE the daemon sends, so a wait longer than the hold time does not expire the daemon's hold timer, and a script never sees a KEEPALIVE that arrived before the file existed. Every other frame that arrives during the wait (an End-of-RIB, an UPDATE) is held in order, up to 1 MiB, and replayed to the script after the wait, which answers and judges it as it always did. See, whose file the interop checker writes. test/plugin/forward-overflow-two-tier.citest/interop/scenarios/bgp-local-pref-strip-gobgp/inject.msg
open value=<behavior> OPEN message behavior
update value=<behavior> UPDATE message behavior
env var=<KEY>:value=<V> Set environment variable
skip-os value=<os>[,<os>] Skip test on listed GOOS values (e.g., darwin, linux)
needs-arch value=<goarch>[,<goarch>] Run only on a host whose CPU arch (GOARCH) is in the list. On any other host the test is SKIPPED, and the SKIP line names both arches: SKIP. -- needs-arch=arm64, host GOARCH=amd64. . A test that builds or boots for a target arch carries it (owner decision 2026-10-13: a target arch is tested only on a CPU of that arch), because Ze builds every kernel on a host of its own arch, with no CPU emulation. Each token must be an arch the kernel builder supports (kernelbuilder.SupportedArches, today amd64 and arm64): an empty token, a typo such as amd46, a kernel spelling such as x86_64 or aarch64, a missing value= or any other key is a parse error on every host. A fixture whose arch is incidental to what it asserts asks for the host's arch instead (hostKernelArch), so it runs everywhere. The test/parse suite has its own parser and does not take needs-arch. internal/test/fixture/install_fixture.go
needs-linux [caps=<tok>[,<tok>]] Linux-only test. It skips on non-Linux hosts and runs in the QEMU guest through ./le test qemu all; the skip reason ends with the host command that runs that one test there, ./le test qemu ci test <path>. caps= declares required capabilities such as net-admin, net-raw, bpf, and sys-time; an unavailable capability produces a visible skip. An unknown token, an unknown key or an empty caps= is refused on every host.
needs-path value=<repo-rel-path>[:hint=<cmd>] Declares an optional heavyweight artifact. The runner resolves the path against the repository root and prints the native hint when the artifact is absent. A malformed or escaping path is a parse error.
netns-link name=<if>[:address=<cidr>] Provisions a dummy interface inside the per-test namespace. The test skips outside the ./le test qemu netns path because the named link must never be created on the host.
exclusive group=<name> Never run concurrently with another test carrying the same group name. Tests outside the group are unaffected and keep running alongside, so this costs far less wall-clock than dropping a whole suite to -p 1. Use it when tests contend for a kernel-global observation surface that unique names or addresses cannot partition: the ddos tests (group=ddos-flood) all flood the same loopback interface, and each daemon's detector picks its victim by top-destination-bytes over that interface's counters, so a sibling's concurrent flood is indistinguishable from the test's own. Applies on every platform and in every runner mode, because the contention is a property of the tests rather than of the host.

Configuration inputs for peer AS derivation

The derivation follows consumers, not filenames or configuration-looking text:

  • option=file:path= selects the named configuration. An embedded file with its basename takes precedence, as it does when the runner launches Ze.
  • A daemon's configuration argument selects a file, or its named stdin block when the argument is -. The runner's registered harness wrappers ending in -- ze ... use the same selection. ze start routing selects routing even though that filename has no extension. Value-less start flags such as --cli and --web-only are not configuration operands.
  • A consumed peer action=rewrite selects its source only when its destination is a selected configuration path. This includes transitive rewrite ancestry. Stdin-backed daemon paths use the same execution-order filename assignment as the runtime, including separate paths for additional daemons.

Initial daemon inputs and rewrite source leaves must be readable. An intermediate created by a rewrite need not exist before execution, but its selected ancestry must reach readable input; a cycle of absent sources is not a producer. Existing selected files must parse, and permissions or other read errors still fail derivation. None of these failures becomes an empty peer list. Unselected tmpfs files and stdin blocks remain raw sidecars, even if they contain valid BGP declarations or unmatched quotes. The former all-block tokenizer incorrectly refused the apostrophe in path-asn-filter-export-reject.ci's receiver acknowledgment. That fixture still publishes its marker only after the subject withdrawal and permitted fence arrive; classification must not rewrite or weaken those receiver checks.

The native derivation corpus gate rejects every native .ci parse failure, including failures before AS derivation. It does not feed other execution dialects into the native parser: test/decode uses DecodingTests.parseCIFile, and test/exabgp-compat uses parseExaBGPCI. The latter's option=file:<name> selects a predecessor configuration relative to the population's own etc/ when it has one (old-4.2, old-5.0), else the shared test/exabgp-compat/etc/; it is not the native option=file:path=<name> directive. Its option=env:<key>=<value> sets one variable in the environment of the daemon under test, ze and --dut exabgp alike, appended after the runner's own variables so a case can override one: api-version-env.ci writes option=env:exabgp.api.version=4 to hold every script to API 4 from its start. A line naming no key or no = is refused at discovery. A helper a configuration runs as le ... reaches the le that runs the suite: that le's directory is first on the daemon's PATH, then the ze under test's, and ZE_LE_BUILD_NAME is removed, so the named build of ./le --name <name> accepts the call. Those populations are chosen by their execution owners, not by tolerating parser errors. Native scenarios that intentionally submit invalid Ze configuration remain in the gate: their .ci itself must still parse. The shared draft boundary also excludes test/draft before discovery; a malformed unpromoted fixture must not redden repository gates.

Observer completion barriers

A peer's last wire expectation is not the observer's completion barrier. In cursor-replay.ci, the observer starts injecting only after eor-sent; the peer MUST linger after receiving that marker. In initial-sync-barrier-raw.ci, the peer MUST linger until the observer has read eor-sent, even when it has already received both expected frames. Without option=linger:value=true, Peer.completed returns and closes the session before the remaining work. When a raw injection races the independent initial-sync marker, put their exact wire expectations in the same seq group: both are required, but neither is required to arrive first. The observer MUST retain its EOR counter barrier before requesting shutdown.

Who stops the daemon decides which side may still be talking to it. For a compiled fixture that does not declare StopsDaemon, the runner waits for its check peers to exit and then signals the daemon, so a non-lingering peer's completion ends ze at once: an engine call the fixture still has in flight, even a request quiesce, then fails on the closed connection. Such a fixture MUST make no engine call after the point at which its peer can complete. When the fixture has work after that point, the peer lingers and the fixture ends the test itself through fixture.Observe, declaring StopsDaemon (initial-sync-marker-beside-process-route.ci, api-eor-during-initial-sync.ci, ipv4-announce-withdraw.ci).

The test peer's 100-millisecond read deadline is a cancellation poll, not a frame boundary. Check, sink, echo and post-success linger consumers MUST retain any partial BGP header or body across a recoverable idle timeout and resume the same frame. Only a complete frame reaches expectations, rejections or echo output; the following frame starts at its own header. Each retry returns through the context check, so an unfinished frame cannot prevent teardown.

Choosing between needs-linux, caps=, and skip-os

The .ci test ... Use
Only validates config (ze config validate -), parses, or runs an offline ze show / ze env Nothing. It runs natively on every OS
Boots a daemon that APPLIES Linux-only config (interface, VLAN, firewall, L2TP kernel) option=needs-linux
The same, and needs privileged network configuration (creates interfaces, brings links up, programs netlink) option=needs-linux:caps=net-admin
The same, and opens a raw or packet socket (resolve ping, traceroute) option=needs-linux:caps=net-raw
The same, and loads eBPF option=needs-linux:caps=bpf
The same, and sets the system clock (the NTP plugin restoring persisted time) option=needs-linux:caps=sys-time
Skips on one non-Linux OS for a reason unrelated to the kernel option=skip-os:value=darwin
Builds or boots for one target arch, which only a CPU of that arch can do option=needs-arch:value=arm64
Needs an optional heavyweight artifact the checkout does not carry option=needs-path:value=<repo-rel>:hint=<cmd>

caps= takes a comma-separated list, so a test that programs netlink and loads eBPF declares caps=net-admin,bpf and is gated on both. An unknown token is a parse error on every host, macOS included, so a typo cannot silently disable the gate.

caps=net-admin exists because Linux alone is not the requirement. On an unprivileged Linux host, a CI runner or a rootless container, a test that applies interface config does not fail cleanly: the interface plugin fails its configure handshake with operation not permitted and the DAEMON exits 1, then the TEST hangs because its check peer waits for a session the exited daemon will never open. The gate reads CapEff from /proc/self/status, not uid 0: a setcap'd binary holds the capability without being root, and a restricted container can be root without it.

An unreadable status file, malformed CapEff, or missing CapEff is an admission failure naming the probe cause, not evidence that capabilities are present or absent. Both .ci and .et discovery retain that failure before execution. NeedsGuest propagates it to feature and RFC placement callers instead of selecting a guest from an unknown host capability set. A valid capability set that lacks a declared bit still produces the documented capability skip.

A caps= test does not run in the merge gate. ./le verify worktree runs unprivileged, so the marker turns an opaque hang into an honest skip there, and the coverage relocates to .github/workflows/qemu-nightly.yml. TestCapabilityGatedTestsHaveANativeVMHome fails when that link is broken: marking a test with a capability nobody's CI has would be a coverage deletion wearing a skip's clothing.

OPEN Behaviors

Value Description
send-unknown-capability Add unknown capability (code 66) to OPEN
inspect-open-message Validate received OPEN against expectations. A NOTIFICATION ze sends in place of the OPEN (a held-down peer refusing a connection) is judged the same way and gets no reply. test/plugin/peer-admin-shutdown-inbound.ci
send-unknown-message Send unknown message type (255) after OPEN
drop-capability Remove a capability from ze-peer's OPEN response
add-capability Add a capability to ze-peer's OPEN response
router-id Send an explicit BGP Identifier instead of the mirrored one
hold-time State the Hold Time of the OPEN body (seconds=<N>)
graceful-restart State ze-peer's own Graceful Restart capability (restart-time=<N>, family=, forward-state=)
llgr State ze-peer's own Long-Lived Graceful Restart capability (stale-time=<N>, family=, forward-state=)
paths-limit State ze-peer's own PATHS-LIMIT entries (family=, limit=<N>)

Sender Facts (hold-time, graceful-restart, llgr, paths-limit)

Each of these four values states a fact about ZE-PEER that a receiver acts on. They exist because a mirror asserts sameness: mirrored, each one made ze read its own configuration back out of the peer's OPEN and believe the peer had said it.

option=open:value=hold-time:seconds=<N>
option=open:value=graceful-restart:restart-time=<N>[:family=<f>[,<f>]][:forward-state=true|false]
option=open:value=llgr:stale-time=<N>[:family=<f>[,<f>]][:forward-state=true|false]
option=open:value=paths-limit:family=<f>[,<f>]:limit=<N>
Key Range What reads it
seconds 0, or 3 to 65535 session_negotiate takes the smaller of this and ze's receive-hold-time, then calls timers.SetHoldTime. RFC 4271 Section 4.2. 1 and 2 fail the file
restart-time 0 to 4095 grStateManager.onSessionDown arms the restart timer on it, runPeer gives it to startEORTimer, and show bgp peer prints it. RFC 4724 Section 3 gives the field 12 bits
stale-time 0 to 16777215 enterLLGRLocked arms one timer per family on it. RFC 9494 Section 3 gives the field 24 bits
limit 0 to 65535 The Max Paths field of the code 76 entry this peer advertises. pathsLimitAdmission.filter drops paths past it, so ze sends this peer no more than limit paths for one prefix
family a family name, or a comma-separated list The <AFI, SAFI, Flags> tuples of code 64, the 7-octet tuples of code 71, and the entries of code 76
forward-state true (default) or false The F bit of every tuple. onSessionReestablished purges the stale routes of a family whose F bit is clear

The families are what make graceful restart act at all. RFC 4724 Section 3 pairs the Restart Time with a tuple list, and onSessionDown builds its stale family set from that list and returns without dispatching anything when the set is empty. Ze's own code 64 carries the time and no tuples (parseGRCapValue), so until ze-peer owned the value, retain-routes, mark-stale and purge-stale were never dispatched in any test.

A file that states none of them gets the harness's own defaults, never ze's.

Fact Default
Hold Time 65535, the largest the two-octet field states. RFC 4271 Section 4.2 takes the smaller of the two, so the negotiated value stays ze's own and no existing test changes cadence. Only a .ci stating a smaller one opts in
Restart Time 300 seconds, longer than any functional test runs
Long-Lived Stale Time 600 seconds
Max Paths 65535, which bounds nothing
Software version (code 75) le test peer. It has no option, because no session decision turns on it: capability.Parse holds no code-75 arm and the only reader is the offline ze bgp decode
Families, for all three capabilities the families ze-peer's own OPEN advertises, read from its Multiprotocol capabilities. An OPEN carrying none is an ipv4/unicast speaker (RFC 4760 Section 8)

Two refusals fail the file rather than dropping a line in silence:

  • Stating a fact AND add-capability or drop-capability for the same code. The stated octets would win, and the typed line would be read, validated and then lost.
  • Stating a fact for a capability ze does not offer. The capability SET le test peer sends mirrors ze's, so the value would have no place to go. Configure ze to offer the capability, or drop the option.

BGP Identifier Control (router-id)

option=open:value=router-id:id=<a.b.c.d>

Ze-peer's default OPEN carries ze's own BGP Identifier with the last octet incremented, which is always a distinct, valid identifier. This option replaces it outright, so a test can present an identifier the default can never produce: 0.0.0.0, or ze's own router-id (RFC 6286 Section 2.2 rejects both, the second only from an internal peer). A malformed or IPv6 value fails the file when it is read. It was ignored until 2026-09-08, which sent the DEFAULT identifier from a test that asked for an invalid one: the file then tested the valid identifier and passed.

Capability Control (drop-capability / add-capability)

Ze-peer mirrors the capability SET of ze's OPEN, so a .ci that says nothing about capabilities still negotiates whatever ze offers. It does NOT mirror the values that describe the SENDER. The AS, the BGP Identifier, the Hold Time, the Role (code 9), the Graceful Restart time, families and flags (code 64), the ADD-PATH directions (code 69), the Long-Lived Graceful Restart stale time and flags (code 71), the FQDN (code 73), the software version (code 75) and the PATHS-LIMIT entries (code 76) are each resolved from the test's own configuration and written into ze-peer's OPEN, because a mirror asserts sameness and every one of those facts is about the speaker rather than about the session. ownedCapabilities (internal/test/peer/open_capability.go) is the list: a code absent from it is mirrored by construction.

The drop-capability and add-capability options act on that reconciled OPEN at wire level, allowing tests to control exactly which capabilities ze-peer advertises.

A capability the .ci states REPLACES the one ze-peer would have resolved. An add-capability naming any resolved code -- 9, 64, 65, 69, 71, 73, 75 or 76 -- is sent as written and ze-peer adds no second capability of that code, so a file that drops a code and adds it back gets exactly what it asked for. Dropping code 65 removes the four-octet AS capability, which leaves the two-octet My Autonomous System field as the only carrier of the peer's AS: above 65535 that field can carry AS_TRANS alone, which is what RFC 6793 Section 3 defines for a speaker with no two-octet AS.

An add-capability:code=65 carrying four octets IS the AS declaration, and the My Autonomous System field follows it. It outranks option=asn and the derivation, because it names the octets that reach the wire and RFC 6793 Section 4.1 makes those the octets a receiver reads. Letting the header field come from anywhere else would put two ASNs in one OPEN, which is the disagreement ze answers with NOTIFICATION 2/2 Bad Peer AS. A stated code 65 of any OTHER length declares no AS: it is a malformed capability the test is driving on purpose, so it is sent as written and the header field keeps the AS the rest of the configuration resolved. That is the one case where the two carriers differ, and they differ because the .ci asked.

Drop a capability:

option=open:value=drop-capability:code=<N>

Removes the capability with the given code from ze-peer's OPEN response. The peer will not see this capability in the mirrored OPEN.

Add a capability:

option=open:value=add-capability:code=<N>:hex=<value-bytes>

Adds a capability with the given code and hex-encoded value bytes to ze-peer's OPEN response.

Key Description
code Capability code (1-255), e.g., 65 for ASN4, 2 for route-refresh
hex Hex-encoded capability value bytes (only for add-capability)

Use case: testing capability mode enforcement:

When Ze is configured with require mode for a capability, it sends a NOTIFICATION if the peer lacks that capability. To test this, use drop-capability to make ze-peer omit the capability from its response:

# Test: Ze requires ASN4, ze-peer drops it → Ze should send NOTIFICATION
option=open:value=drop-capability:code=65

When Ze is configured with refuse mode, it sends a NOTIFICATION if the peer has a capability. To test this, the default mirror behavior already includes the capability, but add-capability can add capabilities not in the original OPEN:

# Test: Add a custom capability for refuse testing
option=open:value=add-capability:code=73:hex=067A652D626770

Multiple overrides can be combined:

option=open:value=drop-capability:code=65
option=open:value=drop-capability:code=2
option=open:value=add-capability:code=73:hex=067A652D626770

UPDATE Behaviors

Routes ze-peer sends after the OPEN handshake, before it starts matching expectations.

Value Keys Description
send-default-route none Send one UPDATE for 192.0.2.1, an address no test host holds, so the route passes the RFC 4271 Section 6.3 next-hop check. 0.0.0.0/32
send-route prefix, origin-as, next-hop, and optionally as-path, as-set, originator-id, cluster-list, label Send one UPDATE for one prefix. Repeat the line for more
send-bulk prefix, count, next-hop, and optionally max-msg, eor Generate count sequential prefixes from prefix and send them as whole BGP messages

send-default-route and send-route build the AS_PATH that a real speaker sends. On an eBGP session, where ze-peer's OPEN AS differs from ze's, the path starts with ze-peer's own AS (RFC 4271 Section 5.1.2). An origin-as that names a different AS follows it, so origin-as=65001 from AS 65002 sends [65002 65001]. On iBGP the path is origin-as alone, or empty. An as-path key is the whole path and goes out as written. Ze drops a route from an eBGP neighbor whose leftmost AS is not the neighbor's (RFC 4271 Section 6.3), so a hand-written hex= UPDATE from an eBGP peer MUST start its AS_PATH with that peer's AS.

send-bulk writes the AS_PATH of a route ze-peer originates (RFC 4271 Section 5.1.2): on eBGP one AS_SEQUENCE holding ze-peer's own OPEN AS, on iBGP an empty AS_PATH, whose length field is zero. It takes no origin-as key, and refuses one as it refuses any key it does not read.

These two generated-route behaviors include LOCAL_PREF=100 on iBGP and omit LOCAL_PREF on eBGP, including labeled send-route UPDATEs. The session kind comes from the OPEN exchange, not an option in the fixture. RFC 4271 Section 5.1.5 requires LOCAL_PREF on internal advertisements and forbids it on external advertisements except within confederations. Raw hex= injection remains unchanged, so a test can still send deliberately malformed attributes.

Generating one oversize UPDATE (max-msg):

option=update:value=send-bulk:prefix=10.0.0.0/24:count=16373:next-hop=10.0.0.1:max-msg=65535

max-msg caps one generated message, header included. It defaults to the RFC 4271 limit of 4096, and accepts 23 to 65535. Raise it to 65535 when the session negotiates the Extended Message capability (RFC 8654), which ze advertises when the peer carries capability { extended-message enable }. Prefixes that do not fit one message spill into the next, so the example above is exactly one 65535 byte message: a 65516 byte body, the largest BGP permits.

Use it whenever a test needs an UPDATE too large to write as a hex literal. A max-size message is 131070 hex characters, which is past the 64 KiB line limit of both .ci scanners and past what a reviewer can check. It is also the only way to hand the daemon a single oversize body: splitting the same prefixes across several standard messages is a different input, because the daemon decides per message.

A malformed key fails the test load rather than sending nothing. That is deliberate: a spec that silently degraded to count=0 would let a test asserting a route was NOT forwarded pass because no route was ever offered.

Commands

cmd=<type>:key=value[:key=value...]

API Commands

cmd=api:conn=<N>:seq=<N>:text=<command>
Key Description
conn Connection number (1-4)
seq Sequence number within connection
text API command text

Example

cmd=api:conn=1:seq=1:text=update text origin igp next-hop 10.0.1.1 nlri add 10.0.0.0/24

Process Commands (Foreground/Background)

For orchestrating multiple processes:

cmd=background:seq=<N>:exec=<command>[:stdin=<name>][:name=<handle>][:timeout=<dur>][:ready=<text>]
cmd=foreground:seq=<N>:exec=<command>[:stdin=<name>][:timeout=<dur>][:exit=<N>]
cmd=stop:seq=<N>:name=<handle>[:signal=kill|term]
Key Description
seq Execution order (lower first)
exec Command to execute
stdin Stdin block name to pipe
timeout Foreground test budget, or background process lifetime (e.g., 10s).
exit Exit code asserted for this command (0..255). See below.
name Handle for a background process, so a later cmd=stop can target it.
ready cmd=background only: text the process prints once it can serve. See below.
signal cmd=stop only: kill (SIGKILL, default) or term (SIGTERM).

Markers may appear in any order; each value runs to the next key in the table above, whichever key that is.

The table is the whole vocabulary. Any other :<word>= on a cmd= line fails the file, naming the key, the accepted set and the line. The scan reads the whole line, exec= included, because that is exactly where a key the parser does not read ends up: a value runs to the next KNOWN key, so an unknown one is swallowed into the value before it rather than dropped. cmd=foreground:seq=2:exec=ze -:stdin=ze-bgp:timeout=15s:env=ZE_FWD_WRITE_DEADLINE=10s parsed with timeout="15s:env=ZE_FWD_WRITE_DEADLINE=10s", so the line got neither the timeout it declared nor the variable, and every assertion still passed. Set an environment variable with option=env:var=<name>:value=<value>.

One consequence: a command carrying a :<word>= span of its own cannot be written on a cmd= line. Put it in a tmpfs= script and run the script.

timeout= must be a Go duration (10s, 1m30s). Both readers of the value keep their own default when it does not parse, so it is refused here instead.

The test deadline ends the whole process tree of every cmd= process. Each one leads its own process group, and the deadline kills that group, so a fixture's daemon dies with the fixture. A descendant that left the group cannot hold the test open either: the runner stops reading its output pipes 10s after the kill.

Every successful start has one Wait owner. Normal completion, named stop, timeout, and early failure join that owner before the runner removes input files or releases ports. A peer drain whose grace expires stops and reaps the peer before reading its final output.

Background: Starts and keeps running until its timeout, an explicit stop, or test completion. Foreground: Setup commands finish before the next step. A ze daemon starts without blocking later steps. Its peers or observer determine when teardown starts. Stop: Terminates a named background process mid-test (see below).

Every foreground helper (any program other than ze and the test peer) is waited for, the last cmd= line included. Its exit= assertion checks its own status before the next command starts. Without exit=, an intermediate helper must exit zero. The final helper can instead declare expect=exit:code=. If it declares both, both must match. With neither, it must exit zero. A helper still running at the test deadline fails it as a timeout that names the command: a process the test does not wait for is cmd=background.

Until 2026-10-14 the last helper was classed with the daemons: started, torn down with no Wait unless expect=exit:code= was declared, so its output reached the accumulators only if it won that race, and a helper that never exited passed as a daemon. test/ui/update-serve.ci started its server that way and now declares it cmd=background.

When an embedded observer sends request shutdown, teardown waits out that daemon's bounded self-stop grace before it stops any other background process, whatever order those processes started in. A helper the daemon still talks to during its own shutdown stays up for that wait.

How an exec= value becomes argv

The runner splits the value on spaces and tabs, and a span inside " or ' stays ONE argument. Quote an argument that carries a space or a pipe, exactly as you would in a shell:

cmd=foreground:seq=1:exec=ze cli -c "show config dump - | json":stdin=config

-c receives show config dump - | json, and the quotes do not reach the process. Three limits apply:

  • A backslash escape is not handled. \" is a backslash and a quote character, never a literal quote inside an argument.
  • An unbalanced quote fails the test with unclosed quote in ....
  • No shell runs, so |, > and $HOME are ordinary characters inside an argument. The runner expands $PORT, $PORT2 and $PORT3, and le test fixture expands the environment in its own arguments.

One splitter serves every suite, so a cmd= line produces the same argv wherever it runs.

The first word selects the program. ze is the daemon under test, and le test <name>, le mock <name> or another harness tool such as le dns query (docs/functional-tests.md, "le dns query") runs the harness, which is the runner's own executable. Any other head, a retired standalone harness name among them, is no head of the runner's: it reaches the PATH lookup, finds no program, and fails the step. A name that the suite compiled into its temporary directory runs from there: the chaos suites compile le, so exec=le chaos run ... runs that build. Any other name is found on PATH. A shim directory on the child's PATH holds ze and le, so a plugin run "le test ..." line in a config reaches the same binaries.

A step that runs an le action names the head le, never ./le. An le head reaches the build the suite chose, as above: the runner's own executable when the suite compiled none. ./le is the repository's launcher script, and the runner drops ZE_LE_BUILD_NAME from every child, so the launcher runs the newest shared build of HEAD in bin/builds/: a suite started with ./le --name <build> would then test a binary built without its working-tree edits. The appliance steps that exec le test qemu ... rely on this.

A non-harness le step runs at the repository root, so a tmpfs= file's bare name does not resolve on it. go, ./le and every le head other than a harness command (le test <name>) or a stand-in server (le mock <name>) run in the checkout, because their arguments name packages and paths there. Every other step runs in the test's own directory, where the runner writes the tmpfs= files. A non-harness le step that passes peer.conf by that name therefore looks for it at the root and fails. Put the le call in a tmpfs= script and run the script: it runs in the test's directory, and its le is the one the shim directory holds.

Provenance: until 2026-09-02 the .ci runner split the value on whitespace alone while the parse suite honored quotes. The example above was already published here, so four test/ui tests were written against it. -c received only the first word of the quoted command, ze cli fell through to its SSH client, and each test failed with no credentials for 127.0.0.1:2222.

ready= -- wait for a background process to serve

A background process that is not ze-peer gets 100ms to start before the next step runs. That is a guess, and a loaded host makes it wrong. A fixture that serves a socket the next step dials (a BMP collector, an RTR cache) then starts after ze has already dialed, ze gets connection refused, and ze waits out its reconnect interval: 30s for BMP (RFC 7854), 600s for RTR. The test fails as a protocol stall.

ready=<text> removes the guess. The runner reads the process's stdout and stderr, and it starts no later step until the text appears:

cmd=background:seq=1:exec=le test fixture plugin/bmp-sender-statistics-collector $PORT2:ready=BMP-COLLECTOR: listening on
cmd=background:seq=2:exec=le test peer --port $PORT:stdin=peer
cmd=foreground:seq=3:exec=ze --plugin ze.bgp-bmp -:stdin=ze-bgp
  • The process MUST print the text AFTER it binds, never before.
  • The wait has no deadline of its own. The test budget bounds it. When the budget ends first, the test fails with background_never_ready, and the message names the command, the text, and all output of the process.
  • The key is refused on cmd=foreground and on a ze-peer line. A foreground command is already awaited, and ze-peer has its own "listening on" barrier. An empty value is refused because it matches any output.
  • The text is a substring match, and any :<word>= span ends it, as with every other key.

cmd=stop -- terminate a background process mid-test

A background process started with name=<handle> can be stopped at a chosen step by cmd=stop:seq=<N>:name=<handle>. The runner looks the process up, signals it, and waits for it to exit before the next step runs, so a later step can deterministically observe what happens after the process dies (e.g. show vpn ipsec sa emptying once an IKE responder is killed and DPD fires).

  • signal=kill (default) sends SIGKILL: the process gets no chance to flush or send a protocol teardown. This is the choice for liveness/dead-peer tests where the peer must go silent (a clean shutdown would take a different code path).
  • signal=term sends SIGTERM, escalating to SIGKILL if the process does not exit within the teardown grace period -- a graceful stop.

Fail-closed: a cmd=stop naming a process that was never started (no matching name=) fails the test with a clear error; it never silently no-ops, and it can only ever signal a process the runner itself started (never an arbitrary PID). Teardown still kills every remaining background process, and tolerates one the stop step already reaped.

Provenance: added by spec-fixit-runner-kill-background to unblock end-to-end peer-death observation (the deleted test/ipsec/ipsec-dpd-timeout.ci needed it).

exit= vs expect=exit:code= (per-command vs file-level)

expect=exit:code= is file-level: Record.ExpectExitCode is a single value (a later expect=exit:code= silently overwrites an earlier one) and the runner uses the final foreground helper's exit when the final command is a helper. Otherwise, it uses the foreground daemon's exit, or the last quick-exit ze command's exit when there is no daemon. Await fences retain their daemon/peer exit selection. A file that runs several ze config validate commands therefore asserts only the final one; every earlier command can exit with any code and the test still passes.

Use exit= on the cmd= line to assert a specific command's own exit code. It is checked the moment that command finishes, and names the offending seq on failure:

cmd seq=2 (ze config validate -): expected exit code 1, got 0

Foreground helpers honor exit= in both intermediate and final positions. A helper mismatch retains its output and stops execution before the next command. Inspect that output, then correct the helper or its exit= assertion. Timeout and cancellation remain failures regardless of the declared exit code.

Prefer exit= whenever a file runs more than one quick-exit ze command. A "quick-exit ze command" is any foreground ze whose verb is not a daemon verb (hub, start, cli, monitor) and which has no config-file argument or --web flag.

For helpers and quick-exit ze, exit=0 requires a successful wait. A nonzero exit compares the child's reported status. A wait error with no readable exit status, including an output pipe that exceeds its drain bound, fails the command instead of being treated as zero. The failure retains output and names the command and wait error. Check that the command runs to completion on this host, then run the test again.

exit= is accepted only where it is checked. Only the two foreground arms that await their process compare it: a foreground helper and a foreground quick-exit ze. Every other command refuses exit= when the file is read, naming the reason and the fix:

Command Why exit= is refused Write instead
cmd=background The runner never waits for it to exit; it is killed at teardown cmd=foreground, or an expect= on what it produced
Foreground ze daemon The runner synchronizes on it and tears it down; its status is the file-level one expect=exit:code=
Foreground le test peer The same as a foreground daemon expect=exit:code=

Until 2026-11-02 these lines parsed, and the assertion was dropped in silence. classifyStepArm names the arm once for the parser and for runOrchestrated, and runOrchestrated checks again before it dispatches, so a record built without the parser cannot slip past either.

Every stream assertion sits under a command and is scoped to it

Every expect=stdout:, reject=stdout:, expect=stderr: and reject=stderr: line belongs to the cmd= line directly above it. Position is the grammar: the line means what its place in the file says it means.

What each line reads depends on its kind, and no reject reads less than it did before the scoping:

Line Reads
expect=stdout:, expect=stderr: (contains=, pattern=) The stream it names, in the span of the command above it
reject=stdout:contains=, reject=stdout:pattern=, reject=stderr:contains= Both streams of the span of the command above it: the stream name selects nothing. Under the last foreground helper, both streams from its start to the end of the run
reject=stderr:pattern= The WHOLE run's stderr, wherever the line sits: a log check

An expect cannot be satisfied by output its author never named. A reject=stderr:pattern= is the file-wide log check it always was (a panic, a fatal error, a level a daemon must not log), so it sees every daemon in the file, one started after it included, and its position is irrelevant. A contains= reject has been scoped to its command since 2026-09-14, so it cannot trip on a sibling command's output. A reject failure names where the text sits, narrowed to the latest command whose span inside the asserting one holds it: cmd seq=1 (ze start): output unexpectedly contains "panic" on stderr, found inside the output span of cmd seq=2 (le test fixture x). That names a span, not an author: a daemon's span covers every byte a later command wrote too.

  • A line with no cmd= above it is refused at parse time, naming the file and the line. It named no command's output, so there was nothing for it to assert. Move it below the cmd= line that produces what it describes.
  • A line under a cmd=stop is refused for the same reason. A stop directive terminates a named background process; it runs no program and writes nothing.
  • On an expect, the stream name selects the stream. expect=stdout: reads the command's stdout span alone and expect=stderr: its stderr span alone. A miss whose needle the command wrote to the other stream says so. Until 2026-10-14 a span covered both streams, so a stdout assertion passed on text written to stderr. A command-scoped reject still reads both streams, because one stream would check less than it did.
  • expect=stderr:pattern= reads what expect=stderr:contains= reads: the stderr span of the cmd= line above it. Until 2026-10-14 it read the whole run's stderr, so a regex under one command matched a line another command wrote. An expected pattern about a daemon's log lines therefore sits under the cmd= line that runs the daemon. reject=stderr:pattern= still reads the whole run's stderr. A pattern inside a stdin=peer block is parsed after the file's cmd= lines, so it belongs to the last of them.

A command's span is the half-open slice of the run's two accumulators it wrote between starting and finishing. A quick-exit ze marks both ends around its own fold, so its span is exact. A background process is still writing when the test ends, and a foreground daemon's bytes arrive throughout its life, so both close at the final lengths: a long-running command's span can therefore contain bytes a later command wrote. A foreground helper's span runs from its start to the end of the run unless it is the last helper or a quick ze command, so an expect under a helper earlier in the file can be satisfied by a later command's output. The last helper's span ends when it exits, so an expect under it reads only what it wrote; a reject under it still reads both streams from its start to the end of the run, so the daemon teardown that follows is still forbidden. A command the run never reached has an EMPTY span, never the whole buffer.

A le test peer command's output is in no span. The runner captures a test peer's stdout and stderr in that peer's own buffers, which feed its verdict and the PEER OUTPUT section of a failure report, and never in the run's two accumulators. An expect=stdout: or expect=stderr: line under a peer's cmd= line therefore reads none of what the peer printed, and fails with "output does not contain" while the text is on screen under PEER OUTPUT. What a peer received is asserted by its expect=bgp lines. What the daemon did about the peer is asserted on the daemon's log, under the daemon's cmd= line (test/plugin/session-peer-stops-reading.ci).

Output snapshots taken between commands do not finish an open span. The runner closes it only after the output writers have joined, including deferred child cleanup on an early error. Until 2026-11-03, saving a foreground helper's output also closed background and daemon spans, so later output reached the report but not the assertions under those commands.

With --save, the orchestrated runner writes the final client and peer stdout and stderr, expected messages, and received wire messages once after the child owners finish. This includes a helper failure or timeout that returns before normal teardown. Saving preserves the original verdict; a write failure is reported as a warning rather than replacing the test's failure.

This replaced a file-level check over one combined buffer. Every expect used to read Record.ClientOutput, the concatenated stdout AND stderr of every command in the file, so an assertion could be satisfied by a command its author never named. Two files did exactly that: test/plugin/kernel-capability-unknown-starts.ci and kernel-capability-doctor-reports.ci each asserted that ze doctor printed a diagnostic code and then ran ze show diagnostic <that code>, whose output satisfied the assertion on a host where the doctor check does not run at all.

Until 2026-09-14 a contains= reject read every command's output, so a negative assertion that must not see another command's output needed a file of its own. test/plugin/vpp-doctor-hugepages.ci and vpp-doctor-hugepages-quiet.ci, and test/appliance/no-install-appliance.ci and appliance-help-not-deprecated.ci, were split for that reason.

A failure names the command: cmd seq=2 (ze config validate -): output does not contain "...".

Known gap: 108 quick-exit ze commands across 50 .ci files predate exit= and are still unasserted (their expect=exit:code= never reaches them). Arming them may surface real defects; tracked in plan/known-failures/.

The parse suite reads its own dialect

test/parse runs under ParsingTests, a second parser with its own execution model. Two differences are load-bearing for an author.

A command's own exit= is its exit assertion here, the same one an expect=exit:code= under it states, because this suite awaits every command, daemon verbs included. A command whose exit= and expect=exit:code= disagree is refused at discovery. Until 2026-11-02 this suite parsed exit= and dropped it, so only the expect=exit:code= form was checked.

Every assertion is scoped to one command, as it is in the generic runner (above). This suite has always worked that way and the generic runner now does too, so the two dialects no longer disagree about what an assertion's position means. One difference survives: here expect=stderr:contains= MAY appear before the first cmd=, because a .ci holding an inline config and no command at all is a legacy negative test whose expected error it carries.

The dialect is a subset of the generic vocabulary, and nothing else parses. A directive no arm reads fails the file at discovery, naming the directive, the file, and every directive the suite does read. It is the same rule as "Unknown Keys Are a Parse Error" above, applied to the whole line rather than to a key inside it.

Read by the parse suite
cmd=, expect=exit:code=
expect=stdout:contains=, expect=stdout:pattern=
expect=stderr:contains=, expect=stderr:pattern=
reject=stdout:contains=, reject=stdout:pattern=, reject=stderr:pattern=
option=skip-os:value=, option=env:

Two spellings this suite once had are deleted, not aliased. Both existed only here, and both are what two parsers over one corpus costs:

  • expect=stdout:regex= is now expect=stdout:pattern=. The generic parser reads pattern= and refuses regex=, so the gates that walk the whole corpus with it could not read three test/parse files at all.
  • expect=stdout:not:contains= is now reject=stdout:contains=, which this suite already read with the identical meaning. This is the one that mattered: the generic parser splits not:contains= at the :contains= key boundary and drops the bare not, so one written line meant "must be absent" to the suite that runs test/parse and "must be present" to every gate that reads it.

cmd=...:timeout=<duration> is honored, and a bare ze - runs as ze start <file>, the same translation the generic runner makes. Substituting the path in place produced ze <path>, which is not a command: the daemon answered unknown command: <path> with its usage and exit 1 before reading a line of config, so a file could assert expect=exit:code=1 and be satisfied by the usage error.

Daemon readiness (ze only): a ze daemon launched either foreground or background is told (via ZE_READY_FILE) to write daemon.ready once startup completes, and the runner publishes its PID to daemon.pid in the tmpfs directory. The PID file appears atomically after its complete numeric contents and any dropped-user ownership are installed, so a fixture that sees it can read the identity immediately. A publication failure fails the command and reaps its owned child. Tests poll both files directly or through a compiled driver in internal/test/fixture before signaling the daemon or asserting on it. The readiness handshake is armed only for Ze daemons.

Example (Decode Test)

stdin=payload:hex=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF003C...
cmd=foreground:seq=1:exec=le test decode --family ipv4/unicast -:stdin=payload
expect=json:json={ "type": "update", ... }

Example (Multi-Process)

stdin=peer:terminator=EOF_PEER
option=asn:value=65000
expect=bgp:conn=1:seq=1:hex=FFFF...
EOF_PEER

stdin=ze-bgp:terminator=EOF_CONF
peer test-peer { remote { ip 127.0.0.1; } ... }
EOF_CONF

cmd=background:seq=1:exec=le test peer --port $PORT:stdin=peer
cmd=foreground:seq=2:exec=ze bgp server -:stdin=ze-bgp:timeout=10s

Example (Multi-Peer)

Tests needing two or more BGP peers use the same $PORT on different loopback addresses. The ze_bgp_tcp_port override sets all peer ports uniformly, which is correct because every peer listens on $PORT.

# Source peer on 127.0.0.1 (default)
stdin=source:terminator=EOF_SOURCE
option=tcp_connections:value=1
action=send:conn=1:seq=1:hex=FFFF...
EOF_SOURCE

# Dest peer on 127.0.0.2 -- sink mode absorbs any UPDATE
stdin=dest:terminator=EOF_DEST
option=tcp_connections:value=1
EOF_DEST

cmd=background:seq=1:exec=le test peer --port $PORT:stdin=source
cmd=background:seq=2:exec=le test peer --bind 127.0.0.2 --mode sink --port $PORT:stdin=dest
cmd=foreground:seq=3:exec=ze -:stdin=ze-bgp:timeout=20s

Each ze-peer gets independent output capture and WaitFor synchronization. The runner waits for each peer's "listening on" message before starting the next command.

On Linux, 127.0.0.2 works automatically (127.0.0.0/8 routes to lo). On macOS and FreeBSD, the test runner adds loopback aliases via the SIOCAIFADDR ioctl.

IPv6 works differently, because a host carries exactly one IPv6 loopback address. A fixture that needs a second one uses fd00::2, which is unique-local (RFC 4193) and never globally routable. ./le setup install adds it, and ./le setup check reports whether it is there. The runner never adds it: the ioctl returns EPERM to an unprivileged process, and ./le verify current runs as an ordinary user. A test that binds an address this host does not carry fails at once with loopback_address_missing and the command to run, rather than timing out on a bind that could not succeed.

The check reads the three places a fixture names an address it binds. One is le test peer --bind <ip> on a cmd= line. The second is connection { local { ip <addr> } } in the config the fixture embeds: Ze sends from that address and listens on it when accept is true, so the host must carry it too. The third is local-address <addr>; in an ExaBGP-syntax config the exabgp-compat suite keeps beside its .ci and names with option=file:. That suite has a parser and a runner of its own, so it calls the same scan through EnsureConfigFileBindAddresses rather than through the embedded path.

A local address outside 127.0.0.0/8 and fc00::/7 is left alone. A config-validation fixture names a routable one (local { ip 192.0.2.1 }), the daemon exits before it binds anything, and ./le setup install adds no such address. The local-link-local fe80::1 leaf that sits beside local-address in an exabgp-compat config is left alone for the same reason.

An exabgp-compat case runs TWO processes, ze and the mock BGP server, and its verdict names which one ended the case:

  • ze exited before the mock BGP server finished its script: <error> is a daemon defect. Read the client stderr in the failure block.
  • the mock BGP server failed: <error> is a wire or document disagreement. The mock prints the frame or the JSON member that did not match.
  • the mock BGP server ended without reporting success means its script did not complete and it said nothing about why.

Both used to print the bare exit status N of whichever process failed, which told a reader nothing about which of the two to open.

Both processes are stopped by a defer beside the start that created them, so a case that returns by any route, a panic included, leaves neither running. The stops written inside the branch arms are ORDERING barriers rather than lifetimes: a process's output must not be read while it is still writing, and its exit status does not exist until it has one. How the suite's forks are created and canceled is in docs/architecture/testing/runner-architecture.md, "How a suite's child processes end".

Expectations

expect=<type>:key=value[:key=value...]

BGP Wire Expectations

expect=bgp:conn=<N>:seq=<N>:hex=<hex-bytes>
expect=bgp:conn=<N>:seq=<N>:prefix=<hex-bytes>
expect=bgp:conn=<N>:seq=<N>:contains=<hex-bytes>
expect=bgp:conn=<N>:seq=<N>:announced=<hex-bytes>
expect=bgp:conn=<N>:seq=<N>:withdrawn=<hex-bytes>
expect=bgp:conn=<N>:seq=<N>:ordered=<hex-bytes>

Validates the BGP wire message received: hex= matches the exact message, prefix= the message start, contains= a substring anywhere in one message. Within one seq group, hex/prefix/contains checks match in any order and each consumes exactly one received message.

ordered= checks in a seq group form a strict FIFO subqueue for asserting in-order delivery across message boundaries: the front needle must appear in the received message, and one message may consume several consecutive needles, each matched at an advancing offset (so order inside a packed message is enforced too). Use ordered= instead of per-message contains= when the sender may legally pack several NLRIs into one UPDATE (the forward rail's bucket merge): per-message framing is not a property ze owes, but delivery order is. A message whose content matches only a non-front needle consumes nothing and is reported as a mismatch.

A received message that meets no expectation fails the test with message mismatch, reported against the first expectation still owed. A hex= expectation is a whole message, so both messages are decoded and their attributes compared. Every other form holds part of a message, so the report names the line's key and its needle, decodes the message that arrived, and ends with Missing: the received message does not match contains=<hex>, so the expected message has not arrived. A needle is never decoded as a message.

announced= matches byte-aligned bytes only in the legacy NLRI or an MP_REACH_NLRI's NLRI field. withdrawn= matches only in Withdrawn Routes or an MP_UNREACH_NLRI's NLRI field. Neither searches path attributes or next hops. Several scoped checks in one seq may match one packed UPDATE or separate UPDATEs; they never consume checks from the next sequence. Keep a fence in its own later sequence when ordering is the subject.

Add attribute=<decimal-type-code>,<hex-value-bytes> to a scoped expectation to require both the NLRI and a value in the named path attribute in the same UPDATE. For example, the LLGR timer receiver uses announced=180A0000:attribute=8,FFFF0006 to require 10.0.0.0/24 with LLGR_STALE. A stale community on another route does not satisfy that assertion. The attribute's compact or extended length encoding does not affect the match. The value is a byte-aligned substring within the named attribute, not a whole-attribute equality check. This qualifier requires announced or withdrawn and cannot be combined with hex, prefix, contains, or ordered; malformed qualifiers fail parsing rather than dropping a condition.

These are raw NLRI byte patterns, not decoded prefix selectors. ADD-PATH identifiers remain present: include the four identifier octets when asserting a particular path. Opaque families need no CIDR conversion. Each field is searched separately, so a pattern cannot bridge two fields.

The corresponding negative assertion is:

reject=bgp:conn=<N>:scope=announced:pattern=<hex-bytes>
reject=bgp:conn=<N>:scope=withdrawn:pattern=<hex-bytes>

Put scope before pattern, whose value consumes the rest of the line. Without scope, the existing rejection still searches every wire byte. An unknown scope is a parse error. A malformed UPDATE fails a scoped rejection closed and cannot satisfy a scoped positive expectation.

Use announcement scope for an export prohibition. RFC 1997 forbids advertising a governed route; it does not forbid withdrawing a previous generation of that route. The egress scenarios therefore require the withdrawal, reject the announcement, and require a permitted fence announcement on the same session. An UPDATE counter alone cannot distinguish those outcomes.

These forms are peer-block directives (inside a stdin=<name>: block); top-level expect=bgp lines support hex= only.

JSON Expectations

expect=json:conn=<N>:seq=<N>:json=<json-object>
expect=json:json=<json-object>

Validates the decoded message matches expected JSON.

Validation rules:

  • Parsed and compared field-by-field (key order independent)
  • Volatile fields removed before comparison: exabgp, ze-bgp, time, host, pid, ppid, counter
  • Neighbor normalization: peer ↔ neighbor treated as equivalent, direction field ignored
  • All non-volatile fields must match exactly

Exit Code Expectations

expect=exit:code=<N>

Validates the foreground process exit code. A test whose ONLY assertion is expect=exit:code=0 is accept-only (weak) and is gated by a lint; see Assertion Strength.

An Unknown Directive or Key Is a Parse Error

Every directive the runner reads is declared: the action= word, the type after it, and the keys inside it. A word or a key outside its declared set fails the file at discovery, naming what was written, the accepted set, and the line. Nothing is dropped in silence, and nothing is guessed.

line 12: unknown action "exepct" (accepts action, await, cmd, command, expect, http, option, reject, stream)
line 12: unknown expect type "stdoutt" (accepts bgp, command-error, event, exit, file, json, output, stderr, stdout, stream, syslog)
line 12: cmd=foreground: unknown key "env" (accepts exec, exit, name, ready, seq, stdin, timeout)

The accepted set in each message is the list the parser gates on, so it cannot describe a vocabulary the runner does not have. The line number is the line in the FILE: comments, blank lines and whole stdin= and tmpfs= blocks are consumed before the directives are parsed, and the refusal used to count only the directives that survived, which named line 2 for a line that was number 69.

The option=, action=, cmd=api and await= families each declare one table that maps a type to the keys its parser reads, and the accepted types and keys in a refusal come from that table. The type is checked first, so option=bogus:65000 is refused as an unknown option type with the accepted list. Three segment shapes are then refused before any key is checked: a segment with no = (option=asn:65000), a key given twice, and a key that does not start with a letter, which would otherwise be read as part of the value before it (path=out.txt:!contains=x). The value of json=, text=, hex= or pattern= runs to the end of the line, so write those keys last. Those four are found by substring search in that order, so a value that holds one of them is cut there: in contains=hex=AB the inner hex= becomes a key of its own and contains= is left empty. The decode suite, the parse suite's option=env, and the option= lines that ze-peer reads in a stdin= block refuse in the same way, and ze-peer's options are one table of entries that pair what is accepted with the parser that reads it. The decode suite does not read the words of its exec= line: it runs them, so ze bgp decode itself refuses a flag it does not know and the test fails on its exit status.

line 3: option=asn: a segment has no key: "65000" has no "="; write each field as key=value (accepts value)
line 4: http=get: key "status" is given twice; keep one

The rule exists because a dropped key takes the whole assertion with it. An expect=stdout:not-contains=X line recorded NO assertion and then passed whatever the command printed. Ten such lines were live across seven .ci files when the check was added, and one of them was the only assertion its test made.

Two spellings get a named message, because both are the right key somewhere else: not-contains= belongs to expect=file:, and !contains= was the expect=stdout spelling until stream non-containment moved to reject=.

Stdout Expectations

expect=stdout:contains=<text>
expect=stdout:pattern=<regex>

Two modes:

  • contains= -- substring match against stdout (multiple allowed, all must match)
  • pattern= -- regex match against stdout (uses Go regexp syntax)

Stdout must NOT contain text is reject=stdout:contains=, below. A stream has ONE negation spelling and it lives on reject=.

Stderr Expectations

expect=stderr:pattern=<regex>
expect=stderr:contains=<text>

Two modes:

  • pattern=: regex match against stderr (uses Go regexp syntax)
  • contains=: substring match against stderr

Both modes read the stderr the command above them wrote, and nothing it wrote to stdout or any other command wrote (see the scope note below). A daemon's log lines are the stderr of the cmd= line that runs it.

Stderr must NOT contain text is reject=stderr:contains=, below.

Await (deterministic stderr fence)

await=stderr:contains=<text>[:timeout=<dur>][:then=stop]

Blocks the runner until the daemon's relayed stderr contains <text>, then tears the daemon down. This is a deterministic replacement for a blind time.sleep that only held the daemon open long enough for a line to appear. timeout= is an optional Go duration (default: 80% of the test budget, at least 10s); on timeout the test fails with a precise message that names the needles still missing. <text> follows the same rule as expect=stderr:contains=.

Several await= lines form ONE fence. It holds once every needle has appeared, in any order: an external plugin's stderr is relayed apart from the daemon's own writes, so an order between them would be a race of its own. A needle may be declared once, and timeout= on one line only.

Without then=stop, the fence is a precondition of the normal end: after it holds, the runner still waits for the peers to exit (a peer script that signals the daemon several times, test/reload/reload-rapid-sighup.ci, relies on this).

then=stop on any await line makes the runner END the test once the fence holds. Use it when a peer lingers (option=linger:value=true), because a lingering peer never exits by itself, so the default end waits forever and a fixture then had to stop the daemon a fixed delay after its signal. Under load that SIGTERM landed while the reload under test was still verifying, and shutdown canceled it. With then=stop the sequence waits on events only:

Step Waits for
1 every awaited needle on the daemon's stderr
2 every lingering check peer to print successful (its last expectation met), or to exit
3 the runner sends the daemon SIGTERM (a kill after the teardown grace) and reaps it; expect=exit:code= asserts that exit
4 every check peer to exit. A lingering peer ends when the daemon closes its session; one still running after the peer drain grace is sent SIGTERM

A plain fence publishes daemon.pid without waiting for daemon.ready, because a plugin that aborts startup may never write it. A then=stop fence keeps that wait: its daemon is expected to run, and a trigger that sends SIGHUP before the handler is installed kills the daemon.

Steps 1 and 2 share the fence's one deadline. A NON-lingering check peer is not waited in step 2, because it ends itself, and its expectation can be the Cease the stop in step 3 produces (test/reload/reload-dynamic-peer-survives.ci). Await the reload's OUTCOME line, never an earlier one: sighup reload complete for a reload that succeeded, reload error for one that was refused.

If either fence step fails, the runner retains that failure while stopping the daemon, signaling the peers immediately, and reaping them. A failed exchange gets no further completion grace that could exhaust the test's deadline before SIGTERM reaches the peers. The runner collects their final stdout and stderr before extracting received messages and writing --save files. This includes a lingering peer's shutdown output: the missing needle must not discard the dialogue that explains why it never appeared.

cmd=foreground:seq=3:exec=ze -:stdin=ze-bgp:timeout=20s
await=stderr:contains=incomplete peer definition:then=stop
await=stderr:contains=reload error
expect=stderr:contains=incomplete peer definition
reject=stderr:contains=sighup reload complete

Pair it with a matching expect=stderr:contains= so the line is both the fence and the assertion. Use it for the reject-fence bucket: an external plugin whose refusal aborts the daemon's plugin-startup coordinator (StartupCoordinator.PluginFailed) leaves no in-daemon observer able to poll it, so the relayed stderr line is the only non-plugin signal.

Choose a needle that is SPECIFIC to the plugin under test, not just a shared phrase. Example (test/plugin/as112-external-refuses.ci): the bare "refusing to start as an external plugin process" is emitted verbatim by three plugins, so the needle includes the as112-only tail -- the address-ownership registry:

cmd=foreground:seq=1:exec=ze -:stdin=ze-bgp:timeout=15s
await=stderr:contains=refusing to start as an external plugin process -- the address-ownership registry
expect=stderr:contains=refusing to start as an external plugin process -- the address-ownership registry

Syslog Expectations

expect=syslog:pattern=<regex>

Validates that captured syslog output matches the regex pattern. When any expect=syslog: line is present, the test runner automatically starts a UDP syslog server and injects ze.log.backend=syslog and ze.log.destination=127.0.0.1:<port> into the test environment.

File Expectations

expect=file:path=<rel>:exists=true
expect=file:path=<rel>:absent=true
expect=file:path=<rel>:contains=<text>
expect=file:path=<rel>:not-contains=<text>
expect=file:glob=<rel-pattern>:count=<N>
expect=file:glob=<rel-pattern>:contains=<text>
expect=file:glob=<rel-pattern>:not-contains=<text>

Validates files after the test process or peer sequence has completed. Paths and glob patterns are relative to the test's own work directory, which is where every child ran and where a daemon writes its artifacts. A test that declares tmpfs= files gets them in that same directory, so the two spellings name one place. For glob contains, at least one matched file must contain the text. For glob not-contains, no matched file may contain it.

Use file expectations for loose source files and exported artifacts such as logs. Persistent config, versions, pointers and plugin state use key expectations:

expect=key:path=meta/config/router.conf/active:exists=true
expect=key:path=meta/config/router.conf/candidate:absent=true
expect=key:glob=file/[0-9]*/router.conf:count=2
expect=key:glob=object/*:count=2
expect=key:glob=object/*:contains=router-id 1.2.3.4

A dated history entry holds sha256:<hex>, not the config: the bytes live in object/<hex> (docs/architecture/storage-backends.md, "Content-addressed history"). Count versions on the entries and assert config text on object/*.

Key expectations resolve beneath database/ in the work directory and decode the complete netcapstring frame before inspecting its value. A bad CRC or trailing frame bytes fail the assertion. They accept the same path, glob, count and containment checks as files. A missing key satisfies absent; an unreadable or corrupt key does not.

Negative Expectations (reject)

reject=stderr:contains=<text>
reject=stderr:pattern=<regex>
reject=stdout:contains=<text>
reject=stdout:pattern=<regex>
reject=syslog:pattern=<regex>
reject=bgp:conn=<N>:pattern=<hex>

Inverse of expect= -- the test fails if the pattern matches. Used to verify that unwanted output (e.g., deprecated warnings, ERROR-level messages) does NOT appear.

Type Description
reject=stderr:contains=<text> Fail if the command's output (stdout or stderr) contains substring
reject=stderr:pattern=<regex> Fail if the whole run's stderr matches regex, wherever the line sits
reject=stdout:contains=<text> Fail if the command's output (stdout or stderr) contains substring
reject=stdout:pattern=<regex> Fail if the command's output (stdout or stderr) matches regex
reject=syslog:pattern=<regex> Fail if syslog output matches regex
reject=bgp:conn=<N>:pattern=<hex> Fail if connection N of a ze-peer receives a frame carrying these wire bytes

"The command" is the cmd= line above the reject (see "Every stream assertion sits under a command and is scoped to it"). A command-scoped reject reads both of its streams, so the stream name selects nothing.

reject=bgp -- bytes a peer must never receive

reject=bgp is the only reject type ze-peer reads, because ze-peer is what sees the wire. It goes inside the peer's stdin block, beside the expect=bgp lines. The hex needle is matched on wire bytes at a byte BOUNDARY, which expect=bgp:contains= is not: that one is a plain substring match over the hex text, so it can also match at an odd nibble offset.

Five properties make it an assertion rather than a hope:

  • It is never consumed. Every frame the peer's message loop reads is checked against it. The option=linger loop keeps checking after completion. Pair the rejection with option=linger:value=true to hold it open for the whole test.
  • A rejection found during linger RETRACTS the success token. Under option=linger the peer prints its success token BEFORE the loop, because teardown is a kill and a post-run print can be lost. The verdict reads that token, so a rejection arriving afterwards has to withdraw it: the peer prints ZE-PEER-REJECTED on its own output, and the verdict reads the retraction after the token and lets it win. Without that channel a linger rejection was detected, returned, and then discarded, which made every negative assertion held open by linger vacuous.
  • Check mode only. Sink and echo peers read every accepted connection concurrently against one checker, so a conn= could not select the session the frame arrived on. The runner refuses the file, and ze-peer refuses to start.
  • The block MUST also carry an expect=bgp:conn=<N> on the same connection, and the runner refuses the file otherwise. That rule is necessary and it is not sufficient. It bounds how long the rejection is checked. It cannot say whether the forbidden bytes would have arrived inside that window. The author owes the second half. Send the delivery LAST on that connection, so a leak of the governed route arrives ahead of it. Or hold the session open with option=linger. A connection stops being read once its expectations complete, and its rejection then passes for the one reason a rejection must not.
  • An odd-length or non-hexadecimal needle is a parse error, because a needle that can never match would pass for that same reason. The runner reads the line with ze-peer's own parser (peer.ParseRejectRule), so a typo fails the FILE rather than the peer: a rejection rejected inside ze-peer would stop it binding, and the runner would report that as a bind timeout.
stdin=external:terminator=EOF_EXTERNAL
option=tcp_connections:value=1
option=linger:value=true
# The delivery that makes the rejection an assertion rather than an empty session.
expect=bgp:conn=1:seq=1:contains=18C00002
# 180A0100 is 10.1.0.0/24, which NO_ADVERTISE forbids on any peer.
reject=bgp:conn=1:pattern=180A0100
EOF_EXTERNAL

Assertion Strength: accept-only tests and readback

A test whose ONLY assertion is expect=exit:code=0 is accept-only (weak): it proves a config or command was ACCEPTED, never that it parsed to the CORRECT tree. A parser that accepts interval 300 but stores 0, or silently drops a source 0.0.0.0/0 block, still passes such a test green. This is the functional-suite analog of the "count-only assertion" mistake class (ai/rules/testing.md).

A lint enforces that the class cannot GROW: TestCIAcceptOnlyLint walks every test/**/*.ci, classifies each with the single accept-only predicate, and FAILS on a NEW accept-only test that is neither strengthened nor annotated. Existing accept-only tests are grandfathered in test/.accept-only-baseline (a sorted allow-list that only shrinks; strengthening or annotating a test removes its line). Correctly EXCLUDED (never weak): a test whose real check lives in a tmpfs set -e script (e.g. test/managed/auth-reject.ci), and any reject= test.

Strengthen with a readback

Add a second cmd= that dumps the parsed tree and assert a representative value with expect=stdout:contains= / pattern=. show config dump - reads a config from stdin and answers the stored tree, and | json renders it, so the assertion observes the parsed VALUE, not just that parsing did not error.

cmd=foreground:seq=1:exec=ze config validate -:stdin=config:exit=0
cmd=foreground:seq=2:exec=ze cli -c "show config dump - | json":stdin=config-dump
expect=exit:code=0
expect=stdout:pattern="interval": "300"

Two gotchas, both load-bearing:

  • ze config dump requires a bgp { } block (it resolves the full BGP tree) where ze config validate does not. To keep proving that a subsystem-only config validates, keep the original validate step on the subsystem-only stdin block and run the dump readback against a second stdin block that prepends a minimal bgp { router-id ... }.
  • A needle containing : followed by something shaped like key= must use pattern=. ~~Only json=/text=/hex=/pattern= preserve colons, and contains= truncates at the first colon.~~
    • Corrected 2026-07-30. That claim was true until ParseKVPairs became boundary-aware. contains= now keeps an ordinary colon, so contains=error: no such peer asserts the whole sentence.
    • What still splits is a colon that introduces a real key token: a letter, then letters, digits, - or _, then =. That split is deliberate, because it is how the engine-step form contains=aes-cbc:timeout=25 keeps working. So contains=note:level=high still splits at :level=, and it is the one shape that needs pattern=.
    • The old behavior was not harmless while it lasted. A sweep on 2026-07-29 found 203 assertions across 15 suites silently reduced to the text before their first colon. The re-armed assertions then exposed a security test (test/appliance/appliance-push-image-escape.ci). That test had never once executed the path-traversal guard it was named for.
  • Keep a pattern= needle free of the substrings json=, text=, and hex=. ParseKVPairs extracts a complex-key value by strings.Index of the first such marker, so a needle that itself contains one of them is mis-split. None of the readback needles above contain these markers. This note is forward guidance for new needles.

Annotate when a unit test already covers the value

When a unit test already asserts the parsed value, a readback would duplicate it. Mark the test accept-only instead, with a comment naming the covering test:

# accept-only: md5 { password; ip } value-parsing is unit-covered by
# TestParsePeerMD5FieldsParsed (internal/component/bgp/reactor/config_test.go).

The marker is a comment line whose content is accept-only: followed by a non-empty reason. A file carrying it is allowlisted by the lint without a baseline entry. Keep the reason greppable and truthful (name the covering test).

Actions

action=<type>:key=value[:key=value...]

Notification

action=notification:conn=<N>:seq=<N>:text=<message>

Sends NOTIFICATION with shutdown message.

Send Raw

action=send:conn=<N>:seq=<N>:hex=<hex-bytes>

Sends raw bytes to peer.

Rewrite Config File

action=rewrite:conn=<N>:seq=<N>:source=<tmpfs-file>:dest=<config-file>

Copies a tmpfs file over the daemon's config file. Used with action=sighup to test config reload.

It also writes a marker a second process can wait on, which is its other use: test/reload/config-apply-ordering-address-swap.ci copies one file to session-returned.txt once the restarted BGP session has re-announced its routes, and the test's own driver waits for that file before it signals the daemon. When the rewrite is the LAST item the peer block queues, the exchange is over and the peer completes on it, exactly as action=sighup and action=sigterm do. Without that the peer kept matching, and the daemon's shutdown NOTIFICATION arrived against an empty expectation list and failed the test as a message mismatch.

Key Description
conn Connection number triggering the rewrite
seq Sequence number (after matching messages)
source Source file name in tmpfs
dest Destination file name in tmpfs (usually ze-bgp.conf)

Send SIGHUP

action=sighup:conn=<N>:seq=<N>

Sends SIGHUP to the daemon process. Reads PID from daemon.pid in the tmpfs directory (written automatically by the test runner).

Key Description
conn Connection number triggering the signal
seq Sequence number (after matching messages)

Send SIGTERM

action=sigterm:conn=<N>:seq=<N>

Sends SIGTERM to the daemon process. Reads PID from daemon.pid in the tmpfs directory (written automatically by the test runner). After sending SIGTERM, the connection is expected to close (daemon shuts down gracefully).

Key Description
conn Connection number triggering the signal
seq Sequence number (after matching messages)

HTTP Checks

HTTP checks validate web endpoint responses after all cmd= processes have started. Executed in seq order with automatic retry on connection errors (server starting up).

Under the per-test namespace mode (./le test qemu netns) the daemon listens on the namespace's own loopback. net/http dials from a goroutine of its own, which does not run on the runner's locked thread, so the runner's transport opens each socket inside the test's namespace (netnsDialContext). Before it did, every http= line in a namespace test was refused.

Assertion Checks (get/post)

http=get:seq=N:url=URL:status=CODE[:contains=TEXT][:bodyfile=PATH][:header=NAME: VALUE]
http=post:seq=N:url=URL:status=CODE[:contains=TEXT][:bodyfile=PATH][:sendfile=PATH][:content-type=TYPE][:header=NAME: VALUE][:insecure-tls=true]
Key Required Description
seq ✓ Execution order (>= 1, lower first)
url ✓ Request URL (supports $PORT and $PORT2 substitution)
status ✓ Expected HTTP status code
contains ✕ Expected body substring
bodyfile ✕ Path to file with expected body (exact match, resolved relative to .ci file)
sendfile ✕ Path to file sent as POST request body, resolved from tmpfs first
content-type ✕ Request body content type for sendfile, defaults to application/json
header ✕ Request header in Name: Value wire form. Repeatable -- the only key that can appear more than once on one line
insecure-tls ✕ Set true for self-signed local HTTPS endpoints

Request Headers (header=)

header= takes the header exactly as it appears on the wire, Name: Value. Repeat header= to set several headers on one check. It works on get, post, and wait.

http=post:seq=1:url=http://127.0.0.1:$PORT/mcp:status=200:sendfile=call.json:header=MCP-Protocol-Version: 2026-07-28:header=Mcp-Method: tools/call:header=Mcp-Name: ze
Behavior Detail
Splitting On the first colon only, so a value can contain colons (header=Referer: http://127.0.0.1:8080/page)
Whitespace Trimmed around both name and value, so header=Foo: bar and header=Foo:bar are equivalent
Precedence Applied after the sendfile default Content-Type, so an explicit header=Content-Type: ... wins
Repeats of one name The first occurrence replaces the value. Each later occurrence adds one more value to that field
Host Routed to the request's Host field, because net/http ignores a Host entry in the header map
Malformed A header= value with no colon is a parse error naming the offending value, never a silent drop
Value stops at The next known key marker (:status=, :contains=, the next :header=, ...), like every other key

Retries up to 20 times at 200ms intervals on transient connection errors (ECONNREFUSED, ECONNRESET, EOF). Non-connection errors (wrong status, missing content) fail immediately.

Readiness Polls (wait)

http=wait:seq=N:url=URL:status=CODE[:contains=TEXT][:header=NAME: VALUE][:timeout=DUR]
Key Required Default Description
seq ✓ - Execution order (>= 1)
url ✓ - Request URL
status ✓ - Expected HTTP status code
contains ✕ - Expected body substring
header ✕ - Request header in Name: Value form, repeatable (see above)
timeout ✕ 15s Poll timeout duration

Unlike assertion checks, wait retries on all failures: connection errors, wrong status codes, and content mismatches. Polls at 500ms intervals until the condition is met or the timeout expires. Wait checks run before assertion checks, making them suitable for waiting until a server has populated data (e.g., routes injected by an async plugin).

Example

cmd=background:seq=1:exec=le test peer --port $PORT:stdin=peer
cmd=background:seq=2:exec=ze -:stdin=ze-bgp

# Wait until routes are available before checking graph output
http=wait:seq=1:url=http://127.0.0.1:$PORT2/lg/graph?prefix=10.10.1.0/24&mode=aspath&format=text:status=200:contains=AS2914:timeout=15s

# Exact SVG match against reference file
http=get:seq=1:url=http://127.0.0.1:$PORT2/lg/graph?prefix=10.10.1.0/24&mode=aspath:status=200:bodyfile=expect/graph.svg

# Substring check
http=get:seq=2:url=http://127.0.0.1:$PORT2/lg/graph?prefix=10.10.1.0/24&mode=nexthop&format=text:status=200:contains=egress

Sleeps and their justification markers

Every time.sleep( in a live .ci carries a marker comment in the form // sleep(<kind>): <reason>, on one # comment line directly above the sleep, indented to match it exactly. The embedded .ci observer body is indentation sensitive. Two producers enforce the marker: checkSleepJustification in internal/le/doc/wiring (run by ./le doc wiring, scoped to changed .ci files, listing every unjustified file:line and exiting 1), and writeCISleep in internal/le/hookruntime/writeedit.go, which blocks a Write or Edit that introduces an unmarked sleep.

The kinds are a closed set, and each one owes a different reason:

Kind What the reason states
poll-interval The real condition the enclosing loop breaks or returns on. This is already a deterministic wait; the sleep is only its granularity
timer The delay itself IS the behavior under test, and the mechanism plus where its period is set
timeout-under-test The fixed internal timeout the sleep waits out, which the test asserts on
needs-linux A dataplane effect (tc, qdisc, nft, kernel FIB) with no readback in the driver, convertible only after a QEMU run
✕ The awaited effect exposes no queryable state to this driver, and what is held until instead

A free-text // settle comment is insufficient: it names no mechanism a reader can check. "The tracker pushes live carrier once a second" is a reason a later reader can overturn; "needs a moment" is the shape that makes a deliberate timer and a guessed duration the same line of code.

A separate ratchet caps how MANY sleeps exist: the total time.sleep( count across test/**/*.ci may not exceed the committed baseline in test/.ci-sleep-baseline, and ./le doc wiring fails when it does. The markers cap how many are unexplained.

Registered engines launched externally

le mock plugin-external <name> runs a registered engine through a genuine TLS connect-back so a test can exercise its external-process refusal or warning. Before configuring the engine logger, the launcher calls crashlog.Flush to restore the stderr pipe managed by the parent daemon. The logger therefore writes each diagnostic into that pipe before returning.

A refusing engine closes its SDK connection on return, and the parent can kill the child as soon as registration fails. Leaving a second asynchronous stderr queue inside the child let that kill discard the refusal before the parent's relay received it. This exception applies to this test launcher only; runtime crash-file capture remains armed. Standalone production external plugins do not use this registered-engine launcher.

The compiled observer API

Compiled .ci observers live under internal/test/fixture. They use pkg/plugin/sdk for the five-stage plugin protocol (Plugin.Run owns it) and the local fixture package for registration, dispatch, polling and failure reporting.

Function Purpose
fixture.Register(name, driver) Register one compiled fixture command
fixture.Run(args) Dispatch le test fixture <name> [args...]
fixture.Observe(...) Connect through the SDK, complete startup, run the scenario after all plugins are ready, then request shutdown
observeConfigured(...) Install callbacks before startup, then run the same observer lifecycle. It is unexported, so only a fixture in this package calls it
fixture.Dispatch(...) Send one command and decode its JSON answer into a Go value
fixture.Poll(...) Retry a predicate until success, exhaustion, or context cancellation
fixture.ReportFailure(err) Emit the ZE-OBSERVER-FAIL sentinel checkObserverSentinel detects. internal/test/runner/runner_validate.go
sdk.Plugin.DispatchCommand(...) Send a typed command request through the plugin connection

fixture.Poll around fixture.Dispatch is the payload-predicate wait: it blocks until the observed payload matches, within a bounded attempt count. It is what replaces a time.Sleep followed by a one-shot assertion.

fixture.Observe can request a clean daemon shutdown even after an assertion failed, so the daemon's exit code does not prove the observer's assertion. A successful scenario must also complete request quiesce with status done. Transport errors, engine refusals and other statuses fail the observer. The observer reports ZE-OBSERVER-FAIL before requesting shutdown, then returns the error to fixture.Run.

For process-status assertions, le test fixture runner/helper-command-exit <stdout-line> <exit-code> writes the line and exits with the requested status from 0 through 255. A requested nonzero status is test input, not an observer failure, so it emits no ZE-OBSERVER-FAIL. Invalid arguments and output errors still follow the normal fixture failure path.

A fixture-driven .ci needs no bgp block and no le test peer to get a daemon that stops. request shutdown reaches a reactorless daemon through the shutdown callback the daemon wires beside its plugin server, before any plugin can dispatch, so a BFD-only or DHCP-only configuration stops on the fixture's request like a BGP one. Adding a BGP peer only to make the daemon stoppable adds a second protocol to the test's failure surface: test/bfd/bfd-detection-interval.ci was red for exactly that reason until 2026-09-07.

option=asn moves both AS numbers. The AS is resolved once and written into the two-octet My Autonomous System field AND the four-octet AS capability, so the two disagree only where a .ci states a malformed capability 65 itself (see "Capability Control" above). RFC 6793 Section 4.1 states the precedence a receiver applies: it "MUST use the AS number encoded in the Capability Value field of the 'support for four-octet AS number capability' in lieu of the 'My Autonomous System' field of the OPEN message". So a half-applied option leaves the capability carrying ze's own AS, ze reads THAT, finds an AS the session is not configured for, and answers OPEN Message Error / Bad Peer AS under RFC 4271 Section 6.2. The refusal is RFC 4271's; RFC 6793 decides which field ze read.

The sentinel is written where the scenario fails, BEFORE request shutdown. The runner reads it out of the DAEMON's stderr, and the daemon relays a plugin's stderr only while both processes live (internal/component/plugin/process, relayStderrFrom). fixture.Run reports the same error after Plugin.Run returns, which is after the daemon was asked to stop, so that line reaches the runner only when it wins the race with the shutdown. Measured 2026-09-02: an observer asserting a route that can never arrive still passed test/plugin/rpki-group-action.ci, and three plugin cases (fib-table, metrics-name-show, modify-increment-localpref) passed with a failing observer. fixture.ReportFailure emits the FIRST failure only, so the report at the failure site and the one on the way out are one line.

Custom shutdown owners retain the same order: complete the successful scenario's final quiesce, report its error, publish its result, then request shutdown. runPlugin13 supplies that barrier when its shutdown flag is true. A started scenario retains its verdict if the plugin transport closes first. The actual runPlugin13 and plugin15ObserveStarted regressions cover successful quiesce, transport loss, engine refusal and warning status through the authenticated SDK transport.

A driver's working directory is the per-test directory, so a driver names its files relatively and fixture.Run refuses to start in the checkout. The runner creates one directory for each test, gives it to every child of that test and removes it at the end (Record.WorkDir, internal/test/runner/runner_exec.go), so daemon.ready, testkey and <case>.conf are written where the test can find them and nothing survives the run. The same names in the checkout land beside tracked source: on 2026-09-05 a run left two ed25519 private keys and five config files at the repository root, none of them ignored, so a git add -A from any session would have committed a private key. fixture.Run therefore reads its own working directory first and exits 1 when that directory holds the ze go.mod (sessionpath.IsRepoRoot). The runner never starts a fixture there, so the refusal is written for a person who typed ./le test fixture <name> at the root: plain text, not a ZE-OBSERVER-FAIL line, ending with the command that runs each .ci test whose step starts the fixture, such as ./le test ui pipe-interactive-save. It reads every test/<dir>/*.ci, skips # comment lines, and asks the one resolver of which runner walks a directory (testfunctional.CIRunnerWords): the functional suite table, then the le test bgp directories, then a registered le test <dir> command. A suite's name is not always its directory (suite installer runs test/install/ through ./le test install), and registered runners need not have a suite row, so the suite table alone cannot resolve every harness. PPPoE has a non-gating suite row and uses its existing le test pppoe runner, including for suite "pppoe --draft" stress reproduction; its platform and namespace requirements remain unchanged. Each fixture command is its runner's words before --all, then the file's stem. When no .ci test starts the fixture, it prints the exec= step to add instead. A driver that needs the checkout reads $ZE_REPO_ROOT, which the runner exports; it never reads the working directory for one.

An observer assertion is therefore worth only what its wait is worth: a value that a plugin fills asynchronously (the Adj-RIB-In after RPKI validation, the SPF table after the first run, a session that drops when the peer completes) needs fixture.Poll around the read, never a fixed settle before it.

Engine Steps

Engine steps drive a live daemon through CLI dispatch, first-class in .ci instead of an embedded Python observer. The runner serializes the parsed steps to engine-steps.json in the test tmpfs, and links it into the daemon-N/ config directory of every further daemon, because a plugin runs in its daemon's config directory; the .ci declares the executor as an external plugin (run "le mock engine-steps ./engine-steps.json"), which runs the steps from OnAllPluginsReady and reports failures via the ZE-OBSERVER-FAIL sentinel the runner gates on.

The executor reads JSON from a file, or stdin when the path is -. Every numeric kind must name an EngineStepKind before any steps are published or TLS is attempted. Missing, zero, and unknown values fail with ZE-OBSERVER-FAIL and an indexed unknown-kind parse error; they are not skipped.

command=<cli command text>
stream=<monitor command text>
expect=output:<predicate>[:timeout=<dur>]
expect=event:namespace=<ns>:name=<name>[:timeout=<dur>]
expect=stream:<predicate>[:timeout=<dur>]
expect=command-error:contains=<text>

command=/stream= keep their full raw text (colons included). expect=output re-dispatches the most recent command= until its predicate holds or the timeout expires; expect=stream matches delivered stream= events; expect=event matches a delivered event by its (namespace, name) identity.

A command= line carrying a pipe chain (show doctor config <file> | json) is split the way the SSH exec channel splits one: the executor dispatches the command alone and renders the daemon's answer through the chain, so the step reads what ze cli -c prints for the same line. The daemon's dispatcher gives | no meaning, so a chain sent whole would reach the handler as arguments. A line with no pipe gets the raw JSON answer, with no default format added, which is what json= predicates walk. A chain the pipe layer refuses fails the step before anything is dispatched, and expect=command-error can read it.

expect=event

The executor declares ONE startup event subscription, derived from the expect=event steps of the file itself, and asks for enveloped delivery. A .ci therefore names each event once, in the step that waits for it, and nothing else declares the subscription.

A startup subscription rides the ready RPC, so it is registered before any plugin's OnAllPluginsReady runs. That is what lets a step observe the FIRST delivery of an event the daemon emits while it starts. test/ipsec/ipsec-sa-installed.ci is the case that forced it: the IKE engine emits vpn-ipsec/sa-up from its startup peer reconciliation, so a subscription dispatched from the step could only ever see a SECOND establishment, and a test with no re-negotiation has none.

Enveloped delivery wraps each event with its (namespace, event) identity (rpc.EventEnvelope), which is what tells one subscribed event from another once several share the one subscription. A bare payload decodes with an empty namespace and event, so a delivery the executor did not ask to be enveloped can never satisfy a step.

Two consequences follow, and both are load-bearing when writing a test:

  • A step list names ONE namespace. rpc.SubscribeEventsInput carries one, so steps naming two are REFUSED before the executor dials, with a message naming both. Split the test rather than subscribing to one and dropping the rest.
  • A step matches ANY delivery of that identity, scrollback included. It cannot assert "another one, after this point". A second sa-up after an operator clear re-matches the first, so re-establishment is asserted by a separate test (test/ipsec/ipsec-clear-reestablish.ci), not by a second expect=event.

expect=command-error

expect=command-error:contains=<text> asserts that the PRECEDING command= FAILED, and that its message contains the text.

Use it for a command that must refuse. The plugin SDK turns a StatusError response into a Go error, so without this directive the command step aborts the run before any expect= is reached, and no .ci can assert an operational error at all. That left one class untestable end to end, and it is the class where a wrong answer costs most: a command that must refuse is exactly the one whose failure mode is answering confidently instead. test/ipsec/ipsec-dataplane-show.ci uses it to prove that a dataplane which cannot be read SAYS so rather than rendering an empty table.

It takes contains= only, and no timeout. An error is the result of one dispatch, so re-dispatching until one appears would wait for a state change no expect= can cause.

A command failure that NO expect=command-error consumes still fails the run, whether the next step is of another kind or the command is the last step in the file. Every .ci written before this directive existed relies on that.

It must not be vacuous. A command= that SUCCEEDS does not satisfy expect=command-error: the step fails with "the preceding command succeeded". Without that rule the directive would pass for the very regression it guards, a refusal quietly becoming a successful empty answer (TestRunEngineStepsCommandErrorRequiresAFailure).

expect=output / expect=stream predicates

The optional trailing :timeout=<dur> is split off the END first, so a predicate operand may itself contain : (a compact-JSON fragment, an IPv6 address). The remainder is one predicate:

Predicate Surfaces Holds when
contains=<text> output, stream the output contains the substring (the default)
matches=<regexp> output, stream the Go regexp matches the output (compiled at parse time, so a bad regexp fails the test immediately, not at timeout)
absent=<text> output only the output does NOT contain the substring
json=<dotted.path>=<value> output only the dotted path into the JSON data field stringifies to <value>

json= walks the raw data field (not status data): each .-segment indexes a JSON object by key or a JSON array by integer index (0..len-1; out-of-range or missing is "not yet", named at timeout). The leaf is compared as a string (numbers/bools stringified via JSON). absent=/json= are expect=output only: they re-dispatch a query, whereas expect=stream is an append-only event stream with no "absent" and no single-event JSON path.

absent= must be non-vacuous. An absent= on output that was never populated passes instantly (false green). Precede it with a step that makes the substring present (a contains=/json= after an inject), then the transition (e.g. a withdraw) the absent= proves. See test/plugin/engine-steps-predicates.ci.

Example

command=request bgp rib inject 10.0.0.1 ipv4/unicast 172.16.0.0/16 origin igp nexthop 10.0.0.2
command=show rib
expect=output:matches=172\.16\.[0-9.]+/16:timeout=10
expect=output:json=0.prefix=172.16.0.0/16:timeout=10
command=request bgp rib withdraw 10.0.0.1 ipv4/unicast 172.16.0.0/16
command=show rib
expect=output:absent=172.16.0.0/16:timeout=10

Complete Example

# Embed config using Tmpfs
tmpfs=test.conf:terminator=EOF_CONF
peer test-peer {
    remote {
        ip 127.0.0.1;
        as 65000;
    }
    router-id 10.0.0.2;
    local-address 127.0.0.1;
    local-as 65533;
    hold-time 180;

    family {
        ipv4/unicast;
    }
    announce {
        ipv4 {
            unicast 10.0.0.0/24 next-hop 10.0.1.254;
        }
    }
}
EOF_CONF

# Test configuration
option=file:path=test.conf
option=asn:value=65000

# Expected API command and wire output
cmd=api:conn=1:seq=1:text=update text origin igp next-hop 10.0.1.254 nlri add 10.0.0.0/24
expect=bgp:conn=1:seq=1:hex=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF002F02000000144001010040020602010000FFFD4003040A0001FE180A0000

# EOR
cmd=api:conn=1:seq=1:text=announce eor ipv4/unicast
expect=bgp:conn=1:seq=1:hex=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF00170200000000

Consumers

Different components consume different line types:

Line Type Consumer
stdin= Test runner (pipes to processes)
tmpfs= Test runner (writes to temp)
option= Test runner + ze-peer
cmd=api: nobody. parseCmd stores the text on the message and only the failure reporter prints it, so the line documents the command that produced the expected bytes and executes nothing. A test that expects it to inject routes asserts against an empty RIB
cmd=foreground:, cmd=background: Test runner (process orchestration)
expect=exit:, stdout:, stderr:, json:, syslog:, file: Test runner
reject=stderr:, reject=syslog: Test runner (negative expectations)
http=get:, http=post: Test runner (HTTP assertion checks)
http=wait: Test runner (HTTP readiness polls)
expect=bgp: le test peer
action=notification:, action=send: le test peer
action=rewrite:, action=sighup:, action=sigterm: le test peer (reload/signal tests)

Lines not recognized by a consumer are ignored.

A check-mode peer block MUST declare a ze-peer-consumed expectation

The consumer split above is load-bearing, not trivia. Only the four le test peer rows reach le test peer. Everything else -- including expect=json -- is validated by the test runner from its own copy of the messages.

A check-mode le test peer with no consumed directive has nothing to check, so it prints no test data available to test against and exits 1 before binding a listening socket. ze then dials a dead port, gets connection refused, and backs off 5->10->20->40s. That looks exactly like a BGP establishment stall and cost a multi-day investigation before the cause was found in the harness.

So a peer block whose only expectation is expect=json runs no BGP at all. The parser now rejects that at discovery time, naming the file and the remedy:

stdin=peer block: check-mode ze-peer (cmd seq=1) declares no ze-peer-consumed
expectation, so it exits with "no test data available to test against" before
binding a listening socket and the test can only pass vacuously.
Want Do
Assert the wire exchange Add expect=bgp:conn=N:seq=N:hex=... (or an action=send/notification/rewrite/close/sighup/sigterm) to the peer block
A peer that is only a dial target for ze (routes injected through an external process plugin, assertions made by that plugin or http=) Run it as le test peer --mode sink -- sink/echo/inject peers legitimately declare nothing

expect=json still works, but only in addition to a consumed directive: it cannot make the peer listen.

A ze-peer always governs its test's result

A test with a check-mode ze-peer is never self-validated: the peer must print successful and its expect=json expectations must match, whatever else the file asserts. An expect=exit:code=0 is additive and does NOT disable the BGP checks.

Until 2026-07-16 it did, which is how a test could pass on ze's exit code while its peer never listened and its JSON assertion never ran. sink/echo/inject peers do not govern (they never report completion), so a test whose only peers are scaffolding is still governed by its own exit/output/file assertions.

A check peer prints successful only when EVERY expectation it holds was met: every expect=bgp line of every conn=N it declares. A peer stopped before that, by SIGTERM from the runner, by the test's timeout, or because a dialing peer's one connection ended while conn=2 was still owed, exits 1 with stopped with expectations unmet: conn=N still owes <expectation> (K expectations unmet). Until 2026-09-24 a peer that SIGTERM reached while it waited for its next connection printed successful, so a test passed with its later connections never opened.

A scaffolding ze-peer is signaled at teardown

A sink, echo or inject le test peer never ends itself: its accept loop runs until its context is cancelled, and le test peer maps SIGTERM to that cancel. The runner sends that SIGTERM at teardown, after the last step and before the barrier that collects peer output. The peer exits with status 0 and its capture is complete, so a .ci author needs no teardown directive and must not add one.

A check-mode peer is never signaled. It exits by itself when its expectations are met, and the runner reads its verdict from the capture a signal would truncate.

Before 2026-08-10 no code signaled a scaffolding peer, so the drain barrier waited out its full 10s grace on every --mode sink or --mode echo test. That cost was not only latency: test/plugin/event-predicate-wait.ci failed at its 15s budget with TYPE: timeout while the daemon itself completed correctly.

Migration from Old Format

Old format (deprecated):

option:file:test.conf
option:asn:65000
1:raw:FFFF...
1:json:{...}

New format:

option=file:path=test.conf
option=asn:value=65000
expect=bgp:conn=1:seq=1:hex=FFFF...
expect=json:conn=1:seq=1:json={...}

Key changes:

  • = instead of : after action
  • Explicit conn= and seq= for message ordering
  • hex= prefix for wire bytes
  • json= prefix for JSON data

Editor Test Format (.et)

Editor resource declarations use the shared .ci grammar for option=needs-linux[:caps=<tokens>] and option=exclusive:group=<name>. Discovery rejects unknown resource keys, duplicate keys, invalid capabilities, missing groups, and invalid or overflowing effective timeouts. A parse failure outranks a capability skip. The native editor runner retains the discovered snapshot and holds its group lease through child reaping and cleanup.

The .et format extends .ci for interactive editor testing. Tests are located in test/editor/.

Overview

Editor tests simulate user input sequences against the headless configuration editor and verify state changes.

Input Actions

Action Purpose Example
input=type:text=<text> Type text input=type:text=edit bgp
input=key:name=<key> Send special key input=key:name=tab
input=tab Tab key (shorthand) input=tab
input=enter Enter key (shorthand) input=enter
input=ctrl:key=<c> Ctrl+key input=ctrl:key=u
input=space Space key input=space

Named keys input=key:name=<key> accepts: tab, enter, esc, escape, up, down, left, right, backspace, delete, home, end, pgup, pgdn, pgdown, space. shift+tab is handled separately as a modifier.

API candidate fixtures

fixture=api-set:path=<full setting path>:value=<value> writes a candidate through the same Editor.SetValue API that config sessions use. The path is absolute and uses the editor's quoted word syntax; the value is the complete text after :value=, including colons or an empty string. The fixture runs in step order on the selected editor, including a named session. Unknown fixtures and declarations missing a path or value are parse errors.

Use it to test commit-time validation of a candidate another surface could leave. A typed set still runs the normal immediate validation and cannot seed a refused value. The next commit, commit verify or show | errors validates the candidate through the product code; the fixture supplies no validation result and writes nothing to the active configuration.

Expectations

Expectation Purpose Example
expect=context:path=<p> Context equals path expect=context:path=bgp.peer.1.1.1.1
expect=context:root Context is root expect=context:root
expect=completion:contains=<list> Completions include all items expect=completion:contains=set,delete,edit
expect=completion:excludes=<list> Completions must NOT include items expect=completion:excludes=vpp,kernel
expect=completion:count=<N> Number of completions expect=completion:count=5
expect=ghost:text=<suffix> Ghost text suggestion expect=ghost:text=-id
expect=dirty:true Has unsaved changes expect=dirty:true
expect=content:contains=<text> Config content includes text expect=content:contains=router-id
expect=content:not-contains=<text> Config content must NOT include expect=content:not-contains=old-value
expect=content:lines=<N> Config content line count expect=content:lines=5
expect=content:pattern=<regex> Config content or displayed output matches a Go regular expression (RE2). . stops at a newline unless the expression starts with (?s); an empty or invalid expression fails the step expect=content:pattern=(?s)peer zulu.*peer alpha
expect=viewport:contains=<text> Displayed output includes text expect=viewport:contains=10.0.0.1
expect=viewport:not-contains=<text> Displayed output must NOT include expect=viewport:not-contains=error
expect=viewport:pattern=<regex> Displayed output matches a Go regular expression, as content:pattern. The way to assert order: (?s)a.*b passes only when a prints before b expect=viewport:pattern=(?s)peer zulu.*peer alpha
expect=errors:count=<N> Validation error count expect=errors:count=0
expect=status:contains=<text> Status message expect=status:contains=committed
expect=hint:contains=<text> Second message line includes text expect=hint:contains=show the peers
expect=hint:empty Second message line is blank expect=hint:empty
expect=explanation:contains=<text> Revealed long explanation includes text expect=explanation:contains=established
expect=explanation:empty ✕ expect=explanation:empty
expect=error:none ✕ expect=error:none
expect=timer:active Confirm timer running expect=timer:active
expect=file:path=<rel>:contains=<text> On-disk file content expect=file:path=test.conf:contains=bgp
expect=file:path=<rel>:not-contains=<text> File must NOT contain expect=file:path=test.conf:not-contains=old
expect=file:path=<rel>:absent=true File does not exist expect=file:path=test.conf:absent=true
expect=key:path=<key>:contains=<text> Decoded persistent value expect=key:path=file/active/test.conf:contains=bgp
expect=key:path=<key>:not-contains=<text> Decoded value excludes text expect=key:path=file/active/test.conf:not-contains=old
expect=key:path=<key>:absent Key does not exist expect=key:path=file/active/test.conf.draft:absent

Pattern expectations match displayed output after stripping ANSI escape sequences, so a line anchor sees the configuration row rather than its color prefix. The viewport is not trimmed or reordered, and missing rows still fail. The working configuration alternative of expect=content:pattern= remains literal. Literal contains and not-contains expectations are unchanged. Failed viewport patterns retain the original output in their diagnostic.

Editor tests use a tree store by default; option=storage:value=tree states that choice explicitly. The tree mode is held in memory (storage.CreateMemory), not on disk: it is the live store's contract (name mapping, guards, versions, metadata, observers) over a map, so no commit waits on a sync, and it closes with the test. option=storage:value=blob runs the test on an on-disk backup artifact the way ze config edit --backup opens one: the runner creates backup.zefs, seeds it, closes it, reopens it writable through storage.OpenBlob, and builds each session with cli.NewOfflineSessionEditor, so a commit publishes into the artifact. Blob mode needs option=session:user=.... Other values are refused. The runner seeds config fixtures once and all sessions and restart= steps share the same lifetime store handle. expect=key reads that handle, while expect=file reads the original loose fixture or a separate exported artifact.

hint reads the second message line with every style stripped. That row carries the completion hint, and the summary of the candidate the operator selected. explanation reads the long explanation Tab reveals, without the box that frames it on screen. Both accept empty and contains=. Both refuse an expectation that names neither key.

A test that runs option=mode:value=command or option=mode:value=operational completes against the real command tree, built from the registered -cmd YANG modules. The tree carries no value hints and no plugin commands. A hint provider reads the live RIB, and a plugin command comes from the live dispatcher. A headless test has neither, so a completion over a VALUE stays empty there.

Dimensions and Timeout

Editor dimensions use option=width:value=<columns> and option=height:value=<rows>. Missing, nonnumeric, nonpositive and overflowing integer values are parse errors. Dimensions default to 80×24 and apply to the initial model, each new session and every restarted model through the terminal resize event. option=timeout:value=<duration> defaults to 30 seconds and refuses nonpositive durations or overflow after concurrency headroom is applied.

restart=editor recreates the selected editor using its own construction mode and session identity. A named session= keeps its user and origin across restart and switch-back; it does not inherit the initial file session's identity.

Wait Actions

Model copies share one session context and clock. Validation, draft polling, confirmation countdown and live-view poll ticks use stoppable timers; closing the model cancels pending ticks. Delivery checks the originating model lifetime before applying the message, so an expired tick cannot affect a restarted editor. Validation also retains its generation check. Authored editor runs use real timers; controlled clocks are confined to focused runtime tests.

The model declares timer work with TimerMsg. Headless execution runs ordinary commands and every state-command result to completion, including nested batches and sequences. Only declared timers run asynchronously; command duration never classifies work as a timer. Expectations settle validation without waiting for recurring display ticks. Restart and case cleanup cancel and join pending model timers before releasing the editor.

Action Purpose Example
wait=ms:<N> Wait N milliseconds wait=ms:200
wait=validation Wait for validation wait=validation
wait=timer:drain Apply timers pending at entry once wait=timer:drain
wait=timer:expire Wait for timer expiry wait=timer:expire

wait=ms:<N> pumps timer deliveries for at least N real milliseconds (zero is allowed). Negative, noninteger and overflowing durations are parse errors. wait=validation applies the current generation before the next state command; older debounce ticks cannot satisfy it. A settled editor returns immediately. wait=timer:drain waits for and applies every declared timer pending at entry, including its complete state-command chain. Timers rearmed by those deliveries remain pending for later steps. No elapsed-time threshold establishes readiness; an empty entry set or cancellation fails the wait. For a fake ping monitor, place it immediately after Enter so the real 50ms first poll drains the prepared replies before viewport or saved-file assertions. wait=timer:expire requires an active confirmation timer on entry and waits for observed timeout rollback. An inactive timer, accept/abort, cancellation, missing countdown/poll or rollback error fails the wait. Session editors observe the real daemon window through draft polling; they do not synthesize its expiry. If the daemon's rollback fails and its window remains open for retry, the wait reports the structured rollback cause at the next poll instead of waiting for the case timeout. The window stays open; reporting failure does not accept it.

Unknown wait actions and malformed arguments are parse errors. Wait failures are recorded on the wait step, including its cause.

Example

# Test: Edit navigation
tmpfs=test.conf:terminator=EOF_CONF
bgp {
  router-id 1.2.3.4;
  peer upstream1 {
    remote {
      ip 1.1.1.1;
      as 65001;
    }
  }
}
EOF_CONF

option=file:path=test.conf

expect=context:root
input=type:text=edit bgp
input=enter
expect=context:path=bgp
expect=error:none

input=type:text=set
input=space
expect=completion:contains=router-id,local-as,peer

Test Categories

Category Location Tests
Navigation test/editor/navigation/ edit, up, top, context
Completion test/editor/completion/ commands, YANG paths, values
Commands test/editor/commands/ set, delete, show, compare
Lifecycle test/editor/lifecycle/ commit, rollback, load, history
Validation test/editor/validation/ hold-time, peer-as
Pipe test/editor/pipe/ grep, head, tail

Full format specification: plan/spec-editor-testing-framework.md