.ci Test File Format
The .ci format is used by Ze's test runner to define functional tests. It supports embedded files (Tmpfs), test options, expectations, and commands.
For the execution architecture (how tests are scheduled and run concurrently) and the web
.wbformat, seerunner-architecture.md.
Syntax Overview
All lines use key=value format with : separators:
action=type:key=value:key=value:...
| Action | Purpose |
|---|---|
stdin= |
Embed stdin content for processes |
tmpfs= |
Embed file content inline |
option= |
Test configuration |
cmd= |
Commands (API, shell, foreground/background) |
expect= |
Expectations to validate |
await= |
Block until the daemon's stderr carries every awaited line, then tear down (deterministic fence); then=stop makes the runner stop the daemon itself |
reject= |
Negative expectations (fail if matched) |
action= |
Actions (send notification, raw bytes) |
http= |
HTTP endpoint checks and readiness polls |
Key Concepts
An unparseable test file fails; it never hides or vanishes
A file that does not parse is recorded as a permanent failure and discovery continues. It is never dropped, and it never aborts the rest of the directory. All three discoverers behave identically.
| Discoverer | Format | Marker |
|---|---|---|
EncodingTests.Discover |
.ci, suites registered by harnesstool.SuiteAnswer and the bgp runner (encode, plugin, ui, ...) |
Record.ParseFailed + State=StateFail + FailureType=parse_error |
ParsingTests.Discover |
.ci (parse suite) |
parsingTest.ParseError |
ParsingTests.Discover |
legacy + with a companion .expect. valid/*.confinvalid/*.conf |
parsingTest.ParseError |
DecodingTests.Discover |
.ci and .test (decode suite) |
decodingTest.ParseError |
The legacy .conf layout has no instances in the tree today, and it is held to
the same contract anyway: a missing, empty, or bad-regex .expect file records
that one fixture as a failure rather than abandoning the directory. An
unreachable abort becomes reachable the moment someone adds the directory, and
the shape is the bug.
Both alternatives are silent-coverage-loss bugs this project has already paid for, which is why neither is permitted:
| Anti-pattern | What it costs |
|---|---|
Return the parse error out of Discover |
The whole directory is abandoned. One bad .ci made the test/ui suite discover and run ZERO tests, and a suite that runs nothing reads as green. |
continue past the bad file |
The file leaves the suite with no warning and no failure record. Its coverage disappears and nothing says so. |
A guard that neither denies nor speaks does not exist
(ai/rules/evidence.md). The runner short-circuits a parse-failed
test before executing anything, so the reported error is the parse error rather
than a confusing downstream symptom.
Unparseable outranks skipped. A file that both fails to parse and carries
a skip marker (option=needs-linux, option=skip-os, option=needs-arch) is
reported FAIL, not SKIP. Its skip
marker was parsed from the same broken file, so it is not trustworthy evidence
that the file need not run: a contradicting directive may sit past the break, or
the marker itself may be what was mis-parsed. Honoring it would mean trusting a
broken file's own claim that it can be ignored.
The consequences are asymmetric, which is what settles the ordering. 158 .ci
files carry one of those markers (12 in test/ui), and on a non-Linux host they
never run. A wrongly-SKIPPED malformed file is invisible indefinitely, which
is exactly how test/ui rotted. A wrongly-FAILED one is loud and costs a single
commit. The check therefore lives in parallel.go's per-test goroutine, ahead of
the skip short-circuit: that is the real entry point, and both
Runner.runTest and parsingRunner.runTest are reached through it.
Suite label, test id, and failure identity
The verify debugging protocol identifies a functional failure with:
| Field | Source | Purpose |
|---|---|---|
| Suite label | le test runner label such as plugin, ui, or managed |
First routing boundary inside ./le test functional |
| Test id | One-based decimal id printed by --list and per-test result lines |
Exact single-test rerun scope |
| Run number | N/TOTAL printed by --list and per-test result lines |
Human progress marker for long suites |
| CI file path | Parsed .ci source path |
Full test definition and embedded fixtures |
| Failure kind | Runner failure type, timeout state, or mismatch class | Conservative grouping key |
| Expected / received evidence | TEST FAILURE block detail |
Full debugging evidence in the stage log |
In ZE_VERIFY_MODE=1, failed suites emit native failure-group metadata before
the full failure blocks. The compact verify index uses that metadata for group
routing and keeps the full TEST FAILURE blocks in the stage log.
Per-step trace output
All three runner families (.ci, .wb, .et) record per-step outcomes during
execution and emit dual-format trace output:
- Human: colored checkmark/cross glyph per step with kind, assert, and detail.
- Machine:
VERIFY STEP: {"file":"...","step":N,"kind":"...","status":"pass|fail",...}-- one JSON line per step, matching theVERIFY FAILURE GROUP:prefix convention.
Trace is emitted automatically for failed tests. Under -v, passing tests also
show their trace. The .ci runner includes the trace in its TEST FAILURE report
block when StepTrace is non-empty.
conn and seq
Most directives use conn=N and seq=N to identify message ordering:
- conn (connection): 1-based TCP connection index. Each
le test peerinstance manages one TCP connection. Multi-peer tests useconn=1for the first peer,conn=2for the second, etc. The maximum is set byoption=tcp_connections:value=N. - seq (sequence): 1-based message sequence within a connection.
seq=1is the first BGP message after OPEN/KEEPALIVE,seq=2is the second, etc.
A test with two peers and one UPDATE each uses conn=1:seq=1 and conn=2:seq=1,
not conn=1:seq=1 and conn=1:seq=2.
Port Substitution
Each test owns three consecutive ports, exposed as variables in commands,
tmpfs= content, stdin blocks, option=env values, and URLs:
| Variable | Meaning |
|---|---|
$PORT |
BGP peer port (leased by the runner, used by le test peer --port $PORT) |
$PORT2 |
Secondary port, $PORT+1 (web UI, looking glass, plugin acceptor) |
$PORT3 |
Third port, $PORT+2 (an RTR cache, a second listener, a session editor) |
runner.ExpandPorts expands these placeholders in tmpfs= content, exec=,
stdin blocks, option=env values, and HTTP check and wait URLs. Assertion
values in expect= and reject= are not expanded. The config sweep in
internal/component/config/cli calls the same function. Any other $PORT<n>
fails the file at discovery, naming the placeholder: it would name a port the
lease does not hold.
Never hardcode port numbers. Use $PORT in cmd= exec values and $PORT2 in http= URLs.
The span is LEASED when the test starts, not when the suite discovers it
(runner.LeaseTestPorts, internal/test/runner/ports.go). Discovery numbers the
Nth test of every suite from the same base, so the preference alone collides
whenever two le test processes run at once. The lease takes per-port advisory
locks in /tmp/le-port-locks-<effective-uid>, independent of TMPDIR, checkout,
or session. The flat per-user directory avoids requiring access through another
user's private lock directory. Directory and file creation modes are 0700 and
0600, respectively; cross-user coordination is unsupported. Lock files MUST NOT
be unlinked: replacing a lock inode splits ownership between old and new users
of the path.
All participating runners MUST be rebuilt for this namespace cutover. Old
runners using le-port-locks under their temporary directory do not coordinate
with this namespace; there is no fallback or hybrid locking mode.
While holding the locks, the runner probes 127.0.0.1 with TCP/IPv4 and ::1
with TCP/IPv6 for every port, keeping all probe listeners open until the whole
span has been checked. IPv6 is omitted only when its family, protocol, or
loopback address is unavailable, not when its port is occupied. Probe listeners
close before children bind; the advisory locks remain until the test releases
its lease. A preferred span that is locked or occupied moves to 25000-32759.
The locks coordinate participating same-user processes on the same host that
share /tmp; they do not provide universal port exclusivity across users or
isolated filesystems. An unrelated process can still bind after the probes
close. A hardcoded number in a .ci file takes part in neither step, which is
why the rule above is a rule.
There is no $PORT4 placeholder. A compiled fixture that starts more
listeners than the three placeholders cover MUST reserve their ports with
runner.ReservePorts or runner.LeaseTestPorts, and MUST retain the
reservation until every child using those ports has stopped and been reaped. An
offset from a placeholder is only a preferred allocation address, never a
reservation.
The live managed-client restore fixture uses the runner's first two ports for the hub's SSH and plugin listeners and reserves a third port for its managed listener. Its hub and client configs declare no BGP peers or dynamic groups, so they bind no BGP listeners and need no BGP port offsets.
Stdin Blocks
A stdin block embeds content the runner pipes to a process's standard input.
Two commands take it as a FILE instead, and the table under "Where the block
goes" says which, why, and what the .ci writes to select each route.
Syntax
Multi-line (with terminator):
stdin=<name>:terminator=<TERM>
<content>
<TERM>
Single-line hex:
stdin=<name>:hex=<hex-value>
Single-line text:
stdin=<name>:text=<text-value>
Parameters
| Parameter | Description |
|---|---|
name |
Identifier referenced by cmd=...:stdin=<name> |
terminator |
End marker for multi-line content |
hex |
Hex-encoded content (single-line) |
text |
Plain text content (single-line, newline appended) |
refused |
The diagnostic code ze config validate refuses this block with, when the test hands ze a config it must refuse (see refused under Tmpfs) |
Examples
Multi-line (config):
stdin=ze-bgp:terminator=EOF_CONF
peer test-peer {
remote {
ip 127.0.0.1;
as 65533;
}
local-as 65533;
}
EOF_CONF
cmd=foreground:seq=1:exec=ze -:stdin=ze-bgp
The block goes to a file and the daemon runs as ze start <file>. The example
said exec=ze bgp server -:stdin=ze until 2026-09-07, naming a command the CLI
does not have and a block the file does not declare.
Single-line hex (decode test):
stdin=payload:hex=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF003C020000001C...
cmd=foreground:seq=1:exec=ze bgp decode --json --family ipv4/unicast -:stdin=payload
expect=json:json={ "type": "update", ... }
The block is PIPED: the - belongs to decode, not to the daemon. The decode
suite runs that exec= line as written, with ze replaced by the binary under
test and the block piped as one line. Until 2026-10-14 it lifted the flags off
the line and ran an argv of its own ending in the hex string, so the command
text was never executed.
Single-line text:
stdin=cmd:text=update text next-hop 10.0.0.1 nlri add 10.0.0.0/24
Where the block goes
The runner pipes the block, with two exceptions. Each one exists because the
child needs a path it can open, and each is selected by what the exec= line
already writes: no new directive chooses between them.
The exec= line |
Where the block goes | Why |
|---|---|---|
ze -, and its flagged forms ze -d -, ze --plugin <p> -, ze --mcp <port> -, ze --web <port> --insecure-web - |
a FILE in a stable per-daemon directory, and argv becomes ze [flags] start <file> |
SIGHUP reads the source again and a restart reuses its tree |
le test peer ... with NO - in argv |
a temporary FILE appended to argv | le test peer takes its expect script as a path argument |
le test peer ... - |
PIPED | LoadExpectFile opens its argument through cliio, so - is standard input there |
every other line, ze bgp decode -, ze config validate -, le test replay -, sh -c ... included |
PIPED | - is the cliio stdin token, and the command reads standard input. ai/rules/cli.md |
The daemon - is recognized by POSITION, not by a list of verbs: the runner
asks DaemonConfigArg which argument is the config, and substitutes only
when that argument is the -. A - belonging to a verb is left in argv and the
block is piped, so ze config validate - and ze bgp decode pcap - test the
form the operator types.
The route changes how the block travels, never what the runner reads from it.
Whether a check peer lingers (option=linger:value=true, which the
then=stop fence below waits on) is read from the .ci block itself with the
peer's own parser, so a peer given the block as a file lingers exactly as one
given it on standard input. TestPeerLingerSurvivesStdinRouting
(internal/test/runner/one_off_proof_test.go) runs both routes through the
fence.
The first daemon block uses ze-bgp.conf in the test work directory, so a
fixture rewriting that bare name still edits the source the daemon reads.
Further distinct blocks use daemon-2/ze-<block>.conf, daemon-3/…, each
with its own sibling database/. Reusing a block reuses its directory.
Numbered directories distinguish even block names that sanitise identically.
Wrapped launches such as le test fail-syscall … -- ze - retain real stdin
and receive an isolated ze.config.dir by the same allocation rule.
Explicit file paths remain authoritative; the runner does not choose a backend.
Until 2026-09-07 the runner took the FIRST - in argv whatever it meant, so
every verb form ran against a file path the author never wrote. A test that
named a stdin block on a verb was testing the path form under a - that said
otherwise, and test/ui/bgp-decode-stdin-hex.ci failed with
invalid hex: encoding/hex: invalid byte: U+002F '/' for that reason.
What a ze-peer block may carry
A block handed to le test peer (named on a cmd=...:exec=le test peer ...:stdin=<name>
line) is read first by ze-peer and then by the test runner. The block named
peer is validated by the same rules even with no such line, because a .ci
with no cmd= at all feeds its expect= lines to ze-peer by another route. A line neither of them
acts on fails the file at parse time, naming the block, the line number and the
directive.
When a file declares cmd= lines, every stdin=peer block must be named by
a cmd=...:exec=le test peer ...:stdin=peer command. An orphan block fails
parsing even without a rejection directive: its wire expectations would have
no consumer. Files with no cmd= lines retain the legacy peer execution path.
Peer command recognition uses the execution tokenizer: quoted command words
such as "le" test 'peer' and quoted mode values retain their peer role.
| Line | Read by |
|---|---|
expect=bgp, reject=bgp, action=*, and the peer's own option= (asn, bind, tcp_connections, conn_map, open, update, linger, remote_close, silent, stop-reading, await_eor, established-file, await-file) |
le test peer |
expect=json, expect=stderr, expect=syslog, reject=stderr, reject=stdout, reject=syslog |
the test runner, where the line stands |
cmd=api |
nobody. It documents the command that produced the expected bytes |
option=timeout |
the test runner parses it, and adopts it only when the file declares none. Its scope is the whole test, not this peer, so a file-level value always wins. 450 tracked peer blocks carry one. Write the one that governs outside the block |
option=env |
refused. It sets the environment of every process the test starts, not of this peer, so it must be written outside the block |
| a line neither parser accepts | refused |
Any OTHER directive the runner parses is accepted where it stands and applies to
the whole test. option=file, expect=exit, await= and http= all work
inside a peer block, and none of them is peer-scoped. Only option=env and
option=timeout are singled out. Those two read as peer scope and are not.
A value the named option does not have is refused as well, not just an unknown
option name. option=update:value=inspect-update-message and
option=asn:value=abc each used to parse into nothing, which is the same silent
drop one level down.
The accepted set is derived from the two parsers rather than written out a third
time (peer.ClaimLine reports what ze-peer did with the line, and the runner's
own line parser answers for the rest), so a directive added to either one cannot
start being dropped here. Before the guard existed the block forwarded only
expect= and action= lines and discarded everything else in silence: eleven
reject=bgp directives across nine RFC-behaviour tests, and the reject=stderr
of a tenth, asserted nothing while reading as the negative half of the proof.
Tmpfs (Virtual File System)
Tmpfs allows embedding multiple files within a single .ci file. Files are extracted to a temp directory at runtime.
File contents are arbitrary text or bytes, not implicitly Ze configuration.
There is no special configuration meaning for a .src or .conf suffix.
For example, a peer's action=rewrite may copy a human-readable acknowledgment
into a completion marker without making either file a daemon configuration.
Syntax
tmpfs=<path>[:mode=<octal>][:encoding=<type>][:refused=<code>]:terminator=<TERM>
<content>
<TERM>
Parameters
| Parameter | Required | Default | Description |
|---|---|---|---|
path |
✓ | - | Relative path (no .., no absolute) |
mode |
✕ | Auto | File permissions (octal: 644, 755) |
encoding |
✕ | text |
Content encoding: text or base64 |
terminator |
✓ | - | End marker (alone on line) |
refused |
✕ | - | The diagnostic code ze config validate refuses this file with (config-parse, config-yang-reference, ...): the test feeds Ze a config it must refuse |
refused= changes nothing at run time: the runner writes the file like any
other. It is the test's declaration that the body is a Ze config Ze refuses on
purpose. The repository config sweep (TestRepositoryConfigsValidate,
internal/component/config/cli) validates every config a .ci carries, and
for a declared file it asserts the opposite: the body must be refused with
that code, and a declared body that validates, or is refused for another
reason, is a finding.
The sweep discovers tracked files and untracked files that Git's ignore rules
permit, using git ls-files --cached --others --exclude-standard. This keeps
launcher exports and other ignored artifacts out while retaining tracked
exceptions and new source files. The runner's draft tree remains outside the
regression corpus. Discovery warnings and source read errors fail the sweep,
including when a narrowed exclusion test would not select that file.
Mode Defaults
A file with no mode= takes its mode from its extension: 0755 for .py,
.sh, .bash, .zsh, .pl, .rb and .run, 0644 for anything else. Use
an explicit mode= when the test needs a different permission, such as an
extensionless shim (wg, ip) put on PATH, which only mode=755 makes
executable.
Both routes a .ci takes write the block's bytes unchanged and keep the mode
the block declared: the parse suite writes the parsed tmpfs as it is, and every
other suite rebuilds it per run (to expand $PORT) with the mode the record
carries, never one derived again from the path.
TestTmpfsContentAndModeReachBothRoutes (internal/test/runner/one_off_proof_test.go)
runs an extensionless mode=755 shim and a control with no mode= through
both routes and compares the bytes and the execute bit that reached disk.
Terminator Rules
- Must be non-empty
- Must be unique within file (no two Tmpfs blocks can share terminator)
- Alphanumeric and underscore only:
[A-Za-z0-9_]+ - Matched exactly (no whitespace trimming)
- Recommended:
EOF_<PURPOSE>(for example,EOF_CONF)
Example
tmpfs=peer.conf:terminator=EOF_CONF
peer test-peer {
remote {
ip 127.0.0.1;
as 65533;
}
local-as 65533;
}
EOF_CONF
option=file:path=peer.conf
option=asn:value=65533
expect=bgp:conn=1:seq=1:hex=FFFF...
Security Constraints
- No absolute paths - must be relative
- No parent traversal - no
..components - No hidden files - no
.prefix in path components - Path length limit - max 256 characters
- Path depth limit - max 10 levels
Limits
Configurable via environment variables:
| Limit | Default | Environment Variable |
|---|---|---|
| Max file size | 1 MB | ze.bgp.ci.max_file_size |
| Max total size | 1 MB | ze.bgp.ci.max_total_size |
| Max files | 100 | ze.bgp.ci.max_files |
| Max path length | 256 | ze.bgp.ci.max_path_length |
| Max path depth | 10 | ze.bgp.ci.max_path_depth |
Options
A peer block may require option=remote_close:value=true when its final
expectation must be followed by the remote ending the transport. The check peer
waits for EOF or a connection reset before reporting success. Extra bytes, any
other read error, or cancellation during runner teardown fail the check. The
runner's command timeout bounds the peer if the remote never closes.
It sends nothing and does not close its own connection to obtain that evidence.
It requires check mode and cannot be combined with linger or conn_map.
Use it after an exact shutdown NOTIFICATION expectation to prove both the
protocol message and the remote close.
option=<type>:key=value[:key=value...]
| Type | Keys | Description |
|---|---|---|
file |
path=<name> |
Config file to use |
asn |
value=<N>[:peer=<ip>] |
The AS ze-peer opens with. It reaches BOTH carriers RFC 6793 defines: the two-octet My Autonomous System field, narrowed to AS_TRANS (23456) above 65535, and the Capability Value of capability 65. The range is 1 to 4294967295 and a value outside it fails the file when it is read, naming the option and the value. peer=<ip> binds the declaration to one of ze's endpoint addresses, for a peer process that serves several of ze's peers at once (option=conn_map). With no option=asn line at all the runner derives one from the session { asn { remote N } } leaf of the ze configuration the .ci names. It reads a tmpfs= block, a stdin= block and the file an option=file:path= points at, and it follows the inheritance chain: the router's own bgp { session { asn {. } } }, then a group or template, then the peer, each level overriding the one above. Three things fail the file at read time rather than being guessed: two peers ze dials at ONE address expecting different ASNs, a declared AS the reader cannot read, and an eBGP peer the derivation reached with nothing. That last refusal knows only what the reader knows. A peer is judged eBGP by comparing the local and remote AS, so a peer for which NO local AS is declared at any level of the chain is not judged eBGP and is not refused; it inherits ze's own AS from the mirror, which is right for an iBGP session and wrong for an eBGP one. The derivation reaches no peer at all when a compiled fixture under internal/test/fixture writes the configuration and launches le test peer itself, because no .ci block is involved; such a fixture passes --asn (cliWirePeerAS). internal/test/fixture/ui_fixture_send_bgp.go |
bind |
value=ipv6 |
Bind to IPv6 |
timeout |
value=<duration> |
Test timeout (e.g., 30s). Overrides auto-timeout. |
tcp_connections |
value=<N> |
The number of TCP connections the peer serves. It is a BOUND, never a witness: the count says how many connections happened and not who caused them. A peer that closes when its expectations are met makes ze dial again on its retry timer, so value=2 is reached by a daemon that did nothing. To assert that the DAEMON dropped and restarted a session, add option=linger, which makes the peer incapable of causing the second connection. |
linger |
value=true |
Peer-block only: the check peer never closes a connection itself. After ALL expectations complete it prints its success token and holds the session open (answering KEEPALIVEs) until test teardown. BETWEEN connections, where one connection's expectations are met and a later connection is still owed, it holds that connection until the REMOTE closes it, and fails the test if teardown comes first. Without it a completed peer closes, which ze correctly treats as session-down; that withdraws the peer's routes and races any forwarding still in flight toward other peers, and it also makes the daemon's OWN close unobservable, because ze dials again on its retry timer whoever closed (asserts a stop and a restart, and needs the peer to be incapable of causing one). A conn_map peer is excluded from the between-connections hold: it serves every connection of one batch in turn and then waits for the daemon to close them all, so holding the first would starve the batch. A reject=bgp that fires during either hold RETRACTS the success token already printed, so the peer still fails the test. test/reload/config-apply-ordering-address-swap.ci |
silent |
value=true |
Peer-block only, check mode only: the peer stops sending the automatic KEEPALIVE reply it otherwise writes for every message it receives. It holds the TCP connection open and keeps reading and matching expectations. Needed to reach ze's receive hold timer: ze sends its own KEEPALIVE every hold/3 seconds, each automatic reply resets ze's hold timer, and "the peer went quiet" is otherwise unexpressible. A closed connection is a different event on a different code path, so action=close does not substitute. Explicit writes still happen: action=send, action=notification, the OPEN handshake itself, and option=linger's post-completion KEEPALIVE loop are unaffected, so silent with linger is not silent. Sink and echo modes ignore it. See. test/plugin/deadpeer-holddown.ci |
stop-reading |
value=true[:conn=<N>] |
Peer-block only, check mode only: the mirror of silent. On connection <N> (counted from 1 like the conn= of an expect=bgp line, default 1) the peer reads the daemon's first UPDATE, judges it like any other frame, and then reads NOTHING more from that connection. It keeps the TCP connection open and writes a KEEPALIVE every second, so ze's receive hold timer never fires, while everything ze writes piles up unread until ze's own write blocks. Needed to reach a session whose peer stopped accepting data: a closed connection and a quiet peer are both different events. The fill does not depend on the host's default buffer sizes on the peer's side: the peer shrinks its receive buffer on that connection before the handshake, which also switches the kernel's receive autotuning off. The daemon's send buffer is still the kernel's, so the test must give ze more to send than a send buffer can autotune to (4 MiB with Linux and macOS defaults). The peer never closes that connection itself. It reads nothing, so it sees the daemon's close only on the write side: a KEEPALIVE written to a closed connection is answered with a reset and the write after it fails, within two seconds. It then accepts the next connection, which therefore exists only because the DAEMON dropped the session. If teardown comes first, the peer fails with the daemon never closed the connection the peer stopped reading. Every other connection is read normally. The first UPDATE MUST meet the last expectation connection <N> holds, because no later frame is read: an expectation left behind fails the peer by name. value=false leaves the option off; any other value, a conn= below 1 or above tcp_connections, and a combination with conn_map, silent or remote_close are refused. See. test/plugin/session-peer-stops-reading.ci |
established-file |
path=<name> |
Peer-block only: ze-peer creates an empty marker file on each session's first UPDATE: <name> for the first session of the run, <name>-2 for the second, <name>-N for the Nth, so a fixture waits for the session a reload restarted (await-file established-2) the way it waits for the first. A relative name resolves in the test's work directory, where ze-peer and every compiled fixture run. The UPDATE, not ze's KEEPALIVE, is the event: RFC 4271 Section 9 permits an UPDATE only in Established, so the file proves the DAEMON is Established, while its KEEPALIVE arrives when it can still be in OpenConfirm. It lets a fixture wait on that state instead of a fixed delay: the reload trigger's awaitEstablished holds the SIGHUP until the file named established exists. A session that sends no UPDATE never writes the file. See. internal/test/fixture/misc_fixture_shellports.gotest/reload/reload-dynamic-peer-survives.ci |
await-file |
path=<name>[:timeout=<duration>] |
Peer-block only: ze-peer sends nothing after the OPEN handshake, not an action=send, not an option=update route, until the file <name> exists. It resolves like established-file, so one peer block can hold its first UPDATE until another block's established-file proves the daemon has that session Established. A relay test needs it when its sender would otherwise race the receiver: ze forwards nothing to a peer it does not yet hold Established, and a route that arrives first reaches the receiver later through a peer-up replay, whose order is not the send order. The wait is bounded at 15 s, or at the timeout= the option declares (a positive Go duration such as 240s; anything else is refused), and fails naming the file and the bound. While it waits, ze-peer keeps the session alive: it answers and consumes each KEEPALIVE the daemon sends, so a wait longer than the hold time does not expire the daemon's hold timer, and a script never sees a KEEPALIVE that arrived before the file existed. Every other frame that arrives during the wait (an End-of-RIB, an UPDATE) is held in order, up to 1 MiB, and replayed to the script after the wait, which answers and judges it as it always did. See, whose file the interop checker writes. test/plugin/forward-overflow-two-tier.citest/interop/scenarios/bgp-local-pref-strip-gobgp/inject.msg |
open |
value=<behavior> |
OPEN message behavior |
update |
value=<behavior> |
UPDATE message behavior |
env |
var=<KEY>:value=<V> |
Set environment variable |
skip-os |
value=<os>[,<os>] |
Skip test on listed GOOS values (e.g., darwin, linux) |
needs-arch |
value=<goarch>[,<goarch>] |
Run only on a host whose CPU arch (GOARCH) is in the list. On any other host the test is SKIPPED, and the SKIP line names both arches: SKIP. -- needs-arch=arm64, host GOARCH=amd64. . A test that builds or boots for a target arch carries it (owner decision 2026-10-13: a target arch is tested only on a CPU of that arch), because Ze builds every kernel on a host of its own arch, with no CPU emulation. Each token must be an arch the kernel builder supports (kernelbuilder.SupportedArches, today amd64 and arm64): an empty token, a typo such as amd46, a kernel spelling such as x86_64 or aarch64, a missing value= or any other key is a parse error on every host. A fixture whose arch is incidental to what it asserts asks for the host's arch instead (hostKernelArch), so it runs everywhere. The test/parse suite has its own parser and does not take needs-arch. internal/test/fixture/install_fixture.go |
needs-linux |
[caps=<tok>[,<tok>]] |
Linux-only test. It skips on non-Linux hosts and runs in the QEMU guest through ./le test qemu all; the skip reason ends with the host command that runs that one test there, ./le test qemu ci test <path>. caps= declares required capabilities such as net-admin, net-raw, bpf, and sys-time; an unavailable capability produces a visible skip. An unknown token, an unknown key or an empty caps= is refused on every host. |
needs-path |
value=<repo-rel-path>[:hint=<cmd>] |
Declares an optional heavyweight artifact. The runner resolves the path against the repository root and prints the native hint when the artifact is absent. A malformed or escaping path is a parse error. |
netns-link |
name=<if>[:address=<cidr>] |
Provisions a dummy interface inside the per-test namespace. The test skips outside the ./le test qemu netns path because the named link must never be created on the host. |
exclusive |
group=<name> |
Never run concurrently with another test carrying the same group name. Tests outside the group are unaffected and keep running alongside, so this costs far less wall-clock than dropping a whole suite to -p 1. Use it when tests contend for a kernel-global observation surface that unique names or addresses cannot partition: the ddos tests (group=ddos-flood) all flood the same loopback interface, and each daemon's detector picks its victim by top-destination-bytes over that interface's counters, so a sibling's concurrent flood is indistinguishable from the test's own. Applies on every platform and in every runner mode, because the contention is a property of the tests rather than of the host. |
Configuration inputs for peer AS derivation
The derivation follows consumers, not filenames or configuration-looking text:
option=file:path=selects the named configuration. An embedded file with its basename takes precedence, as it does when the runner launches Ze.- A daemon's configuration argument selects a file, or its named stdin block
when the argument is
-. The runner's registered harness wrappers ending in-- ze ...use the same selection.ze start routingselectsroutingeven though that filename has no extension. Value-less start flags such as--cliand--web-onlyare not configuration operands. - A consumed peer
action=rewriteselects its source only when its destination is a selected configuration path. This includes transitive rewrite ancestry. Stdin-backed daemon paths use the same execution-order filename assignment as the runtime, including separate paths for additional daemons.
Initial daemon inputs and rewrite source leaves must be readable. An intermediate
created by a rewrite need not exist before execution, but its selected ancestry
must reach readable input; a cycle of absent sources is not a producer. Existing
selected files must parse, and permissions or other read errors still fail
derivation. None of these failures becomes an empty peer list.
Unselected tmpfs files and stdin blocks remain raw sidecars,
even if they contain valid BGP declarations or unmatched quotes. The former
all-block tokenizer incorrectly refused the apostrophe in
path-asn-filter-export-reject.ci's receiver acknowledgment. That fixture still
publishes its marker only after the subject withdrawal and permitted fence
arrive; classification must not rewrite or weaken those receiver checks.
The native derivation corpus gate rejects every native .ci parse failure,
including failures before AS derivation. It does not feed other execution
dialects into the native parser: test/decode uses DecodingTests.parseCIFile,
and test/exabgp-compat uses parseExaBGPCI. The latter's
option=file:<name> selects a predecessor configuration relative to the
population's own etc/ when it has one (old-4.2, old-5.0), else the shared
test/exabgp-compat/etc/; it is
not the native option=file:path=<name> directive. Its option=env:<key>=<value>
sets one variable in the environment of the daemon under test, ze and
--dut exabgp alike, appended after the runner's own variables so a case can
override one: api-version-env.ci writes option=env:exabgp.api.version=4 to
hold every script to API 4 from its start. A line naming no key or no = is
refused at discovery. A helper a configuration runs as le ...
reaches the le that runs the suite: that le's directory is first on the daemon's
PATH, then the ze under test's, and ZE_LE_BUILD_NAME is removed, so the named
build of ./le --name <name> accepts the call. Those populations are chosen
by their execution owners, not by tolerating parser errors. Native scenarios
that intentionally submit invalid Ze configuration remain in the gate: their
.ci itself must still parse.
The shared draft boundary also excludes test/draft before discovery; a
malformed unpromoted fixture must not redden repository gates.
Observer completion barriers
A peer's last wire expectation is not the observer's completion barrier. In
cursor-replay.ci, the observer starts injecting only after eor-sent; the
peer MUST linger after receiving that marker. In initial-sync-barrier-raw.ci,
the peer MUST linger until the observer has read eor-sent, even when it has
already received both expected frames. Without option=linger:value=true,
Peer.completed returns and closes the session before the remaining work.
When a raw injection races the independent initial-sync marker, put their
exact wire expectations in the same seq group: both are required, but neither
is required to arrive first. The observer MUST retain its EOR counter barrier
before requesting shutdown.
Who stops the daemon decides which side may still be talking to it. For a
compiled fixture that does not declare StopsDaemon, the runner waits for its
check peers to exit and then signals the daemon, so a non-lingering peer's
completion ends ze at once: an engine call the fixture still has in flight,
even a request quiesce, then fails on the closed connection. Such a fixture
MUST make no engine call after the point at which its peer can complete. When
the fixture has work after that point, the peer lingers and the fixture ends
the test itself through fixture.Observe, declaring StopsDaemon
(initial-sync-marker-beside-process-route.ci, api-eor-during-initial-sync.ci,
ipv4-announce-withdraw.ci).
The test peer's 100-millisecond read deadline is a cancellation poll, not a frame boundary. Check, sink, echo and post-success linger consumers MUST retain any partial BGP header or body across a recoverable idle timeout and resume the same frame. Only a complete frame reaches expectations, rejections or echo output; the following frame starts at its own header. Each retry returns through the context check, so an unfinished frame cannot prevent teardown.
Choosing between needs-linux, caps=, and skip-os
The .ci test ... |
Use |
|---|---|
Only validates config (ze config validate -), parses, or runs an offline ze show / ze env |
Nothing. It runs natively on every OS |
| Boots a daemon that APPLIES Linux-only config (interface, VLAN, firewall, L2TP kernel) | option=needs-linux |
| The same, and needs privileged network configuration (creates interfaces, brings links up, programs netlink) | option=needs-linux:caps=net-admin |
The same, and opens a raw or packet socket (resolve ping, traceroute) |
option=needs-linux:caps=net-raw |
| The same, and loads eBPF | option=needs-linux:caps=bpf |
| The same, and sets the system clock (the NTP plugin restoring persisted time) | option=needs-linux:caps=sys-time |
| Skips on one non-Linux OS for a reason unrelated to the kernel | option=skip-os:value=darwin |
| Builds or boots for one target arch, which only a CPU of that arch can do | option=needs-arch:value=arm64 |
| Needs an optional heavyweight artifact the checkout does not carry | option=needs-path:value=<repo-rel>:hint=<cmd> |
caps= takes a comma-separated list, so a test that programs netlink and loads
eBPF declares caps=net-admin,bpf and is gated on both. An unknown token is a
parse error on every host, macOS included, so a typo cannot silently disable the
gate.
caps=net-admin exists because Linux alone is not the requirement. On an
unprivileged Linux host, a CI runner or a rootless container, a test that
applies interface config does not fail cleanly: the interface plugin fails its
configure handshake with operation not permitted and the DAEMON exits 1, then
the TEST hangs because its check peer waits for a session the exited daemon will
never open. The gate reads CapEff from /proc/self/status, not uid 0: a
setcap'd binary holds the capability without being root, and a restricted
container can be root without it.
An unreadable status file, malformed CapEff, or missing CapEff is an admission
failure naming the probe cause, not evidence that capabilities are present or
absent. Both .ci and .et discovery retain that failure before execution.
NeedsGuest propagates it to feature and RFC placement callers instead of
selecting a guest from an unknown host capability set. A valid capability set
that lacks a declared bit still produces the documented capability skip.
A caps= test does not run in the merge gate. ./le verify worktree runs
unprivileged, so the marker turns an opaque hang into an honest skip there, and
the coverage relocates to .github/workflows/qemu-nightly.yml.
TestCapabilityGatedTestsHaveANativeVMHome fails when that link is broken:
marking a test with a capability nobody's CI has would be a coverage deletion
wearing a skip's clothing.
OPEN Behaviors
| Value | Description |
|---|---|
send-unknown-capability |
Add unknown capability (code 66) to OPEN |
inspect-open-message |
Validate received OPEN against expectations. A NOTIFICATION ze sends in place of the OPEN (a held-down peer refusing a connection) is judged the same way and gets no reply. test/plugin/peer-admin-shutdown-inbound.ci |
send-unknown-message |
Send unknown message type (255) after OPEN |
drop-capability |
Remove a capability from ze-peer's OPEN response |
add-capability |
Add a capability to ze-peer's OPEN response |
router-id |
Send an explicit BGP Identifier instead of the mirrored one |
hold-time |
State the Hold Time of the OPEN body (seconds=<N>) |
graceful-restart |
State ze-peer's own Graceful Restart capability (restart-time=<N>, family=, forward-state=) |
llgr |
State ze-peer's own Long-Lived Graceful Restart capability (stale-time=<N>, family=, forward-state=) |
paths-limit |
State ze-peer's own PATHS-LIMIT entries (family=, limit=<N>) |
Sender Facts (hold-time, graceful-restart, llgr, paths-limit)
Each of these four values states a fact about ZE-PEER that a receiver acts on. They exist because a mirror asserts sameness: mirrored, each one made ze read its own configuration back out of the peer's OPEN and believe the peer had said it.
option=open:value=hold-time:seconds=<N>
option=open:value=graceful-restart:restart-time=<N>[:family=<f>[,<f>]][:forward-state=true|false]
option=open:value=llgr:stale-time=<N>[:family=<f>[,<f>]][:forward-state=true|false]
option=open:value=paths-limit:family=<f>[,<f>]:limit=<N>
| Key | Range | What reads it |
|---|---|---|
seconds |
0, or 3 to 65535 | session_negotiate takes the smaller of this and ze's receive-hold-time, then calls timers.SetHoldTime. RFC 4271 Section 4.2. 1 and 2 fail the file |
restart-time |
0 to 4095 | grStateManager.onSessionDown arms the restart timer on it, runPeer gives it to startEORTimer, and show bgp peer prints it. RFC 4724 Section 3 gives the field 12 bits |
stale-time |
0 to 16777215 | enterLLGRLocked arms one timer per family on it. RFC 9494 Section 3 gives the field 24 bits |
limit |
0 to 65535 | The Max Paths field of the code 76 entry this peer advertises. pathsLimitAdmission.filter drops paths past it, so ze sends this peer no more than limit paths for one prefix |
family |
a family name, or a comma-separated list | The <AFI, SAFI, Flags> tuples of code 64, the 7-octet tuples of code 71, and the entries of code 76 |
forward-state |
true (default) or false |
The F bit of every tuple. onSessionReestablished purges the stale routes of a family whose F bit is clear |
The families are what make graceful restart act at all. RFC 4724 Section 3
pairs the Restart Time with a tuple list, and onSessionDown builds its stale
family set from that list and returns without dispatching anything when the set is
empty. Ze's own code 64 carries the time and no tuples (parseGRCapValue), so
until ze-peer owned the value, retain-routes, mark-stale and purge-stale
were never dispatched in any test.
A file that states none of them gets the harness's own defaults, never ze's.
| Fact | Default |
|---|---|
| Hold Time | 65535, the largest the two-octet field states. RFC 4271 Section 4.2 takes the smaller of the two, so the negotiated value stays ze's own and no existing test changes cadence. Only a .ci stating a smaller one opts in |
| Restart Time | 300 seconds, longer than any functional test runs |
| Long-Lived Stale Time | 600 seconds |
| Max Paths | 65535, which bounds nothing |
| Software version (code 75) | le test peer. It has no option, because no session decision turns on it: capability.Parse holds no code-75 arm and the only reader is the offline ze bgp decode |
| Families, for all three capabilities | the families ze-peer's own OPEN advertises, read from its Multiprotocol capabilities. An OPEN carrying none is an ipv4/unicast speaker (RFC 4760 Section 8) |
Two refusals fail the file rather than dropping a line in silence:
- Stating a fact AND
add-capabilityordrop-capabilityfor the same code. The stated octets would win, and the typed line would be read, validated and then lost. - Stating a fact for a capability ze does not offer. The capability SET le test peer sends mirrors ze's, so the value would have no place to go. Configure ze to offer the capability, or drop the option.
BGP Identifier Control (router-id)
option=open:value=router-id:id=<a.b.c.d>
Ze-peer's default OPEN carries ze's own BGP Identifier with the last octet incremented, which is always a distinct, valid identifier. This option replaces it outright, so a test can present an identifier the default can never produce: 0.0.0.0, or ze's own router-id (RFC 6286 Section 2.2 rejects both, the second only from an internal peer). A malformed or IPv6 value fails the file when it is read. It was ignored until 2026-09-08, which sent the DEFAULT identifier from a test that asked for an invalid one: the file then tested the valid identifier and passed.
Capability Control (drop-capability / add-capability)
Ze-peer mirrors the capability SET of ze's OPEN, so a .ci that says nothing about capabilities still negotiates whatever ze offers. It does NOT mirror the values that describe the SENDER. The AS, the BGP Identifier, the Hold Time, the Role (code 9), the Graceful Restart time, families and flags (code 64), the ADD-PATH directions (code 69), the Long-Lived Graceful Restart stale time and flags (code 71), the FQDN (code 73), the software version (code 75) and the PATHS-LIMIT entries (code 76) are each resolved from the test's own configuration and written into ze-peer's OPEN, because a mirror asserts sameness and every one of those facts is about the speaker rather than about the session. ownedCapabilities (internal/test/peer/open_capability.go) is the list: a code absent from it is mirrored by construction.
The drop-capability and add-capability options act on that reconciled OPEN at wire level, allowing tests to control exactly which capabilities ze-peer advertises.
A capability the .ci states REPLACES the one ze-peer would have resolved. An add-capability naming any resolved code -- 9, 64, 65, 69, 71, 73, 75 or 76 -- is sent as written and ze-peer adds no second capability of that code, so a file that drops a code and adds it back gets exactly what it asked for. Dropping code 65 removes the four-octet AS capability, which leaves the two-octet My Autonomous System field as the only carrier of the peer's AS: above 65535 that field can carry AS_TRANS alone, which is what RFC 6793 Section 3 defines for a speaker with no two-octet AS.
An add-capability:code=65 carrying four octets IS the AS declaration, and the My Autonomous System field follows it. It outranks option=asn and the derivation, because it names the octets that reach the wire and RFC 6793 Section 4.1 makes those the octets a receiver reads. Letting the header field come from anywhere else would put two ASNs in one OPEN, which is the disagreement ze answers with NOTIFICATION 2/2 Bad Peer AS. A stated code 65 of any OTHER length declares no AS: it is a malformed capability the test is driving on purpose, so it is sent as written and the header field keeps the AS the rest of the configuration resolved. That is the one case where the two carriers differ, and they differ because the .ci asked.
Drop a capability:
option=open:value=drop-capability:code=<N>
Removes the capability with the given code from ze-peer's OPEN response. The peer will not see this capability in the mirrored OPEN.
Add a capability:
option=open:value=add-capability:code=<N>:hex=<value-bytes>
Adds a capability with the given code and hex-encoded value bytes to ze-peer's OPEN response.
| Key | Description |
|---|---|
code |
Capability code (1-255), e.g., 65 for ASN4, 2 for route-refresh |
hex |
Hex-encoded capability value bytes (only for add-capability) |
Use case: testing capability mode enforcement:
When Ze is configured with require mode for a capability, it sends a NOTIFICATION if the peer lacks that capability. To test this, use drop-capability to make ze-peer omit the capability from its response:
# Test: Ze requires ASN4, ze-peer drops it → Ze should send NOTIFICATION
option=open:value=drop-capability:code=65
When Ze is configured with refuse mode, it sends a NOTIFICATION if the peer has a capability. To test this, the default mirror behavior already includes the capability, but add-capability can add capabilities not in the original OPEN:
# Test: Add a custom capability for refuse testing
option=open:value=add-capability:code=73:hex=067A652D626770
Multiple overrides can be combined:
option=open:value=drop-capability:code=65
option=open:value=drop-capability:code=2
option=open:value=add-capability:code=73:hex=067A652D626770
UPDATE Behaviors
Routes ze-peer sends after the OPEN handshake, before it starts matching expectations.
| Value | Keys | Description |
|---|---|---|
send-default-route |
none | Send one UPDATE for 192.0.2.1, an address no test host holds, so the route passes the RFC 4271 Section 6.3 next-hop check. 0.0.0.0/32 |
send-route |
prefix, origin-as, next-hop, and optionally as-path, as-set, originator-id, cluster-list, label |
Send one UPDATE for one prefix. Repeat the line for more |
send-bulk |
prefix, count, next-hop, and optionally max-msg, eor |
Generate count sequential prefixes from prefix and send them as whole BGP messages |
send-default-route and send-route build the AS_PATH that a real speaker
sends. On an eBGP session, where ze-peer's OPEN AS differs from ze's, the path
starts with ze-peer's own AS (RFC 4271 Section 5.1.2). An origin-as that
names a different AS follows it, so origin-as=65001 from AS 65002 sends
[65002 65001]. On iBGP the path is origin-as alone, or empty. An as-path
key is the whole path and goes out as written. Ze drops a route from an eBGP
neighbor whose leftmost AS is not the neighbor's (RFC 4271 Section 6.3), so a
hand-written hex= UPDATE from an eBGP peer MUST start its AS_PATH with that
peer's AS.
send-bulk writes the AS_PATH of a route ze-peer originates (RFC 4271 Section
5.1.2): on eBGP one AS_SEQUENCE holding ze-peer's own OPEN AS, on iBGP an empty
AS_PATH, whose length field is zero. It takes no origin-as key, and refuses
one as it refuses any key it does not read.
These two generated-route behaviors include LOCAL_PREF=100 on iBGP and omit
LOCAL_PREF on eBGP, including labeled send-route UPDATEs. The session kind
comes from the OPEN exchange, not an option in the fixture. RFC 4271 Section
5.1.5 requires LOCAL_PREF on internal advertisements and forbids it on external
advertisements except within confederations. Raw hex= injection remains
unchanged, so a test can still send deliberately malformed attributes.
Generating one oversize UPDATE (max-msg):
option=update:value=send-bulk:prefix=10.0.0.0/24:count=16373:next-hop=10.0.0.1:max-msg=65535
max-msg caps one generated message, header included. It defaults to the RFC
4271 limit of 4096, and accepts 23 to 65535. Raise it to 65535 when the session
negotiates the Extended Message capability (RFC 8654), which ze advertises when
the peer carries capability { extended-message enable }. Prefixes that do not
fit one message spill into the next, so the example above is exactly one 65535
byte message: a 65516 byte body, the largest BGP permits.
Use it whenever a test needs an UPDATE too large to write as a hex literal. A
max-size message is 131070 hex characters, which is past the 64 KiB line limit
of both .ci scanners and past what a reviewer can check. It is also the only
way to hand the daemon a single oversize body: splitting the same prefixes
across several standard messages is a different input, because the daemon
decides per message.
A malformed key fails the test load rather than sending nothing. That is
deliberate: a spec that silently degraded to count=0 would let a test asserting
a route was NOT forwarded pass because no route was ever offered.
Commands
cmd=<type>:key=value[:key=value...]
API Commands
cmd=api:conn=<N>:seq=<N>:text=<command>
| Key | Description |
|---|---|
conn |
Connection number (1-4) |
seq |
Sequence number within connection |
text |
API command text |
Example
cmd=api:conn=1:seq=1:text=update text origin igp next-hop 10.0.1.1 nlri add 10.0.0.0/24
Process Commands (Foreground/Background)
For orchestrating multiple processes:
cmd=background:seq=<N>:exec=<command>[:stdin=<name>][:name=<handle>][:timeout=<dur>][:ready=<text>]
cmd=foreground:seq=<N>:exec=<command>[:stdin=<name>][:timeout=<dur>][:exit=<N>]
cmd=stop:seq=<N>:name=<handle>[:signal=kill|term]
| Key | Description |
|---|---|
seq |
Execution order (lower first) |
exec |
Command to execute |
stdin |
Stdin block name to pipe |
timeout |
Foreground test budget, or background process lifetime (e.g., 10s). |
exit |
Exit code asserted for this command (0..255). See below. |
name |
Handle for a background process, so a later cmd=stop can target it. |
ready |
cmd=background only: text the process prints once it can serve. See below. |
signal |
cmd=stop only: kill (SIGKILL, default) or term (SIGTERM). |
Markers may appear in any order; each value runs to the next key in the table above, whichever key that is.
The table is the whole vocabulary. Any other :<word>= on a cmd= line fails
the file, naming the key, the accepted set and the line. The scan reads the
whole line, exec= included, because that is exactly where a key the parser
does not read ends up: a value runs to the next KNOWN key, so an unknown one is
swallowed into the value before it rather than dropped.
cmd=foreground:seq=2:exec=ze -:stdin=ze-bgp:timeout=15s:env=ZE_FWD_WRITE_DEADLINE=10s
parsed with timeout="15s:env=ZE_FWD_WRITE_DEADLINE=10s", so the line got
neither the timeout it declared nor the variable, and every assertion still
passed. Set an environment variable with option=env:var=<name>:value=<value>.
One consequence: a command carrying a :<word>= span of its own cannot be
written on a cmd= line. Put it in a tmpfs= script and run the script.
timeout= must be a Go duration (10s, 1m30s). Both readers of the value
keep their own default when it does not parse, so it is refused here instead.
The test deadline ends the whole process tree of every cmd= process. Each one
leads its own process group, and the deadline kills that group, so a fixture's
daemon dies with the fixture. A descendant that left the group cannot hold the
test open either: the runner stops reading its output pipes 10s after the kill.
Every successful start has one Wait owner. Normal completion, named stop, timeout, and early failure join that owner before the runner removes input files or releases ports. A peer drain whose grace expires stops and reaps the peer before reading its final output.
Background: Starts and keeps running until its timeout, an explicit stop, or test completion.
Foreground: Setup commands finish before the next step. A ze daemon starts
without blocking later steps. Its peers or observer determine when teardown starts.
Stop: Terminates a named background process mid-test (see below).
Every foreground helper (any program other than ze and the test peer) is
waited for, the last cmd= line included. Its exit= assertion checks its own
status before the next command starts. Without exit=, an intermediate helper
must exit zero. The final helper can instead declare expect=exit:code=.
If it declares both, both must match. With neither, it must exit zero.
A helper still running at the test deadline fails it as a timeout that
names the command: a process the test does not wait for is cmd=background.
Until 2026-10-14 the last helper was classed with the daemons: started, torn
down with no Wait unless expect=exit:code= was declared, so its output
reached the accumulators only if it won that race, and a helper that never
exited passed as a daemon. test/ui/update-serve.ci started its server that
way and now declares it cmd=background.
When an embedded observer sends request shutdown, teardown waits out that
daemon's bounded self-stop grace before it stops any other background process,
whatever order those processes started in. A helper the daemon still talks to
during its own shutdown stays up for that wait.
How an exec= value becomes argv
The runner splits the value on spaces and tabs, and a span inside " or '
stays ONE argument. Quote an argument that carries a space or a pipe, exactly as
you would in a shell:
cmd=foreground:seq=1:exec=ze cli -c "show config dump - | json":stdin=config
-c receives show config dump - | json, and the quotes do not reach the
process. Three limits apply:
- A backslash escape is not handled.
\"is a backslash and a quote character, never a literal quote inside an argument. - An unbalanced quote fails the test with
unclosed quote in .... - No shell runs, so
|,>and$HOMEare ordinary characters inside an argument. The runner expands$PORT,$PORT2and$PORT3, andle test fixtureexpands the environment in its own arguments.
One splitter serves every suite, so a cmd= line produces the same argv
wherever it runs.
The first word selects the program. ze is the daemon under test, and
le test <name>, le mock <name> or another harness tool such as le dns query
(docs/functional-tests.md, "le dns query") runs the harness, which is the runner's own executable. Any
other head, a retired standalone harness name among them, is no head of the
runner's: it reaches the PATH lookup, finds no program, and fails the step.
A name that the suite compiled into its temporary directory
runs from there: the chaos suites compile le, so exec=le chaos run ...
runs that build. Any other name is found on PATH. A shim directory on the
child's PATH holds ze and le, so a plugin run "le test ..."
line in a config reaches the same binaries.
A step that runs an le action names the head le, never ./le. An le
head reaches the build the suite chose, as above: the runner's own executable
when the suite compiled none. ./le is the
repository's launcher script, and the runner drops ZE_LE_BUILD_NAME from
every child, so the launcher runs the newest shared build of HEAD in
bin/builds/: a suite started with ./le --name <build> would then test a
binary built without its working-tree edits. The
appliance steps that exec le test qemu ... rely on this.
A non-harness le step runs at the repository root, so a tmpfs= file's bare
name does not resolve on it. go, ./le and every le head other than a
harness command (le test <name>) or a stand-in server (le mock <name>) run in
the checkout, because their arguments name packages and paths there. Every other
step runs in the test's own directory, where the runner writes the tmpfs=
files. A non-harness le step that passes peer.conf by that name therefore
looks for it at the root and fails. Put the le call in a tmpfs= script and run
the script: it runs in the test's directory, and its le is the one the shim
directory holds.
Provenance: until 2026-09-02 the .ci runner split the value on whitespace
alone while the parse suite honored quotes. The example above was already
published here, so four test/ui tests were written against it. -c received
only the first word of the quoted command, ze cli fell through to its SSH
client, and each test failed with no credentials for 127.0.0.1:2222.
ready= -- wait for a background process to serve
A background process that is not ze-peer gets 100ms to start before the next step runs. That is a guess, and a loaded host makes it wrong. A fixture that serves a socket the next step dials (a BMP collector, an RTR cache) then starts after ze has already dialed, ze gets connection refused, and ze waits out its reconnect interval: 30s for BMP (RFC 7854), 600s for RTR. The test fails as a protocol stall.
ready=<text> removes the guess. The runner reads the process's stdout and
stderr, and it starts no later step until the text appears:
cmd=background:seq=1:exec=le test fixture plugin/bmp-sender-statistics-collector $PORT2:ready=BMP-COLLECTOR: listening on
cmd=background:seq=2:exec=le test peer --port $PORT:stdin=peer
cmd=foreground:seq=3:exec=ze --plugin ze.bgp-bmp -:stdin=ze-bgp
- The process MUST print the text AFTER it binds, never before.
- The wait has no deadline of its own. The test budget bounds it. When the
budget ends first, the test fails with
background_never_ready, and the message names the command, the text, and all output of the process. - The key is refused on
cmd=foregroundand on a ze-peer line. A foreground command is already awaited, and ze-peer has its own "listening on" barrier. An empty value is refused because it matches any output. - The text is a substring match, and any
:<word>=span ends it, as with every other key.
cmd=stop -- terminate a background process mid-test
A background process started with name=<handle> can be stopped at a chosen step
by cmd=stop:seq=<N>:name=<handle>. The runner looks the process up, signals it,
and waits for it to exit before the next step runs, so a later step can
deterministically observe what happens after the process dies (e.g. show vpn ipsec sa emptying once an IKE responder is killed and DPD fires).
signal=kill(default) sends SIGKILL: the process gets no chance to flush or send a protocol teardown. This is the choice for liveness/dead-peer tests where the peer must go silent (a clean shutdown would take a different code path).signal=termsends SIGTERM, escalating to SIGKILL if the process does not exit within the teardown grace period -- a graceful stop.
Fail-closed: a cmd=stop naming a process that was never started (no matching
name=) fails the test with a clear error; it never silently no-ops, and it
can only ever signal a process the runner itself started (never an arbitrary PID).
Teardown still kills every remaining background process, and tolerates one the stop
step already reaped.
Provenance: added by spec-fixit-runner-kill-background to unblock end-to-end
peer-death observation (the deleted test/ipsec/ipsec-dpd-timeout.ci needed it).
exit= vs expect=exit:code= (per-command vs file-level)
expect=exit:code= is file-level: Record.ExpectExitCode is a single value
(a later expect=exit:code= silently overwrites an earlier one) and the runner
uses the final foreground helper's exit when the final command is a helper.
Otherwise, it uses the foreground daemon's exit, or the last quick-exit ze
command's exit when there is no daemon. Await fences retain their daemon/peer
exit selection. A file that runs several ze config validate commands
therefore asserts only the final one; every earlier command can exit with any code
and the test still passes.
Use exit= on the cmd= line to assert a specific command's own exit code. It is
checked the moment that command finishes, and names the offending seq on failure:
cmd seq=2 (ze config validate -): expected exit code 1, got 0
Foreground helpers honor exit= in both intermediate and final positions.
A helper mismatch retains its output and stops execution before the next command.
Inspect that output, then correct the helper or its exit= assertion.
Timeout and cancellation remain failures regardless of the declared exit code.
Prefer exit= whenever a file runs more than one quick-exit ze command. A
"quick-exit ze command" is any foreground ze whose verb is not a daemon verb
(hub, start, cli, monitor) and which has no config-file argument or
--web flag.
For helpers and quick-exit ze, exit=0 requires a successful wait. A nonzero
exit compares the child's reported status. A wait error with no readable exit
status, including an output pipe that exceeds its drain bound, fails the command
instead of being treated as zero. The failure retains output and names the
command and wait error. Check that the command runs to completion on this host,
then run the test again.
exit= is accepted only where it is checked. Only the two foreground arms
that await their process compare it: a foreground helper and a foreground
quick-exit ze. Every other command refuses exit= when the file is read,
naming the reason and the fix:
| Command | Why exit= is refused |
Write instead |
|---|---|---|
cmd=background |
The runner never waits for it to exit; it is killed at teardown | cmd=foreground, or an expect= on what it produced |
Foreground ze daemon |
The runner synchronizes on it and tears it down; its status is the file-level one | expect=exit:code= |
Foreground le test peer |
The same as a foreground daemon | expect=exit:code= |
Until 2026-11-02 these lines parsed, and the assertion was dropped in silence.
classifyStepArm names the arm once for the parser and for runOrchestrated,
and runOrchestrated checks again before it dispatches, so a record built
without the parser cannot slip past either.
Every stream assertion sits under a command and is scoped to it
Every expect=stdout:, reject=stdout:, expect=stderr: and reject=stderr:
line belongs to the cmd= line directly above it. Position is the grammar: the
line means what its place in the file says it means.
What each line reads depends on its kind, and no reject reads less than it did before the scoping:
| Line | Reads |
|---|---|
expect=stdout:, expect=stderr: (contains=, pattern=) |
The stream it names, in the span of the command above it |
reject=stdout:contains=, reject=stdout:pattern=, reject=stderr:contains= |
Both streams of the span of the command above it: the stream name selects nothing. Under the last foreground helper, both streams from its start to the end of the run |
reject=stderr:pattern= |
The WHOLE run's stderr, wherever the line sits: a log check |
An expect cannot be satisfied by output its author never named. A
reject=stderr:pattern= is the file-wide log check it always was (a panic, a
fatal error, a level a daemon must not log), so it sees every daemon in the
file, one started after it included, and its position is irrelevant. A
contains= reject has been scoped to its command since 2026-09-14, so it cannot
trip on a sibling command's output. A reject failure names where the text sits,
narrowed to the latest command whose span inside the asserting one holds it:
cmd seq=1 (ze start): output unexpectedly contains "panic" on stderr, found inside the output span of cmd seq=2 (le test fixture x). That names a span,
not an author: a daemon's span covers every byte a later command wrote too.
- A line with no
cmd=above it is refused at parse time, naming the file and the line. It named no command's output, so there was nothing for it to assert. Move it below thecmd=line that produces what it describes. - A line under a
cmd=stopis refused for the same reason. A stop directive terminates a named background process; it runs no program and writes nothing. - On an expect, the stream name selects the stream.
expect=stdout:reads the command's stdout span alone andexpect=stderr:its stderr span alone. A miss whose needle the command wrote to the other stream says so. Until 2026-10-14 a span covered both streams, so a stdout assertion passed on text written to stderr. A command-scoped reject still reads both streams, because one stream would check less than it did.
expect=stderr:pattern=reads whatexpect=stderr:contains=reads: the stderr span of thecmd=line above it. Until 2026-10-14 it read the whole run's stderr, so a regex under one command matched a line another command wrote. An expected pattern about a daemon's log lines therefore sits under thecmd=line that runs the daemon.reject=stderr:pattern=still reads the whole run's stderr. A pattern inside astdin=peerblock is parsed after the file'scmd=lines, so it belongs to the last of them.
A command's span is the half-open slice of the run's two accumulators it wrote
between starting and finishing. A quick-exit ze marks both ends around its own
fold, so its span is exact. A background process is still writing when the test
ends, and a foreground daemon's bytes arrive throughout its life, so both close
at the final lengths: a long-running command's span can therefore contain bytes
a later command wrote. A foreground helper's span runs from its start to the
end of the run unless it is the last helper or a quick ze command, so an
expect under a helper earlier in the file can be satisfied by a later
command's output. The last helper's span ends when it exits, so an expect
under it reads only what it wrote; a reject under it still reads both streams
from its start to the end of the run, so the daemon teardown that follows is
still forbidden. A command the run never reached has an EMPTY span, never the
whole buffer.
A le test peer command's output is in no span. The runner captures a test
peer's stdout and stderr in that peer's own buffers, which feed its verdict
and the PEER OUTPUT section of a failure report, and never in the run's two
accumulators. An expect=stdout: or expect=stderr: line under a peer's
cmd= line therefore reads none of what the peer printed, and fails with
"output does not contain" while the text is on screen under PEER OUTPUT. What
a peer received is asserted by its expect=bgp lines. What the daemon did
about the peer is asserted on the daemon's log, under the daemon's cmd= line
(test/plugin/session-peer-stops-reading.ci).
Output snapshots taken between commands do not finish an open span. The runner closes it only after the output writers have joined, including deferred child cleanup on an early error. Until 2026-11-03, saving a foreground helper's output also closed background and daemon spans, so later output reached the report but not the assertions under those commands.
With --save, the orchestrated runner writes the final client and peer stdout
and stderr, expected messages, and received wire messages once after the child
owners finish. This includes a helper failure or timeout that returns before
normal teardown. Saving preserves the original verdict; a write failure is
reported as a warning rather than replacing the test's failure.
This replaced a file-level check over one combined buffer. Every expect
used to read Record.ClientOutput, the concatenated stdout AND stderr of every
command in the file, so an assertion could be satisfied by a command its author
never named. Two files did
exactly that: test/plugin/kernel-capability-unknown-starts.ci and
kernel-capability-doctor-reports.ci each asserted that ze doctor printed a
diagnostic code and then ran ze show diagnostic <that code>, whose output satisfied the
assertion on a host where the doctor check does not run at all.
Until 2026-09-14 a contains= reject read every command's output, so a
negative assertion that must not see another command's output needed a file of
its own. test/plugin/vpp-doctor-hugepages.ci and
vpp-doctor-hugepages-quiet.ci, and test/appliance/no-install-appliance.ci
and appliance-help-not-deprecated.ci, were split for that reason.
A failure names the command: cmd seq=2 (ze config validate -): output does not contain "...".
Known gap: 108 quick-exit ze commands across 50 .ci files predate exit=
and are still unasserted (their expect=exit:code= never reaches them). Arming
them may surface real defects; tracked in plan/known-failures/.
The parse suite reads its own dialect
test/parse runs under ParsingTests, a second parser with its own execution
model. Two differences are load-bearing for an author.
A command's own exit= is its exit assertion here, the same one an
expect=exit:code= under it states, because this suite awaits every command,
daemon verbs included. A command whose exit= and expect=exit:code= disagree
is refused at discovery. Until 2026-11-02 this suite parsed exit= and dropped
it, so only the expect=exit:code= form was checked.
Every assertion is scoped to one command, as it is in the generic runner
(above). This suite has always worked that way and the generic runner now does
too, so the two dialects no longer disagree about what an assertion's position
means. One difference survives: here expect=stderr:contains= MAY appear before
the first cmd=, because a .ci holding an inline config and no command at all
is a legacy negative test whose expected error it carries.
The dialect is a subset of the generic vocabulary, and nothing else parses. A directive no arm reads fails the file at discovery, naming the directive, the file, and every directive the suite does read. It is the same rule as "Unknown Keys Are a Parse Error" above, applied to the whole line rather than to a key inside it.
| Read by the parse suite |
|---|
cmd=, expect=exit:code= |
expect=stdout:contains=, expect=stdout:pattern= |
expect=stderr:contains=, expect=stderr:pattern= |
reject=stdout:contains=, reject=stdout:pattern=, reject=stderr:pattern= |
option=skip-os:value=, option=env: |
Two spellings this suite once had are deleted, not aliased. Both existed only here, and both are what two parsers over one corpus costs:
expect=stdout:regex=is nowexpect=stdout:pattern=. The generic parser readspattern=and refusesregex=, so the gates that walk the whole corpus with it could not read threetest/parsefiles at all.expect=stdout:not:contains=is nowreject=stdout:contains=, which this suite already read with the identical meaning. This is the one that mattered: the generic parser splitsnot:contains=at the:contains=key boundary and drops the barenot, so one written line meant "must be absent" to the suite that runstest/parseand "must be present" to every gate that reads it.
cmd=...:timeout=<duration> is honored, and a bare ze - runs as
ze start <file>, the same translation the generic runner makes. Substituting
the path in place produced ze <path>, which is not a command: the daemon
answered unknown command: <path> with its usage and exit 1 before reading a
line of config, so a file could assert expect=exit:code=1 and be satisfied by
the usage error.
Daemon readiness (ze only): a ze daemon launched either foreground or
background is told (via ZE_READY_FILE) to write daemon.ready once startup
completes, and the runner publishes its PID to daemon.pid in the tmpfs directory.
The PID file appears atomically after its complete numeric contents and any
dropped-user ownership are installed, so a fixture that sees it can read the
identity immediately. A publication failure fails the command and reaps its
owned child.
Tests poll both files directly or through a compiled driver in
internal/test/fixture before signaling the daemon or asserting on it. The
readiness handshake is armed only for Ze daemons.
Example (Decode Test)
stdin=payload:hex=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF003C...
cmd=foreground:seq=1:exec=le test decode --family ipv4/unicast -:stdin=payload
expect=json:json={ "type": "update", ... }
Example (Multi-Process)
stdin=peer:terminator=EOF_PEER
option=asn:value=65000
expect=bgp:conn=1:seq=1:hex=FFFF...
EOF_PEER
stdin=ze-bgp:terminator=EOF_CONF
peer test-peer { remote { ip 127.0.0.1; } ... }
EOF_CONF
cmd=background:seq=1:exec=le test peer --port $PORT:stdin=peer
cmd=foreground:seq=2:exec=ze bgp server -:stdin=ze-bgp:timeout=10s
Example (Multi-Peer)
Tests needing two or more BGP peers use the same $PORT on different loopback
addresses. The ze_bgp_tcp_port override sets all peer ports uniformly, which
is correct because every peer listens on $PORT.
# Source peer on 127.0.0.1 (default)
stdin=source:terminator=EOF_SOURCE
option=tcp_connections:value=1
action=send:conn=1:seq=1:hex=FFFF...
EOF_SOURCE
# Dest peer on 127.0.0.2 -- sink mode absorbs any UPDATE
stdin=dest:terminator=EOF_DEST
option=tcp_connections:value=1
EOF_DEST
cmd=background:seq=1:exec=le test peer --port $PORT:stdin=source
cmd=background:seq=2:exec=le test peer --bind 127.0.0.2 --mode sink --port $PORT:stdin=dest
cmd=foreground:seq=3:exec=ze -:stdin=ze-bgp:timeout=20s
Each ze-peer gets independent output capture and WaitFor synchronization.
The runner waits for each peer's "listening on" message before starting the
next command.
On Linux, 127.0.0.2 works automatically (127.0.0.0/8 routes to lo). On macOS
and FreeBSD, the test runner adds loopback aliases via the SIOCAIFADDR ioctl.
IPv6 works differently, because a host carries exactly one IPv6 loopback
address. A fixture that needs a second one uses fd00::2, which is unique-local
(RFC 4193) and never globally routable. ./le setup install adds it, and
./le setup check reports whether it is there. The runner never adds it:
the ioctl returns EPERM to an unprivileged process, and ./le verify current runs as
an ordinary user. A test that binds an address this host does not carry fails at
once with loopback_address_missing and the command to run, rather than timing
out on a bind that could not succeed.
The check reads the three places a fixture names an address it binds. One is
le test peer --bind <ip> on a cmd= line. The second is connection { local { ip <addr> } } in the config the fixture embeds: Ze sends from that address and
listens on it when accept is true, so the host must carry it too. The third is
local-address <addr>; in an ExaBGP-syntax config the exabgp-compat suite keeps
beside its .ci and names with option=file:. That suite has a parser and a
runner of its own, so it calls the same scan through
EnsureConfigFileBindAddresses rather than through the embedded path.
A local address outside 127.0.0.0/8 and fc00::/7 is left alone. A
config-validation fixture names a routable one (local { ip 192.0.2.1 }), the
daemon exits before it binds anything, and ./le setup install adds no such
address. The local-link-local fe80::1 leaf that sits beside local-address in
an exabgp-compat config is left alone for the same reason.
An exabgp-compat case runs TWO processes, ze and the mock BGP server, and its verdict names which one ended the case:
ze exited before the mock BGP server finished its script: <error>is a daemon defect. Read the client stderr in the failure block.the mock BGP server failed: <error>is a wire or document disagreement. The mock prints the frame or the JSON member that did not match.the mock BGP server ended without reporting successmeans its script did not complete and it said nothing about why.
Both used to print the bare exit status N of whichever process failed, which
told a reader nothing about which of the two to open.
Both processes are stopped by a defer beside the start that created them, so
a case that returns by any route, a panic included, leaves neither running. The
stops written inside the branch arms are ORDERING barriers rather than
lifetimes: a process's output must not be read while it is still writing, and
its exit status does not exist until it has one. How the suite's forks are
created and canceled is in
docs/architecture/testing/runner-architecture.md, "How a suite's child
processes end".
Expectations
expect=<type>:key=value[:key=value...]
BGP Wire Expectations
expect=bgp:conn=<N>:seq=<N>:hex=<hex-bytes>
expect=bgp:conn=<N>:seq=<N>:prefix=<hex-bytes>
expect=bgp:conn=<N>:seq=<N>:contains=<hex-bytes>
expect=bgp:conn=<N>:seq=<N>:announced=<hex-bytes>
expect=bgp:conn=<N>:seq=<N>:withdrawn=<hex-bytes>
expect=bgp:conn=<N>:seq=<N>:ordered=<hex-bytes>
Validates the BGP wire message received: hex= matches the exact message,
prefix= the message start, contains= a substring anywhere in one message.
Within one seq group, hex/prefix/contains checks match in any order and
each consumes exactly one received message.
ordered= checks in a seq group form a strict FIFO subqueue for asserting
in-order delivery across message boundaries: the front needle must appear in
the received message, and one message may consume several consecutive needles,
each matched at an advancing offset (so order inside a packed message is
enforced too). Use ordered= instead of per-message contains= when the
sender may legally pack several NLRIs into one UPDATE (the forward rail's
bucket merge): per-message framing is not a property ze owes, but delivery
order is. A message whose content matches only a non-front needle consumes
nothing and is reported as a mismatch.
A received message that meets no expectation fails the test with message mismatch, reported against the first expectation still owed. A hex=
expectation is a whole message, so both messages are decoded and their
attributes compared. Every other form holds part of a message, so the report
names the line's key and its needle, decodes the message that arrived, and ends
with Missing: the received message does not match contains=<hex>, so the expected message has not arrived. A needle is never decoded as a message.
announced= matches byte-aligned bytes only in the legacy NLRI or an
MP_REACH_NLRI's NLRI field. withdrawn= matches only in Withdrawn Routes or an
MP_UNREACH_NLRI's NLRI field. Neither searches path attributes or next hops.
Several scoped checks in one seq may match one packed UPDATE or separate
UPDATEs; they never consume checks from the next sequence. Keep a fence in its
own later sequence when ordering is the subject.
Add attribute=<decimal-type-code>,<hex-value-bytes> to a scoped expectation to
require both the NLRI and a value in the named path attribute in the same
UPDATE. For example, the LLGR timer receiver uses
announced=180A0000:attribute=8,FFFF0006 to require 10.0.0.0/24 with LLGR_STALE.
A stale community on another route does not satisfy that assertion. The
attribute's compact or extended length encoding does not affect the match.
The value is a byte-aligned substring within the named attribute, not a
whole-attribute equality check. This qualifier requires announced or
withdrawn and cannot be combined with hex, prefix, contains, or
ordered; malformed qualifiers fail parsing rather than dropping a condition.
These are raw NLRI byte patterns, not decoded prefix selectors. ADD-PATH identifiers remain present: include the four identifier octets when asserting a particular path. Opaque families need no CIDR conversion. Each field is searched separately, so a pattern cannot bridge two fields.
The corresponding negative assertion is:
reject=bgp:conn=<N>:scope=announced:pattern=<hex-bytes>
reject=bgp:conn=<N>:scope=withdrawn:pattern=<hex-bytes>
Put scope before pattern, whose value consumes the rest of the line.
Without scope, the existing rejection still searches every wire byte.
An unknown scope is a parse error. A malformed UPDATE fails a scoped rejection
closed and cannot satisfy a scoped positive expectation.
Use announcement scope for an export prohibition. RFC 1997 forbids advertising a governed route; it does not forbid withdrawing a previous generation of that route. The egress scenarios therefore require the withdrawal, reject the announcement, and require a permitted fence announcement on the same session. An UPDATE counter alone cannot distinguish those outcomes.
These forms are peer-block directives (inside a stdin=<name>: block);
top-level expect=bgp lines support hex= only.
JSON Expectations
expect=json:conn=<N>:seq=<N>:json=<json-object>
expect=json:json=<json-object>
Validates the decoded message matches expected JSON.
Validation rules:
- Parsed and compared field-by-field (key order independent)
- Volatile fields removed before comparison:
exabgp,ze-bgp,time,host,pid,ppid,counter - Neighbor normalization:
peer↔neighbortreated as equivalent,directionfield ignored - All non-volatile fields must match exactly
Exit Code Expectations
expect=exit:code=<N>
Validates the foreground process exit code. A test whose ONLY assertion is
expect=exit:code=0 is accept-only (weak) and is gated by a lint; see
Assertion Strength.
An Unknown Directive or Key Is a Parse Error
Every directive the runner reads is declared: the action= word, the type after
it, and the keys inside it. A word or a key outside its declared set fails the
file at discovery, naming what was written, the accepted set, and the line.
Nothing is dropped in silence, and nothing is guessed.
line 12: unknown action "exepct" (accepts action, await, cmd, command, expect, http, option, reject, stream)
line 12: unknown expect type "stdoutt" (accepts bgp, command-error, event, exit, file, json, output, stderr, stdout, stream, syslog)
line 12: cmd=foreground: unknown key "env" (accepts exec, exit, name, ready, seq, stdin, timeout)
The accepted set in each message is the list the parser gates on, so it cannot
describe a vocabulary the runner does not have. The line number is the line in
the FILE: comments, blank lines and whole stdin= and tmpfs= blocks are
consumed before the directives are parsed, and the refusal used to count only
the directives that survived, which named line 2 for a line that was number 69.
The option=, action=, cmd=api and await= families each declare one table
that maps a type to the keys its parser reads, and the accepted types and keys
in a refusal come from that table. The type is checked first, so
option=bogus:65000 is refused as an unknown option type with the accepted
list. Three segment shapes are then refused before any key is checked: a segment
with no = (option=asn:65000), a key given twice, and a key that does not
start with a letter, which would otherwise be read as part of the value before
it (path=out.txt:!contains=x). The value of json=, text=, hex= or
pattern= runs to the end of the line, so write those keys last. Those four are
found by substring search in that order, so a value that holds one of them is
cut there: in contains=hex=AB the inner hex= becomes a key of its own and
contains= is left empty. The decode suite, the parse suite's option=env, and
the option= lines that ze-peer reads in a stdin= block refuse in the same
way, and ze-peer's options are one table of entries that pair what is accepted
with the parser that reads it. The decode suite does not read the words of its
exec= line: it runs them, so ze bgp decode itself refuses a flag it does
not know and the test fails on its exit status.
line 3: option=asn: a segment has no key: "65000" has no "="; write each field as key=value (accepts value)
line 4: http=get: key "status" is given twice; keep one
The rule exists because a dropped key takes the whole assertion with it. An
expect=stdout:not-contains=X line recorded NO assertion and then passed
whatever the command printed. Ten such lines were live across seven .ci files
when the check was added, and one of them was the only assertion its test made.
Two spellings get a named message, because both are the right key somewhere
else: not-contains= belongs to expect=file:, and !contains= was the
expect=stdout spelling until stream non-containment moved to reject=.
Stdout Expectations
expect=stdout:contains=<text>
expect=stdout:pattern=<regex>
Two modes:
contains=-- substring match against stdout (multiple allowed, all must match)pattern=-- regex match against stdout (uses Goregexpsyntax)
Stdout must NOT contain text is reject=stdout:contains=, below. A stream has
ONE negation spelling and it lives on reject=.
Stderr Expectations
expect=stderr:pattern=<regex>
expect=stderr:contains=<text>
Two modes:
pattern=: regex match against stderr (uses Goregexpsyntax)contains=: substring match against stderr
Both modes read the stderr the command above them wrote, and nothing it wrote
to stdout or any other command wrote (see the scope note below). A daemon's log
lines are the stderr of the cmd= line that runs it.
Stderr must NOT contain text is reject=stderr:contains=, below.
Await (deterministic stderr fence)
await=stderr:contains=<text>[:timeout=<dur>][:then=stop]
Blocks the runner until the daemon's relayed stderr contains <text>, then tears
the daemon down. This is a deterministic replacement for a blind time.sleep that only
held the daemon open long enough for a line to appear. timeout= is an optional
Go duration (default: 80% of the test budget, at least 10s); on timeout the test
fails with a precise message that names the needles still missing.
<text> follows the same rule as expect=stderr:contains=.
Several await= lines form ONE fence. It holds once every needle has appeared,
in any order: an external plugin's stderr is relayed apart from the daemon's own
writes, so an order between them would be a race of its own. A needle may be
declared once, and timeout= on one line only.
Without then=stop, the fence is a precondition of the normal end: after it
holds, the runner still waits for the peers to exit (a peer script that signals
the daemon several times, test/reload/reload-rapid-sighup.ci, relies on this).
then=stop on any await line makes the runner END the test once the fence
holds. Use it when a peer lingers (option=linger:value=true), because a
lingering peer never exits by itself, so the default end waits forever and a
fixture then had to stop the daemon a fixed delay after its signal. Under load
that SIGTERM landed while the reload under test was still verifying, and shutdown
canceled it. With then=stop the sequence waits on events only:
| Step | Waits for |
|---|---|
| 1 | every awaited needle on the daemon's stderr |
| 2 | every lingering check peer to print successful (its last expectation met), or to exit |
| 3 | the runner sends the daemon SIGTERM (a kill after the teardown grace) and reaps it; expect=exit:code= asserts that exit |
| 4 | every check peer to exit. A lingering peer ends when the daemon closes its session; one still running after the peer drain grace is sent SIGTERM |
A plain fence publishes daemon.pid without waiting for daemon.ready,
because a plugin that aborts startup may never write it. A then=stop fence
keeps that wait: its daemon is expected to run, and a trigger that sends SIGHUP
before the handler is installed kills the daemon.
Steps 1 and 2 share the fence's one deadline. A NON-lingering check peer is not
waited in step 2, because it ends itself, and its expectation can be the Cease
the stop in step 3 produces (test/reload/reload-dynamic-peer-survives.ci).
Await the reload's OUTCOME line, never an earlier one: sighup reload complete
for a reload that succeeded, reload error for one that was refused.
If either fence step fails, the runner retains that failure while stopping the
daemon, signaling the peers immediately, and reaping them. A failed exchange
gets no further completion grace that could exhaust the test's deadline before
SIGTERM reaches the peers. The runner collects their final stdout and stderr
before extracting received messages and writing --save files. This includes a
lingering peer's shutdown output: the missing needle must not discard the
dialogue that explains why it never appeared.
cmd=foreground:seq=3:exec=ze -:stdin=ze-bgp:timeout=20s
await=stderr:contains=incomplete peer definition:then=stop
await=stderr:contains=reload error
expect=stderr:contains=incomplete peer definition
reject=stderr:contains=sighup reload complete
Pair it with a matching expect=stderr:contains= so the line is both the fence
and the assertion. Use it for the reject-fence bucket: an external plugin whose
refusal aborts the daemon's plugin-startup coordinator
(StartupCoordinator.PluginFailed) leaves no in-daemon observer able to poll it,
so the relayed stderr line is the only non-plugin signal.
Choose a needle that is SPECIFIC to the plugin under test, not just a shared
phrase. Example (test/plugin/as112-external-refuses.ci): the bare "refusing to
start as an external plugin process" is emitted verbatim by three plugins, so the
needle includes the as112-only tail -- the address-ownership registry:
cmd=foreground:seq=1:exec=ze -:stdin=ze-bgp:timeout=15s
await=stderr:contains=refusing to start as an external plugin process -- the address-ownership registry
expect=stderr:contains=refusing to start as an external plugin process -- the address-ownership registry
Syslog Expectations
expect=syslog:pattern=<regex>
Validates that captured syslog output matches the regex pattern. When any expect=syslog: line is present, the test runner automatically starts a UDP syslog server and injects ze.log.backend=syslog and ze.log.destination=127.0.0.1:<port> into the test environment.
File Expectations
expect=file:path=<rel>:exists=true
expect=file:path=<rel>:absent=true
expect=file:path=<rel>:contains=<text>
expect=file:path=<rel>:not-contains=<text>
expect=file:glob=<rel-pattern>:count=<N>
expect=file:glob=<rel-pattern>:contains=<text>
expect=file:glob=<rel-pattern>:not-contains=<text>
Validates files after the test process or peer sequence has completed. Paths and
glob patterns are relative to the test's own work directory, which is where every
child ran and where a daemon writes its artifacts. A test that declares tmpfs=
files gets them in that same directory, so the two spellings name one place. For
glob contains, at least one matched file must contain the text. For glob
not-contains, no matched file may contain it.
Use file expectations for loose source files and exported artifacts such as logs. Persistent config, versions, pointers and plugin state use key expectations:
expect=key:path=meta/config/router.conf/active:exists=true
expect=key:path=meta/config/router.conf/candidate:absent=true
expect=key:glob=file/[0-9]*/router.conf:count=2
expect=key:glob=object/*:count=2
expect=key:glob=object/*:contains=router-id 1.2.3.4
A dated history entry holds sha256:<hex>, not the config: the bytes live in
object/<hex> (docs/architecture/storage-backends.md, "Content-addressed
history"). Count versions on the entries and assert config text on object/*.
Key expectations resolve beneath database/ in the work directory and decode
the complete netcapstring frame before inspecting its value. A bad CRC or
trailing frame bytes fail the assertion. They accept the same path, glob,
count and containment checks as files. A missing key satisfies absent;
an unreadable or corrupt key does not.
Negative Expectations (reject)
reject=stderr:contains=<text>
reject=stderr:pattern=<regex>
reject=stdout:contains=<text>
reject=stdout:pattern=<regex>
reject=syslog:pattern=<regex>
reject=bgp:conn=<N>:pattern=<hex>
Inverse of expect= -- the test fails if the pattern matches. Used to verify that unwanted output (e.g., deprecated warnings, ERROR-level messages) does NOT appear.
| Type | Description |
|---|---|
reject=stderr:contains=<text> |
Fail if the command's output (stdout or stderr) contains substring |
reject=stderr:pattern=<regex> |
Fail if the whole run's stderr matches regex, wherever the line sits |
reject=stdout:contains=<text> |
Fail if the command's output (stdout or stderr) contains substring |
reject=stdout:pattern=<regex> |
Fail if the command's output (stdout or stderr) matches regex |
reject=syslog:pattern=<regex> |
Fail if syslog output matches regex |
reject=bgp:conn=<N>:pattern=<hex> |
Fail if connection N of a ze-peer receives a frame carrying these wire bytes |
"The command" is the cmd= line above the reject (see "Every stream assertion
sits under a command and is scoped to it"). A command-scoped reject reads both
of its streams, so the stream name selects nothing.
reject=bgp -- bytes a peer must never receive
reject=bgp is the only reject type ze-peer reads, because ze-peer is what sees
the wire. It goes inside the peer's stdin block, beside the expect=bgp lines.
The hex needle is matched on wire bytes at a byte BOUNDARY, which
expect=bgp:contains= is not: that one is a plain substring match over the hex
text, so it can also match at an odd nibble offset.
Five properties make it an assertion rather than a hope:
- It is never consumed. Every frame the peer's message loop reads is checked
against it. The
option=lingerloop keeps checking after completion. Pair the rejection withoption=linger:value=trueto hold it open for the whole test. - A rejection found during linger RETRACTS the success token. Under
option=lingerthe peer prints its success token BEFORE the loop, because teardown is a kill and a post-run print can be lost. The verdict reads that token, so a rejection arriving afterwards has to withdraw it: the peer printsZE-PEER-REJECTEDon its own output, and the verdict reads the retraction after the token and lets it win. Without that channel a linger rejection was detected, returned, and then discarded, which made every negative assertion held open by linger vacuous. - Check mode only. Sink and echo peers read every accepted connection
concurrently against one checker, so a
conn=could not select the session the frame arrived on. The runner refuses the file, and ze-peer refuses to start. - The block MUST also carry an
expect=bgp:conn=<N>on the same connection, and the runner refuses the file otherwise. That rule is necessary and it is not sufficient. It bounds how long the rejection is checked. It cannot say whether the forbidden bytes would have arrived inside that window. The author owes the second half. Send the delivery LAST on that connection, so a leak of the governed route arrives ahead of it. Or hold the session open withoption=linger. A connection stops being read once its expectations complete, and its rejection then passes for the one reason a rejection must not. - An odd-length or non-hexadecimal needle is a parse error, because a needle
that can never match would pass for that same reason. The runner reads the
line with ze-peer's own parser (
peer.ParseRejectRule), so a typo fails the FILE rather than the peer: a rejection rejected inside ze-peer would stop it binding, and the runner would report that as a bind timeout.
stdin=external:terminator=EOF_EXTERNAL
option=tcp_connections:value=1
option=linger:value=true
# The delivery that makes the rejection an assertion rather than an empty session.
expect=bgp:conn=1:seq=1:contains=18C00002
# 180A0100 is 10.1.0.0/24, which NO_ADVERTISE forbids on any peer.
reject=bgp:conn=1:pattern=180A0100
EOF_EXTERNAL
Assertion Strength: accept-only tests and readback
A test whose ONLY assertion is expect=exit:code=0 is accept-only (weak): it
proves a config or command was ACCEPTED, never that it parsed to the CORRECT tree.
A parser that accepts interval 300 but stores 0, or silently drops a source 0.0.0.0/0 block, still passes such a test green. This is the functional-suite
analog of the "count-only assertion" mistake class (ai/rules/testing.md).
A lint enforces that the class cannot GROW: TestCIAcceptOnlyLint walks every
test/**/*.ci, classifies each with the single accept-only predicate, and FAILS on
a NEW accept-only test that is neither strengthened nor annotated. Existing
accept-only tests are grandfathered in test/.accept-only-baseline (a sorted
allow-list that only shrinks; strengthening or annotating a test removes its line).
Correctly EXCLUDED (never weak): a test whose real check lives in a tmpfs set -e
script (e.g. test/managed/auth-reject.ci), and any reject= test.
Strengthen with a readback
Add a second cmd= that dumps the parsed tree and assert a representative value
with expect=stdout:contains= / pattern=. show config dump - reads a config
from stdin and answers the stored tree, and | json renders it, so the assertion
observes the parsed VALUE, not just that parsing did not error.
cmd=foreground:seq=1:exec=ze config validate -:stdin=config:exit=0
cmd=foreground:seq=2:exec=ze cli -c "show config dump - | json":stdin=config-dump
expect=exit:code=0
expect=stdout:pattern="interval": "300"
Two gotchas, both load-bearing:
ze config dumprequires abgp { }block (it resolves the full BGP tree) whereze config validatedoes not. To keep proving that a subsystem-only config validates, keep the originalvalidatestep on the subsystem-only stdin block and run thedumpreadback against a second stdin block that prepends a minimalbgp { router-id ... }.- A needle containing
:followed by something shaped likekey=must usepattern=. ~~Onlyjson=/text=/hex=/pattern=preserve colons, andcontains=truncates at the first colon.~~- Corrected 2026-07-30. That claim was true until
ParseKVPairsbecame boundary-aware.contains=now keeps an ordinary colon, socontains=error: no such peerasserts the whole sentence. - What still splits is a colon that introduces a real key token: a letter,
then letters, digits,
-or_, then=. That split is deliberate, because it is how the engine-step formcontains=aes-cbc:timeout=25keeps working. Socontains=note:level=highstill splits at:level=, and it is the one shape that needspattern=. - The old behavior was not harmless while it lasted. A sweep on 2026-07-29
found 203 assertions across 15 suites silently reduced to the text
before their first colon. The re-armed assertions then exposed a security
test (
test/appliance/appliance-push-image-escape.ci). That test had never once executed the path-traversal guard it was named for.
- Corrected 2026-07-30. That claim was true until
- Keep a
pattern=needle free of the substringsjson=,text=, andhex=.ParseKVPairsextracts a complex-key value bystrings.Indexof the first such marker, so a needle that itself contains one of them is mis-split. None of the readback needles above contain these markers. This note is forward guidance for new needles.
Annotate when a unit test already covers the value
When a unit test already asserts the parsed value, a readback would duplicate it. Mark the test accept-only instead, with a comment naming the covering test:
# accept-only: md5 { password; ip } value-parsing is unit-covered by
# TestParsePeerMD5FieldsParsed (internal/component/bgp/reactor/config_test.go).
The marker is a comment line whose content is accept-only: followed by a
non-empty reason. A file carrying it is allowlisted by the lint without a baseline
entry. Keep the reason greppable and truthful (name the covering test).
Actions
action=<type>:key=value[:key=value...]
Notification
action=notification:conn=<N>:seq=<N>:text=<message>
Sends NOTIFICATION with shutdown message.
Send Raw
action=send:conn=<N>:seq=<N>:hex=<hex-bytes>
Sends raw bytes to peer.
Rewrite Config File
action=rewrite:conn=<N>:seq=<N>:source=<tmpfs-file>:dest=<config-file>
Copies a tmpfs file over the daemon's config file. Used with action=sighup to test config reload.
It also writes a marker a second process can wait on, which is its other use:
test/reload/config-apply-ordering-address-swap.ci copies one file to
session-returned.txt once the restarted BGP session has re-announced its
routes, and the test's own driver waits for that file before it signals the
daemon. When the rewrite is the LAST item the peer block queues, the exchange
is over and the peer completes on it, exactly as action=sighup and
action=sigterm do. Without that the peer kept matching, and the daemon's
shutdown NOTIFICATION arrived against an empty expectation list and failed the
test as a message mismatch.
| Key | Description |
|---|---|
conn |
Connection number triggering the rewrite |
seq |
Sequence number (after matching messages) |
source |
Source file name in tmpfs |
dest |
Destination file name in tmpfs (usually ze-bgp.conf) |
Send SIGHUP
action=sighup:conn=<N>:seq=<N>
Sends SIGHUP to the daemon process. Reads PID from daemon.pid in the tmpfs directory (written automatically by the test runner).
| Key | Description |
|---|---|
conn |
Connection number triggering the signal |
seq |
Sequence number (after matching messages) |
Send SIGTERM
action=sigterm:conn=<N>:seq=<N>
Sends SIGTERM to the daemon process. Reads PID from daemon.pid in the tmpfs directory (written automatically by the test runner). After sending SIGTERM, the connection is expected to close (daemon shuts down gracefully).
| Key | Description |
|---|---|
conn |
Connection number triggering the signal |
seq |
Sequence number (after matching messages) |
HTTP Checks
HTTP checks validate web endpoint responses after all cmd= processes have started.
Executed in seq order with automatic retry on connection errors (server starting up).
Under the per-test namespace mode (./le test qemu netns) the daemon listens on
the namespace's own loopback. net/http dials from a goroutine of its own, which
does not run on the runner's locked thread, so the runner's transport opens each
socket inside the test's namespace (netnsDialContext). Before it did, every
http= line in a namespace test was refused.
Assertion Checks (get/post)
http=get:seq=N:url=URL:status=CODE[:contains=TEXT][:bodyfile=PATH][:header=NAME: VALUE]
http=post:seq=N:url=URL:status=CODE[:contains=TEXT][:bodyfile=PATH][:sendfile=PATH][:content-type=TYPE][:header=NAME: VALUE][:insecure-tls=true]
| Key | Required | Description |
|---|---|---|
seq |
✓ | Execution order (>= 1, lower first) |
url |
✓ | Request URL (supports $PORT and $PORT2 substitution) |
status |
✓ | Expected HTTP status code |
contains |
✕ | Expected body substring |
bodyfile |
✕ | Path to file with expected body (exact match, resolved relative to .ci file) |
sendfile |
✕ | Path to file sent as POST request body, resolved from tmpfs first |
content-type |
✕ | Request body content type for sendfile, defaults to application/json |
header |
✕ | Request header in Name: Value wire form. Repeatable -- the only key that can appear more than once on one line |
insecure-tls |
✕ | Set true for self-signed local HTTPS endpoints |
Request Headers (header=)
header= takes the header exactly as it appears on the wire, Name: Value.
Repeat header= to set several headers on one check. It works on get, post,
and wait.
http=post:seq=1:url=http://127.0.0.1:$PORT/mcp:status=200:sendfile=call.json:header=MCP-Protocol-Version: 2026-07-28:header=Mcp-Method: tools/call:header=Mcp-Name: ze
| Behavior | Detail |
|---|---|
| Splitting | On the first colon only, so a value can contain colons (header=Referer: http://127.0.0.1:8080/page) |
| Whitespace | Trimmed around both name and value, so header=Foo: bar and header=Foo:bar are equivalent |
| Precedence | Applied after the sendfile default Content-Type, so an explicit header=Content-Type: ... wins |
| Repeats of one name | The first occurrence replaces the value. Each later occurrence adds one more value to that field |
Host |
Routed to the request's Host field, because net/http ignores a Host entry in the header map |
| Malformed | A header= value with no colon is a parse error naming the offending value, never a silent drop |
| Value stops at | The next known key marker (:status=, :contains=, the next :header=, ...), like every other key |
Retries up to 20 times at 200ms intervals on transient connection errors (ECONNREFUSED, ECONNRESET, EOF). Non-connection errors (wrong status, missing content) fail immediately.
Readiness Polls (wait)
http=wait:seq=N:url=URL:status=CODE[:contains=TEXT][:header=NAME: VALUE][:timeout=DUR]
| Key | Required | Default | Description |
|---|---|---|---|
seq |
✓ | - | Execution order (>= 1) |
url |
✓ | - | Request URL |
status |
✓ | - | Expected HTTP status code |
contains |
✕ | - | Expected body substring |
header |
✕ | - | Request header in Name: Value form, repeatable (see above) |
timeout |
✕ | 15s |
Poll timeout duration |
Unlike assertion checks, wait retries on all failures: connection errors, wrong status codes,
and content mismatches. Polls at 500ms intervals until the condition is met or the timeout expires.
Wait checks run before assertion checks, making them suitable for waiting until a server has
populated data (e.g., routes injected by an async plugin).
Example
cmd=background:seq=1:exec=le test peer --port $PORT:stdin=peer
cmd=background:seq=2:exec=ze -:stdin=ze-bgp
# Wait until routes are available before checking graph output
http=wait:seq=1:url=http://127.0.0.1:$PORT2/lg/graph?prefix=10.10.1.0/24&mode=aspath&format=text:status=200:contains=AS2914:timeout=15s
# Exact SVG match against reference file
http=get:seq=1:url=http://127.0.0.1:$PORT2/lg/graph?prefix=10.10.1.0/24&mode=aspath:status=200:bodyfile=expect/graph.svg
# Substring check
http=get:seq=2:url=http://127.0.0.1:$PORT2/lg/graph?prefix=10.10.1.0/24&mode=nexthop&format=text:status=200:contains=egress
Sleeps and their justification markers
Every time.sleep( in a live .ci carries a marker comment in the form
// sleep(<kind>): <reason>, on one # comment line directly above the sleep,
indented to match it exactly. The embedded .ci observer body is indentation
sensitive. Two producers enforce the marker: checkSleepJustification in
internal/le/doc/wiring (run by ./le doc wiring, scoped to changed .ci
files, listing every unjustified file:line and exiting 1), and writeCISleep
in internal/le/hookruntime/writeedit.go, which blocks a Write or Edit that
introduces an unmarked sleep.
The kinds are a closed set, and each one owes a different reason:
| Kind | What the reason states |
|---|---|
poll-interval |
The real condition the enclosing loop breaks or returns on. This is already a deterministic wait; the sleep is only its granularity |
timer |
The delay itself IS the behavior under test, and the mechanism plus where its period is set |
timeout-under-test |
The fixed internal timeout the sleep waits out, which the test asserts on |
needs-linux |
A dataplane effect (tc, qdisc, nft, kernel FIB) with no readback in the driver, convertible only after a QEMU run |
| ✕ | The awaited effect exposes no queryable state to this driver, and what is held until instead |
A free-text // settle comment is insufficient: it names no mechanism a reader
can check. "The tracker pushes live carrier once a second" is a reason a later
reader can overturn; "needs a moment" is the shape that makes a deliberate timer
and a guessed duration the same line of code.
A separate ratchet caps how MANY sleeps exist: the total time.sleep( count
across test/**/*.ci may not exceed the committed baseline in
test/.ci-sleep-baseline, and ./le doc wiring fails when it does. The markers
cap how many are unexplained.
Registered engines launched externally
le mock plugin-external <name> runs a registered engine through a genuine TLS
connect-back so a test can exercise its external-process refusal or warning.
Before configuring the engine logger, the launcher calls crashlog.Flush to
restore the stderr pipe managed by the parent daemon. The logger therefore
writes each diagnostic into that pipe before returning.
A refusing engine closes its SDK connection on return, and the parent can kill the child as soon as registration fails. Leaving a second asynchronous stderr queue inside the child let that kill discard the refusal before the parent's relay received it. This exception applies to this test launcher only; runtime crash-file capture remains armed. Standalone production external plugins do not use this registered-engine launcher.
The compiled observer API
Compiled .ci observers live under internal/test/fixture. They use
pkg/plugin/sdk for the five-stage plugin protocol (Plugin.Run owns it) and
the local fixture package for registration, dispatch, polling and failure
reporting.
| Function | Purpose |
|---|---|
fixture.Register(name, driver) |
Register one compiled fixture command |
fixture.Run(args) |
Dispatch le test fixture <name> [args...] |
fixture.Observe(...) |
Connect through the SDK, complete startup, run the scenario after all plugins are ready, then request shutdown |
observeConfigured(...) |
Install callbacks before startup, then run the same observer lifecycle. It is unexported, so only a fixture in this package calls it |
fixture.Dispatch(...) |
Send one command and decode its JSON answer into a Go value |
fixture.Poll(...) |
Retry a predicate until success, exhaustion, or context cancellation |
fixture.ReportFailure(err) |
Emit the ZE-OBSERVER-FAIL sentinel checkObserverSentinel detects. internal/test/runner/runner_validate.go |
sdk.Plugin.DispatchCommand(...) |
Send a typed command request through the plugin connection |
fixture.Poll around fixture.Dispatch is the payload-predicate wait: it blocks
until the observed payload matches, within a bounded attempt count. It is what
replaces a time.Sleep followed by a one-shot assertion.
fixture.Observe can request a clean daemon shutdown even after an assertion
failed, so the daemon's exit code does not prove the observer's assertion. A
successful scenario must also complete request quiesce with status done.
Transport errors, engine refusals and other statuses fail the observer. The
observer reports ZE-OBSERVER-FAIL before requesting shutdown, then returns the
error to fixture.Run.
For process-status assertions, le test fixture runner/helper-command-exit <stdout-line> <exit-code> writes the line and exits with the requested status
from 0 through 255. A requested nonzero status is test input, not an observer
failure, so it emits no ZE-OBSERVER-FAIL. Invalid arguments and output errors
still follow the normal fixture failure path.
A fixture-driven .ci needs no bgp block and no le test peer to get a daemon
that stops. request shutdown reaches a reactorless daemon through the
shutdown callback the daemon wires beside its plugin server, before any plugin
can dispatch, so a BFD-only or DHCP-only configuration stops on the fixture's
request like a BGP one. Adding a BGP peer only to make the daemon stoppable adds
a second protocol to the test's failure surface: test/bfd/bfd-detection-interval.ci
was red for exactly that reason until 2026-09-07.
option=asn moves both AS numbers. The AS is resolved once and written into
the two-octet My Autonomous System field AND the four-octet AS capability, so the
two disagree only where a .ci states a malformed capability 65 itself (see
"Capability Control" above). RFC 6793 Section 4.1 states the precedence a receiver
applies: it "MUST use the AS number encoded in the Capability Value field of the
'support for four-octet AS number capability' in lieu of the 'My Autonomous
System' field of the OPEN message". So a half-applied option leaves the
capability carrying ze's own AS, ze reads THAT, finds an AS the session is not
configured for, and answers OPEN Message Error / Bad Peer AS under RFC 4271
Section 6.2. The refusal is RFC 4271's; RFC 6793 decides which field ze read.
The sentinel is written where the scenario fails, BEFORE request shutdown.
The runner reads it out of the DAEMON's stderr, and the daemon relays a plugin's
stderr only while both processes live (internal/component/plugin/process,
relayStderrFrom). fixture.Run reports the same error after Plugin.Run
returns, which is after the daemon was asked to stop, so that line reaches the
runner only when it wins the race with the shutdown. Measured 2026-09-02: an
observer asserting a route that can never arrive still passed
test/plugin/rpki-group-action.ci, and three plugin cases (fib-table,
metrics-name-show, modify-increment-localpref) passed with a failing
observer. fixture.ReportFailure emits the FIRST failure only, so the report at
the failure site and the one on the way out are one line.
Custom shutdown owners retain the same order: complete the successful
scenario's final quiesce, report its error, publish its result, then request
shutdown. runPlugin13 supplies that barrier when its shutdown flag is true.
A started scenario retains its verdict if the plugin transport closes first.
The actual runPlugin13 and
plugin15ObserveStarted regressions cover successful quiesce, transport loss,
engine refusal and warning status through the authenticated SDK transport.
A driver's working directory is the per-test directory, so a driver names its
files relatively and fixture.Run refuses to start in the checkout. The
runner creates one directory for each test, gives it to every child of that test
and removes it at the end (Record.WorkDir, internal/test/runner/runner_exec.go),
so daemon.ready, testkey and <case>.conf are written where the test can
find them and nothing survives the run. The same names in the checkout land
beside tracked source: on 2026-09-05 a run left two ed25519 private keys and
five config files at the repository root, none of them ignored, so a git add -A
from any session would have committed a private key. fixture.Run therefore
reads its own working directory first and exits 1 when that directory holds the
ze go.mod (sessionpath.IsRepoRoot). The runner never starts a fixture
there, so the refusal is written for a person who typed
./le test fixture <name> at the root: plain text, not a ZE-OBSERVER-FAIL
line, ending with the command that runs each .ci test whose step starts the
fixture, such as ./le test ui pipe-interactive-save. It reads every
test/<dir>/*.ci, skips # comment lines, and asks the one resolver of which
runner walks a directory (testfunctional.CIRunnerWords): the functional suite
table, then the le test bgp directories, then a registered le test <dir>
command. A suite's name is not always its directory (suite installer runs
test/install/ through ./le test install), and registered runners need not
have a suite row, so the suite table alone cannot resolve every harness.
PPPoE has a non-gating suite row and uses its existing le test pppoe runner,
including for suite "pppoe --draft" stress reproduction; its platform and
namespace requirements remain unchanged. Each fixture command is its runner's
words before --all, then the file's stem. When no .ci test starts the
fixture, it prints the exec= step to add instead. A driver that needs the checkout reads $ZE_REPO_ROOT, which the
runner exports; it never reads the working directory for one.
An observer assertion is therefore worth only what its wait is worth: a value
that a plugin fills asynchronously (the Adj-RIB-In after RPKI validation, the
SPF table after the first run, a session that drops when the peer completes)
needs fixture.Poll around the read, never a fixed settle before it.
Engine Steps
Engine steps drive a live daemon through CLI dispatch, first-class in .ci
instead of an embedded Python observer. The runner serializes the parsed steps
to engine-steps.json in the test tmpfs, and links it into the daemon-N/
config directory of every further daemon, because a plugin runs in its daemon's
config directory; the .ci declares the executor as an
external plugin (run "le mock engine-steps ./engine-steps.json"), which runs
the steps from OnAllPluginsReady and reports failures via the
ZE-OBSERVER-FAIL sentinel the runner gates on.
The executor reads JSON from a file, or stdin when the path is -. Every
numeric kind must name an EngineStepKind before any steps are published or
TLS is attempted. Missing, zero, and unknown values fail with
ZE-OBSERVER-FAIL and an indexed unknown-kind parse error; they are not skipped.
command=<cli command text>
stream=<monitor command text>
expect=output:<predicate>[:timeout=<dur>]
expect=event:namespace=<ns>:name=<name>[:timeout=<dur>]
expect=stream:<predicate>[:timeout=<dur>]
expect=command-error:contains=<text>
command=/stream= keep their full raw text (colons included). expect=output
re-dispatches the most recent command= until its predicate holds or the
timeout expires; expect=stream matches delivered stream= events;
expect=event matches a delivered event by its (namespace, name) identity.
A command= line carrying a pipe chain (show doctor config <file> | json) is
split the way the SSH exec channel splits one: the executor dispatches the
command alone and renders the daemon's answer through the chain, so the step
reads what ze cli -c prints for the same line. The daemon's dispatcher gives
| no meaning, so a chain sent whole would reach the handler as arguments. A
line with no pipe gets the raw JSON answer, with no default format added, which
is what json= predicates walk. A chain the pipe layer refuses fails the step
before anything is dispatched, and expect=command-error can read it.
expect=event
The executor declares ONE startup event subscription, derived from the
expect=event steps of the file itself, and asks for enveloped delivery. A .ci
therefore names each event once, in the step that waits for it, and nothing else
declares the subscription.
A startup subscription rides the ready RPC, so it is registered before any
plugin's OnAllPluginsReady runs. That is what lets a step observe the FIRST
delivery of an event the daemon emits while it starts. test/ipsec/ipsec-sa-installed.ci
is the case that forced it: the IKE engine emits vpn-ipsec/sa-up from its
startup peer reconciliation, so a subscription dispatched from the step could
only ever see a SECOND establishment, and a test with no re-negotiation has none.
Enveloped delivery wraps each event with its (namespace, event) identity
(rpc.EventEnvelope), which is what tells one subscribed event from another once
several share the one subscription. A bare payload decodes with an empty
namespace and event, so a delivery the executor did not ask to be enveloped can
never satisfy a step.
Two consequences follow, and both are load-bearing when writing a test:
- A step list names ONE namespace.
rpc.SubscribeEventsInputcarries one, so steps naming two are REFUSED before the executor dials, with a message naming both. Split the test rather than subscribing to one and dropping the rest. - A step matches ANY delivery of that identity, scrollback included. It cannot
assert "another one, after this point". A second
sa-upafter an operatorclearre-matches the first, so re-establishment is asserted by a separate test (test/ipsec/ipsec-clear-reestablish.ci), not by a secondexpect=event.
expect=command-error
expect=command-error:contains=<text> asserts that the PRECEDING command=
FAILED, and that its message contains the text.
Use it for a command that must refuse. The plugin SDK turns a StatusError
response into a Go error, so without this directive the command step aborts the
run before any expect= is reached, and no .ci can assert an operational
error at all. That left one class untestable end to end, and it is the class
where a wrong answer costs most: a command that must refuse is exactly the one
whose failure mode is answering confidently instead. test/ipsec/ipsec-dataplane-show.ci
uses it to prove that a dataplane which cannot be read SAYS so rather than
rendering an empty table.
It takes contains= only, and no timeout. An error is the result of one
dispatch, so re-dispatching until one appears would wait for a state change no
expect= can cause.
A command failure that NO expect=command-error consumes still fails the run,
whether the next step is of another kind or the command is the last step in the
file. Every .ci written before this directive existed relies on that.
It must not be vacuous. A command= that SUCCEEDS does not satisfy
expect=command-error: the step fails with "the preceding command succeeded".
Without that rule the directive would pass for the very regression it guards, a
refusal quietly becoming a successful empty answer
(TestRunEngineStepsCommandErrorRequiresAFailure).
expect=output / expect=stream predicates
The optional trailing :timeout=<dur> is split off the END first, so a predicate
operand may itself contain : (a compact-JSON fragment, an IPv6 address). The
remainder is one predicate:
| Predicate | Surfaces | Holds when |
|---|---|---|
contains=<text> |
output, stream | the output contains the substring (the default) |
matches=<regexp> |
output, stream | the Go regexp matches the output (compiled at parse time, so a bad regexp fails the test immediately, not at timeout) |
absent=<text> |
output only | the output does NOT contain the substring |
json=<dotted.path>=<value> |
output only | the dotted path into the JSON data field stringifies to <value> |
json= walks the raw data field (not status data): each .-segment indexes
a JSON object by key or a JSON array by integer index (0..len-1; out-of-range or
missing is "not yet", named at timeout). The leaf is compared as a string
(numbers/bools stringified via JSON). absent=/json= are expect=output only:
they re-dispatch a query, whereas expect=stream is an append-only event stream
with no "absent" and no single-event JSON path.
absent= must be non-vacuous. An absent= on output that was never populated
passes instantly (false green). Precede it with a step that makes the substring
present (a contains=/json= after an inject), then the transition (e.g. a
withdraw) the absent= proves. See test/plugin/engine-steps-predicates.ci.
Example
command=request bgp rib inject 10.0.0.1 ipv4/unicast 172.16.0.0/16 origin igp nexthop 10.0.0.2
command=show rib
expect=output:matches=172\.16\.[0-9.]+/16:timeout=10
expect=output:json=0.prefix=172.16.0.0/16:timeout=10
command=request bgp rib withdraw 10.0.0.1 ipv4/unicast 172.16.0.0/16
command=show rib
expect=output:absent=172.16.0.0/16:timeout=10
Complete Example
# Embed config using Tmpfs
tmpfs=test.conf:terminator=EOF_CONF
peer test-peer {
remote {
ip 127.0.0.1;
as 65000;
}
router-id 10.0.0.2;
local-address 127.0.0.1;
local-as 65533;
hold-time 180;
family {
ipv4/unicast;
}
announce {
ipv4 {
unicast 10.0.0.0/24 next-hop 10.0.1.254;
}
}
}
EOF_CONF
# Test configuration
option=file:path=test.conf
option=asn:value=65000
# Expected API command and wire output
cmd=api:conn=1:seq=1:text=update text origin igp next-hop 10.0.1.254 nlri add 10.0.0.0/24
expect=bgp:conn=1:seq=1:hex=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF002F02000000144001010040020602010000FFFD4003040A0001FE180A0000
# EOR
cmd=api:conn=1:seq=1:text=announce eor ipv4/unicast
expect=bgp:conn=1:seq=1:hex=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF00170200000000
Consumers
Different components consume different line types:
| Line Type | Consumer |
|---|---|
stdin= |
Test runner (pipes to processes) |
tmpfs= |
Test runner (writes to temp) |
option= |
Test runner + ze-peer |
cmd=api: |
nobody. parseCmd stores the text on the message and only the failure reporter prints it, so the line documents the command that produced the expected bytes and executes nothing. A test that expects it to inject routes asserts against an empty RIB |
cmd=foreground:, cmd=background: |
Test runner (process orchestration) |
expect=exit:, stdout:, stderr:, json:, syslog:, file: |
Test runner |
reject=stderr:, reject=syslog: |
Test runner (negative expectations) |
http=get:, http=post: |
Test runner (HTTP assertion checks) |
http=wait: |
Test runner (HTTP readiness polls) |
expect=bgp: |
le test peer |
action=notification:, action=send: |
le test peer |
action=rewrite:, action=sighup:, action=sigterm: |
le test peer (reload/signal tests) |
Lines not recognized by a consumer are ignored.
A check-mode peer block MUST declare a ze-peer-consumed expectation
The consumer split above is load-bearing, not trivia. Only the four
le test peer rows reach le test peer. Everything else -- including expect=json --
is validated by the test runner from its own copy of the messages.
A check-mode le test peer with no consumed directive has nothing to check, so it
prints no test data available to test against and exits 1 before binding a
listening socket. ze then dials a dead port, gets connection refused, and backs
off 5->10->20->40s. That looks exactly like a BGP establishment stall and cost a
multi-day investigation before the cause was found in the harness.
So a peer block whose only expectation is expect=json runs no BGP at all. The
parser now rejects that at discovery time, naming the file and the remedy:
stdin=peer block: check-mode ze-peer (cmd seq=1) declares no ze-peer-consumed
expectation, so it exits with "no test data available to test against" before
binding a listening socket and the test can only pass vacuously.
| Want | Do |
|---|---|
| Assert the wire exchange | Add expect=bgp:conn=N:seq=N:hex=... (or an action=send/notification/rewrite/close/sighup/sigterm) to the peer block |
A peer that is only a dial target for ze (routes injected through an external process plugin, assertions made by that plugin or http=) |
Run it as le test peer --mode sink -- sink/echo/inject peers legitimately declare nothing |
expect=json still works, but only in addition to a consumed directive: it
cannot make the peer listen.
A ze-peer always governs its test's result
A test with a check-mode ze-peer is never self-validated: the peer must print
successful and its expect=json expectations must match, whatever else the file
asserts. An expect=exit:code=0 is additive and does NOT disable the BGP checks.
Until 2026-07-16 it did, which is how a test could pass on ze's exit code while its peer never listened and its JSON assertion never ran. sink/echo/inject peers do not govern (they never report completion), so a test whose only peers are scaffolding is still governed by its own exit/output/file assertions.
A check peer prints successful only when EVERY expectation it holds was met:
every expect=bgp line of every conn=N it declares. A peer stopped before that,
by SIGTERM from the runner, by the test's timeout, or because a dialing peer's one
connection ended while conn=2 was still owed, exits 1 with
stopped with expectations unmet: conn=N still owes <expectation> (K expectations unmet).
Until 2026-09-24 a peer that SIGTERM reached while it waited for its next
connection printed successful, so a test passed with its later connections
never opened.
A scaffolding ze-peer is signaled at teardown
A sink, echo or inject le test peer never ends itself: its accept loop runs until its
context is cancelled, and le test peer maps SIGTERM to that cancel. The runner
sends that SIGTERM at teardown, after the last step and before the barrier that
collects peer output. The peer exits with status 0 and its capture is complete, so
a .ci author needs no teardown directive and must not add one.
A check-mode peer is never signaled. It exits by itself when its expectations are met, and the runner reads its verdict from the capture a signal would truncate.
Before 2026-08-10 no code signaled a scaffolding peer, so the drain barrier waited
out its full 10s grace on every --mode sink or --mode echo test. That cost was
not only latency: test/plugin/event-predicate-wait.ci failed at its 15s budget
with TYPE: timeout while the daemon itself completed correctly.
Migration from Old Format
Old format (deprecated):
option:file:test.conf
option:asn:65000
1:raw:FFFF...
1:json:{...}
New format:
option=file:path=test.conf
option=asn:value=65000
expect=bgp:conn=1:seq=1:hex=FFFF...
expect=json:conn=1:seq=1:json={...}
Key changes:
=instead of:after action- Explicit
conn=andseq=for message ordering hex=prefix for wire bytesjson=prefix for JSON data
Editor Test Format (.et)
Editor resource declarations use the shared .ci grammar for
option=needs-linux[:caps=<tokens>] and option=exclusive:group=<name>.
Discovery rejects unknown resource keys, duplicate keys, invalid capabilities,
missing groups, and invalid or overflowing effective timeouts. A parse failure
outranks a capability skip. The native editor runner retains the discovered
snapshot and holds its group lease through child reaping and cleanup.
The .et format extends .ci for interactive editor testing. Tests are located in test/editor/.
Overview
Editor tests simulate user input sequences against the headless configuration editor and verify state changes.
Input Actions
| Action | Purpose | Example |
|---|---|---|
input=type:text=<text> |
Type text | input=type:text=edit bgp |
input=key:name=<key> |
Send special key | input=key:name=tab |
input=tab |
Tab key (shorthand) | input=tab |
input=enter |
Enter key (shorthand) | input=enter |
input=ctrl:key=<c> |
Ctrl+key | input=ctrl:key=u |
input=space |
Space key | input=space |
Named keys input=key:name=<key> accepts: tab, enter, esc, escape,
up, down, left, right, backspace, delete, home, end, pgup,
pgdn, pgdown, space. shift+tab is handled separately as a modifier.
API candidate fixtures
fixture=api-set:path=<full setting path>:value=<value> writes a candidate
through the same Editor.SetValue API that config sessions use. The path is
absolute and uses the editor's quoted word syntax; the value is the complete
text after :value=, including colons or an empty string. The fixture runs
in step order on the selected editor, including a named session. Unknown
fixtures and declarations missing a path or value are parse errors.
Use it to test commit-time validation of a candidate another surface could
leave. A typed set still runs the normal immediate validation and cannot
seed a refused value. The next commit, commit verify or show | errors
validates the candidate through the product code; the fixture supplies no
validation result and writes nothing to the active configuration.
Expectations
| Expectation | Purpose | Example |
|---|---|---|
expect=context:path=<p> |
Context equals path | expect=context:path=bgp.peer.1.1.1.1 |
expect=context:root |
Context is root | expect=context:root |
expect=completion:contains=<list> |
Completions include all items | expect=completion:contains=set,delete,edit |
expect=completion:excludes=<list> |
Completions must NOT include items | expect=completion:excludes=vpp,kernel |
expect=completion:count=<N> |
Number of completions | expect=completion:count=5 |
expect=ghost:text=<suffix> |
Ghost text suggestion | expect=ghost:text=-id |
expect=dirty:true |
Has unsaved changes | expect=dirty:true |
expect=content:contains=<text> |
Config content includes text | expect=content:contains=router-id |
expect=content:not-contains=<text> |
Config content must NOT include | expect=content:not-contains=old-value |
expect=content:lines=<N> |
Config content line count | expect=content:lines=5 |
expect=content:pattern=<regex> |
Config content or displayed output matches a Go regular expression (RE2). . stops at a newline unless the expression starts with (?s); an empty or invalid expression fails the step |
expect=content:pattern=(?s)peer zulu.*peer alpha |
expect=viewport:contains=<text> |
Displayed output includes text | expect=viewport:contains=10.0.0.1 |
expect=viewport:not-contains=<text> |
Displayed output must NOT include | expect=viewport:not-contains=error |
expect=viewport:pattern=<regex> |
Displayed output matches a Go regular expression, as content:pattern. The way to assert order: (?s)a.*b passes only when a prints before b |
expect=viewport:pattern=(?s)peer zulu.*peer alpha |
expect=errors:count=<N> |
Validation error count | expect=errors:count=0 |
expect=status:contains=<text> |
Status message | expect=status:contains=committed |
expect=hint:contains=<text> |
Second message line includes text | expect=hint:contains=show the peers |
expect=hint:empty |
Second message line is blank | expect=hint:empty |
expect=explanation:contains=<text> |
Revealed long explanation includes text | expect=explanation:contains=established |
expect=explanation:empty |
✕ | expect=explanation:empty |
expect=error:none |
✕ | expect=error:none |
expect=timer:active |
Confirm timer running | expect=timer:active |
expect=file:path=<rel>:contains=<text> |
On-disk file content | expect=file:path=test.conf:contains=bgp |
expect=file:path=<rel>:not-contains=<text> |
File must NOT contain | expect=file:path=test.conf:not-contains=old |
expect=file:path=<rel>:absent=true |
File does not exist | expect=file:path=test.conf:absent=true |
expect=key:path=<key>:contains=<text> |
Decoded persistent value | expect=key:path=file/active/test.conf:contains=bgp |
expect=key:path=<key>:not-contains=<text> |
Decoded value excludes text | expect=key:path=file/active/test.conf:not-contains=old |
expect=key:path=<key>:absent |
Key does not exist | expect=key:path=file/active/test.conf.draft:absent |
Pattern expectations match displayed output after stripping ANSI escape sequences,
so a line anchor sees the configuration row rather than its color prefix. The
viewport is not trimmed or reordered, and missing rows still fail. The working
configuration alternative of expect=content:pattern= remains literal. Literal
contains and not-contains expectations are unchanged. Failed viewport patterns
retain the original output in their diagnostic.
Editor tests use a tree store by default; option=storage:value=tree states
that choice explicitly. The tree mode is held in memory (storage.CreateMemory),
not on disk: it is the live store's contract (name mapping, guards, versions,
metadata, observers) over a map, so no commit waits on a sync, and it closes
with the test. option=storage:value=blob runs the test on an on-disk backup
artifact the way ze config edit --backup opens one: the runner creates
backup.zefs, seeds it, closes it, reopens it writable through
storage.OpenBlob, and builds each session with
cli.NewOfflineSessionEditor, so a commit publishes into the artifact. Blob
mode needs option=session:user=.... Other values are refused. The runner seeds config
fixtures once and all sessions and restart= steps share the same lifetime
store handle. expect=key reads that handle, while expect=file reads the
original loose fixture or a separate exported artifact.
hint reads the second message line with every style stripped. That row carries
the completion hint, and the summary of the candidate the operator selected.
explanation reads the long explanation Tab reveals, without the box that frames
it on screen. Both accept empty and contains=. Both refuse an expectation
that names neither key.
A test that runs option=mode:value=command or option=mode:value=operational
completes against the real command tree, built from the registered -cmd YANG
modules. The tree carries no value hints and no plugin commands. A hint provider
reads the live RIB, and a plugin command comes from the live dispatcher. A
headless test has neither, so a completion over a VALUE stays empty there.
Dimensions and Timeout
Editor dimensions use option=width:value=<columns> and
option=height:value=<rows>. Missing, nonnumeric, nonpositive and overflowing
integer values are parse errors. Dimensions default to 80×24 and apply to the
initial model, each new session and every restarted model through the terminal
resize event. option=timeout:value=<duration> defaults to 30 seconds and
refuses nonpositive durations or overflow after concurrency headroom is applied.
restart=editor recreates the selected editor using its own construction mode
and session identity. A named session= keeps its user and origin across
restart and switch-back; it does not inherit the initial file session's identity.
Wait Actions
Model copies share one session context and clock. Validation, draft polling, confirmation countdown and live-view poll ticks use stoppable timers; closing the model cancels pending ticks. Delivery checks the originating model lifetime before applying the message, so an expired tick cannot affect a restarted editor. Validation also retains its generation check. Authored editor runs use real timers; controlled clocks are confined to focused runtime tests.
The model declares timer work with TimerMsg. Headless execution runs ordinary
commands and every state-command result to completion, including nested batches
and sequences. Only declared timers run asynchronously; command duration never
classifies work as a timer. Expectations settle validation without waiting for
recurring display ticks. Restart and case cleanup cancel and join pending model
timers before releasing the editor.
| Action | Purpose | Example |
|---|---|---|
wait=ms:<N> |
Wait N milliseconds | wait=ms:200 |
wait=validation |
Wait for validation | wait=validation |
wait=timer:drain |
Apply timers pending at entry once | wait=timer:drain |
wait=timer:expire |
Wait for timer expiry | wait=timer:expire |
wait=ms:<N> pumps timer deliveries for at least N real milliseconds (zero is
allowed). Negative, noninteger and overflowing durations are parse errors.
wait=validation applies the current generation before the next state command;
older debounce ticks cannot satisfy it. A settled editor returns immediately.
wait=timer:drain waits for and applies every declared timer pending at entry,
including its complete state-command chain. Timers rearmed by those deliveries
remain pending for later steps. No elapsed-time threshold establishes readiness;
an empty entry set or cancellation fails the wait. For a fake ping monitor,
place it immediately after Enter so the real 50ms first poll drains the prepared
replies before viewport or saved-file assertions.
wait=timer:expire requires an active confirmation timer on entry and waits for
observed timeout rollback. An inactive timer, accept/abort, cancellation, missing
countdown/poll or rollback error fails the wait. Session editors observe the
real daemon window through draft polling; they do not synthesize its expiry.
If the daemon's rollback fails and its window remains open for retry, the wait
reports the structured rollback cause at the next poll instead of waiting for
the case timeout. The window stays open; reporting failure does not accept it.
Unknown wait actions and malformed arguments are parse errors. Wait failures are recorded on the wait step, including its cause.
Example
# Test: Edit navigation
tmpfs=test.conf:terminator=EOF_CONF
bgp {
router-id 1.2.3.4;
peer upstream1 {
remote {
ip 1.1.1.1;
as 65001;
}
}
}
EOF_CONF
option=file:path=test.conf
expect=context:root
input=type:text=edit bgp
input=enter
expect=context:path=bgp
expect=error:none
input=type:text=set
input=space
expect=completion:contains=router-id,local-as,peer
Test Categories
| Category | Location | Tests |
|---|---|---|
| Navigation | test/editor/navigation/ |
edit, up, top, context |
| Completion | test/editor/completion/ |
commands, YANG paths, values |
| Commands | test/editor/commands/ |
set, delete, show, compare |
| Lifecycle | test/editor/lifecycle/ |
commit, rollback, load, history |
| Validation | test/editor/validation/ |
hold-time, peer-as |
| Pipe | test/editor/pipe/ |
grep, head, tail |
Full format specification: plan/spec-editor-testing-framework.md