.ci Test File Format

For the execution architecture (how tests are scheduled and run concurrently) and the web .wb format, see runner-architecture.md.

Syntax Overview

All lines use key=value format with : separators:

action=type:key=value:key=value:...

Key Concepts

An unparseable test file fails; it never hides or vanishes

A file that does not parse is recorded as a permanent failure and discovery continues. It is never dropped, and it never aborts the rest of the directory. All three discoverers behave identically.

The legacy .conf layout has no instances in the tree today, and it is held to the same contract anyway: a missing, empty, or bad-regex .expect file records that one fixture as a failure rather than abandoning the directory. An unreachable abort becomes reachable the moment someone adds the directory, and the shape is the bug.

Both alternatives are silent-coverage-loss bugs this project has already paid for, which is why neither is permitted:

A guard that neither denies nor speaks does not exist (ai/rules/evidence.md). The runner short-circuits a parse-failed test before executing anything, so the reported error is the parse error rather than a confusing downstream symptom.

Unparseable outranks skipped. A file that both fails to parse and carries option=needs-linux / option=skip-os is reported FAIL, not SKIP. Its skip marker was parsed from the same broken file, so it is not trustworthy evidence that the file need not run: a contradicting directive may sit past the break, or the marker itself may be what was mis-parsed. Honoring it would mean trusting a broken file's own claim that it can be ignored.

The consequences are asymmetric, which is what settles the ordering. 158 .ci files carry one of those markers (12 in test/ui), and on a non-Linux host they never run. A wrongly-SKIPPED malformed file is invisible indefinitely, which is exactly how test/ui rotted. A wrongly-FAILED one is loud and costs a single commit. The check therefore lives in parallel.go's per-test goroutine, ahead of the skip short-circuit: that is the real entry point, and both Runner.runTest and parsingRunner.runTest are reached through it.

Suite label, test id, and failure identity

The verify debugging protocol identifies a functional failure with:

In ZE_VERIFY_MODE=1, failed suites emit native failure-group metadata before the full failure blocks. The compact verify index uses that metadata for group routing and keeps the full TEST FAILURE blocks in the stage log.

Per-step trace output

All three runner families (.ci, .wb, .et) record per-step outcomes during execution and emit dual-format trace output:

• Human: colored checkmark/cross glyph per step with kind, assert, and detail.
• Machine: VERIFY STEP: {"file":"...","step":N,"kind":"...","status":"pass|fail",...} -- one JSON line per step, matching the VERIFY FAILURE GROUP: prefix convention.

Trace is emitted automatically for failed tests. Under -v, passing tests also show their trace. The .ci runner includes the trace in its TEST FAILURE report block when StepTrace is non-empty.

conn and seq

Most directives use conn=N and seq=N to identify message ordering:

• conn (connection): 1-based TCP connection index. Each le test peer instance manages one TCP connection. Multi-peer tests use conn=1 for the first peer, conn=2 for the second, etc. The maximum is set by option=tcp_connections:value=N.
• seq (sequence): 1-based message sequence within a connection. seq=1 is the first BGP message after OPEN/KEEPALIVE, seq=2 is the second, etc.

A test with two peers and one UPDATE each uses conn=1:seq=1 and conn=2:seq=1, not conn=1:seq=1 and conn=1:seq=2.

Port Substitution

Each test owns a pair of ports, exposed as variables in commands, tmpfs= content, option=env values, and URLs:

Never hardcode port numbers. Use $PORT in cmd= exec values and $PORT2 in http= URLs.

The pair is LEASED when the test starts, not when the suite discovers it (runner.LeaseTestPorts, internal/test/runner/ports.go). Discovery numbers the Nth test of every suite from the same base, so the preference alone collides whenever two le test processes run at once; the lease takes a machine-wide advisory lock in $TMPDIR/ze-test-port-locks and probes the pair, and a test whose preferred pair is locked or occupied gets one from 25000-32759 instead. A hardcoded number in a .ci file takes part in neither step, which is why the rule above is a rule.

Stdin Blocks

A stdin block embeds content the runner pipes to a process's standard input. Two commands take it as a FILE instead, and the table under "Where the block goes" says which, why, and what the .ci writes to select each route.

Syntax

Multi-line (with terminator):

stdin=<name>:terminator=<TERM>
<content>
<TERM>

Single-line hex:

stdin=<name>:hex=<hex-value>

Single-line text:

stdin=<name>:text=<text-value>

Parameters

Examples

Multi-line (config):

stdin=ze-bgp:terminator=EOF_CONF
peer test-peer {
    remote {
        ip 127.0.0.1;
        as 65533;
    }
    local-as 65533;
}
EOF_CONF

cmd=foreground:seq=1:exec=ze -:stdin=ze-bgp

The block goes to a file and the daemon runs as ze start <file>. The example said exec=ze bgp server -:stdin=ze until 2026-09-07, naming a command the CLI does not have and a block the file does not declare.

Single-line hex (decode test):

stdin=payload:hex=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF003C020000001C...
cmd=foreground:seq=1:exec=ze bgp decode --json --family ipv4/unicast -:stdin=payload
expect=json:json={ "type": "update", ... }

The block is PIPED: the - belongs to decode, not to the daemon.

Single-line text:

stdin=cmd:text=update text nhop set 10.0.0.1 nlri ipv4/unicast add 10.0.0.0/24

Where the block goes

The runner pipes the block, with two exceptions. Each one exists because the child needs a path it can open, and each is selected by what the exec= line already writes: no new directive chooses between them.

The daemon - is recognized by POSITION, not by a list of verbs: the runner asks zeDaemonConfigArgIndex which argument is the config, and substitutes only when that argument is the -. A - belonging to a verb is left in argv and the block is piped, so ze config validate - and ze bgp decode pcap - test the form the operator types.

The first daemon block uses ze-bgp.conf in the test work directory, so a fixture rewriting that bare name still edits the source the daemon reads. Further distinct blocks use daemon-2/ze-<block>.conf, daemon-3/…, each with its own sibling database/. Reusing a block reuses its directory. Numbered directories distinguish even block names that sanitise identically. Wrapped launches such as le test fail-syscall … -- ze - retain real stdin and receive an isolated ze.config.dir by the same allocation rule. Explicit file paths remain authoritative; the runner does not choose a backend.

Until 2026-09-07 the runner took the FIRST - in argv whatever it meant, so every verb form ran against a file path the author never wrote. A test that named a stdin block on a verb was testing the path form under a - that said otherwise, and test/ui/bgp-decode-stdin-hex.ci failed with invalid hex: encoding/hex: invalid byte: U+002F '/' for that reason.

What a ze-peer block may carry

A block handed to le test peer (named on a cmd=...:exec=le test peer ...:stdin=<name> line) is read first by ze-peer and then by the test runner. The block named peer is validated by the same rules even with no such line, because a .ci with no cmd= at all feeds its expect= lines to ze-peer by another route. A line neither of them acts on fails the file at parse time, naming the block, the line number and the directive.

Any OTHER directive the runner parses is accepted where it stands and applies to the whole test. option=file, expect=exit, await= and http= all work inside a peer block, and none of them is peer-scoped. Only option=env and option=timeout are singled out. Those two read as peer scope and are not.

A value the named option does not have is refused as well, not just an unknown option name. option=update:value=inspect-update-message and option=asn:value=abc each used to parse into nothing, which is the same silent drop one level down.

The accepted set is derived from the two parsers rather than written out a third time (peer.ClaimLine reports what ze-peer did with the line, and the runner's own line parser answers for the rest), so a directive added to either one cannot start being dropped here. Before the guard existed the block forwarded only expect= and action= lines and discarded everything else in silence: eleven reject=bgp directives across nine RFC-behaviour tests, and the reject=stderr of a tenth, asserted nothing while reading as the negative half of the proof.

Tmpfs (Virtual File System)

Tmpfs allows embedding multiple files within a single .ci file. Files are extracted to a temp directory at runtime.

Syntax

tmpfs=<path>[:mode=<octal>][:encoding=<type>]:terminator=<TERM>
<content>
<TERM>

Parameters

Mode Defaults

Files default to 0644. Use an explicit mode= only when the test needs a different permission.

Terminator Rules

• Must be non-empty
• Must be unique within file (no two Tmpfs blocks can share terminator)
• Alphanumeric and underscore only: [A-Za-z0-9_]+
• Matched exactly (no whitespace trimming)
• Recommended: EOF_<PURPOSE> (for example, EOF_CONF)

Example

tmpfs=peer.conf:terminator=EOF_CONF
peer test-peer {
    remote {
        ip 127.0.0.1;
        as 65533;
    }
    local-as 65533;
}
EOF_CONF


option=file:path=peer.conf
option=asn:value=65533
expect=bgp:conn=1:seq=1:hex=FFFF...

Security Constraints

1. No absolute paths - must be relative
2. No parent traversal - no .. components
3. No hidden files - no . prefix in path components
4. Path length limit - max 256 characters
5. Path depth limit - max 10 levels

Limits

Configurable via environment variables:

Options

option=<type>:key=value[:key=value...]

Choosing between needs-linux, caps=, and skip-os

caps= takes a comma-separated list, so a test that programs netlink and loads eBPF declares caps=net-admin,bpf and is gated on both. An unknown token is a parse error on every host, macOS included, so a typo cannot silently disable the gate.

caps=net-admin exists because Linux alone is not the requirement. On an unprivileged Linux host, a CI runner or a rootless container, a test that applies interface config does not fail cleanly: the interface plugin fails its configure handshake with operation not permitted and the DAEMON exits 1, then the TEST hangs because its check peer waits for a session the exited daemon will never open. The gate reads CapEff from /proc/self/status, not uid 0: a setcap'd binary holds the capability without being root, and a restricted container can be root without it.

A caps= test does not run in the merge gate. ./le verify worktree runs unprivileged, so the marker turns an opaque hang into an honest skip there, and the coverage relocates to .github/workflows/qemu-nightly.yml. TestCapabilityGatedTestsHaveANativeVMHome fails when that link is broken: marking a test with a capability nobody's CI has would be a coverage deletion wearing a skip's clothing.

OPEN Behaviors

Sender Facts (hold-time, graceful-restart, llgr, paths-limit)

Each of these four values states a fact about ZE-PEER that a receiver acts on. They exist because a mirror asserts sameness: mirrored, each one made ze read its own configuration back out of the peer's OPEN and believe the peer had said it.

option=open:value=hold-time:seconds=<N>
option=open:value=graceful-restart:restart-time=<N>[:family=<f>[,<f>]][:forward-state=true|false]
option=open:value=llgr:stale-time=<N>[:family=<f>[,<f>]][:forward-state=true|false]
option=open:value=paths-limit:family=<f>[,<f>]:limit=<N>

The families are what make graceful restart act at all. RFC 4724 Section 3 pairs the Restart Time with a tuple list, and onSessionDown builds its stale family set from that list and returns without dispatching anything when the set is empty. Ze's own code 64 carries the time and no tuples (parseGRCapValue), so until ze-peer owned the value, retain-routes, mark-stale and purge-stale were never dispatched in any test.

A file that states none of them gets the harness's own defaults, never ze's.

Two refusals fail the file rather than dropping a line in silence:

• Stating a fact AND add-capability or drop-capability for the same code. The stated octets would win, and the typed line would be read, validated and then lost.
• Stating a fact for a capability ze does not offer. The capability SET le test peer sends mirrors ze's, so the value would have no place to go. Configure ze to offer the capability, or drop the option.

BGP Identifier Control (router-id)

option=open:value=router-id:id=<a.b.c.d>

Ze-peer's default OPEN carries ze's own BGP Identifier with the last octet incremented, which is always a distinct, valid identifier. This option replaces it outright, so a test can present an identifier the default can never produce: 0.0.0.0, or ze's own router-id (RFC 6286 Section 2.2 rejects both, the second only from an internal peer). A malformed or IPv6 value fails the file when it is read. It was ignored until 2026-09-08, which sent the DEFAULT identifier from a test that asked for an invalid one: the file then tested the valid identifier and passed.

Capability Control (drop-capability / add-capability)

Ze-peer mirrors the capability SET of ze's OPEN, so a .ci that says nothing about capabilities still negotiates whatever ze offers. It does NOT mirror the values that describe the SENDER. The AS, the BGP Identifier, the Hold Time, the Role (code 9), the Graceful Restart time, families and flags (code 64), the ADD-PATH directions (code 69), the Long-Lived Graceful Restart stale time and flags (code 71), the FQDN (code 73), the software version (code 75) and the PATHS-LIMIT entries (code 76) are each resolved from the test's own configuration and written into ze-peer's OPEN, because a mirror asserts sameness and every one of those facts is about the speaker rather than about the session. ownedCapabilities (internal/test/peer/open_capability.go) is the list: a code absent from it is mirrored by construction.

The drop-capability and add-capability options act on that reconciled OPEN at wire level, allowing tests to control exactly which capabilities ze-peer advertises.

A capability the .ci states REPLACES the one ze-peer would have resolved. An add-capability naming any resolved code -- 9, 64, 65, 69, 71, 73, 75 or 76 -- is sent as written and ze-peer adds no second capability of that code, so a file that drops a code and adds it back gets exactly what it asked for. Dropping code 65 removes the four-octet AS capability, which leaves the two-octet My Autonomous System field as the only carrier of the peer's AS: above 65535 that field can carry AS_TRANS alone, which is what RFC 6793 Section 3 defines for a speaker with no two-octet AS.

An add-capability:code=65 carrying four octets IS the AS declaration, and the My Autonomous System field follows it. It outranks option=asn and the derivation, because it names the octets that reach the wire and RFC 6793 Section 4.1 makes those the octets a receiver reads. Letting the header field come from anywhere else would put two ASNs in one OPEN, which is the disagreement ze answers with NOTIFICATION 2/2 Bad Peer AS. A stated code 65 of any OTHER length declares no AS: it is a malformed capability the test is driving on purpose, so it is sent as written and the header field keeps the AS the rest of the configuration resolved. That is the one case where the two carriers differ, and they differ because the .ci asked.

Drop a capability:

option=open:value=drop-capability:code=<N>

Removes the capability with the given code from ze-peer's OPEN response. The peer will not see this capability in the mirrored OPEN.

Add a capability:

option=open:value=add-capability:code=<N>:hex=<value-bytes>

Adds a capability with the given code and hex-encoded value bytes to ze-peer's OPEN response.

Use case: testing capability mode enforcement:

When Ze is configured with require mode for a capability, it sends a NOTIFICATION if the peer lacks that capability. To test this, use drop-capability to make ze-peer omit the capability from its response:

# Test: Ze requires ASN4, ze-peer drops it → Ze should send NOTIFICATION
option=open:value=drop-capability:code=65

When Ze is configured with refuse mode, it sends a NOTIFICATION if the peer has a capability. To test this, the default mirror behavior already includes the capability, but add-capability can add capabilities not in the original OPEN:

# Test: Add a custom capability for refuse testing
option=open:value=add-capability:code=73:hex=067A652D626770

Multiple overrides can be combined:

option=open:value=drop-capability:code=65
option=open:value=drop-capability:code=2
option=open:value=add-capability:code=73:hex=067A652D626770

UPDATE Behaviors

Routes ze-peer sends after the OPEN handshake, before it starts matching expectations.

send-default-route and send-route build the AS_PATH that a real speaker sends. On an eBGP session, where ze-peer's OPEN AS differs from ze's, the path starts with ze-peer's own AS (RFC 4271 Section 5.1.2). An origin-as that names a different AS follows it, so origin-as=65001 from AS 65002 sends [65002 65001]. On iBGP the path is origin-as alone, or empty. An as-path key is the whole path and goes out as written. Ze drops a route from an eBGP neighbor whose leftmost AS is not the neighbor's (RFC 4271 Section 6.3), so a hand-written hex= UPDATE from an eBGP peer MUST start its AS_PATH with that peer's AS.

Generating one oversize UPDATE (max-msg):

option=update:value=send-bulk:prefix=10.0.0.0/24:count=16373:next-hop=10.0.0.1:origin-as=65001:max-msg=65535

max-msg caps one generated message, header included. It defaults to the RFC 4271 limit of 4096, and accepts 23 to 65535. Raise it to 65535 when the session negotiates the Extended Message capability (RFC 8654), which ze advertises when the peer carries capability { extended-message enable }. Prefixes that do not fit one message spill into the next, so the example above is exactly one 65535 byte message: a 65516 byte body, the largest BGP permits.

Use it whenever a test needs an UPDATE too large to write as a hex literal. A max-size message is 131070 hex characters, which is past the 64 KiB line limit of both .ci scanners and past what a reviewer can check. It is also the only way to hand the daemon a single oversize body: splitting the same prefixes across several standard messages is a different input, because the daemon decides per message.

A malformed key fails the test load rather than sending nothing. That is deliberate: a spec that silently degraded to count=0 would let a test asserting a route was NOT forwarded pass because no route was ever offered.

Commands

cmd=<type>:key=value[:key=value...]

API Commands

cmd=api:conn=<N>:seq=<N>:text=<command>

Example

cmd=api:conn=1:seq=1:text=update text origin set igp nhop set 10.0.1.1 nlri ipv4/unicast add 10.0.0.0/24

Process Commands (Foreground/Background)

For orchestrating multiple processes:

cmd=background:seq=<N>:exec=<command>[:stdin=<name>][:name=<handle>][:timeout=<dur>][:ready=<text>]
cmd=foreground:seq=<N>:exec=<command>[:stdin=<name>][:timeout=<dur>][:exit=<N>]
cmd=stop:seq=<N>:name=<handle>[:signal=kill|term]

Markers may appear in any order; each value runs to the next key in the table above, whichever key that is.

The table is the whole vocabulary. Any other :<word>= on a cmd= line fails the file, naming the key, the accepted set and the line. The scan reads the whole line, exec= included, because that is exactly where a key the parser does not read ends up: a value runs to the next KNOWN key, so an unknown one is swallowed into the value before it rather than dropped. cmd=foreground:seq=2:exec=ze -:stdin=ze-bgp:timeout=15s:env=ZE_FWD_WRITE_DEADLINE=10s parsed with timeout="15s:env=ZE_FWD_WRITE_DEADLINE=10s", so the line got neither the timeout it declared nor the variable, and every assertion still passed. Set an environment variable with option=env:var=<name>:value=<value>.

One consequence: a command carrying a :<word>= span of its own cannot be written on a cmd= line. Put it in a tmpfs= script and run the script.

timeout= must be a Go duration (10s, 1m30s). Both readers of the value keep their own default when it does not parse, so it is refused here instead.

The test deadline ends the whole process tree of every cmd= process. Each one leads its own process group, and the deadline kills that group, so a fixture's daemon dies with the fixture. A descendant that left the group cannot hold the test open either: the runner stops reading its output pipes 10s after the kill.

Background: Starts and keeps running until its timeout, an explicit stop, or test completion. Foreground: Setup commands finish before the next step. A ze daemon starts without blocking later steps. Its peers or observer determine when teardown starts. Stop: Terminates a named background process mid-test (see below).

A foreground helper that is the LAST cmd= line is the exception, and a file that asserts on its output MUST declare expect=exit:code=. The runner waits for a foreground helper only while another command still follows it; as the last line it is classed with the daemons, so it is started and torn down with no Wait, and its output reaches the accumulators only if it wins that race. The exit assertion is what makes the runner wait for it (fgProc.Wait()), and the code it asserts is that helper's own. Without it an expect=stdout:contains= over the helper's span reads an empty buffer, and a reject= over the same span can never fail.

When an embedded observer sends request shutdown, teardown waits out that daemon's bounded self-stop grace before it stops any other background process, whatever order those processes started in. A helper the daemon still talks to during its own shutdown stays up for that wait.

How an exec= value becomes argv

The runner splits the value on spaces and tabs, and a span inside " or ' stays ONE argument. Quote an argument that carries a space or a pipe, exactly as you would in a shell:

cmd=foreground:seq=1:exec=ze cli -c "show config dump - | json":stdin=config

-c receives show config dump - | json, and the quotes do not reach the process. Three limits apply:

• A backslash escape is not handled. \" is a backslash and a quote character, never a literal quote inside an argument.
• An unbalanced quote fails the test with unclosed quote in ....
• No shell runs, so |, > and $HOME are ordinary characters inside an argument. The runner expands $PORT and $PORT2 and nothing else, and le test fixture expands the environment in its own arguments.

One splitter serves every suite, so a cmd= line produces the same argv wherever it runs.

The first word selects the program. ze is the daemon under test, and le test <name> runs the harness, which is the runner's own executable. Any other head, a retired standalone harness name among them, is no head of the runner's: it reaches the PATH lookup, finds no program, and fails the step. A name that the suite compiled into its temporary directory runs from there: the chaos suites compile le, so exec=le chaos run ... runs that build. Any other name is found on PATH. A shim directory on the child's PATH holds ze and le, so a plugin run "le test ..." line in a config reaches the same binaries.

Provenance: until 2026-09-02 the .ci runner split the value on whitespace alone while the parse suite honored quotes. The example above was already published here, so four test/ui tests were written against it. -c received only the first word of the quoted command, ze cli fell through to its SSH client, and each test failed with no credentials for 127.0.0.1:2222.

ready= -- wait for a background process to serve

A background process that is not ze-peer gets 100ms to start before the next step runs. That is a guess, and a loaded host makes it wrong. A fixture that serves a socket the next step dials (a BMP collector, an RTR cache) then starts after ze has already dialed, ze gets connection refused, and ze waits out its reconnect interval: 30s for BMP (RFC 7854), 600s for RTR. The test fails as a protocol stall.

ready=<text> removes the guess. The runner reads the process's stdout and stderr, and it starts no later step until the text appears:

cmd=background:seq=1:exec=le test fixture plugin/bmp-sender-statistics-collector $PORT2:ready=BMP-COLLECTOR: listening on
cmd=background:seq=2:exec=le test peer --port $PORT:stdin=peer
cmd=foreground:seq=3:exec=ze --plugin ze.bgp-bmp -:stdin=ze-bgp

• The process MUST print the text AFTER it binds, never before.
• The wait has no deadline of its own. The test budget bounds it. When the budget ends first, the test fails with background_never_ready, and the message names the command, the text, and all output of the process.
• The key is refused on cmd=foreground and on a ze-peer line. A foreground command is already awaited, and ze-peer has its own "listening on" barrier. An empty value is refused because it matches any output.
• The text is a substring match, and any :<word>= span ends it, as with every other key.

cmd=stop -- terminate a background process mid-test

A background process started with name=<handle> can be stopped at a chosen step by cmd=stop:seq=<N>:name=<handle>. The runner looks the process up, signals it, and waits for it to exit before the next step runs, so a later step can deterministically observe what happens after the process dies (e.g. show vpn ipsec sa emptying once an IKE responder is killed and DPD fires).

• signal=kill (default) sends SIGKILL: the process gets no chance to flush or send a protocol teardown. This is the choice for liveness/dead-peer tests where the peer must go silent (a clean shutdown would take a different code path).
• signal=term sends SIGTERM, escalating to SIGKILL if the process does not exit within the teardown grace period -- a graceful stop.

Fail-closed: a cmd=stop naming a process that was never started (no matching name=) fails the test with a clear error; it never silently no-ops, and it can only ever signal a process the runner itself started (never an arbitrary PID). Teardown still kills every remaining background process, and tolerates one the stop step already reaped.

Provenance: added by spec-fixit-runner-kill-background to unblock end-to-end peer-death observation (the deleted test/ipsec/ipsec-dpd-timeout.ci needed it).

exit= vs expect=exit:code= (per-command vs file-level)

expect=exit:code= is file-level: Record.ExpectExitCode is a single value (a later expect=exit:code= silently overwrites an earlier one) and the runner compares it against lastQuickZeErr -- the exit status of the last quick-exit ze command in the file. A file that runs several ze config validate commands therefore asserts only the final one; every earlier command can exit with any code and the test still passes.

Use exit= on the cmd= line to assert a specific command's own exit code. It is checked the moment that command finishes, and names the offending seq on failure:

cmd seq=2 (ze config validate -): expected exit code 1, got 0

Prefer exit= whenever a file runs more than one quick-exit ze command. A "quick-exit ze command" is any foreground ze whose verb is not a daemon verb (hub, start, cli, monitor) and which has no config-file argument or --web flag.

Every stream assertion is scoped to the command above it

expect=stdout:, reject=stdout:, expect=stderr:contains= and reject=stderr:contains= are checked against the output of the cmd= line directly above them, and against nothing else. Position is the grammar: the line means what its place in the file says it means.

• A line with no cmd= above it is refused at parse time, naming the file and the line. It named no command's output, so there was nothing for it to assert. Move it below the cmd= line that produces what it describes.
• A line under a cmd=stop is refused for the same reason. A stop directive terminates a named background process; it runs no program and writes nothing.
• The stream name still selects nothing for contains=. A command's span covers both its streams, because a program is free to write a message to either and a test that pins which one is asserting the wrong thing.
• expect=stderr:pattern= and reject=stderr:pattern= are NOT scoped. Those are the daemon-logging mechanism validateLogging owns, over the daemon's relayed stderr, and they stay file-level.

A command's span is the half-open slice of the run's two accumulators it wrote between starting and finishing. A quick-exit ze marks both ends around its own fold, so its span is exact. A background process is still writing when the test ends, and a foreground daemon's bytes arrive throughout its life, so both close at the final lengths: a long-running command's span can therefore contain bytes a later command wrote. A command the run never reached has an EMPTY span, never the whole buffer.

This replaced a file-level check over one combined buffer. Every assertion used to read Record.ClientOutput, the concatenated stdout AND stderr of every command in the file, so an assertion could be satisfied by a command its author never named and a reject could trip on any command's output. Two files did exactly that: test/plugin/kernel-capability-unknown-starts.ci and kernel-capability-doctor-reports.ci each asserted that ze doctor printed a diagnostic code and then ran ze explain <that code>, whose output satisfied the assertion on a host where the doctor check does not run at all.

So a file no longer needs splitting to make a negative assertion mean something. test/plugin/vpp-doctor-hugepages.ci and vpp-doctor-hugepages-quiet.ci, and test/appliance/no-install-appliance.ci and appliance-help-not-deprecated.ci, are each one scenario in two files for a reason that no longer applies; they are correct as they stand and splitting a new file for that reason is not.

A failure names the command: cmd seq=2 (ze config validate -): output does not contain "...".

Known gap: 108 quick-exit ze commands across 50 .ci files predate exit= and are still unasserted (their expect=exit:code= never reaches them). Arming them may surface real defects; tracked in plan/known-failures/.

The parse suite reads its own dialect

test/parse runs under ParsingTests, a second parser with its own execution model. Two differences are load-bearing for an author.

Every assertion is scoped to one command, as it is in the generic runner (above). This suite has always worked that way and the generic runner now does too, so the two dialects no longer disagree about what an assertion's position means. One difference survives: here expect=stderr:contains= MAY appear before the first cmd=, because a .ci holding an inline config and no command at all is a legacy negative test whose expected error it carries.

The dialect is a subset of the generic vocabulary, and nothing else parses. A directive no arm reads fails the file at discovery, naming the directive, the file, and every directive the suite does read. It is the same rule as "Unknown Keys Are a Parse Error" above, applied to the whole line rather than to a key inside it.

Two spellings this suite once had are deleted, not aliased. Both existed only here, and both are what two parsers over one corpus costs:

• expect=stdout:regex= is now expect=stdout:pattern=. The generic parser reads pattern= and refuses regex=, so the gates that walk the whole corpus with it could not read three test/parse files at all.
• expect=stdout:not:contains= is now reject=stdout:contains=, which this suite already read with the identical meaning. This is the one that mattered: the generic parser splits not:contains= at the :contains= key boundary and drops the bare not, so one written line meant "must be absent" to the suite that runs test/parse and "must be present" to every gate that reads it.

cmd=...:timeout=<duration> is honored, and a bare ze - runs as ze start <file>, the same translation the generic runner makes. Substituting the path in place produced ze <path>, which is not a command: the daemon answered unknown command: <path> with its usage and exit 1 before reading a line of config, so a file could assert expect=exit:code=1 and be satisfied by the usage error.

Daemon readiness (ze only): a ze daemon launched either foreground or background is told (via ZE_READY_FILE) to write daemon.ready once startup completes, and the runner publishes its PID to daemon.pid in the tmpfs directory. Tests poll both files directly or through a compiled driver in internal/test/fixture before signaling the daemon or asserting on it. The readiness handshake is armed only for Ze daemons.

Example (Decode Test)

stdin=payload:hex=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF003C...
cmd=foreground:seq=1:exec=le test decode --family ipv4/unicast -:stdin=payload
expect=json:json={ "type": "update", ... }

Example (Multi-Process)

stdin=peer:terminator=EOF_PEER
option=asn:value=65000
expect=bgp:conn=1:seq=1:hex=FFFF...
EOF_PEER

stdin=ze-bgp:terminator=EOF_CONF
peer test-peer { remote { ip 127.0.0.1; } ... }
EOF_CONF

cmd=background:seq=1:exec=le test peer --port $PORT:stdin=peer
cmd=foreground:seq=2:exec=ze bgp server -:stdin=ze-bgp:timeout=10s

Example (Multi-Peer)

Tests needing two or more BGP peers use the same $PORT on different loopback addresses. The ze_bgp_tcp_port override sets all peer ports uniformly, which is correct because every peer listens on $PORT.

# Source peer on 127.0.0.1 (default)
stdin=source:terminator=EOF_SOURCE
option=tcp_connections:value=1
action=send:conn=1:seq=1:hex=FFFF...
EOF_SOURCE

# Dest peer on 127.0.0.2 -- sink mode absorbs any UPDATE
stdin=dest:terminator=EOF_DEST
option=tcp_connections:value=1
EOF_DEST

cmd=background:seq=1:exec=le test peer --port $PORT:stdin=source
cmd=background:seq=2:exec=le test peer --bind 127.0.0.2 --mode sink --port $PORT:stdin=dest
cmd=foreground:seq=3:exec=ze -:stdin=ze-bgp:timeout=20s

Each ze-peer gets independent output capture and WaitFor synchronization. The runner waits for each peer's "listening on" message before starting the next command.

On Linux, 127.0.0.2 works automatically (127.0.0.0/8 routes to lo). On macOS and FreeBSD, the test runner adds loopback aliases via the SIOCAIFADDR ioctl.

IPv6 works differently, because a host carries exactly one IPv6 loopback address. A fixture that needs a second one uses fd00::2, which is unique-local (RFC 4193) and never globally routable. ./le setup install adds it, and ./le setup check reports whether it is there. The runner never adds it: the ioctl returns EPERM to an unprivileged process, and ./le verify current mode full runs as an ordinary user. A test that binds an address this host does not carry fails at once with loopback_address_missing and the command to run, rather than timing out on a bind that could not succeed.

The check reads the three places a fixture names an address it binds. One is le test peer --bind <ip> on a cmd= line. The second is connection { local { ip <addr> } } in the config the fixture embeds: Ze sends from that address and listens on it when accept is true, so the host must carry it too. The third is local-address <addr>; in an ExaBGP-syntax config the exabgp-compat suite keeps beside its .ci and names with option=file:. That suite has a parser and a runner of its own, so it calls the same scan through EnsureConfigFileBindAddresses rather than through the embedded path.

A local address outside 127.0.0.0/8 and fc00::/7 is left alone. A config-validation fixture names a routable one (local { ip 192.0.2.1 }), the daemon exits before it binds anything, and ./le setup install adds no such address. The local-link-local fe80::1 leaf that sits beside local-address in an exabgp-compat config is left alone for the same reason.

An exabgp-compat case runs TWO processes, ze and the mock BGP server, and its verdict names which one ended the case:

• ze exited before the mock BGP server finished its script: <error> is a daemon defect. Read the client stderr in the failure block.
• the mock BGP server failed: <error> is a wire or document disagreement. The mock prints the frame or the JSON member that did not match.
• the mock BGP server ended without reporting success means its script did not complete and it said nothing about why.

Both used to print the bare exit status N of whichever process failed, which told a reader nothing about which of the two to open.

Both processes are stopped by a defer beside the start that created them, so a case that returns by any route, a panic included, leaves neither running. The stops written inside the branch arms are ORDERING barriers rather than lifetimes: a process's output must not be read while it is still writing, and its exit status does not exist until it has one. How the suite's forks are created and canceled is in docs/architecture/testing/runner-architecture.md, "How a suite's child processes end".

Expectations

expect=<type>:key=value[:key=value...]

BGP Wire Expectations

expect=bgp:conn=<N>:seq=<N>:hex=<hex-bytes>
expect=bgp:conn=<N>:seq=<N>:prefix=<hex-bytes>
expect=bgp:conn=<N>:seq=<N>:contains=<hex-bytes>
expect=bgp:conn=<N>:seq=<N>:ordered=<hex-bytes>

Validates the BGP wire message received: hex= matches the exact message, prefix= the message start, contains= a substring anywhere in one message. Within one seq group, hex/prefix/contains checks match in any order and each consumes exactly one received message.

ordered= checks in a seq group form a strict FIFO subqueue for asserting in-order delivery across message boundaries: the front needle must appear in the received message, and one message may consume several consecutive needles, each matched at an advancing offset (so order inside a packed message is enforced too). Use ordered= instead of per-message contains= when the sender may legally pack several NLRIs into one UPDATE (the forward rail's bucket merge): per-message framing is not a property ze owes, but delivery order is. A message whose content matches only a non-front needle consumes nothing and is reported as a mismatch.

These forms are peer-block directives (inside a stdin=<name>: block); top-level expect=bgp lines support hex= only.

JSON Expectations

expect=json:conn=<N>:seq=<N>:json=<json-object>
expect=json:json=<json-object>

Validates the decoded message matches expected JSON.

Validation rules:

• Parsed and compared field-by-field (key order independent)
• Volatile fields removed before comparison: exabgp, ze-bgp, time, host, pid, ppid, counter
• Neighbor normalization: peer ↔ neighbor treated as equivalent, direction field ignored
• All non-volatile fields must match exactly

Exit Code Expectations

expect=exit:code=<N>

Validates the foreground process exit code. A test whose ONLY assertion is expect=exit:code=0 is accept-only (weak) and is gated by a lint; see Assertion Strength.

An Unknown Directive or Key Is a Parse Error

Every directive the runner reads is declared: the action= word, the type after it, and the keys inside it. A word or a key outside its declared set fails the file at discovery, naming what was written, the accepted set, and the line. Nothing is dropped in silence, and nothing is guessed.

line 12: unknown action "exepct" (accepts action, await, cmd, command, expect, http, option, reject, stream)
line 12: unknown expect type "stdoutt" (accepts bgp, command-error, event, exit, file, json, output, stderr, stdout, stream, syslog)
line 12: cmd=foreground: unknown key "env" (accepts exec, exit, name, ready, seq, stdin, timeout)

The accepted set in each message is the list the parser gates on, so it cannot describe a vocabulary the runner does not have. The line number is the line in the FILE: comments, blank lines and whole stdin= and tmpfs= blocks are consumed before the directives are parsed, and the refusal used to count only the directives that survived, which named line 2 for a line that was number 69.

The rule exists because a dropped key takes the whole assertion with it. An expect=stdout:not-contains=X line recorded NO assertion and then passed whatever the command printed. Ten such lines were live across seven .ci files when the check was added, and one of them was the only assertion its test made.

Two spellings get a named message, because both are the right key somewhere else: not-contains= belongs to expect=file:, and !contains= was the expect=stdout spelling until stream non-containment moved to reject=.

Stdout Expectations

expect=stdout:contains=<text>
expect=stdout:pattern=<regex>

Two modes:

• contains= -- substring match against stdout (multiple allowed, all must match)
• pattern= -- regex match against stdout (uses Go regexp syntax)

Stdout must NOT contain text is reject=stdout:contains=, below. A stream has ONE negation spelling and it lives on reject=.

Stderr Expectations

expect=stderr:pattern=<regex>
expect=stderr:contains=<text>

Two modes:

• pattern=: regex match against stderr (uses Go regexp syntax)
• contains=: substring match against stderr

pattern= reads the daemon's stderr alone. contains= reads the same combined buffer every stdout assertion reads (see the scope note below), so it matches a needle the command wrote to stdout. Reach for pattern= when the stream matters.

Stderr must NOT contain text is reject=stderr:contains=, below.

Await (deterministic stderr fence)

await=stderr:contains=<text>[:timeout=<dur>][:then=stop]

Blocks the runner until the daemon's relayed stderr contains <text>, then tears the daemon down. This is a deterministic replacement for a blind time.sleep that only held the daemon open long enough for a line to appear. timeout= is an optional Go duration (default: 80% of the test budget, at least 10s); on timeout the test fails with a precise message that names the needles still missing. <text> follows the same rule as expect=stderr:contains=.

Several await= lines form ONE fence. It holds once every needle has appeared, in any order: an external plugin's stderr is relayed apart from the daemon's own writes, so an order between them would be a race of its own. A needle may be declared once, and timeout= on one line only.

Without then=stop, the fence is a precondition of the normal end: after it holds, the runner still waits for the peers to exit (a peer script that signals the daemon several times, test/reload/reload-rapid-sighup.ci, relies on this).

then=stop on any await line makes the runner END the test once the fence holds. Use it when a peer lingers (option=linger:value=true), because a lingering peer never exits by itself, so the default end waits forever and a fixture then had to stop the daemon a fixed delay after its signal. Under load that SIGTERM landed while the reload under test was still verifying, and shutdown canceled it. With then=stop the sequence waits on events only:

A plain fence publishes daemon.pid without waiting for daemon.ready, because a plugin that aborts startup may never write it. A then=stop fence keeps that wait: its daemon is expected to run, and a trigger that sends SIGHUP before the handler is installed kills the daemon.

Steps 1 and 2 share the fence's one deadline. A NON-lingering check peer is not waited in step 2, because it ends itself, and its expectation can be the Cease the stop in step 3 produces (test/reload/reload-dynamic-peer-survives.ci). Await the reload's OUTCOME line, never an earlier one: sighup reload complete for a reload that succeeded, reload error for one that was refused.

cmd=foreground:seq=3:exec=ze -:stdin=ze-bgp:timeout=20s
await=stderr:contains=incomplete peer definition:then=stop
await=stderr:contains=reload error
expect=stderr:contains=incomplete peer definition
reject=stderr:contains=sighup reload complete

Pair it with a matching expect=stderr:contains= so the line is both the fence and the assertion. Use it for the reject-fence bucket: an external plugin whose refusal aborts the daemon's plugin-startup coordinator (StartupCoordinator.PluginFailed) leaves no in-daemon observer able to poll it, so the relayed stderr line is the only non-plugin signal.

Choose a needle that is SPECIFIC to the plugin under test, not just a shared phrase. Example (test/plugin/as112-external-refuses.ci): the bare "refusing to start as an external plugin process" is emitted verbatim by three plugins, so the needle includes the as112-only tail -- the address-ownership registry:

cmd=foreground:seq=1:exec=ze -:stdin=ze-bgp:timeout=15s
await=stderr:contains=refusing to start as an external plugin process -- the address-ownership registry
expect=stderr:contains=refusing to start as an external plugin process -- the address-ownership registry

Syslog Expectations

expect=syslog:pattern=<regex>

Validates that captured syslog output matches the regex pattern. When any expect=syslog: line is present, the test runner automatically starts a UDP syslog server and injects ze.log.backend=syslog and ze.log.destination=127.0.0.1:<port> into the test environment.

File Expectations

expect=file:path=<rel>:exists=true
expect=file:path=<rel>:absent=true
expect=file:path=<rel>:contains=<text>
expect=file:path=<rel>:not-contains=<text>
expect=file:glob=<rel-pattern>:count=<N>
expect=file:glob=<rel-pattern>:contains=<text>
expect=file:glob=<rel-pattern>:not-contains=<text>

Validates files after the test process or peer sequence has completed. Paths and glob patterns are relative to the test's own work directory, which is where every child ran and where a daemon writes its artifacts. A test that declares tmpfs= files gets them in that same directory, so the two spellings name one place. For glob contains, at least one matched file must contain the text. For glob not-contains, no matched file may contain it.

Use file expectations for loose source files and exported artifacts such as logs. Persistent config, versions, pointers and plugin state use key expectations:

expect=key:path=meta/config/router.conf/active:exists=true
expect=key:path=meta/config/router.conf/candidate:absent=true
expect=key:glob=file/[0-9]*/router.conf:count=2
expect=key:glob=object/*:count=2
expect=key:glob=object/*:contains=router-id 1.2.3.4

A dated history entry holds sha256:<hex>, not the config: the bytes live in object/<hex> (docs/architecture/storage-backends.md, "Content-addressed history"). Count versions on the entries and assert config text on object/*.

Key expectations resolve beneath database/ in the work directory and decode the complete netcapstring frame before inspecting its value. A bad CRC or trailing frame bytes fail the assertion. They accept the same path, glob, count and containment checks as files. A missing key satisfies absent; an unreadable or corrupt key does not.

Negative Expectations (reject)

reject=stderr:contains=<text>
reject=stderr:pattern=<regex>
reject=stdout:contains=<text>
reject=stdout:pattern=<regex>
reject=syslog:pattern=<regex>
reject=bgp:conn=<N>:pattern=<hex>

Inverse of expect= -- the test fails if the pattern matches. Used to verify that unwanted output (e.g., deprecated warnings, ERROR-level messages) does NOT appear.

reject=bgp -- bytes a peer must never receive

reject=bgp is the only reject type ze-peer reads, because ze-peer is what sees the wire. It goes inside the peer's stdin block, beside the expect=bgp lines. The hex needle is matched on wire bytes at a byte BOUNDARY, which expect=bgp:contains= is not: that one is a plain substring match over the hex text, so it can also match at an odd nibble offset.

Five properties make it an assertion rather than a hope:

• It is never consumed. Every frame the peer's message loop reads is checked against it. The option=linger loop keeps checking after completion. Pair the rejection with option=linger:value=true to hold it open for the whole test.
• A rejection found during linger RETRACTS the success token. Under option=linger the peer prints its success token BEFORE the loop, because teardown is a kill and a post-run print can be lost. The verdict reads that token, so a rejection arriving afterwards has to withdraw it: the peer prints ZE-PEER-REJECTED on its own output, and the verdict reads the retraction after the token and lets it win. Without that channel a linger rejection was detected, returned, and then discarded, which made every negative assertion held open by linger vacuous.
• Check mode only. Sink and echo peers read every accepted connection concurrently against one checker, so a conn= could not select the session the frame arrived on. The runner refuses the file, and ze-peer refuses to start.
• The block MUST also carry an expect=bgp:conn=<N> on the same connection, and the runner refuses the file otherwise. That rule is necessary and it is not sufficient. It bounds how long the rejection is checked. It cannot say whether the forbidden bytes would have arrived inside that window. The author owes the second half. Send the delivery LAST on that connection, so a leak of the governed route arrives ahead of it. Or hold the session open with option=linger. A connection stops being read once its expectations complete, and its rejection then passes for the one reason a rejection must not.
• An odd-length or non-hexadecimal needle is a parse error, because a needle that can never match would pass for that same reason. The runner reads the line with ze-peer's own parser (peer.ParseRejectRule), so a typo fails the FILE rather than the peer: a rejection rejected inside ze-peer would stop it binding, and the runner would report that as a bind timeout.

stdin=external:terminator=EOF_EXTERNAL
option=tcp_connections:value=1
option=linger:value=true
# The delivery that makes the rejection an assertion rather than an empty session.
expect=bgp:conn=1:seq=1:contains=18C00002
# 180A0100 is 10.1.0.0/24, which NO_ADVERTISE forbids on any peer.
reject=bgp:conn=1:pattern=180A0100
EOF_EXTERNAL

Assertion Strength: accept-only tests and readback

A test whose ONLY assertion is expect=exit:code=0 is accept-only (weak): it proves a config or command was ACCEPTED, never that it parsed to the CORRECT tree. A parser that accepts interval 300 but stores 0, or silently drops a source 0.0.0.0/0 block, still passes such a test green. This is the functional-suite analog of the "count-only assertion" mistake class (ai/rules/testing.md).

A lint enforces that the class cannot GROW: TestCIAcceptOnlyLint walks every test/**/*.ci, classifies each with the single accept-only predicate, and FAILS on a NEW accept-only test that is neither strengthened nor annotated. Existing accept-only tests are grandfathered in test/.accept-only-baseline (a sorted allow-list that only shrinks; strengthening or annotating a test removes its line). Correctly EXCLUDED (never weak): a test whose real check lives in a tmpfs set -e script (e.g. test/managed/auth-reject.ci), and any reject= test.

Strengthen with a readback

Add a second cmd= that dumps the parsed tree and assert a representative value with expect=stdout:contains= / pattern=. show config dump - reads a config from stdin and answers the stored tree, and | json renders it, so the assertion observes the parsed VALUE, not just that parsing did not error.

cmd=foreground:seq=1:exec=ze config validate -:stdin=config:exit=0
cmd=foreground:seq=2:exec=ze cli -c "show config dump - | json":stdin=config-dump
expect=exit:code=0
expect=stdout:pattern="interval": "300"

Two gotchas, both load-bearing:

• ze config dump requires a bgp { } block (it resolves the full BGP tree) where ze config validate does not. To keep proving that a subsystem-only config validates, keep the original validate step on the subsystem-only stdin block and run the dump readback against a second stdin block that prepends a minimal bgp { router-id ... }.
• A needle containing : followed by something shaped like key= must use pattern=. ~~Only json=/text=/hex=/pattern= preserve colons, and contains= truncates at the first colon.~~
  • Corrected 2026-07-30. That claim was true until ParseKVPairs became boundary-aware. contains= now keeps an ordinary colon, so contains=error: no such peer asserts the whole sentence.
  • What still splits is a colon that introduces a real key token: a letter, then letters, digits, - or _, then =. That split is deliberate, because it is how the engine-step form contains=aes-cbc:timeout=25 keeps working. So contains=note:level=high still splits at :level=, and it is the one shape that needs pattern=.
  • The old behavior was not harmless while it lasted. A sweep on 2026-07-29 found 203 assertions across 15 suites silently reduced to the text before their first colon. The re-armed assertions then exposed a security test (test/appliance/appliance-push-image-escape.ci). That test had never once executed the path-traversal guard it was named for.
• Keep a pattern= needle free of the substrings json=, text=, and hex=. ParseKVPairs extracts a complex-key value by strings.Index of the first such marker, so a needle that itself contains one of them is mis-split. None of the readback needles above contain these markers. This note is forward guidance for new needles.

Annotate when a unit test already covers the value

When a unit test already asserts the parsed value, a readback would duplicate it. Mark the test accept-only instead, with a comment naming the covering test:

# accept-only: md5 { password; ip } value-parsing is unit-covered by
# TestParsePeerMD5FieldsParsed (internal/component/bgp/reactor/config_test.go).

The marker is a comment line whose content is accept-only: followed by a non-empty reason. A file carrying it is allowlisted by the lint without a baseline entry. Keep the reason greppable and truthful (name the covering test).

Actions

action=<type>:key=value[:key=value...]

Notification

action=notification:conn=<N>:seq=<N>:text=<message>

Sends NOTIFICATION with shutdown message.

Send Raw

action=send:conn=<N>:seq=<N>:hex=<hex-bytes>

Sends raw bytes to peer.

Rewrite Config File

action=rewrite:conn=<N>:seq=<N>:source=<tmpfs-file>:dest=<config-file>

Copies a tmpfs file over the daemon's config file. Used with action=sighup to test config reload.

It also writes a marker a second process can wait on, which is its other use: test/reload/config-apply-ordering-address-swap.ci copies one file to session-returned.txt once the restarted BGP session has re-announced its routes, and the test's own driver waits for that file before it signals the daemon. When the rewrite is the LAST item the peer block queues, the exchange is over and the peer completes on it, exactly as action=sighup and action=sigterm do. Without that the peer kept matching, and the daemon's shutdown NOTIFICATION arrived against an empty expectation list and failed the test as a message mismatch.

Send SIGHUP

action=sighup:conn=<N>:seq=<N>

Sends SIGHUP to the daemon process. Reads PID from daemon.pid in the tmpfs directory (written automatically by the test runner).

Send SIGTERM

action=sigterm:conn=<N>:seq=<N>

Sends SIGTERM to the daemon process. Reads PID from daemon.pid in the tmpfs directory (written automatically by the test runner). After sending SIGTERM, the connection is expected to close (daemon shuts down gracefully).

HTTP Checks

HTTP checks validate web endpoint responses after all cmd= processes have started. Executed in seq order with automatic retry on connection errors (server starting up).

Assertion Checks (get/post)

http=get:seq=N:url=URL:status=CODE[:contains=TEXT][:bodyfile=PATH][:header=NAME: VALUE]
http=post:seq=N:url=URL:status=CODE[:contains=TEXT][:bodyfile=PATH][:sendfile=PATH][:content-type=TYPE][:header=NAME: VALUE][:insecure-tls=true]

Request Headers (header=)

header= takes the header exactly as it appears on the wire, Name: Value. Repeat header= to set several headers on one check. It works on get, post, and wait.

http=post:seq=1:url=http://127.0.0.1:$PORT/mcp:status=200:sendfile=call.json:header=MCP-Protocol-Version: 2026-07-28:header=Mcp-Method: tools/call:header=Mcp-Name: ze

Retries up to 20 times at 200ms intervals on transient connection errors (ECONNREFUSED, ECONNRESET, EOF). Non-connection errors (wrong status, missing content) fail immediately.

Readiness Polls (wait)

http=wait:seq=N:url=URL:status=CODE[:contains=TEXT][:header=NAME: VALUE][:timeout=DUR]

Unlike assertion checks, wait retries on all failures: connection errors, wrong status codes, and content mismatches. Polls at 500ms intervals until the condition is met or the timeout expires. Wait checks run before assertion checks, making them suitable for waiting until a server has populated data (e.g., routes injected by an async plugin).

Example

cmd=background:seq=1:exec=le test peer --port $PORT:stdin=peer
cmd=background:seq=2:exec=ze -:stdin=ze-bgp

# Wait until routes are available before checking graph output
http=wait:seq=1:url=http://127.0.0.1:$PORT2/lg/graph?prefix=10.10.1.0/24&mode=aspath&format=text:status=200:contains=AS2914:timeout=15s

# Exact SVG match against reference file
http=get:seq=1:url=http://127.0.0.1:$PORT2/lg/graph?prefix=10.10.1.0/24&mode=aspath:status=200:bodyfile=expect/graph.svg

# Substring check
http=get:seq=2:url=http://127.0.0.1:$PORT2/lg/graph?prefix=10.10.1.0/24&mode=nexthop&format=text:status=200:contains=egress

Sleeps and their justification markers

Every time.sleep( in a live .ci carries a marker comment in the form // sleep(<kind>): <reason>, on one # comment line directly above the sleep, indented to match it exactly. The embedded .ci observer body is indentation sensitive. Two producers enforce the marker: checkSleepJustification in internal/le/doc/wiring (run by ./le doc wiring, scoped to changed .ci files, listing every unjustified file:line and exiting 1), and writeCISleep in internal/le/hookruntime/writeedit.go, which blocks a Write or Edit that introduces an unmarked sleep.

The kinds are a closed set, and each one owes a different reason:

A free-text // settle comment is insufficient: it names no mechanism a reader can check. "The tracker pushes live carrier once a second" is a reason a later reader can overturn; "needs a moment" is the shape that makes a deliberate timer and a guessed duration the same line of code.

A separate ratchet caps how MANY sleeps exist: the total time.sleep( count across test/**/*.ci may not exceed the committed baseline in test/.ci-sleep-baseline, and ./le doc wiring fails when it does. The markers cap how many are unexplained.

The compiled observer API

Compiled .ci observers live under internal/test/fixture. They use pkg/plugin/sdk for the five-stage plugin protocol (Plugin.Run owns it) and the local fixture package for registration, dispatch, polling and failure reporting.

fixture.Poll around fixture.Dispatch is the payload-predicate wait: it blocks until the observed payload matches, within a bounded attempt count. It is what replaces a time.Sleep followed by a one-shot assertion.

fixture.Observe can request a clean daemon shutdown even after an assertion failed, so the daemon's exit code does not prove the observer's assertion. A failing observer returns an error, which fixture.Run hands to fixture.ReportFailure.

A fixture-driven .ci needs no bgp block and no le test peer to get a daemon that stops. request shutdown reaches a reactorless daemon through the shutdown callback the daemon wires beside its plugin server, before any plugin can dispatch, so a BFD-only or DHCP-only configuration stops on the fixture's request like a BGP one. Adding a BGP peer only to make the daemon stoppable adds a second protocol to the test's failure surface: test/bfd/bfd-detection-interval.ci was red for exactly that reason until 2026-09-07.

option=asn moves both AS numbers. The AS is resolved once and written into the two-octet My Autonomous System field AND the four-octet AS capability, so the two disagree only where a .ci states a malformed capability 65 itself (see "Capability Control" above). RFC 6793 Section 4.1 states the precedence a receiver applies: it "MUST use the AS number encoded in the Capability Value field of the 'support for four-octet AS number capability' in lieu of the 'My Autonomous System' field of the OPEN message". So a half-applied option leaves the capability carrying ze's own AS, ze reads THAT, finds an AS the session is not configured for, and answers OPEN Message Error / Bad Peer AS under RFC 4271 Section 6.2. The refusal is RFC 4271's; RFC 6793 decides which field ze read.

The sentinel is written where the scenario fails, BEFORE request shutdown. The runner reads it out of the DAEMON's stderr, and the daemon relays a plugin's stderr only while both processes live (internal/component/plugin/process, relayStderrFrom). fixture.Run reports the same error after Plugin.Run returns, which is after the daemon was asked to stop, so that line reaches the runner only when it wins the race with the shutdown. Measured 2026-09-02: an observer asserting a route that can never arrive still passed test/plugin/rpki-group-action.ci, and three plugin cases (fib-table, metrics-name-show, modify-increment-localpref) passed with a failing observer. fixture.ReportFailure emits the FIRST failure only, so the report at the failure site and the one on the way out are one line.

A driver's working directory is the per-test directory, so a driver names its files relatively and fixture.Run refuses to start in the checkout. The runner creates one directory for each test, gives it to every child of that test and removes it at the end (Record.WorkDir, internal/test/runner/runner_exec.go), so daemon.ready, testkey and <case>.conf are written where the test can find them and nothing survives the run. The same names in the checkout land beside tracked source: on 2026-09-05 a run left two ed25519 private keys and five config files at the repository root, none of them ignored, so a git add -A from any session would have committed a private key. fixture.Run therefore reads its own working directory first and exits 1 with the ZE-OBSERVER-FAIL sentinel when that directory holds the ze go.mod (sessionpath.IsRepoRoot). A driver that needs the checkout reads $ZE_REPO_ROOT, which the runner exports; it never reads the working directory for one.

An observer assertion is therefore worth only what its wait is worth: a value that a plugin fills asynchronously (the Adj-RIB-In after RPKI validation, the SPF table after the first run, a session that drops when the peer completes) needs fixture.Poll around the read, never a fixed settle before it.

Engine Steps

Engine steps drive a live daemon through CLI dispatch, first-class in .ci instead of an embedded Python observer. The runner serializes the parsed steps to engine-steps.json in the test tmpfs, and links it into the daemon-N/ config directory of every further daemon, because a plugin runs in its daemon's config directory; the .ci declares the executor as an external plugin (run "le test engine-steps ./engine-steps.json"), which runs the steps from OnAllPluginsReady and reports failures via the ZE-OBSERVER-FAIL sentinel the runner gates on.

command=<cli command text>
stream=<monitor command text>
expect=output:<predicate>[:timeout=<dur>]
expect=event:namespace=<ns>:name=<name>[:timeout=<dur>]
expect=stream:<predicate>[:timeout=<dur>]
expect=command-error:contains=<text>

command=/stream= keep their full raw text (colons included). expect=output re-dispatches the most recent command= until its predicate holds or the timeout expires; expect=stream matches delivered stream= events; expect=event matches a delivered event by its (namespace, name) identity.

expect=event

The executor declares ONE startup event subscription, derived from the expect=event steps of the file itself, and asks for enveloped delivery. A .ci therefore names each event once, in the step that waits for it, and nothing else declares the subscription.

A startup subscription rides the ready RPC, so it is registered before any plugin's OnAllPluginsReady runs. That is what lets a step observe the FIRST delivery of an event the daemon emits while it starts. test/ipsec/ipsec-sa-installed.ci is the case that forced it: the IKE engine emits vpn-ipsec/sa-up from its startup peer reconciliation, so a subscription dispatched from the step could only ever see a SECOND establishment, and a test with no re-negotiation has none.

Enveloped delivery wraps each event with its (namespace, event) identity (rpc.EventEnvelope), which is what tells one subscribed event from another once several share the one subscription. A bare payload decodes with an empty namespace and event, so a delivery the executor did not ask to be enveloped can never satisfy a step.

Two consequences follow, and both are load-bearing when writing a test:

• A step list names ONE namespace. rpc.SubscribeEventsInput carries one, so steps naming two are REFUSED before the executor dials, with a message naming both. Split the test rather than subscribing to one and dropping the rest.
• A step matches ANY delivery of that identity, scrollback included. It cannot assert "another one, after this point". A second sa-up after an operator clear re-matches the first, so re-establishment is asserted by a separate test (test/ipsec/ipsec-clear-reestablish.ci), not by a second expect=event.

expect=command-error

expect=command-error:contains=<text> asserts that the PRECEDING command= FAILED, and that its message contains the text.

Use it for a command that must refuse. The plugin SDK turns a StatusError response into a Go error, so without this directive the command step aborts the run before any expect= is reached, and no .ci can assert an operational error at all. That left one class untestable end to end, and it is the class where a wrong answer costs most: a command that must refuse is exactly the one whose failure mode is answering confidently instead. test/ipsec/ipsec-dataplane-show.ci uses it to prove that a dataplane which cannot be read SAYS so rather than rendering an empty table.

It takes contains= only, and no timeout. An error is the result of one dispatch, so re-dispatching until one appears would wait for a state change no expect= can cause.

A command failure that NO expect=command-error consumes still fails the run, whether the next step is of another kind or the command is the last step in the file. Every .ci written before this directive existed relies on that.

It must not be vacuous. A command= that SUCCEEDS does not satisfy expect=command-error: the step fails with "the preceding command succeeded". Without that rule the directive would pass for the very regression it guards, a refusal quietly becoming a successful empty answer (TestRunEngineStepsCommandErrorRequiresAFailure).

expect=output / expect=stream predicates

The optional trailing :timeout=<dur> is split off the END first, so a predicate operand may itself contain : (a compact-JSON fragment, an IPv6 address). The remainder is one predicate:

json= walks the raw data field (not status data): each .-segment indexes a JSON object by key or a JSON array by integer index (0..len-1; out-of-range or missing is "not yet", named at timeout). The leaf is compared as a string (numbers/bools stringified via JSON). absent=/json= are expect=output only: they re-dispatch a query, whereas expect=stream is an append-only event stream with no "absent" and no single-event JSON path.

absent= must be non-vacuous. An absent= on output that was never populated passes instantly (false green). Precede it with a step that makes the substring present (a contains=/json= after an inject), then the transition (e.g. a withdraw) the absent= proves. See test/plugin/engine-steps-predicates.ci.

Example

command=request bgp rib inject 10.0.0.1 ipv4/unicast 172.16.0.0/16 origin igp nexthop 10.0.0.2
command=show rib
expect=output:matches=172\.16\.[0-9.]+/16:timeout=10
expect=output:json=0.prefix=172.16.0.0/16:timeout=10
command=request bgp rib withdraw 10.0.0.1 ipv4/unicast 172.16.0.0/16
command=show rib
expect=output:absent=172.16.0.0/16:timeout=10

Complete Example

# Embed config using Tmpfs
tmpfs=test.conf:terminator=EOF_CONF
peer test-peer {
    remote {
        ip 127.0.0.1;
        as 65000;
    }
    router-id 10.0.0.2;
    local-address 127.0.0.1;
    local-as 65533;
    hold-time 180;

    family {
        ipv4/unicast;
    }
    announce {
        ipv4 {
            unicast 10.0.0.0/24 next-hop 10.0.1.254;
        }
    }
}
EOF_CONF

# Test configuration
option=file:path=test.conf
option=asn:value=65000

# Expected API command and wire output
cmd=api:conn=1:seq=1:text=update text origin set igp nhop set 10.0.1.254 nlri ipv4/unicast add 10.0.0.0/24
expect=bgp:conn=1:seq=1:hex=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF002F02000000144001010040020602010000FFFD4003040A0001FE180A0000

# EOR
cmd=api:conn=1:seq=1:text=announce eor ipv4/unicast
expect=bgp:conn=1:seq=1:hex=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF00170200000000

Consumers

Different components consume different line types:

Lines not recognized by a consumer are ignored.

A check-mode peer block MUST declare a ze-peer-consumed expectation

The consumer split above is load-bearing, not trivia. Only the four le test peer rows reach le test peer. Everything else -- including expect=json -- is validated by the test runner from its own copy of the messages.

A check-mode le test peer with no consumed directive has nothing to check, so it prints no test data available to test against and exits 1 before binding a listening socket. ze then dials a dead port, gets connection refused, and backs off 5->10->20->40s. That looks exactly like a BGP establishment stall and cost a multi-day investigation before the cause was found in the harness.

So a peer block whose only expectation is expect=json runs no BGP at all. The parser now rejects that at discovery time, naming the file and the remedy:

stdin=peer block: check-mode ze-peer (cmd seq=1) declares no ze-peer-consumed
expectation, so it exits with "no test data available to test against" before
binding a listening socket and the test can only pass vacuously.

expect=json still works, but only in addition to a consumed directive: it cannot make the peer listen.

A ze-peer always governs its test's result

A test with a check-mode ze-peer is never self-validated: the peer must print successful and its expect=json expectations must match, whatever else the file asserts. An expect=exit:code=0 is additive and does NOT disable the BGP checks.

Until 2026-07-16 it did, which is how a test could pass on ze's exit code while its peer never listened and its JSON assertion never ran. sink/echo/inject peers do not govern (they never report completion), so a test whose only peers are scaffolding is still governed by its own exit/output/file assertions.

A check peer prints successful only when EVERY expectation it holds was met: every expect=bgp line of every conn=N it declares. A peer stopped before that, by SIGTERM from the runner, by the test's timeout, or because a dialing peer's one connection ended while conn=2 was still owed, exits 1 with stopped with expectations unmet: conn=N still owes <expectation> (K expectations unmet). Until 2026-09-24 a peer that SIGTERM reached while it waited for its next connection printed successful, so a test passed with its later connections never opened.

A scaffolding ze-peer is signaled at teardown

A sink, echo or inject le test peer never ends itself: its accept loop runs until its context is cancelled, and le test peer maps SIGTERM to that cancel. The runner sends that SIGTERM at teardown, after the last step and before the barrier that collects peer output. The peer exits with status 0 and its capture is complete, so a .ci author needs no teardown directive and must not add one.

A check-mode peer is never signaled. It exits by itself when its expectations are met, and the runner reads its verdict from the capture a signal would truncate.

Before 2026-08-10 no code signaled a scaffolding peer, so the drain barrier waited out its full 10s grace on every --mode sink or --mode echo test. That cost was not only latency: test/plugin/event-predicate-wait.ci failed at its 15s budget with TYPE: timeout while the daemon itself completed correctly.

Migration from Old Format

Old format (deprecated):

option:file:test.conf
option:asn:65000
1:raw:FFFF...
1:json:{...}

New format:

option=file:path=test.conf
option=asn:value=65000
expect=bgp:conn=1:seq=1:hex=FFFF...
expect=json:conn=1:seq=1:json={...}

Key changes:

• = instead of : after action
• Explicit conn= and seq= for message ordering
• hex= prefix for wire bytes
• json= prefix for JSON data

Editor Test Format (.et)

The .et format extends .ci for interactive editor testing. Tests are located in test/editor/.

Overview

Editor tests simulate user input sequences against the headless configuration editor and verify state changes.

Input Actions

Named keys input=key:name=<key> accepts: tab, enter, esc, escape, up, down, left, right, backspace, delete, home, end, pgup, pgdn, pgdown, space. shift+tab is handled separately as a modifier.

Expectations

Editor tests use a tree store by default; option=storage:value=tree states that choice explicitly. option=storage:value=blob runs the test on a backup artifact the way ze config edit --backup opens one: the runner creates backup.zefs, seeds it, closes it, reopens it writable through storage.OpenBlob, and builds each session with cli.NewOfflineSessionEditor, so a commit publishes into the artifact. Blob mode needs option=session:user=.... Other values are refused. The runner seeds config fixtures once and all sessions and restart= steps share the same lifetime store handle. expect=key reads that handle, while expect=file reads the original loose fixture or a separate exported artifact.

hint reads the second message line with every style stripped. That row carries the completion hint, and the summary of the candidate the operator selected. explanation reads the long explanation Tab reveals, without the box that frames it on screen. Both accept empty and contains=. Both refuse an expectation that names neither key.

A test that runs option=mode:value=command or option=mode:value=operational completes against the real command tree, built from the registered -cmd YANG modules. The tree carries no value hints and no plugin commands. A hint provider reads the live RIB, and a plugin command comes from the live dispatcher. A headless test has neither, so a completion over a VALUE stays empty there.

Wait Actions

Example

# Test: Edit navigation
tmpfs=test.conf:terminator=EOF_CONF
bgp {
  router-id 1.2.3.4;
  peer upstream1 {
    remote {
      ip 1.1.1.1;
      as 65001;
    }
  }
}
EOF_CONF

option=file:path=test.conf

expect=context:root
input=type:text=edit bgp
input=enter
expect=context:path=bgp
expect=error:none

input=type:text=set
input=space
expect=completion:contains=router-id,local-as,peer

Test Categories

Full format specification: plan/spec-editor-testing-framework.md