.ci Test File Format

For the execution architecture (how tests are scheduled and run concurrently) and the web .wb format, see runner-architecture.md.

Syntax Overview

All lines use key=value format with : separators:

action=type:key=value:key=value:...

Key Concepts

An unparseable test file fails; it never hides or vanishes

A file that does not parse is recorded as a permanent failure and discovery continues. It is never dropped, and it never aborts the rest of the directory. All three discoverers behave identically.

The legacy .conf layout has no instances in the tree today, and it is held to the same contract anyway: a missing, empty, or bad-regex .expect file records that one fixture as a failure rather than abandoning the directory. An unreachable abort becomes reachable the moment someone adds the directory, and the shape is the bug.

Both alternatives are silent-coverage-loss bugs this project has already paid for, which is why neither is permitted:

A guard that neither denies nor speaks does not exist (ai/rules/evidence.md). The runner short-circuits a parse-failed test before executing anything, so the reported error is the parse error rather than a confusing downstream symptom.

Unparseable outranks skipped. A file that both fails to parse and carries option=needs-linux / option=skip-os is reported FAIL, not SKIP. Its skip marker was parsed from the same broken file, so it is not trustworthy evidence that the file need not run: a contradicting directive may sit past the break, or the marker itself may be what was mis-parsed. Honoring it would mean trusting a broken file's own claim that it can be ignored.

The consequences are asymmetric, which is what settles the ordering. 158 .ci files carry one of those markers (12 in test/ui), and on a non-Linux host they never run. A wrongly-SKIPPED malformed file is invisible indefinitely, which is exactly how test/ui rotted. A wrongly-FAILED one is loud and costs a single commit. The check therefore lives in parallel.go's per-test goroutine, ahead of the skip short-circuit: that is the real entry point, and both Runner.runTest and parsingRunner.runTest are reached through it.

Suite label, test id, and failure identity

The verify debugging protocol identifies a functional failure with:

In ZE_VERIFY_MODE=1, failed suites emit native failure-group metadata before the full failure blocks. The compact verify index uses that metadata for group routing and keeps the full TEST FAILURE blocks in the stage log.

Per-step trace output

All three runner families (.ci, .wb, .et) record per-step outcomes during execution and emit dual-format trace output:

• Human: colored checkmark/cross glyph per step with kind, assert, and detail.
• Machine: VERIFY STEP: {"file":"...","step":N,"kind":"...","status":"pass|fail",...} -- one JSON line per step, matching the VERIFY FAILURE GROUP: prefix convention.

Trace is emitted automatically for failed tests. Under -v, passing tests also show their trace. The .ci runner includes the trace in its TEST FAILURE report block when StepTrace is non-empty.

conn and seq

Most directives use conn=N and seq=N to identify message ordering:

• conn (connection): 1-based TCP connection index. Each ze-peer instance manages one TCP connection. Multi-peer tests use conn=1 for the first peer, conn=2 for the second, etc. The maximum is set by option=tcp_connections:value=N.
• seq (sequence): 1-based message sequence within a connection. seq=1 is the first BGP message after OPEN/KEEPALIVE, seq=2 is the second, etc.

A test with two peers and one UPDATE each uses conn=1:seq=1 and conn=2:seq=1, not conn=1:seq=1 and conn=1:seq=2.

Port Substitution

Each test owns a pair of ports, exposed as variables in commands, tmpfs= content, option=env values, and URLs:

Never hardcode port numbers. Use $PORT in cmd= exec values and $PORT2 in http= URLs.

The pair is LEASED when the test starts, not when the suite discovers it (runner.LeaseTestPorts, internal/test/runner/ports.go). Discovery numbers the Nth test of every suite from the same base, so the preference alone collides whenever two ze-test processes run at once; the lease takes a machine-wide advisory lock in $TMPDIR/ze-test-port-locks and probes the pair, and a test whose preferred pair is locked or occupied gets one from 25000-32759 instead. A hardcoded number in a .ci file takes part in neither step, which is why the rule above is a rule.

Stdin Blocks

Stdin blocks embed content that will be piped to a process's stdin.

Syntax

Multi-line (with terminator):

stdin=<name>:terminator=<TERM>
<content>
<TERM>

Single-line hex:

stdin=<name>:hex=<hex-value>

Single-line text:

stdin=<name>:text=<text-value>

Parameters

Examples

Multi-line (config):

stdin=ze-bgp:terminator=EOF_CONF
peer test-peer {
    remote {
        ip 127.0.0.1;
        as 65533;
    }
    local-as 65533;
}
EOF_CONF

cmd=foreground:seq=1:exec=ze bgp server -:stdin=ze

Single-line hex (decode test):

stdin=payload:hex=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF003C020000001C...
cmd=foreground:seq=1:exec=ze-test decode --family ipv4/unicast -:stdin=payload
expect=json:json={ "type": "update", ... }

Single-line text:

stdin=cmd:text=update text nhop set 10.0.0.1 nlri ipv4/unicast add 10.0.0.0/24

What a ze-peer block may carry

A block handed to ze-peer (named on a cmd=...:exec=ze-peer ...:stdin=<name> line) is read first by ze-peer and then by the test runner. The block named peer is validated by the same rules even with no such line, because a .ci with no cmd= at all feeds its expect= lines to ze-peer by another route. A line neither of them acts on fails the file at parse time, naming the block, the line number and the directive.

Any OTHER directive the runner parses is accepted where it stands and applies to the whole test. option=file, expect=exit, await= and http= all work inside a peer block, and none of them is peer-scoped. Only option=env and option=timeout are singled out. Those two read as peer scope and are not.

A value the named option does not have is refused as well, not just an unknown option name. option=update:value=inspect-update-message and option=asn:value=abc each used to parse into nothing, which is the same silent drop one level down.

The accepted set is derived from the two parsers rather than written out a third time (peer.ClaimLine reports what ze-peer did with the line, and the runner's own line parser answers for the rest), so a directive added to either one cannot start being dropped here. Before the guard existed the block forwarded only expect= and action= lines and discarded everything else in silence: eleven reject=bgp directives across nine RFC-behaviour tests, and the reject=stderr of a tenth, asserted nothing while reading as the negative half of the proof.

Tmpfs (Virtual File System)

Tmpfs allows embedding multiple files within a single .ci file. Files are extracted to a temp directory at runtime.

Syntax

tmpfs=<path>[:mode=<octal>][:encoding=<type>]:terminator=<TERM>
<content>
<TERM>

Parameters

Mode Defaults

Files default to 0644. Use an explicit mode= only when the test needs a different permission.

Terminator Rules

• Must be non-empty
• Must be unique within file (no two Tmpfs blocks can share terminator)
• Alphanumeric and underscore only: [A-Za-z0-9_]+
• Matched exactly (no whitespace trimming)
• Recommended: EOF_<PURPOSE> (for example, EOF_CONF)

Example

tmpfs=peer.conf:terminator=EOF_CONF
peer test-peer {
    remote {
        ip 127.0.0.1;
        as 65533;
    }
    local-as 65533;
}
EOF_CONF


option=file:path=peer.conf
option=asn:value=65533
expect=bgp:conn=1:seq=1:hex=FFFF...

Security Constraints

1. No absolute paths - must be relative
2. No parent traversal - no .. components
3. No hidden files - no . prefix in path components
4. Path length limit - max 256 characters
5. Path depth limit - max 10 levels

Limits

Configurable via environment variables:

Options

option=<type>:key=value[:key=value...]

Choosing between needs-linux, caps=, and skip-os

caps= takes a comma-separated list, so a test that programs netlink and loads eBPF declares caps=net-admin,bpf and is gated on both. An unknown token is a parse error on every host, macOS included, so a typo cannot silently disable the gate.

caps=net-admin exists because Linux alone is not the requirement. On an unprivileged Linux host, a CI runner or a rootless container, a test that applies interface config does not fail cleanly: the interface plugin fails its configure handshake with operation not permitted and the DAEMON exits 1, then the TEST hangs because its check peer waits for a session the exited daemon will never open. The gate reads CapEff from /proc/self/status, not uid 0: a setcap'd binary holds the capability without being root, and a restricted container can be root without it.

A caps= test does not run in the merge gate. ./le verify worktree runs unprivileged, so the marker turns an opaque hang into an honest skip there, and the coverage relocates to .github/workflows/qemu-nightly.yml. TestCapabilityGatedTestsHaveANativeVMHome fails when that link is broken: marking a test with a capability nobody's CI has would be a coverage deletion wearing a skip's clothing.

OPEN Behaviors

BGP Identifier Control (router-id)

option=open:value=router-id:id=<a.b.c.d>

Ze-peer's default OPEN carries ze's own BGP Identifier with the last octet incremented, which is always a distinct, valid identifier. This option replaces it outright, so a test can present an identifier the default can never produce: 0.0.0.0, or ze's own router-id (RFC 6286 Section 2.2 rejects both, the second only from an internal peer). A malformed or IPv6 value is ignored and the default mirror stands.

Capability Control (drop-capability / add-capability)

Ze-peer mirrors the peer's OPEN message back (with a modified router-id). The drop-capability and add-capability options modify this mirrored OPEN at wire level, allowing tests to control exactly which capabilities ze-peer advertises.

Drop a capability:

option=open:value=drop-capability:code=<N>

Removes the capability with the given code from ze-peer's OPEN response. The peer will not see this capability in the mirrored OPEN.

Add a capability:

option=open:value=add-capability:code=<N>:hex=<value-bytes>

Adds a capability with the given code and hex-encoded value bytes to ze-peer's OPEN response.

Use case: testing capability mode enforcement:

When Ze is configured with require mode for a capability, it sends a NOTIFICATION if the peer lacks that capability. To test this, use drop-capability to make ze-peer omit the capability from its response:

# Test: Ze requires ASN4, ze-peer drops it → Ze should send NOTIFICATION
option=open:value=drop-capability:code=65

When Ze is configured with refuse mode, it sends a NOTIFICATION if the peer has a capability. To test this, the default mirror behavior already includes the capability, but add-capability can add capabilities not in the original OPEN:

# Test: Add a custom capability for refuse testing
option=open:value=add-capability:code=73:hex=067A652D626770

Multiple overrides can be combined:

option=open:value=drop-capability:code=65
option=open:value=drop-capability:code=2
option=open:value=add-capability:code=73:hex=067A652D626770

UPDATE Behaviors

Routes ze-peer sends after the OPEN handshake, before it starts matching expectations.

Generating one oversize UPDATE (max-msg):

option=update:value=send-bulk:prefix=10.0.0.0/24:count=16373:next-hop=10.0.0.1:origin-as=65001:max-msg=65535

max-msg caps one generated message, header included. It defaults to the RFC 4271 limit of 4096, and accepts 23 to 65535. Raise it to 65535 when the session negotiates the Extended Message capability (RFC 8654), which ze advertises when the peer carries capability { extended-message enable }. Prefixes that do not fit one message spill into the next, so the example above is exactly one 65535 byte message: a 65516 byte body, the largest BGP permits.

Use it whenever a test needs an UPDATE too large to write as a hex literal. A max-size message is 131070 hex characters, which is past the 64 KiB line limit of both .ci scanners and past what a reviewer can check. It is also the only way to hand the daemon a single oversize body: splitting the same prefixes across several standard messages is a different input, because the daemon decides per message.

A malformed key fails the test load rather than sending nothing. That is deliberate: a spec that silently degraded to count=0 would let a test asserting a route was NOT forwarded pass because no route was ever offered.

Commands

cmd=<type>:key=value[:key=value...]

API Commands

cmd=api:conn=<N>:seq=<N>:text=<command>

Example

cmd=api:conn=1:seq=1:text=update text origin set igp nhop set 10.0.1.1 nlri ipv4/unicast add 10.0.0.0/24

Process Commands (Foreground/Background)

For orchestrating multiple processes:

cmd=background:seq=<N>:exec=<command>[:stdin=<name>][:name=<handle>]
cmd=foreground:seq=<N>:exec=<command>[:stdin=<name>][:timeout=<dur>][:exit=<N>]
cmd=stop:seq=<N>:name=<handle>[:signal=kill|term]

Markers may appear in any order; each value runs to the next known marker.

Background: Starts and keeps running until test ends. Foreground: Starts and waits for completion. Stop: Terminates a named background process mid-test (see below).

cmd=stop -- terminate a background process mid-test

A background process started with name=<handle> can be stopped at a chosen step by cmd=stop:seq=<N>:name=<handle>. The runner looks the process up, signals it, and waits for it to exit before the next step runs, so a later step can deterministically observe what happens after the process dies (e.g. show vpn ipsec sa emptying once an IKE responder is killed and DPD fires).

• signal=kill (default) sends SIGKILL: the process gets no chance to flush or send a protocol teardown. This is the choice for liveness/dead-peer tests where the peer must go silent (a clean shutdown would take a different code path).
• signal=term sends SIGTERM, escalating to SIGKILL if the process does not exit within the teardown grace period -- a graceful stop.

Fail-closed: a cmd=stop naming a process that was never started (no matching name=) fails the test with a clear error; it never silently no-ops, and it can only ever signal a process the runner itself started (never an arbitrary PID). Teardown still kills every remaining background process, and tolerates one the stop step already reaped.

Provenance: added by spec-fixit-runner-kill-background to unblock end-to-end peer-death observation (the deleted test/ipsec/ipsec-dpd-timeout.ci needed it).

exit= vs expect=exit:code= (per-command vs file-level)

expect=exit:code= is file-level: Record.ExpectExitCode is a single value (a later expect=exit:code= silently overwrites an earlier one) and the runner compares it against lastQuickZeErr -- the exit status of the last quick-exit ze command in the file. A file that runs several ze config validate commands therefore asserts only the final one; every earlier command can exit with any code and the test still passes.

Use exit= on the cmd= line to assert a specific command's own exit code. It is checked the moment that command finishes, and names the offending seq on failure:

cmd seq=2 (ze config validate -): expected exit code 1, got 0

Prefer exit= whenever a file runs more than one quick-exit ze command. A "quick-exit ze command" is any foreground ze whose verb is not a daemon verb (hub, start, cli, monitor) and which has no config-file argument or --web flag.

Note that stdout/stderr expectations are file-level in the same way: they match the accumulated output of every command in the file, so expect=stdout:contains= can be satisfied by a different command than the one intended, and reject=stdout:pattern= trips on any command's output. When a reject must apply to one command, keep that command in its own file (see test/vrrp/vrrp-doctor-quiet.ci).

Known gap: 108 quick-exit ze commands across 50 .ci files predate exit= and are still unasserted (their expect=exit:code= never reaches them). Arming them may surface real defects; tracked in plan/known-failures/.

Daemon readiness (ze only): a ze daemon launched either foreground or background is told (via ZE_READY_FILE) to write daemon.ready once startup completes, and the runner publishes its PID to daemon.pid in the tmpfs directory. Tests poll both files directly or through a compiled driver in internal/test/fixture before signaling the daemon or asserting on it. The readiness handshake is armed only for Ze daemons.

Example (Decode Test)

stdin=payload:hex=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF003C...
cmd=foreground:seq=1:exec=ze-test decode --family ipv4/unicast -:stdin=payload
expect=json:json={ "type": "update", ... }

Example (Multi-Process)

stdin=peer:terminator=EOF_PEER
option=asn:value=65000
expect=bgp:conn=1:seq=1:hex=FFFF...
EOF_PEER

stdin=ze-bgp:terminator=EOF_CONF
peer test-peer { remote { ip 127.0.0.1; } ... }
EOF_CONF

cmd=background:seq=1:exec=ze-peer --port $PORT:stdin=peer
cmd=foreground:seq=2:exec=ze bgp server -:stdin=ze-bgp:timeout=10s

Example (Multi-Peer)

Tests needing two or more BGP peers use the same $PORT on different loopback addresses. The ze_bgp_tcp_port override sets all peer ports uniformly, which is correct because every peer listens on $PORT.

# Source peer on 127.0.0.1 (default)
stdin=source:terminator=EOF_SOURCE
option=tcp_connections:value=1
action=send:conn=1:seq=1:hex=FFFF...
EOF_SOURCE

# Dest peer on 127.0.0.2 -- sink mode absorbs any UPDATE
stdin=dest:terminator=EOF_DEST
option=tcp_connections:value=1
EOF_DEST

cmd=background:seq=1:exec=ze-peer --port $PORT:stdin=source
cmd=background:seq=2:exec=ze-peer --bind 127.0.0.2 --mode sink --port $PORT:stdin=dest
cmd=foreground:seq=3:exec=ze -:stdin=ze-bgp:timeout=20s

Each ze-peer gets independent output capture and WaitFor synchronization. The runner waits for each peer's "listening on" message before starting the next command.

On Linux, 127.0.0.2 works automatically (127.0.0.0/8 routes to lo). On macOS and FreeBSD, the test runner adds loopback aliases via the SIOCAIFADDR ioctl.

IPv6 works differently, because a host carries exactly one IPv6 loopback address. A fixture that needs a second one uses fd00::2, which is unique-local (RFC 4193) and never globally routable. ./le setup adds it, and ./le setup check reports whether it is there. The runner never adds it: the ioctl returns EPERM to an unprivileged process, and ./le verify current mode full runs as an ordinary user. A test that binds an address this host does not carry fails at once with loopback_address_missing and the command to run, rather than timing out on a bind that could not succeed.

The check reads both places a fixture names an address it binds. One is ze-peer --bind <ip> on a cmd= line. The other is connection { local { ip <addr> } } in the config the fixture embeds: Ze sends from that address and listens on it when accept is true, so the host must carry it too. A local address outside 127.0.0.0/8 and fc00::/7 is left alone. A config-validation fixture names a routable one (local { ip 192.0.2.1 }), the daemon exits before it binds anything, and ./le setup adds no such address.

Expectations

expect=<type>:key=value[:key=value...]

BGP Wire Expectations

expect=bgp:conn=<N>:seq=<N>:hex=<hex-bytes>
expect=bgp:conn=<N>:seq=<N>:prefix=<hex-bytes>
expect=bgp:conn=<N>:seq=<N>:contains=<hex-bytes>
expect=bgp:conn=<N>:seq=<N>:ordered=<hex-bytes>

Validates the BGP wire message received: hex= matches the exact message, prefix= the message start, contains= a substring anywhere in one message. Within one seq group, hex/prefix/contains checks match in any order and each consumes exactly one received message.

ordered= checks in a seq group form a strict FIFO subqueue for asserting in-order delivery across message boundaries: the front needle must appear in the received message, and one message may consume several consecutive needles, each matched at an advancing offset (so order inside a packed message is enforced too). Use ordered= instead of per-message contains= when the sender may legally pack several NLRIs into one UPDATE (the forward rail's bucket merge): per-message framing is not a property ze owes, but delivery order is. A message whose content matches only a non-front needle consumes nothing and is reported as a mismatch.

These forms are peer-block directives (inside a stdin=<name>: block); top-level expect=bgp lines support hex= only.

JSON Expectations

expect=json:conn=<N>:seq=<N>:json=<json-object>
expect=json:json=<json-object>

Validates the decoded message matches expected JSON.

Validation rules:

• Parsed and compared field-by-field (key order independent)
• Volatile fields removed before comparison: exabgp, ze-bgp, time, host, pid, ppid, counter
• Neighbor normalization: peer ↔ neighbor treated as equivalent, direction field ignored
• All non-volatile fields must match exactly

Exit Code Expectations

expect=exit:code=<N>

Validates the foreground process exit code. A test whose ONLY assertion is expect=exit:code=0 is accept-only (weak) and is gated by a lint; see Assertion Strength.

Stdout Expectations

expect=stdout:contains=<text>
expect=stdout:!contains=<text>
expect=stdout:pattern=<regex>

Three modes:

• contains= -- substring match against stdout (multiple allowed, all must match)
• !contains= -- negative substring match (stdout must NOT contain text)
• pattern= -- regex match against stdout (uses Go regexp syntax)

Stderr Expectations

expect=stderr:pattern=<regex>
expect=stderr:contains=<text>

Two modes:

• pattern=: regex match against stderr (uses Go regexp syntax)
• contains=: substring match against stderr

Await (deterministic stderr fence)

await=stderr:contains=<text>[:timeout=<dur>]

Blocks the runner until the daemon's relayed stderr contains <text>, then tears the daemon down. This is a deterministic replacement for a blind time.sleep that only held the daemon open long enough for a line to appear. timeout= is an optional Go duration (default 10s); on timeout the test fails with a precise message. <text> follows the same rule as expect=stderr:contains= (the needle must not contain a literal :).

Pair it with a matching expect=stderr:contains= so the line is both the fence and the assertion. Use it for the reject-fence bucket: an external plugin whose refusal aborts the daemon's plugin-startup coordinator (StartupCoordinator.PluginFailed) leaves no in-daemon observer able to poll it, so the relayed stderr line is the only non-plugin signal.

Choose a needle that is SPECIFIC to the plugin under test, not just a shared phrase. Example (test/plugin/as112-external-refuses.ci): the bare "refusing to start as an external plugin process" is emitted verbatim by three plugins, so the needle includes the as112-only tail -- the address-ownership registry:

cmd=foreground:seq=1:exec=ze -:stdin=ze-bgp:timeout=15s
await=stderr:contains=refusing to start as an external plugin process -- the address-ownership registry
expect=stderr:contains=refusing to start as an external plugin process -- the address-ownership registry

Syslog Expectations

expect=syslog:pattern=<regex>

Validates that captured syslog output matches the regex pattern. When any expect=syslog: line is present, the test runner automatically starts a UDP syslog server and injects ze.log.backend=syslog and ze.log.destination=127.0.0.1:<port> into the test environment.

File Expectations

expect=file:path=<rel>:exists=true
expect=file:path=<rel>:absent=true
expect=file:path=<rel>:contains=<text>
expect=file:path=<rel>:not-contains=<text>
expect=file:glob=<rel-pattern>:count=<N>
expect=file:glob=<rel-pattern>:contains=<text>
expect=file:glob=<rel-pattern>:not-contains=<text>

Validates files after the test process or peer sequence has completed. Paths and glob patterns are relative to the tmpfs directory when the test uses tmpfs=, otherwise relative to the .ci file directory. For glob contains, at least one matched file must contain the text. For glob not-contains, no matched file may contain it.

Use file expectations for post-run artifacts such as generated configs, pointer files, and logs. Do not write shell just to inspect files.

Negative Expectations (reject)

reject=stderr:pattern=<regex>
reject=stdout:contains=<text>
reject=stdout:pattern=<regex>
reject=syslog:pattern=<regex>
reject=bgp:conn=<N>:pattern=<hex>

Inverse of expect= -- the test fails if the pattern matches. Used to verify that unwanted output (e.g., deprecated warnings, ERROR-level messages) does NOT appear.

reject=bgp -- bytes a peer must never receive

reject=bgp is the only reject type ze-peer reads, because ze-peer is what sees the wire. It goes inside the peer's stdin block, beside the expect=bgp lines. The hex needle is matched on wire bytes at a byte BOUNDARY, which expect=bgp:contains= is not: that one is a plain substring match over the hex text, so it can also match at an odd nibble offset.

Five properties make it an assertion rather than a hope:

• It is never consumed. Every frame the peer's message loop reads is checked against it. The option=linger loop keeps checking after completion. Pair the rejection with option=linger:value=true to hold it open for the whole test.
• A rejection found during linger RETRACTS the success token. Under option=linger the peer prints its success token BEFORE the loop, because teardown is a kill and a post-run print can be lost. The verdict reads that token, so a rejection arriving afterwards has to withdraw it: the peer prints ZE-PEER-REJECTED on its own output, and the verdict reads the retraction after the token and lets it win. Without that channel a linger rejection was detected, returned, and then discarded, which made every negative assertion held open by linger vacuous.
• Check mode only. Sink and echo peers read every accepted connection concurrently against one checker, so a conn= could not select the session the frame arrived on. The runner refuses the file, and ze-peer refuses to start.
• The block MUST also carry an expect=bgp:conn=<N> on the same connection, and the runner refuses the file otherwise. That rule is necessary and it is not sufficient. It bounds how long the rejection is checked. It cannot say whether the forbidden bytes would have arrived inside that window. The author owes the second half. Send the delivery LAST on that connection, so a leak of the governed route arrives ahead of it. Or hold the session open with option=linger. A connection stops being read once its expectations complete, and its rejection then passes for the one reason a rejection must not.
• An odd-length or non-hexadecimal needle is a parse error, because a needle that can never match would pass for that same reason. The runner reads the line with ze-peer's own parser (peer.ParseRejectRule), so a typo fails the FILE rather than the peer: a rejection rejected inside ze-peer would stop it binding, and the runner would report that as a bind timeout.

stdin=external:terminator=EOF_EXTERNAL
option=tcp_connections:value=1
option=linger:value=true
# The delivery that makes the rejection an assertion rather than an empty session.
expect=bgp:conn=1:seq=1:contains=18C00002
# 180A0100 is 10.1.0.0/24, which NO_ADVERTISE forbids on any peer.
reject=bgp:conn=1:pattern=180A0100
EOF_EXTERNAL

Assertion Strength: accept-only tests and readback

A test whose ONLY assertion is expect=exit:code=0 is accept-only (weak): it proves a config or command was ACCEPTED, never that it parsed to the CORRECT tree. A parser that accepts interval 300 but stores 0, or silently drops a source 0.0.0.0/0 block, still passes such a test green. This is the functional-suite analog of the "count-only assertion" mistake class (ai/rules/testing.md).

A lint enforces that the class cannot GROW: TestCIAcceptOnlyLint walks every test/**/*.ci, classifies each with the single accept-only predicate, and FAILS on a NEW accept-only test that is neither strengthened nor annotated. Existing accept-only tests are grandfathered in test/.accept-only-baseline (a sorted allow-list that only shrinks; strengthening or annotating a test removes its line). Correctly EXCLUDED (never weak): a test whose real check lives in a tmpfs set -e script (e.g. test/managed/auth-reject.ci), and any reject= test.

Strengthen with a readback

Add a second cmd= that dumps the parsed tree and assert a representative value with expect=stdout:contains= / pattern=. show config dump - reads a config from stdin and answers the stored tree, and | json renders it, so the assertion observes the parsed VALUE, not just that parsing did not error.

cmd=foreground:seq=1:exec=ze config validate -:stdin=config:exit=0
cmd=foreground:seq=2:exec=ze cli -c "show config dump - | json":stdin=config-dump
expect=exit:code=0
expect=stdout:pattern="interval": "300"

Two gotchas, both load-bearing:

• ze config dump requires a bgp { } block (it resolves the full BGP tree) where ze config validate does not. To keep proving that a subsystem-only config validates, keep the original validate step on the subsystem-only stdin block and run the dump readback against a second stdin block that prepends a minimal bgp { router-id ... }.
• A needle containing : followed by something shaped like key= must use pattern=. ~~Only json=/text=/hex=/pattern= preserve colons, and contains= truncates at the first colon.~~
  • Corrected 2026-07-30. That claim was true until ParseKVPairs became boundary-aware. contains= now keeps an ordinary colon, so contains=error: no such peer asserts the whole sentence.
  • What still splits is a colon that introduces a real key token: a letter, then letters, digits, - or _, then =. That split is deliberate, because it is how the engine-step form contains=aes-cbc:timeout=25 keeps working. So contains=note:level=high still splits at :level=, and it is the one shape that needs pattern=.
  • The old behavior was not harmless while it lasted. A sweep on 2026-07-29 found 203 assertions across 15 suites silently reduced to the text before their first colon. The re-armed assertions then exposed a security test (test/appliance/appliance-push-image-escape.ci). That test had never once executed the path-traversal guard it was named for.
• Keep a pattern= needle free of the substrings json=, text=, and hex=. ParseKVPairs extracts a complex-key value by strings.Index of the first such marker, so a needle that itself contains one of them is mis-split. None of the readback needles above contain these markers. This note is forward guidance for new needles.

Annotate when a unit test already covers the value

When a unit test already asserts the parsed value, a readback would duplicate it. Mark the test accept-only instead, with a comment naming the covering test:

# accept-only: md5 { password; ip } value-parsing is unit-covered by
# TestParsePeerMD5FieldsParsed (internal/component/bgp/reactor/config_test.go).

The marker is a comment line whose content is accept-only: followed by a non-empty reason. A file carrying it is allowlisted by the lint without a baseline entry. Keep the reason greppable and truthful (name the covering test).

Actions

action=<type>:key=value[:key=value...]

Notification

action=notification:conn=<N>:seq=<N>:text=<message>

Sends NOTIFICATION with shutdown message.

Send Raw

action=send:conn=<N>:seq=<N>:hex=<hex-bytes>

Sends raw bytes to peer.

Rewrite Config File

action=rewrite:conn=<N>:seq=<N>:source=<tmpfs-file>:dest=<config-file>

Copies a tmpfs file over the daemon's config file. Used with action=sighup to test config reload.

Send SIGHUP

action=sighup:conn=<N>:seq=<N>

Sends SIGHUP to the daemon process. Reads PID from daemon.pid in the tmpfs directory (written automatically by the test runner).

Send SIGTERM

action=sigterm:conn=<N>:seq=<N>

Sends SIGTERM to the daemon process. Reads PID from daemon.pid in the tmpfs directory (written automatically by the test runner). After sending SIGTERM, the connection is expected to close (daemon shuts down gracefully).

HTTP Checks

HTTP checks validate web endpoint responses after all cmd= processes have started. Executed in seq order with automatic retry on connection errors (server starting up).

Assertion Checks (get/post)

http=get:seq=N:url=URL:status=CODE[:contains=TEXT][:bodyfile=PATH][:header=NAME: VALUE]
http=post:seq=N:url=URL:status=CODE[:contains=TEXT][:bodyfile=PATH][:sendfile=PATH][:content-type=TYPE][:header=NAME: VALUE][:insecure-tls=true]

Request Headers (header=)

header= takes the header exactly as it appears on the wire, Name: Value. Repeat header= to set several headers on one check. It works on get, post, and wait.

http=post:seq=1:url=http://127.0.0.1:$PORT/mcp:status=200:sendfile=call.json:header=MCP-Protocol-Version: 2026-07-28:header=Mcp-Method: tools/call:header=Mcp-Name: ze

Retries up to 20 times at 200ms intervals on transient connection errors (ECONNREFUSED, ECONNRESET, EOF). Non-connection errors (wrong status, missing content) fail immediately.

Readiness Polls (wait)

http=wait:seq=N:url=URL:status=CODE[:contains=TEXT][:header=NAME: VALUE][:timeout=DUR]

Unlike assertion checks, wait retries on all failures: connection errors, wrong status codes, and content mismatches. Polls at 500ms intervals until the condition is met or the timeout expires. Wait checks run before assertion checks, making them suitable for waiting until a server has populated data (e.g., routes injected by an async plugin).

Example

cmd=background:seq=1:exec=ze-peer --port $PORT:stdin=peer
cmd=background:seq=2:exec=ze -:stdin=ze-bgp

# Wait until routes are available before checking graph output
http=wait:seq=1:url=http://127.0.0.1:$PORT2/lg/graph?prefix=10.10.1.0/24&mode=aspath&format=text:status=200:contains=AS2914:timeout=15s

# Exact SVG match against reference file
http=get:seq=1:url=http://127.0.0.1:$PORT2/lg/graph?prefix=10.10.1.0/24&mode=aspath:status=200:bodyfile=expect/graph.svg

# Substring check
http=get:seq=2:url=http://127.0.0.1:$PORT2/lg/graph?prefix=10.10.1.0/24&mode=nexthop&format=text:status=200:contains=egress

Sleeps and their justification markers

Every time.sleep( in a live .ci carries a marker comment in the form // sleep(<kind>): <reason>, on one # comment line directly above the sleep, indented to match it exactly. The embedded .ci observer body is indentation sensitive. Two producers enforce the marker: checkSleepJustification in internal/le/doc/wiring (run by ./le doc wiring, scoped to changed .ci files, listing every unjustified file:line and exiting 1), and writeCISleep in internal/le/hookruntime/writeedit.go, which blocks a Write or Edit that introduces an unmarked sleep.

The kinds are a closed set, and each one owes a different reason:

A free-text // settle comment is insufficient: it names no mechanism a reader can check. "The tracker pushes live carrier once a second" is a reason a later reader can overturn; "needs a moment" is the shape that makes a deliberate timer and a guessed duration the same line of code.

A separate ratchet caps how MANY sleeps exist: the total time.sleep( count across test/**/*.ci may not exceed the committed baseline in test/.ci-sleep-baseline, and ./le doc wiring fails when it does. The markers cap how many are unexplained.

The compiled observer API

Compiled .ci observers live under internal/test/fixture. They use pkg/plugin/sdk for the five-stage plugin protocol (Plugin.Run owns it) and the local fixture package for registration, dispatch, polling and failure reporting.

fixture.Poll around fixture.Dispatch is the payload-predicate wait: it blocks until the observed payload matches, within a bounded attempt count. It is what replaces a time.Sleep followed by a one-shot assertion.

fixture.Observe can request a clean daemon shutdown even after an assertion failed, so the daemon's exit code does not prove the observer's assertion. A failing observer returns an error, which fixture.Run hands to fixture.ReportFailure.

Engine Steps

Engine steps drive a live daemon through CLI dispatch, first-class in .ci instead of an embedded Python observer. The runner serializes the parsed steps to engine-steps.json in the test tmpfs; the .ci declares the executor as an external plugin (run "ze-test engine-steps ./engine-steps.json"), which runs the steps from OnAllPluginsReady and reports failures via the ZE-OBSERVER-FAIL sentinel the runner gates on.

command=<cli command text>
stream=<monitor command text>
expect=output:<predicate>[:timeout=<dur>]
expect=event:namespace=<ns>:name=<name>[:timeout=<dur>]
expect=stream:<predicate>[:timeout=<dur>]
expect=command-error:contains=<text>

command=/stream= keep their full raw text (colons included). expect=output re-dispatches the most recent command= until its predicate holds or the timeout expires; expect=stream matches delivered stream= events; expect=event matches a delivered event by exclusive subscription.

expect=command-error

expect=command-error:contains=<text> asserts that the PRECEDING command= FAILED, and that its message contains the text.

Use it for a command that must refuse. The plugin SDK turns a StatusError response into a Go error, so without this directive the command step aborts the run before any expect= is reached, and no .ci can assert an operational error at all. That left one class untestable end to end, and it is the class where a wrong answer costs most: a command that must refuse is exactly the one whose failure mode is answering confidently instead. test/ipsec/ipsec-dataplane-show.ci uses it to prove that a dataplane which cannot be read SAYS so rather than rendering an empty table.

It takes contains= only, and no timeout. An error is the result of one dispatch, so re-dispatching until one appears would wait for a state change no expect= can cause.

A command failure that NO expect=command-error consumes still fails the run, whether the next step is of another kind or the command is the last step in the file. Every .ci written before this directive existed relies on that.

It must not be vacuous. A command= that SUCCEEDS does not satisfy expect=command-error: the step fails with "the preceding command succeeded". Without that rule the directive would pass for the very regression it guards, a refusal quietly becoming a successful empty answer (TestRunEngineStepsCommandErrorRequiresAFailure).

expect=output / expect=stream predicates

The optional trailing :timeout=<dur> is split off the END first, so a predicate operand may itself contain : (a compact-JSON fragment, an IPv6 address). The remainder is one predicate:

json= walks the raw data field (not status data): each .-segment indexes a JSON object by key or a JSON array by integer index (0..len-1; out-of-range or missing is "not yet", named at timeout). The leaf is compared as a string (numbers/bools stringified via JSON). absent=/json= are expect=output only: they re-dispatch a query, whereas expect=stream is an append-only event stream with no "absent" and no single-event JSON path.

absent= must be non-vacuous. An absent= on output that was never populated passes instantly (false green). Precede it with a step that makes the substring present (a contains=/json= after an inject), then the transition (e.g. a withdraw) the absent= proves. See test/plugin/engine-steps-predicates.ci.

Example

command=request bgp rib inject 10.0.0.1 ipv4/unicast 172.16.0.0/16 origin igp nexthop 10.0.0.2
command=show rib
expect=output:matches=172\.16\.[0-9.]+/16:timeout=10
expect=output:json=0.prefix=172.16.0.0/16:timeout=10
command=request bgp rib withdraw 10.0.0.1 ipv4/unicast 172.16.0.0/16
command=show rib
expect=output:absent=172.16.0.0/16:timeout=10

Complete Example

# Embed config using Tmpfs
tmpfs=test.conf:terminator=EOF_CONF
peer test-peer {
    remote {
        ip 127.0.0.1;
        as 65000;
    }
    router-id 10.0.0.2;
    local-address 127.0.0.1;
    local-as 65533;
    hold-time 180;

    family {
        ipv4/unicast;
    }
    announce {
        ipv4 {
            unicast 10.0.0.0/24 next-hop 10.0.1.254;
        }
    }
}
EOF_CONF

# Test configuration
option=file:path=test.conf
option=asn:value=65000

# Expected API command and wire output
cmd=api:conn=1:seq=1:text=update text origin set igp nhop set 10.0.1.254 nlri ipv4/unicast add 10.0.0.0/24
expect=bgp:conn=1:seq=1:hex=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF002F02000000144001010040020602010000FFFD4003040A0001FE180A0000

# EOR
cmd=api:conn=1:seq=1:text=announce eor ipv4/unicast
expect=bgp:conn=1:seq=1:hex=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF00170200000000

Consumers

Different components consume different line types:

Lines not recognized by a consumer are ignored.

A check-mode peer block MUST declare a ze-peer-consumed expectation

The consumer split above is load-bearing, not trivia. Only the four ze-peer rows reach ze-peer. Everything else -- including expect=json -- is validated by the test runner from its own copy of the messages.

A check-mode ze-peer with no consumed directive has nothing to check, so it prints no test data available to test against and exits 1 before binding a listening socket. ze then dials a dead port, gets connection refused, and backs off 5->10->20->40s. That looks exactly like a BGP establishment stall and cost a multi-day investigation before the cause was found in the harness.

So a peer block whose only expectation is expect=json runs no BGP at all. The parser now rejects that at discovery time, naming the file and the remedy:

stdin=peer block: check-mode ze-peer (cmd seq=1) declares no ze-peer-consumed
expectation, so it exits with "no test data available to test against" before
binding a listening socket and the test can only pass vacuously.

expect=json still works, but only in addition to a consumed directive: it cannot make the peer listen.

A ze-peer always governs its test's result

A test with a check-mode ze-peer is never self-validated: the peer must print successful and its expect=json expectations must match, whatever else the file asserts. An expect=exit:code=0 is additive and does NOT disable the BGP checks.

Until 2026-07-16 it did, which is how a test could pass on ze's exit code while its peer never listened and its JSON assertion never ran. sink/echo/inject peers do not govern (they never report completion), so a test whose only peers are scaffolding is still governed by its own exit/output/file assertions.

A scaffolding ze-peer is signaled at teardown

A sink, echo or inject ze-peer never ends itself: its accept loop runs until its context is cancelled, and ze-test peer maps SIGTERM to that cancel. The runner sends that SIGTERM at teardown, after the last step and before the barrier that collects peer output. The peer exits with status 0 and its capture is complete, so a .ci author needs no teardown directive and must not add one.

A check-mode peer is never signaled. It exits by itself when its expectations are met, and the runner reads its verdict from the capture a signal would truncate.

Before 2026-08-10 no code signaled a scaffolding peer, so the drain barrier waited out its full 10s grace on every --mode sink or --mode echo test. That cost was not only latency: test/plugin/event-predicate-wait.ci failed at its 15s budget with TYPE: timeout while the daemon itself completed correctly.

Migration from Old Format

Old format (deprecated):

option:file:test.conf
option:asn:65000
1:raw:FFFF...
1:json:{...}

New format:

option=file:path=test.conf
option=asn:value=65000
expect=bgp:conn=1:seq=1:hex=FFFF...
expect=json:conn=1:seq=1:json={...}

Key changes:

• = instead of : after action
• Explicit conn= and seq= for message ordering
• hex= prefix for wire bytes
• json= prefix for JSON data

Editor Test Format (.et)

The .et format extends .ci for interactive editor testing. Tests are located in test/editor/.

Overview

Editor tests simulate user input sequences against the headless configuration editor and verify state changes.

Input Actions

Named keys input=key:name=<key> accepts: tab, enter, esc, escape, up, down, left, right, backspace, delete, home, end, pgup, pgdn, pgdown, space. shift+tab is handled separately as a modifier.

Expectations

Wait Actions

Example

# Test: Edit navigation
tmpfs=test.conf:terminator=EOF_CONF
bgp {
  router-id 1.2.3.4;
  peer upstream1 {
    remote {
      ip 1.1.1.1;
      as 65001;
    }
  }
}
EOF_CONF

option=file:path=test.conf

expect=context:root
input=type:text=edit bgp
input=enter
expect=context:path=bgp
expect=error:none

input=type:text=set
input=space
expect=completion:contains=router-id,local-as,peer

Test Categories

Full format specification: plan/spec-editor-testing-framework.md