Architecture

Interoperability Testing

Ze validates protocol correctness against production BGP daemons in two complementary ways: live session interop tests (Docker containers running real daemons) and byte-level wire format validation against ExaBGP (Ze's predecessor, a BGP implementation in Python).

For BGP terminology used in this document, see BGP protocol.

ai/rules/interop-and-goal-validation.md states when an interop test is owed. This page is the infrastructure it is owed against.

The suites, one per protocol area

Protocol area Peer implementation Scenario directory Native action
BGP (session, capability, NLRI, community, policy) Docker: FRR, BIRD, GoBGP, StayRTR test/interop/scenarios/ ./le test integration interop
IPsec (IKEv2, EAP, MOBIKE) Docker: strongSwan test/interop-ipsec/ ./le test integration interop-ipsec
L2TP Docker test/interop-l2tp/ ./le test deployment l2tp, and ./le test deployment l2tp-ppp for the full PPP and NCP path
PPPoE (Ze as client) Docker: accel-ppp test/interop-pppoe/ ./le test deployment docker-pppoe-accel
RADIUS (admin login: PAP, CHAP, EAP, Filter-Id) Docker: FreeRADIUS test/interop-radius/scenarios/ ./le test integration interop-radius
RADIUS (subscriber: accounting, CoA and Disconnect) Docker: FreeRADIUS, an xl2tpd + pppd LAC test/interop-radius-subscriber/scenarios/ ./le test integration interop-radius-subscriber
RSVP-TE (Ze as transit) Docker: freeRouter, ingress and egress test/interop-rsvpte/scenarios/ ./le test integration interop-rsvpte
Multi-node topologies (several Ze nodes, any suite's images, named links) Docker: any registered image test/interop-topology/scenarios/ ./le test integration interop-topology

The L2TP lab's lcp-restart-timer-pppd scenario first proves a working PPP data path, then loses pppd's LCP replies after a live restart. It measures Ze's configured 500 ms retransmission, exact three-request budget and subsequent CDN from the relay's monotonic wire observations. The lab page describes the peer source and the final kernel cleanup assertion.

Every suite discovers its scenarios the same way. Discover (internal/le/interoplab/discover.go) reads the scenario directory, keeps the subdirectories, sorts the names lexically, and joins each one against the owning checker registry. A scenario with no checker, or a nil checker, is an ERROR rather than a skipped test, so a fixture and its registry cannot silently disagree. Nothing depends on the order: each scenario gets its own setup, check and teardown.

A reader outside a suite asks for its scenarios through the suite's catalog. Each lab registers one in its register_catalog.go (interoplab.RegisterCatalog, internal/le/interoplab/catalog.go), built on the same Discover call and checker map its runner uses, so a name cited elsewhere is a name the runner runs. ./le feature check resolves a feature's Interop entry <suite>/<scenario> this way, and names the registered suites when it refuses one it does not know. The catalog also declares RunScenario, which runs ONE scenario through the lab's own RunAt with the name as its selector, winning over the lab's selector variable. ./le feature run records an interop green run through it, against the scenario directory's git tree id, so a feature's Supported claim needs a current green run of each scenario it counts (docs/contributing/feature-maturity.md, "Recorded runs").

The catalog is also where a lab declares where it lives and how the repository runs it: Package (the checker directory), ScenarioRoot, SelectorVariable and Action (<command> <area>/<verb>). The lab's own runner reads the same scenarioDirectory and selectorVariable constants, so the declaration and the run cannot disagree. ./le rfc derives its interop carriers from these fields, one per registered suite, and ./le rfc record runs one scenario through the declared action and selector.

A scenario directory carries only declarative inputs its runner reads: ze.conf plus the peer configuration and argument files that topology needs. Assertions never live there. They are typed Go checkers under internal/le/interoplab/: BGP builds its registry from scenarioOperations in internal/le/interoplab/bgp/checkers.go, and IPsec declares scenarioCheckers in internal/le/interoplab/ipsec/checkers.go. A checker waits for readiness, asserts the protocol behaviour, verifies stability where the scenario needs it, and returns an error on failure.

A scenario directory is NAMED and carries no numeric prefix. The name is the scenario's identity: Discover matches it exactly, ./le test integration takes it as a scenario selector, and specs, journal rows and code comments cite it.

Role export policy on a roleless destination

bgp-role-export-roleless-destination-frr runs FRR as a Customer whose Ze-side session deliberately declares no role. A raw Customer source permits customer and its route must reach FRR; BIRD's source permits only provider and its route must remain absent there. Ze's received RIB must hold both source prefixes, and FRR must also hold Ze's independent control route. The positive source detects failure to observe the destination's OPEN; the negative source detects failure to insert egress on a roleless destination. Before reading FRR's negative result, the checker waits for Ze's completed export-set rejection naming FRR's address and advertised Customer role. The source peers have receive-only forwarding bindings, making FRR the sole send-granted target. The received RIB alone is not a forwarding fence. BIRD's bgp_fire_tx drains queued UPDATEs before its initial End-of-RIB, so the source worker's FIFO cannot produce that decision from an earlier EOR.

FRR 10.7.1's bgp_open_capability emits a configured local_role, and bgp_role_violation rejects an absent remote Role only with strict mode. Its bgp_otc_filter accepts an unmarked route as Customer and sets OTC locally. BIRD 3.4.0's bgp_update_attrs adds OTC only for a configured Provider, Peer or RS, so this unconfigured sender originates no OTC. These peer-source contracts are in FRR's OPEN code, FRR's route code and BIRD's attribute code and packet sender.

Tested Daemons

Daemon Version Image Query Method What It Validates
FRR 10.7.1 pinned by digest (interoplab.FRRImage, one declaration for the BGP, IPsec and L2TP labs and the perf runner). quay.io/frrouting/frr:10.7.1 vtysh eBGP, iBGP, route exchange, GR, communities, MD5, route server
BIRD 3.4.0 Built from the upstream v3.4.0 tag on Debian trixie (which refuses a tag moved off the pinned commit). BIRD 3 line only; Alpine 3.24 packages 3.2.4, and 3.4.0 built on Alpine 3.24.3 dies with SIGILL in rte_announce_to on the first route it imports. test/interop/Dockerfile.birdai/META-DECISIONS.md birdc eBGP, route exchange, triangle topologies
GoBGP 4.11.0 with the export-reload source patch Checksum-verified release and build-time source diff. golang:1.25-alpinetest/interop/Dockerfile.gobgp gobgp CLI, and gobgp-paths (a gRPC ListPath reader) for the families the CLI cannot name; it prints each attribute through GoBGP's String(), and an Extended Communities attribute also through its MarshalJSON, because String() writes an IPv4-address-specific Route Target and Route Origin alike eBGP, route injection and verification
StayRTR 0.6.4, patched to write its queued Error Report before it closes Go builder. golang:1.27-alpine3.24alpine:3.24.3test/interop/Dockerfile.stayrtr HTTP. /rpki.json RTR (RFC 8210) as the CACHE, so Ze is the client of an implementation that is not its own. Origin validation answers (RFC 6811) against VRPs a third party encoded
pmacct pmbmpd 1.8.0 pmacct/pmbmpd:v1.8.0 JSON msglog file BMP (RFC 7854, RFC 9069) as the COLLECTOR. pmacct decodes the per-peer header, the Peer Up Information TLVs and the Peer Down reason itself, so a scenario reads another implementation's reading of Ze's bytes rather than a string Ze emitted
Unbound 1.26.2 Built from the upstream release on (Alpine 3.24 packages 1.25.2). alpine:3.24.3test/interop/Dockerfile.unbound kdig from the dnsclient container; the query log on stderr (log-queries) Upstream recursive resolver of Ze's LAN DNS resolver (dns-resolver-forward-unbound): Ze's answer per query equals Unbound's own, a repeated query reaches Unbound once, and a truncated UDP reply is retried over TCP
Knot DNS 3.6.2 Built from the upstream release on (Alpine 3.24 packages 3.5.10). The same image runs the dnsclient LAN host, with Knot's kdig and Alpine's dhcpcd 10.5.2, the latest upstream release. alpine:3.24.3test/interop/Dockerfile.knot kdig Authoritative server of the lab zone lab.example. behind Unbound, and behind Ze directly in dns-resolver-dhcp-lan and in the L2TP lab's dns-resolver-l2tp-subscriber (172.29.0.7), where the xl2tpd + pppd LAC learns Ze's resolver through IPCP (usepeerdns) and resolves with busybox nslookup. docs/labs/l2tp-interop.md
dhcpcd 10.5.2 Alpine package in. test/interop/Dockerfile.knot dhcpcd --dumplease DHCPv4 client of Ze's DHCP server: the lease and its option 6 (dns-resolver-dhcp-lan, and the topology suite's access-up, where option 6 must hold the CPE's IPCP servers); the lease, its server identifier (dhcp_server_identifier) and the DHCPNAK it logs on a reboot (dhcpv4-server-shared-segment-dhcpcd)
ExaBGP API 6.0.0 contract fixtures Compiled Go wire server Wire byte comparison Byte-for-byte encoding across all address families

The conditional-MED collector check requires a single AS number in the received AS_PATH. It uses the shared AS-number parser, so decimal and dotted spellings identify the same source without accepting a multi-AS path.

GoBGP export-reload successor proof

bgp-export-reload-gobgp and bgp-export-reload-secondary-gobgp are new successor scenarios, not recovery of the historical four-peer experiment. Their registered leaf checker uses GoBGP's actual SIGHUP configuration reload. Both exact scenario identities also register their native peer preparation with the shared preparation registry; the common dispatcher contains no scenario-specific feature calls or rendered-directory-name heuristics. The ordinary scenario uses the global RIB; the secondary scenario uses route-server clients, with secondary selection enabled only toward FRR and disabled toward the independent FRR sink. Ze and BIRD originate fixed routes. Neither source changes its routes after the baseline fence. Ordinary export-policy AS_PATH matching includes GoBGP's already-prepended AS65005 (65005 65001 for the primary source); route-server matching sees the unprepended source path. Their fallback expectations must not be interchanged.

The phase configurations exercise reject-to-accept, accept-to-reject, attribute-only unchanged-best changes, export-rejected best with an acceptable backup, import-driven replacement, last-candidate removal, simultaneous import removal and export rejection, repeated reload, and restoration. Exact FRR and sink route objects are the delivery oracle. GoBGP's computed Adj-RIB-Out is retained only as diagnostic policy state, never as delivery evidence.

Repeat retains a byte-identical configuration and the native no-policy-difference reload path. Two separately armed, one-shot native CLOSE_NOWRITE watches observe the actual running configuration file, one per SIGHUP. Only the daemon reads that file while either watch is armed. The second read proves that GoBGP's serial signal loop returned from the first UpdateConfig and consumed the unchanged file again before restore overwrites it. The first reload has completed; the second read does not claim its own UpdateConfig completion or acknowledge a wire queue. Both receiver tables, source counters and original-session checks remain required. Native CLI capability collections are normalized once at their parser boundary as canonical JSON multisets, retaining duplicate multiplicity and exact values. GoBGP emits remote capabilities by iterating a map, so array order is not session identity. The original OPEN bytes remain independently retained in the packet capture; normalization does not replace that wire evidence.

The GoBGP container starts a bounded native packet capture before starting the patched v4.11.0 daemon against a writable container-local config copy. Only these two topologies start GoBGP first and await its native CLI readiness before starting the active source peers. This retains the original handshakes without collecting avoidable connection-refused attempts during daemon startup. The observation runtime is Alpine 3.24.3 with GNU coreutils 9.11-r0, tcpdump and inotify-tools 4.23.9.0-r0. The shared native-capture lifecycle owns and reaps the packet capture; each one-shot file observer also requires joined completion.

The daemon and CLI build from the checksum-verified v4.11.0 release with gobgp-v4.11.0-export-reload.patch applied before compilation. Policy reload performs retained-input processing followed by export reconciliation. A second change withdraws a secondary destination that was actually sent when import removes its last candidate and the new export policy rejects every old candidate. Both changes use GoBGP's existing outgoing queue and sent-route bookkeeping.

The image retains the exact patch input that its builder applied. Before reading configuration files, the checker reads those bytes and their SHA256 from the running GoBGP peer. They must match this checkout's declared patch and the named runtime artifact. Missing, empty, different or inconsistent evidence refuses the proof; an upstream version string cannot replace the applied source identity. The observed bytes and digest remain in proof.json as gobgp-patch and gobgp-patch-sha256, beside the actual image and binary identities.

The checker retains original OPENs, the complete captured interval, exact source Adj-RIB-In attributes, receiver tables, daemon identities and counter fences. Unknown route documents, failed queries, missing capture completion, capture loss, a reconnect, or source UPDATE/ROUTE-REFRESH after the baseline cannot establish a reload proof. Never-sent routes must remain absent without spurious withdrawals. All phases remain proof debt until the native run actually reaches them; a first-phase failure proves no later phase.

The scope is stable-session, non-ADD-PATH IPv4 unicast only. Original four-peer configurations, OPENs and captures remain missing; ADD-PATH IDs and send-max, VRFs, deferred families and preexisting orphan advertisements remain unverified. These successors neither replace that evidence nor discharge the fifteen historical live-checker obligations.

bgp-local-pref-strip-gobgp reuses the bounded single-session relay below in transparent mode: it changes neither OPEN nor UPDATE nor KEEPALIVE bytes. The existing GoBGP destination remains AS 65003 with external peer AS 65001; only its transport neighbor moves to the relay at .10. A RIB cannot prove LOCAL_PREF stripping: GoBGP v4.11.0 ProcessMessage removes the attribute before storage for an external, non-route-server-client peer, including this fixture's peer.

The wire proof covers a declared interval on that original connection, not its future traffic. Ze's direction must contain the original OPEN and KEEPALIVE, all three injected subjects (the third follows EOR and arrives in a later UPDATE than the first two), and the 10.54.255.0/24 FIFO fence after them. RFC 4271 Section 4.3: "An UPDATE message is used to advertise feasible routes that share common path attributes to a peer, or to withdraw multiple unfeasible routes from service (see 3.1)." Routes in one UPDATE share one attribute set and arrive together, so the fence is accepted in the UPDATE that completes the three subjects, in either NLRI order inside it. The whole UPDATE is validated before its subjects and fence count. A fence in an UPDATE that leaves a subject outstanding, a second fence, and a subject in an UPDATE after the fence's are refused, and the attribute checks apply to every NLRI an UPDATE carries. Only after forwarding and recording that full fence can the reverse worker begin reading its subsequent KEEPALIVE barrier. Each closed capture ends with its actual source/target endpoints, exact frame count and declared boundary. Timeout, EOF, truncation or the 256-frame limit cannot produce a completion marker. The checker revalidates both intervals, exact speaker identities, AS 65004/injector next hop and every LOCAL_PREF code 5, while retaining all three arrivals and both surviving native sessions. The workers keep forwarding later bytes, which are explicitly outside the proof. When the relay session ends on a transport error, the relay records it as transport-error in its -result.json file and also prints extended relay transport error: <cause> to stdout, so the speaker's docker logs carry it. A failed checker's dump reads both ("Debugging Failures").

The Extended Message FRR scenarios use a bounded, single-session relay that changes capability 6 in OPEN only. It records Ze's original OPEN separately, leaves UPDATE and KEEPALIVE bytes unchanged, caps each direction at 256 frames, and joins both socket workers before returning. FRR 10.3.1 gates extension bilaterally, so its facing OPEN always advertises capability 6 while Ze's facing advertisement varies independently. This isolates Ze's directional behavior without replacing FRR's UPDATE producer or consumer.

The producer is frr at host .3; the consumer is a separate frr-sink at host .15, both in AS 65002, with distinct native router IDs matching those addresses. Relays .10 and .11 connect them to Ze independently. The consumer originates no test routes: its decoded path must come from .11 with AS_PATH 65001 65002, and allowas-in 1 admits the producer's AS once. The old one-daemon topology presented the same (AS, router ID) on two configured Ze peers. Ze's default identifier-claim policy rejected the second OPEN with 2/3 through Peer.validateOpen, routerIDClaims.claim, and Session.runOpenValidator. This was not RFC 4271 Section 6.8 TCP collision resolution, which closes the losing connection with Cease. RFC 6286 Section 2.1 says an identifier "should be unique within an AS"; Section 2.2 mandates Bad BGP Identifier for zero or a self-identifier from an internal peer, not this case. Section 3 explicitly mentions parallel sessions between the same speakers. The fixture therefore uses distinct native speakers compatible with Ze's default policy; it does not disable that policy or rewrite identifiers in transit. FRR's bgp_open_make writes each peer's native local_id into its OPEN.

No test route is originated from the producer's startup configuration. The checker first requires both native FRR sessions to be Established and both Ze peers to report established through show bgp peer list | json. These two-sided observations share one 60-second readiness deadline. Only then does it originate the baseline once through FRR's existing vtysh path and require its decoded path at the consumer. Large and later control routes follow through the same origin path. This is a first-origination barrier, not a route retry or session replacement. The source relay starts before the sink relay. A startup network announcement could therefore reach reactorForwardRSSection before the sink has live forwardFacts, when that destination is correctly skipped. These live fast-path fixtures do not attach the route-server/Adj-RIB-In event consumers for retained peer-up replay, so waiting for the sink afterward cannot recover that earlier announcement. FRR's Established state alone is insufficient: FRR can receive Ze's initial KEEPALIVE before Ze processes FRR's KEEPALIVE. Ze's peer-list producer handleBgpPeerList serializes reactorAPIAdapter.Peers's Peer.State(), not the raw session FSM state. Peer.runOnce publishes negotiated encoding contexts and forwarding facts before publishing PeerStateEstablished, so that peer state is the local admission barrier. The checker reuses waitZePeerState for each original relay peer and retains the independent FRR observations. It changes neither AS-loop policy nor capability permissions.

The FRR-facing Extended Message relays start only after the native configuration probes pass. Ze's CLI must list both relay peers. Each destination FRR daemon's last passive neighbor must reach Active (passive), after peer_unshut_after_cfg clears the configuration hold. A listening TCP socket or successful show version is not sufficient: FRR bgp_accept can close a connection while configuration is still loading. These barriers use the existing ordered ReadyProbe hooks, with probe-specific environment and required stdout strings. Probes with no required strings retain exit-only readiness; relays to other daemons and other speaker personalities retain their startup order. No readiness probe opens a BGP connection, and the relay never reconnects a connected socket.

Ze's own probe waits for a listener whenever Ze answers a peer it never dials. The runner starts the peers in order, Ze before the third-party BGP daemons, and waits on each probe before it starts the next. When the config a Ze starts from (ze.conf, or ze-peer.conf for a second Ze) gives a peer connection remote connect false, or holds a dynamic group, the probe runs ss -ltn and requires :<port> for each such peer's listen port: 179 unless connection local port sets another. The question is answered from the parsed config, so a comment that names connect false decides nothing. Without the barrier, BIRD or FRR can dial before Ze listens, and the refused dial waits the daemon's connect-retry (120 seconds by default), which outlasts the session budget. A Ze that dials every peer keeps the probe that only runs true, because it redials the daemons started after it on its own backoff. A relay barrier keeps the listener check: one sh runs ss -ltn and then the peer list, and the probe requires both sets of strings.

A GoBGP-facing relay, as in bgp-local-pref-strip-gobgp, is gated the same way. GoBGP's InitialConfig calls StartBgp, which opens the listener, before it adds its neighbors, and passConnToPeer closes a connection from an address it has no peer for ("Can't find configuration for a new passive connection"). GoBGP's barrier is gobgp neighbor <relay address>, which must print BGP neighbor is <relay address>: it fails with "not found neighbor" until AddPeer has run. gobgp global -j is no barrier, because it answers as soon as StartBgp has run. Ze's CLI must also list the relay peer, and the relay starts after GoBGP.

That barrier admits the relay; it does not make GoBGP's session come up in time for the injector. Under load GoBGP's session to the relay has reached Established well over a minute after Ze started, while the injector used to announce after four of Ze's KEEPALIVEs (about 20 s), so its burst reached GoBGP, if at all, through the route server's peer-up replay rather than the forward rail the wire capture judges. The injector therefore waits on a state: its inject.msg declares option=await-file with its own 240 s bound, and the checker writes that file into the injector's container only after FRR and GoBGP report their sessions Established and Ze's show bgp peer list holds both destination peers (.3 and the relay .10) established, the state the forward rail reads. While it waits the injector answers Ze's KEEPALIVEs, so the 15 s hold does not expire. After the burst and End-of-RIB it waits for two more of Ze's KEEPALIVEs before the third subject, a full interval wherever the release fell in Ze's cycle, so the third reaches GoBGP in a later UPDATE than the first two.

Bounded readiness waits check cancellation before each probe, including the first. A simultaneously ready poll tick cannot admit another probe after cancellation is observed. Failure retains the last measured state or probe error and does not start dependent peers or the scenario checker.

A failed check names what the peer printed, not only what the checker expected. A wait that measured the peer but timed out ends its error with last output: and the last value its probe returned: text as the peer printed it, any other value in Go's %v form. A wait whose value is a verdict, such as a bool or a count, also ends with last answer: and the text its probe last read. A one-shot assertion over an answer it read ends with searched output: and that answer. Every quoted answer is trimmed and cut to its last 40 lines and its last 4096 bytes, with a note on its first line saying how much was cut, so a full routing table cannot flood the report. An empty answer is named as empty.

Lab.Query refuses a command that exited 0 and printed nothing with an EmptyAnswerError, kept apart from an exec that failed or was cut short. The BGP suite's positive waits (waitContains, waitContainsFold, waitContainsAny) read that refusal as a measured empty answer, so a route that never arrives times out with last output: (the peer printed nothing). FRR's vtysh prints nothing for an empty show ipv6 route ospf6, and before this the red read "never measured peer state", which names a peer that never answered. An empty answer still satisfies no positive wait. A negative wait keeps the refusal, because there an empty answer would read as absence.

The shared FRR BGP route wait reads an exact-prefix show bgp <family> <prefix> json answer for presence: the top-level prefix must match and its paths must be a nonempty array of objects. Absence instead reads show bgp <family> json and requires FRR's default-VRF table envelope with a routes object. Only that table's missing exact-prefix key, or an empty array under that key, proves absence. A prefix query's {} cannot: FRR 10.7.1 also emits it when no BGP process exists. Errors, missing fields, unrelated JSON, and a different prefix's per-prefix answer are not measured absence.

Installed-route assertions use FRR's native zebra JSON for BGP and OSPF, including IPv6 link-local installation. One entry under the exact prefix key must carry the matching prefix and expected protocol, with selected, destSelected, and installed true. Its matching next-hop object must itself carry both active and fib true. A protocol, state, address substring, or next hop found in another entry cannot satisfy the proof. These fields come from FRR 10.7.1 vty_show_ip_route and nexthop_json_helper. The pure boundary regressions supplement, not replace, live scenario proof; source identity alone does not establish the identity of a running image.

bgp-ospf-redistribute-default-frr follows FRR's real OSPF dummy subnet 10.82.0.0/24 through Ze's redistribution consumer and back to FRR over eBGP. It requires the producer address 10.82.0.3/24, OSPF point-to-point network type on ze-redist0, Full adjacency, Ze's exact protocol-250 kernel route via FRR, and established BGP before judging the received path. FRR's native BGP JSON must report the exact prefix, one valid best path from external peer Ze, AS_PATH 65001, and exactly Ze's session-local address as next hop. The connected subnet remains owned by FRR, so this assertion reads its BGP RIB. It does not require zebra to replace that connected route. Ze's final originated-route send guard withholds a next hop owned by the destination peer. Reverting the consumer to explicit IGP-gateway origination therefore meets that guard here; the useful path must use Ze's endpoint. The counterfactual requires the real OSPF and BGP prerequisites alongside the missing accepted path. Receiver absence alone does not establish where the route stopped. FRR 10.7.1's bgp_update independently refuses a host-route NLRI matching one of its local addresses, so an accepted path for its own /32 is an invalid expectation. Its point-to-point OSPF producer advertises the configured /24 subnet, which preserves the same next-hop discrimination. The existing registered peer-preparation hook creates and raises ze-redist0 before FRR starts; zebra configures its address. FRR explicitly refuses network-type changes on lo, so the producer is an ordinary dummy interface.

This scenario enables debug only for the consumer's bgp.redistribute and the upstream redistribute.orchestrator logger at startup, so its actual injection command survives. After failure, read-only diagnostics capture FRR's BGP table, neighbor counters and routes, and Ze's log rings, peer state, advertised RIB, OSPF routes, system-RIB candidates and metrics. The reads share an independent 15-second deadline and retain query errors alongside the original assertion failure. The recorder keeps full query answers; the error report carries bounded excerpts.

The FRR readiness producers are peer_unshut_after_cfg, bgp_start, and bgp_accept.

All five bgp-extended-message-*-frr fixtures build the native FRR attribute through four EXTENDED route-map sequences of 100 distinct large communities. The first sets the attribute; the next three use additive, with on-match next on the first three sequences. FRR therefore accumulates all 400 values (4800 attribute-value octets) on one route without exceeding its 4096-byte VTY input buffer. Repeating set large-community within one sequence is not equivalent: FRR replaces the previous set rule of that kind. The FRR producers are config_from_file, route_map_add_set and route_map_apply_ext, and route_set_lcommunity.

Before teardown, a failed Extended Message checker reads both relays' result JSON, original/delivered frame captures, TCP socket tables, and last 80 log lines. The diagnostic queries share a 15-second deadline and preserve read errors and the original assertion failure. They do not retry or replace the BGP session.

bgp-parsed-empty-mp-unreach-frr drives the parsed forwarding path from an ASN4 source to a two-octet recipient, with real FRR behind the OPEN-only relay. Both sessions negotiate IPv4 and IPv6 without ADD-PATH. The FRR fixture supplies a local IPv6 address: IPv6-family negotiation still needs it on IPv4 transport. After initial synchronization, the source seeds a route, withdraws it in an UPDATE carrying an empty IPv6 MP_UNREACH, then sends a genuine standalone IPv6 EOR in a separate epoch. Distinct-MED announcements fence each epoch.

For the parsed-empty-mp-source speaker, explicit --result <path> selects the JSONL capture path and places its readiness marker at <path>.ready. Both files are created exclusively with mode 0600. Each run needs fresh paths: an existing file or symlink causes an error rather than an overwrite.

Without --result, the defaults remain /tmp/parsed-empty-mp-source.jsonl for the capture and /tmp/parsed-empty-mp-ready for readiness. The scenario checker still reads these fixed defaults inside the isolated lab.

This capture-path meaning belongs to the parsed-empty-MP source. The generic runOracleSpeaker uses --result for its textual verdict instead.

The checker requires FRR installation and removal, complete source and recipient wire histories, the negotiated AS_PATH widths, and unchanged FRR daemon/session identity. A mixed withdrawal must not manufacture an EOR; the later genuine marker must survive. Before teardown, pass and failure both retain proof.json under the native session scratch directory of the checkout the suite runs from, including fixture inputs, captures, FRR table, PID, version and logs. The checker resolves that directory against CheckContext.Checkout, never the working directory, and refuses when no checkout is given. Required capture or write failures fail the proof. Diagnostic collection has a cancellation-independent fifteen-second bound.

Prerequisites

Requirement Used By Notes
Docker Interop tests Containers for FRR, BIRD, GoBGP, Ze
A Docker daemon kernel with every feature Ze enrolls Interop tests, docker-* deployment proofs, the l2tp, vpp and vpp-iface proofs, TestVPPSRv6ServiceRoute, the terminal-demo render and validate mode, the ./le perf ze DUT, ./le verify evidence release-candidate Checked before any image build or container start; see "The Docker host kernel check"
~1.5 GB disk Interop tests Docker images (Go builder, FRR, Alpine)
Docker ./le test integration clean, ./le test integration docker-peers ✕

The interop test network uses 172.30.0.0/24. MD5 authentication scenarios require NET_ADMIN capability (granted automatically by the orchestrator).

Real VPP SRv6 service-route proof

Run the focused service-route proof through native admission:

./le job run label vpp-srv6 command go test -tags integration -count=1 \
  -timeout 30m ./internal/le/test/deployment \
  -run '^TestVPPSRv6ServiceRoute$' -v

This test reuses the deployment VPP image, its container (NET_ADMIN, /dev/net/tun and /dev/vhost-net, never --privileged), build recipes, sockets, and staged Ze configuration. It does not replace the legacy ./le test deployment vpp population. Docker, Go, a VPP image with the SRv6 API and AF_PACKET plugin, and a Docker kernel with veth support are required. The managed phase also needs /usr/bin/vpp, writable /etc/vpp, and sufficient already-reserved hugepage resources for its configured 64M heap, 1024 buffers, and 16M stats segment. The harness never changes global hugepage reservations. ZE_VPP_DOCKER_IMAGE, ZE_VPP_DOCKER_PLATFORM, and ZE_VPP_DOCKER_GOARCH select the image and matching target architecture through the existing VPP settings. The test prints the selected image, VPP version, and retained scratch directory.

These Docker-host integration tests are excluded from the QEMU guest package population: the guest has no Docker daemon. The native admission command above runs their real container and packet probes rather than substituting guest mocks.

The compiled peer sends IPv4/unicast MP_REACH with an IPv6 next hop and an RFC 9252 L3 Service TLV through a real BGP session. Only Ze installs service policies and steering. The probe reads VPP's policy and steering dumps and CLI output. It requires an encapsulation policy whose single segment is the received SID, a distinct local BSID, and matching prefix steering in table zero. Two prefixes share a policy. A normal Ze restart against the still-running external VPP must retain the confirmed BSID identity without duplicates and resume packet forwarding. Withdrawal of one, replacement of one SID, last-user withdrawal, and reinstallation each have state and packet assertions.

The same harness then switches to Ze-managed VPP. A test-only external plugin uses the existing sdk.Plugin.EmitEvent ingress to publish best-change entries for the same prefix in tenant tables 10 and 20. This is a separate producer contract from the BGP/sysrib path above, not a claim of BGP VRF configuration. Steering must use the tenant table while policy/outer lookup remains in backend table zero. The probe kills the real managed VPP child, requires a different PID and a production replay request, restores only interface/underlay prerequisites, then answers that request through the ordinary event ingress. Restored forwarding, repeated replay without duplicate BSIDs, replacement in one table, independent withdrawal, shared-policy retention, and final removal are checked. The runtime publisher also decodes the exact bytes it sends and requires their replay marker to match the requested replay, rather than treating an ordinary incremental event as a replay proof. Before querying VPP, the probe waits for the binary API handshake's Connected event using the production connector's bounded connection mechanism. AllPluginsReady starts the plugin population; it does not prove the managed VPP child has opened its API socket. The harness retains the complete managed daemon/child stdout and stderr in srv6-managed-daemon.log and copies the generated configuration and VPP logfile before container cleanup, including failures before API readiness.

The packet oracle injects a unique Ethernet/IPv4 datagram through a veth and captures VPP's emitted frame. It checks the IPv6 source and received-SID destination, reduced encapsulation without an SRH, and preserved inner payload with one TTL decrement. Withdrawal must stop forwarding the corresponding datagram; adjacent positive probes prevent a dead capture from passing. The managed phase uses VPP's built-in packet generator for real graph ingress and Ethernet egress capture, with the same independent byte oracle. It requires the generated-packet counter to reach one before a negative assertion. Input and output pcaps and the external plugin's complete test transcript remain in the scratch directory. Prerequisite mutations must return their successful CLI response: vppctl exit status zero alone is not acceptance. A failed prerequisite or positive packet assertion records bounded VPP logging, interface, error, IPv6 FIB, and trace diagnostics. Packet observations log at most eight frame prefixes per injection; these diagnostics do not replace the forwarding assertion or extend its deadline.

The external-VPP configuration activates the real connected producer and netlink interface monitor. A test-only SDK plugin waits for all plugins to be ready, adds the Linux covering addresses through real netlink events, and queries show rib until both next-hop and SID covering prefixes have protocol connected. The BGP probe waits for that result before sending service announcements. After Ze restarts, the SDK probe re-adds the addresses to the new monitor and publishes a new process-generation readiness record; the old record cannot satisfy the barrier. Its actual RIB result remains in srv6-connected.log.

The fixture additionally supplies the VPP IPv6 underlay route, neighbor, tenant tables, and encapsulation source. These prerequisites are not evidence that Ze added underlay provisioning, SID reachability validation, remote decapsulation, or external-VPP reconnect support.

Synthetic oracle tests check rejection of wrong policy bindings, tables, SIDs, and packet fields. They are not dataplane evidence. A real proof requires an executed, non-skipped TestVPPSRv6ServiceRoute result. For discrimination, omit policy creation in (*govppSRv6Backend).acquirePolicy, rerun the command above to rebuild the daemon, and require failure at the first installed-state assertion. Restore the source and require the packet proof to pass before claiming closure.

Real VPP label pop proof (mpls-label-pop)

./le test deployment vpp runs nine scenarios in order against one VPP container. mpls-label-pop is the fourth, after mpls-fib, and it proves that an LDP egress pop reaches VPP's MPLS table 0 when the engine, not a fib { } block, chooses the label writer.

Before Ze starts, the harness creates the packet-generator interface pg0, so the VPP interface backend's first interface dump names it. Ze then runs with interface { backend vpp; ethernet pg0 { unit default { mpls { enable true; } } } } and ldp { lsr-id 10.0.0.1; transport-address 10.0.0.1; interfaces lo; }, and no fib { } block. The container's Linux loopback offers LDP no advertisable prefix, so the LSR-ID host route is the only FEC LDP originates. Every verdict comes from VPP's own CLI:

Check VPP query What passes
installed show mpls fib table 0 Exactly one label of 16 or above holds an eos entry whose path is an ip4 deag and a neos entry whose path is an mpls deag
mpls-input show mpls interface pg0 is followed by MPLS enabled
popped show trace A frame carrying that label, S set, reaches mpls-lookup and then lookup-ip4-dst or ip4-lookup
unprogrammed-drop show trace A frame carrying label 999999 reaches mpls-lookup, then mpls-drop, and no IPv4 lookup
removed show mpls fib table 0 After the configuration loses ldp { } and the running daemon gets SIGHUP, the label has neither entry

The parsers follow VPP's formatters in FDio/vpp v26.06, the release of the ligato/vpp-base:26.06-release image. mpls_fib_show prints each entry with format_fib_entry, whose first token is format_fib_prefix: for an MPLS prefix, "%U:%U" of the label and of format_mpls_eos_bit, then "/%d". format_mpls_eos_bit writes MPLS_EOS_BITS, "eos" or "neos", and a reserved label prints by name, so only an allocated label matches. format_fib_path writes "%U weight=%d " with format_dpo_proto (ip4, mpls) and then fib_path_type_names, where a lookup path is deag. show_mpls_interface_one prints the interface name, then " MPLS enabled". A table 0 miss forwards through mpls_fib_drop_dpo_index, whose node is mpls-drop. A frame on an interface without MPLS input meets the mpls-not-enabled feature and never reaches mpls-lookup, so popped also proves that Ze enforced mpls { enable true; } in VPP.

The frames go in through the packet generator from a pcap file in the scratch mount, the pattern of vppSRv6PGPacket. show packet-generator must count one generated frame before the trace is judged, so a stream that never ran cannot pass the drop check.

The scenario is red when fib-vpp no longer consumes label entries, for example with the (mpls-fib, entry) subscription in fib-vpp's register.go removed, or when LDP names fib-kernel in Dependencies again: installed then fails. The daemon binary is rebuilt by the run itself, so reverting the source and rerunning ./le test deployment vpp is enough to show it.

Live Interop Tests (test/interop/)

Each scenario runs Ze and one or more peer daemons in Docker containers on a shared network (172.30.0.0/24), establishes real BGP sessions, and asserts correct behavior via each daemon's native CLI.

How It Works

The native internal/le/interoplab/bgp package discovers scenario directories in test/interop/scenarios/. Before any scenario, the shared interoplab.Suite engine reports what a killed earlier run left behind ("Crash-safe teardown" below), runs the lab's preflight (which stages ze and le, and runs le's start probe: "The start probe" below), checks the Docker daemon's kernel with that ze ("The Docker host kernel check" below), and builds the images. For each scenario it then:

  1. Creates an isolated Docker network, after refusing an existing container or network of the same name that this process did not create.
  2. Starts Ze and the peer daemons declared by the scenario's config files, in the order scenarioPeers lists them.
  3. Waits for each peer's declared readiness probe before it starts the next.
  4. Runs the scenario's typed checker from the package-local BGP registry, recording every call it makes to the lab ("Recording and replaying a checker" below).
  5. Tears down the containers and the network this scenario created, including after setup or checker failure and after a Ctrl-C or SIGTERM, which cancel the run rather than end the process. A resource is recorded for teardown only once its create succeeds, or for a container once docker run was issued and failed for a reason other than the daemon refusing the name as taken (ContainerMayExist): the daemon may have created it. A failure before docker ran (an address, a mount or an AppArmor option the wrapper could not render) records nothing. So a name the pre-clean refused, one another run took first, or one never created is never removed. Docker.CreateNetwork refuses a network name that already exists rather than join it. A SIGKILL, a lost machine or a panic runs no teardown. The next lab start reports the leftovers. The explicit ./le test integration clean command can remove them.

An exact BGP-lab scenario selector, whether passed explicitly or through INTEROP_SCENARIO, changes image preparation to the selected peers' dependency closure. After the network is created and dynamic configuration preparation has returned its final peers, Suite.preparePeerImages matches their logical image names against the suite's existing declarations. It builds Ze and pulls the actual FRR reference those peers use, including a scenario-specific version pin, without preparing unrelated BIRD, Kea or other peer images. An undeclared image or a failed dependency fails the scenario before any peer starts; an optional image failure affects only scenarios that depend on it. Successful builds are shared within the run and their owned retention tags are released once, including after setup failure or cancellation. The selected-image progress line follows the scenario's running line. Without a selector, the lab retains the eager full-suite image preparation described above; the kernel probe, binary staging and deadlines are unchanged.

The start probe

A staged binary can build and still fail to start. On 2026-11-10 every le process panicked at start on a help line over the 60-character limit (commit c77a8dd2f3). The cross-compile passed, so the preflight passed, and every helper container that runs le (speaker, relay, injector) then failed its readiness wait. Four scenarios failed one after another, each after its image build, before anyone saw that no scenario had reached its checker.

A LabBinary may declare a StartProbe: the arguments the staged binary is started with once, right after its build. StageBinaries runs it in a one-shot alpine:3.24.3 container through RunOneShot, with the binary mounted read-only at /lab/<name>. The file keeps its declared name because cmd/ze selects its personality from that name. The container is built the same way as the kernel probe's (stagedBinaryContainer), so a darwin host runs the linux binary, and a probe that is cut short is removed by name.

Every lab that stages le declares linuxle.StartProbe, which is le help. le runs its registration checks before it reads its first word (leroot.DispatchWithStartup), so help reaches the check that panicked, and help needs no checkout. A binary that declares no probe is not run: a staged daemon needs a config to start, and the kernel check already runs the staged ze.

A probe that exits non-zero, or that Docker cannot run to an exit, stops the preflight before any image build. The refusal names the binary, its output path, the probe argv, the exit status and the first twelve lines of its stderr, then the fix: correct the start failure the stderr names, and run ./le test integration <lab> again. Each probe prints one progress line with its verdict and elapsed time:

interop: start probe le (`le help`): pass (<elapsed>)

Crash-safe teardown

Every container, one-shot container (the kernel probe included), network and built image the wrapper (interoplab.Docker) creates carries owner labels, all from one helper, ownerIdentity.labelArguments:

Label Value
ze.lab 1, the marker required by every cleanup verdict
ze.lab.run a random id unique to the creating process
ze.lab.launch an optional inherited id from the parent that launched the creating process
ze.lab.pid the creating process's pid
ze.lab.pid-start that pid's start token, which changes when the pid is reused (procscan)
ze.lab.boot-id the client machine's boot id
ze.lab.machine-id the client machine's id (/etc/machine-id, or the Mac's IOPlatformUUID)
ze.lab.started when the run started, RFC 3339
ze.lab.purpose the creating command line, for the operator
ze.lab.kind the object the create made: container, network or image
ze.lab.uid the creating process's uid
ze.lab.os the client's operating system, as Go names it (linux, darwin)
ze.lab.pid-ns on Linux, the inode of the creating process's pid namespace (/proc/self/ns/pid). Absent on macOS, which has no pid namespace

A container inherits the labels of its image, so one a person starts by hand from a lab-built image carries the builder's owner labels with ze.lab.kind image. The wrapper sets the kind on every create, overriding the inherited one, and the cleanup judges a resource only when its kind label names the object it is: such a container is kept and reported as carrying image labels only, not created by a lab run.

Every owner fact is read on the client, the machine running le, never from the daemon: on a Mac the daemon runs in a VM whose pids are not the Mac's.

Automatic startup only inspects leftovers and reports a dry cleanup preview. A same-name container or network is reclaimed only when every owner label equals this process's own labels. An exited foreign owner still causes a name refusal.

A parent that launches test children passes Docker.ChildEnvironment to each child and uses the same client for Docker.CleanupOwnedOrphans after they exit. The private bootstrap variable ZE_LAB_LAUNCH carries the parent's launch id. Descendants keep that id but generate their own ze.lab.run and process labels. The owned cleanup filters and rechecks ze.lab.launch before the ordinary owner checks below. It retains resources with no launch label or another launch id.

This attributes cooperating children to their launcher. It does not protect against copied labels or isolate suites within one launch.

The explicit Docker.CleanupOrphans removes a labeled resource only when its owner is provably gone. Every path to "gone" first requires the owner's machine id, operating system, uid and pid namespace to equal this client's: a client in another pid namespace (a devcontainer mounting the host's docker.sock) or under another uid (a /proc mounted hidepid=2) sees a live owner's pid as absent. Then one of three paths proves it: the same boot with the pid absent; the same boot with the pid held by a process with another start token; or another boot id with the owner started before this machine booted (btime in /proc/stat, kern.boottime on macOS). A pid held by a process that has exited and is not yet reaped (a zombie, Z or X in /proc/<pid>/stat or in ps -o state=) is absent: the go test child a docker-peers Ctrl-C kills is one until le reaps it, and it owns nothing. An owner under another boot id that started after this boot is a cloned machine sharing the machine id, which may still be running, so it is kept. It removes containers, then networks, each by the id the labeled listing gave, then the run-owned ze-interop-retained: tag by its name (Docker.ReleaseImage). Removing a tag by name untags only that reference, never the image another tag holds, and only a tag in the ze-interop-retained repository is a candidate, so the shared cache tag is never named. Everything else is kept and reported with its reason and the command that resolves it: an owner still running, an owner on another machine (run the cleanup there), in another pid namespace (run it there) or under another uid (run it as that uid), labels it cannot read, a boot time it cannot read, or a network another run's container is still attached to. An unlabeled resource, a volume and the shared cache tag that NO_BUILD=1 reuses are never candidates.

A removal call that fails, or that outlasts the 30 s client bound while the daemon completes it, is not reported failed on its own say. The wrapper inspects that exact resource again under the same bound (Docker.confirmRemoval): gone is "removed", still present is "failed" with the removal's error, and an inspect that fails too is "unknown", naming both errors.

A docker rm -f the daemon refuses with "removal of container is already in progress" is not a failure either: the daemon is already removing it, as --rm does for a one-shot whose bound ended. The wrapper inspects the container at once, then every createSettleInterval until createSettleWindow ends (Docker.awaitRemoval). Gone is "removed", an inspect that fails is "unknown", and a container still present when the window ends is reported by name as still present, with ./le test integration cleanup. Only a container removal gets that answer: it is the one "already in progress" refusal in the dockerd 29.9 binary, so a network or image removal keeps the single inspect above.

A one-shot container (Docker.RunOneShot: the kernel probe, a host preflight, a terminal-demo render) runs in the foreground with --rm and a ze-oneshot-<random> name. Killing the docker client stops nothing the daemon runs, so when the caller's context is canceled or the one-shot's bound ends, the wrapper removes the container by that name on a context the cancellation does not reach, and reports the removal's outcome with the run's error. Its free-form OneShotContainer.Arguments, like a lab peer's PeerConfig.Arguments, are refused before docker runs when they carry --name (which would replace the name the removal finds it by), a ze.lab label in any spelling (which would override the owner labels), or --label-file, with a BUG: error naming the field. Any spelling includes -l at the end of a short-flag cluster such as -dl <v>, -dl=<v> or -dl<v>, read the way docker's flag parser reads one: past the value-free flags of docker run (-d -i -P -q -t), stopping at the first flag that takes a value. --name has no short form.

A caller outside a lab Suite that needs a long-running container, such as the deployment proofs (./le test deployment l2tp, vpp-iface, vpp), starts it with Docker.RunDetached. Its typed spec (DetachedContainer: platform, network and address, capabilities, devices, mounts, environment, working directory, entrypoint, command) carries no raw docker flag, and the container carries the same owner labels, so a run killed before its own removal is reaped like a lab's. Commands inside it run through Docker.ExecAttached, with attached streams and their own bound. Those verbs root on SIGINT and SIGTERM, remove their container on a context the signal does not reach, and name a failed removal in the report's container-removal-error, exiting 1. The removal is owed from the moment docker run -d was issued, under the same ContainerMayExist test a lab uses: the daemon creates the container before it starts it, so a start it refuses (a device the host lacks, such as /dev/vhost-net) or a Ctrl-C during the start still removes the container. A run that returned an error (a step it could not perform, or an interrupt) reached no verdict, so it prints none: the error, then a removal that failed with the command that finishes it, and it exits 1.

Killing the docker client does not cancel a create the daemon already accepted, so after a Ctrl-C during docker run the container can appear after the first docker rm -f answered that no container has that name. Docker 29.9 gives that answer as exit 0 with no output, and prints the name only for a container it removed; older releases refuse with "No such container", and both answers count as absent. When a start failed or was canceled, Docker.RemoveContainer does not accept that first answer. It runs docker rm -f again every createSettleInterval until createSettleWindow ends, and removes the container if it appears. If the container appears and cannot be removed, or the wait is stopped, the error names the container and ./le test integration clean. This applies to every start that marks its name: Docker.RunDetached (the deployment proofs, the benchmark), a lab peer's run, and a one-shot container. A create that lands after the window still carries the owner labels, so the cleanup removes it. Removing a container whose start succeeded takes one docker rm -f. So does any later removal of a name: the first removal after a failed start uses the mark up, whatever its docker rm -f answered.

A docker command le stopped is named for what stopped it. One whose own bound ended reads "did not finish within " and carries exit 124, as timeout(1) answers. One that a deadline of its caller's context ended first reads "was cut short by its caller's deadline after , before its own bound": a Wait window that closed while its probe's exec ran is the usual case, and the command did not hang. It carries exit 124 and still wraps context.DeadlineExceeded, so a caller that branches on a deadline takes the same branch; only the text differs. The runner tells the two apart by the cause the command's own bound records, not by reading the caller's context afterwards, which a deadline passing just after the bound fired would fool. A lab cross-compile its caller cut short is refused the same way, without the slow-build reasons. One whose context was canceled, by an interrupt reaching the verb's signal root or by the run stopping, reads "was canceled before it finished" and carries no exit code, because the code would be le's own kill.

Every docker command runs in a process group of its own, so a terminal Ctrl-C reaches le and not the docker client: the signal root cancels the run, and the removals it then issues on a context the cancel does not reach are not killed by the same signal, or by a second Ctrl-C. A command handed a terminal as one of its streams stays in the group of le, where reading the terminal raises no SIGTTIN ("How a suite's child processes end" in runner-architecture.md).

The multi-DUT benchmark (./le perf run, ./le perf record evidence) is the other such caller. Its network goes through Docker.CreateNetwork and Docker.RemoveNetwork with the owner labels, each DUT and the sender through Docker.RunDetached, the readiness wait through Docker.ContainerRunning, and each image through Docker.Build (a named build context, such as FreeRtr's source checkout, is a typed NamedBuildContext) followed by Docker.ReleaseImage, which drops the run-owned tag and keeps the DUT's own. Only BuildKit accepts --build-context, so a build that carries a named context first asks docker buildx version. When the docker CLI has no buildx plugin, docker build would run the legacy builder and answer with its usage text, so Docker.Build refuses before docker build runs: the error names BuildKit and the missing plugin, and names ./le setup install as the fix. That run's docker-buildx step installs the plugin on macOS and names the package to install as root on Linux (docs/guide/developer-setup.md, "Docker labs"). The error then gives the same install by hand: brew install docker-buildx and the cliPluginsExtraDirs entry Homebrew prints on macOS, added to the config file this docker client reads ($DOCKER_CONFIG/config.json from the client's environment, else ~/.docker/config.json, resolved by setup.DockerConfigPath, the helper the setup step edits it through), and the distribution's buildx package on Linux. An environment that sets DOCKER_BUILDKIT to a false value forces the legacy builder even with buildx installed, so that build is refused too, naming the variable and unset DOCKER_BUILDKIT; Ze never overrides the setting. The named context is never dropped to make the build run.

Docker.ContainerRunning answers an inspect it cannot read as an error, never as "not running". It removes only a DUT or sender name it recorded, and records one only when interoplab.ContainerMayExist says its RunDetached may have left a container: never for a name the daemon refused as another container's.

Suite.Run inspects resources after the Docker probe and before Preflight. It prints each proposed removal or skip, then interop: cleanup: N to remove, N kept, N failed, N unknown. An empty inspection prints interop: cleanup: no Ze lab resource on this Docker daemon. A failed listing stops setup and names the failing Docker command to diagnose before another run.

Before a scenario creates resources, its pre-clean reclaims a same-name resource only when all labels equal this process's labels (ownerIdentity.created). This includes an earlier scenario's shared network whose removal outlasted its bound. Every other owner causes a refusal, including an exited owner. The refusal names the lab's suffix variable, such as ZE_L2TP_INTEROP_SUFFIX, when available, or another action that avoids the conflicting name. The scenario's teardown removes only resources the scenario itself created.

./le test integration clean runs the cleanup by hand and exits 1 when a removal failed, naming the failing command. ./le test integration clean dry lists what would go and what stays, removes nothing, and also lists ze-interop-retained: tags left from before the labels existed, which no mode removes. Both render through | json.

The Docker host kernel check

A container runs on the Docker daemon's kernel, so a kernel feature Ze needs and that kernel lacks fails scenarios for a fact about the host. mobike-initiator and mobike-responder showed the cost: on a 6.8 kernel they spent fifteen minutes and reported two protocol failures that were the missing XFRM_MSG_MIGRATE_STATE. Every Docker run that runs Ze therefore checks the daemon's kernel first, and refuses a kernel that lacks any feature. The owner's ruling is that a wrong kernel "should not be possible - fail": there is no skip, no warning mode and no exemption.

The required set is not listed in le. Suite.Run starts the ze the lab staged for the daemon's architecture (Suite.StagedZe, from StagedZePath) in a throwaway container and reads its answer:

docker run --rm --cap-add NET_ADMIN --cap-add SYS_ADMIN --network none \
  -v <staged ze>:/ze:ro alpine:3.24.3 /ze doctor --json kernel-capabilities

ze doctor kernel-capabilities probes every capability a package enrolls in internal/component/kernelcap, with configuration ignored (docs/architecture/doctor-and-health-checks.md). A capability enrolled later is required of every Docker host with no le change. NET_ADMIN lets the netlink probes ask; SYS_ADMIN lets the xfrm-interface probe unshare a network namespace. Without it that row reads unknown, which also refuses.

On a daemon whose docker info security options list name=apparmor, the probe container runs under Ze's AppArmor profile instead of docker-default (--security-opt apparmor=ze-kernel-probe): docker-default denies the MPLS transit MTU probe's mount and its /proc/sys write, and that row would read denied (owner D-7: the host is set up for Ze, never unconfined). On a Linux client whose /sys/kernel/security/apparmor/profiles lacks the profile, the check refuses before any container starts and names ./le setup docker kernel apparmor confirm ze-kernel-probe. An unreadable list (apparmorfs refuses it to a reader without policy-view privilege, such as a non-root user) is never read as "loaded": the probe runs, and when runc answers "unable to apply apparmor profile" the check refuses with the same command rather than reporting an unreadable answer. Both refusals also name the daemon's kernel release and the kernel route below, because the profile lets the probe answer and adds no kernel feature: a stock kernel may still lack some (MOBIKE's xfrm migrate, for example), and the operator reads the whole path in one run. On a Mac both refusals, and a denied row's fix, name the same command and say that there it loads the profile into the Colima VM behind the Docker endpoint, and that the labs' ze profile boots Ze's kernel, which applies no AppArmor, so a daemon applying it is one DOCKER_HOST names (appArmorColimaLoad); then the macOS route, ./le setup docker host start. Colima's Ubuntu default VM applies AppArmor and holds only the profiles loaded inside it, so runc's "write ... attr/apparmor/exec: no such file or directory" is the kernel's answer for a profile it has not loaded (colima 29.5.2, 6.8.0-117-generic, 2026-10-21: docker-default applies, an unknown name fails with the same text). On a client that is neither Linux nor macOS neither refusal names the load command, which refuses there (appArmorLoadPlatform): the refusal says so, still names the daemon's kernel release, and names only the kernel route. The client system is the one the Docker wrapper records in its owner labels (ze.lab.os); it is read from the client even when the machine or boot identity is not, so a host with no /etc/machine-id still gets the Linux text (readHostFacts).

A security options query that fails or prints no JSON refuses, because the argv depends on it. Ze's own kernel carries no AppArmor, so the QEMU guest's daemon lists none and the guest route is unchanged.

The probe is the first container to run under a Ze profile, and the same rule holds for every lab container: one that needs more than docker-default grants runs under a Ze profile that grants exactly that, never apparmor=unconfined. Each profile registers in interoplab (RegisterAppArmorProfile, from a lab's register_apparmor.go) with its name, what runs under it, what docker-default denies it, its text and the command to run once it is loaded. A peer names its profile in PeerConfig.AppArmorProfile. On a daemon that applies AppArmor the container starts with --security-opt apparmor=<name>; on a Linux client whose profile list lacks it, the lab refuses before the container starts, naming ./le setup docker kernel apparmor confirm <name>; where that list is unreadable the container runs, and when runc answers "unable to apply apparmor profile" the lab refuses naming the same command (labAppArmorUnapplied); on a Mac both refusals name the command, which loads the profile into Colima's VM, and then the macOS route, ./le setup docker host start; on any other client they name the route instead of the command, which refuses there (appArmorLoadUnreachable); on a daemon that applies no AppArmor the option is left out. ze-kernel-probe parses with apparmor_parser -Q 4.0.1 (Ubuntu 24.04).

A lab peer runs with no more than its daemon needs, never --privileged (plan/release/validation/spec-lab-containers-least-privilege.md). A sysctl set once before the daemon starts is a --sysctl argument, written by Docker into the peer's own network namespace. A sysctl written while the peer runs takes interoplab.NetSysctlWriteArguments (systempaths=unconfined) together with the generic profile ze-lab-net-sysctl, which is docker-default with every /proc/sys write denied outside net/ and every mount denied; any lab names it, none defines its own copy (owner, 2026-10-10: one shared profile granting only what the labs together need). The VRRP scenarios' ze (any scenario carrying keepalived.conf) is such a peer: once it knows its interface VRRP writes the per-device knobs under net/ipv4/conf/ and net/ipv6/conf/ and the namespace-wide net/ipv4/conf/all/rp_filter and net/ipv4/icmp_errors_use_inbound_ifaddr (applyDataplaneSysctls, internal/plugins/vrrp/dataplane_linux.go), and TestVRRPWritesOnlyLabGrantedSysctls records every write it makes and checks the profile grants each. In the IPsec lab the NAT box, strongSwan and ze hold NET_ADMIN, and ze also holds BPF, because vpn ipsec application-override deny (the default) loads and attaches its setsockopt guard and fails the apply without it (capabilityBPF, internal/le/interoplab/ipsec/ipsec.go). In every scenario ze also holds the host's /dev/net/tun (interoplab.TUNDevice, --device) and runs under ze-lab-net-sysctl with /proc/sys writable: a Child SA that receives UDP-encapsulated ESP makes ze create the ze-espform TUN device and write its disable_policy and rp_filter (openESPFormTUN, internal/component/ike/dataplane/espform_tun_linux.go), and the install is refused when either fails. Whether an SA is encapsulated is decided at run time (NAT detection, MOBIKE, which floats IKE to port 4500 before IKE_AUTH, or a peer that came in on 4500), so no scenario is exempt. TUNSETIFF needs CAP_NET_ADMIN in the container's namespace, which ze already holds. The NAT box sets its three sysctls through --sysctl; strongSwan runs under ze-lab-net-sysctl only in a scenario whose scenarioCheckers entry sets writesPeerSysctl (internal/le/interoplab/ipsec/checkers.go), because its checker writes strongSwan's sysctls at run time (it cuts the reassembly marks through dropFragmentsAtPeer, or enables IPv6 on the lab interface through vtiIPv6PeerInner), and TestIPsecLabPeersRunWithoutPrivilege checks the profile grants every one of them; TestOnlyCheckersThatWritePeerSysctlsAreFlagged reads the source and refuses a checker that reaches one of those helpers without the flag, or the flag without the write. In the L2TP lab ze and the xl2tpd LAC hold NET_ADMIN and --device /dev/ppp, and the preflight probes with the same grants and loads no module, so it passes only where the peers can run. The PPPoE lab grants ze, accel-ppp and the pppd client the same two, its preflight probes the same way, and no PPPoE image installs kmod or runs modprobe. The RADIUS subscriber lab grants ze and its LAC the same two and FreeRADIUS neither.

The /dev/ppp half of those preflights is declared once, in internal/le/interoplab/pppdevice.go, because the Docker host kernel check below cannot see a character device. Each lab still runs one preflight container: it puts PPPDeviceProbe (the line printing DEV_PPP=ok or DEV_PPP=missing) first in its sh -c command and adds its own probes, parses the output with ParsePreflight, takes the device entry of its refusal from PPPDeviceMissing, and refuses a container that did not run through PPPPreflightRunRefusal. That refusal always wraps Docker's error, keeping the operation, exit code, stderr and cause, and adds sudo modprobe ppp_generic only when Docker's stderr names /dev/ppp.

The TUN clone device every lab Ze running IPsec holds is declared once too, as TUNDevice and TUNDeviceArguments in internal/le/interoplab/tundevice.go; the IPsec lab's ze and the home-CPE lab's CPE take it from there. It has no preflight: Ze's kernel builds tun in (CONFIG_TUN=y, gokrazy/kernel/runtime.config), so a Docker host on Ze's kernel always has the node, and on any other host Docker refuses the container start with an error gathering device information line naming /dev/net/tun. TestIPsecLabPeersRunWithoutPrivilege holds the IPsec lab's ze to exactly NET_ADMIN, BPF, that device and the net sysctl grant.

The check runs after the preflight, because it needs the staged ze, and before the first image build, so a refused host costs no build and counts no scenario. It refuses when:

Answer Refusal names
a row is unknown the probe failure, every non-present row by subsystem, CONFIG_ symbol, state and reason, and the daemon's kernel release. Unknown does not establish a missing kernel feature and never prescribes an install, kernel rebuild or reboot. Resolve the reported probe failure and rerun the same /ze doctor --json kernel-capabilities probe through ./le setup docker kernel check ze '<staged-ze>', with the actual supplied executable shell-quoted in the refusal so the retry does not cross-build a different capability population
✕ every such row by subsystem, CONFIG_ symbol, state and reason, the daemon's kernel release, and the next step for the platform (DockerKernelRoute, owner D-6), its command on a line of its own: on Linux a kernel with every feature or ./le setup docker kernel install and a reboot; on a daemon whose kernel release already ends in -ze (Ze's CONFIG_LOCALVERSION: the QEMU guest of ./le test qemu docker-lab, or a host after the install), on any client system but macOS, Ze's runtime kernel's own fix instead (dockerKernelRouteFor: add the options to runtime.require, then rebuild the kernel cache entry with ./le build host-driver and ./ze-host appliance kernel --target runtime --arch <arch>); on macOS the ze Colima profile, which boots Ze's docker-host image on Ze's runtime kernel: unset a DOCKER_HOST that names another daemon, or add the options to runtime.require and rebuild the image with ./le setup docker host build, then ./le setup docker host start; on any other system the refused run's own le words against the Docker daemon of the Ze-kernel QEMU guest, ./le test qemu docker-lab lab '<the words>' printed with the words filled in, followed by env '<NAME=value>' with each scenario selector variable the suite declared to ReadEnvironment and the run set (IPSEC_INTEROP_SCENARIO=mobike-responder, for example), so the guest reruns the one scenario rather than the whole suite; or ./le test qemu docker-lab (boot and check only) when the process is not le or a word or an assignment holds white space or a quote. A denied row adds, on a daemon applying AppArmor, ./le setup docker kernel apparmor confirm ze-kernel-probe; on any other daemon it says the policy the row names must grant the probe its steps, and under SELinux names sudo ausearch -m avc -ts recent to read the denial. gokrazy/kernel/runtime.configgokrazy/kernel/runtime.config
the container fails, or prints no JSON the command it ran, its exit code, stdout and stderr
the answer holds no row the command it ran
every row is present but the probe exits non-zero or answers ready false the command it ran, its exit code and the ready it read: the producer's verdict and its rows disagree, so the answer is unread, never a pass
the suite names no staged ze Suite.StagedZe

The refusal is the suite's setup error: interop: setup: <reason>, exit 1. The docker-l2tp-ppp and docker-pppoe-accel deployment proofs run the L2TP and PPPoE suites, so they reach the same check. A suite with no kernel module of its own, such as the FreeRADIUS admin-login suite, is checked too: the owner's rule is about the host, not about the scenario.

The vpp-iface VLAN QoS scenario keeps one Ze daemon running through two SIGHUP reloads. A successful empty-map preflight establishes a fresh pool. VPP's own show qos egress map output must contain exactly one map, whose external ID is the VLAN index and whose mark refers to the first pool entry. Its output must first contain the full VLAN row with priorities 3 and 5 mapped to themselves, then only priority 3 after replacement. The next reload removes the egress map from config, and show qos mark must report no mark on the same existing VLAN interface. A failed CLI query, a changed interface index or an exited daemon cannot satisfy the proof. The signal targets the exact daemon command in the proof's private container, as the other VPP deployment scenarios do.

The l2tp, vpp and vpp-iface deployment proofs run no suite, and neither does TestVPPSRv6ServiceRoute (build tag integration), which prepares its daemon through the step vpp-iface uses. Each cross-compiles its own daemon to tmp/evidence/bin/ze-linux-<arch>, and buildCheckedDaemon runs the same check with that binary right after the build, before the proof inspects or pulls its peer image and before its first container, so a refused host downloads and starts nothing. With NO_BUILD=1 the proof skips the build and probes with the daemon already at that path, and refuses by name when there is none.

The terminal-demo render (le site terminal-demo render and render all) and the renderer's validate mode run Ze in validator and recorder containers, so both make the same check, through one shared preflight, with the demo binary (tmp/terminal-demos/bin/ze) after confirming that binary exists and before the renderer image is inspected or any container starts.

./le perf run and ./le perf record evidence prepare the ze DUT's inputs through the BGP lab's own staging whenever the selected build or test includes Ze (every DUT, or ze named). That staging cross-compiles the same three binaries test/interop/Dockerfile.ze consumes: ze-linux, le-linux and ze-le-linux. A test then makes the same kernel check with the staged test/interop/ze-linux before the benchmark starts. A build-only invocation stages before image construction, but starts no Ze and does not probe the runtime kernel. Selecting only other DUTs neither stages Ze nor probes for it.

Every route refuses a staged ze path that holds no file, or holds a directory, naming the path, before any docker command runs. Docker answers a bind mount of a missing path by creating a root-owned directory there, which a later staging then cannot replace. A missing file names, on a line of its own, the run's own le words again: env -u NO_BUILD ./le <words> when the run skipped its build, ./le <words> otherwise (perf stages without NO_BUILD).

./le verify evidence release-candidate runs the whole verify gate, Ze's functional tests included, inside a container on the daemon's kernel. After the clean-tree check and before the container starts, it cross-compiles a linux ze for the daemon's architecture into tmp/verify-evidence/ze-linux, which git ignores so the tree stays clean, and makes the same check with it. The container never runs --privileged: it holds one --cap-add for each capability the option=needs-linux:caps= vocabulary names and one --device for each node a production path reached by a test opens (/dev/ppp, opened by ppp.DevPPPSetup, and /dev/net/tun, opened by openESPFormTUN for an IPsec Child SA that receives UDP-encapsulated ESP), read from the runner's own tables (runner.ContainerCapsDockerNames, runner.ContainerDevicePaths), and keeps Docker's default seccomp and AppArmor profiles. A test declares no device: TestContainerDevicesCoverWhatTestedPathsOpen reads the L2TP and IPsec dataplane sources for every /dev node they open, as a literal or a package constant, and checks the table holds exactly those. A token added to either table reaches the container with no other edit. sys-time is left out: a container shares the Docker host's clock, so a verify run must not set it, and the NTP restore test skips there through its own caps gate. A container never loads a kernel module, so a Docker host without one of those nodes makes Docker refuse the start; the run then answers an error naming the node and the host command that creates it (sudo modprobe ppp_generic for /dev/ppp, sudo modprobe tun for /dev/net/tun) rather than a bare exit code.

The run is a release check only Thomas starts. The verb declares leaction.Action.WholeTree, so before its body runs, and so before the clean-tree check, the kernel probe or any docker command, the dispatcher refuses it with exit 2 unless stdin and stdout are both terminals (human.RequireTerminal). An agent's tool call reaches le through pipes, so it is refused, and the refusal gives the exact line to hand Thomas and ./le verify changed for the agent's own evidence. The container is started with docker run -t, which gives the inner ./le verify current, itself a whole-tree verb, a pseudo-terminal on stdin and stdout: the host side already proved a person started the run, and the terminal carries that fact in without an override keyword another caller could type.

The container is a Docker.RunOneShot, so it carries the owner labels and a name, and runs under a four-hour bound (owner D-1, 2026-10-19): the gate takes 25 to 53 minutes, so the bound never ends a normal run and only stops a wedged daemon. SIGINT and SIGTERM cancel the action's root context; the wrapper then removes the container by name, and the run answers the interruption as an error, never as a failed gate. The three named cache volumes (ze-gomod-cache, ze-gobuild-cache, ze-apt-cache) are mounted, never removed: they are what keeps a second run minutes long.

On macOS the daemon in hand is the ze Colima profile's, which boots Ze's docker-host image (ai/META-DECISIONS.md). With DOCKER_HOST unset or empty, every docker command le runs for a lab, the kernel check included, carries DOCKER_HOST=unix://<colima home>/ze/docker.sock, where the Colima home is resolved as Colima resolves it ($HOME/.colima by default, COLIMA_HOME or $XDG_CONFIG_HOME/colima when set; dockerhost.ColimaHome) (dockerhost.Environment, through Docker.run, the one route to the process runner), whatever /var/run/docker.sock or the current Docker context names. When that socket is absent the first docker command refuses, before any image build or container, naming the missing socket, that the labs on macOS run on the ze profile, and ./le setup docker host start on a line of its own; Colima's default profile and the context are never used in its place. A DOCKER_HOST the operator set is kept as it is, because it names a daemon on purpose (the QEMU docker-lab's forwarded socket is one), and the kernel check stays the gate that judges that daemon's kernel. ze appliance build, which every QEMU verb that builds an appliance runs (./le test qemu installer and its siblings, vpp-hugepages, mpls-boot, appliance-hostname, the two crash-capture-* verbs), runs under the same selection. ffmpeg, which le site terminal-demo runs, is no Docker client and runs under le's own environment.

./le setup docker kernel check asks the daemon in hand the same question. With no ze it cross-builds tmp/qemu/linux-<arch>/ze for the daemon's architecture (docker version's server arch) through BuildLinuxZe, the producer ./le test qemu docker-lab uses; ze <path> names another. Either way the file must be an ELF executable for that architecture, or the check refuses before any container runs, naming the path, what it is instead (a directory, not ELF, another machine), and the build route. ./le setup docker kernel apparmor runs on Linux, against this machine's kernel, and on macOS, against the Colima VM behind the client's Docker endpoint (docker context inspect, dockerhost.ColimaProfile): a docker.sock beside a colima.yaml names that directory's profile, under any Colima home (~/.colima, $XDG_CONFIG_HOME/colima or COLIMA_HOME), and the docker.sock at a home's root, which Colima forwards for the default profile (its Lima instance _lima/colima), names default. It refuses any other macOS endpoint, such as Docker Desktop's, naming the macOS route, ./le setup docker host start. On macOS every step runs inside the VM through colima ssh --profile <profile> --: a VM that does not answer true refuses naming colima start --profile <profile>, a missing colima or docker names its brew install, and a question that hangs for 30 s names colima status --profile <profile>. The profile is staged under the checkout's tmp/, which Colima's home mount shows the VM at the same path; a VM that cannot read it refuses before any root step, naming colima start --profile <profile> --edit. It refuses where /sys/module/apparmor/parameters/enabled does not read Y or apparmor_parser is absent on that machine, installs the registered profile named after confirm as /etc/apparmor.d/<name> (so it loads again at boot) and loads it with apparmor_parser -r -W; without confirm naming a registered profile it prints the two sudo steps of every registered profile, runs none, and names the command that loads each. ./le setup docker kernel install is the Linux route: it installs the cached runtime kernel for the host's architecture under /boot and /lib/modules, rebuilds the initramfs and the GRUB menu with the Debian tools, and saves the new entry as GRUB's default by its title. It refuses before any step unless GRUB_DEFAULT=saved, and unless the cached release is the release the version internal/appliance/kernel.version pins builds (7.3.0 for pin 7.3; see docs/architecture/appliance/kernel-profiles.md) followed by the CONFIG_LOCALVERSION suffix its config declares (-ze, so never a bare upstream release another kernel could own, nor an entry built before a version bump), and unless the cached kernel's config builds in, or as a module, every option this host relies on its running kernel for: the PCI hotplug drivers when an interface in /sys/class/net, or one netplan's generated /run/systemd/network/10-netplan-<id>.network names, is a slot name (ens18, which the kernel without them renames enp0s18); the option behind each iptables table, match or target module in /proc/modules; and UFW's stock matches and targets when /etc/ufw/ufw.conf exists. One table maps each module to its option, and a loaded netfilter module it does not map is refused by name. The refusal names each dependency, what the kernel would break, the missing options, and the rebuild (hostdeps.go). After the steps it reads /boot/grub/grub.cfg, through sudo cat when the host keeps it root-only, to find the entry's title. Without confirm <release> it prints the steps and runs none, because passwordless sudo asks nothing: ./le setup docker kernel install confirm <release>, where <release> is the version internal/appliance/kernel.version pins with the -ze suffix. The cache entry is keyed by the backend that built the kernel, and ze-host answers docker's entry whenever the docker binary is on PATH, so a kernel built with QEMU on such a host is named with builder qemu (./le setup docker kernel install builder qemu confirm <release>); the dry run's confirming command and every rebuild it prints keep that backend. A reinstall replaces: it removes /lib/modules/<release> and /boot/initrd.img-<release> before writing them again, so cp -a never nests the new modules inside the old tree. The copy drops the cache's ownership, so the installed modules belong to root, not to the user who built them. It prints each sudo step before it runs it, and never reboots. Off Linux it refuses: on macOS it names the ze Colima profile, ./le setup docker host start; on any other system the guest route, ./le test qemu docker-lab (docs/architecture/testing/qemu-integration.md). An interrupted or failed confirmed plan lists its completed command prefix and unfinished work, including the GRUB default selection. A stop before its first step says no install step started. A command that might have changed files instead leaves an explicit do-not-reboot warning until installation finishes, including the gap between removing and replacing modules. The refusal states that no automatic reboot was requested and prints the exact command to finish, with the same builder and confirmed release.

To ask a host by hand, stage ze with any lab once and run the command above. TestDockerKernelCheckOnThisHost (build tag integration) does that against the daemon this machine uses: it stages ze, starts the probe container and nothing else, and judges the verdict against the rows it read. On colima's 6.8.0-117-generic (2026-10-10) the check refuses ipsec-mobike, l2tp, l2tp-ppp and mpls as absent and mpls-transit-mtu as unknown, and an unconfined probe on 2026-10-21 answered the same five rows: that kernel's config leaves CONFIG_XFRM_MIGRATE unset, and its L2TP and MPLS routing modules are in linux-modules-extra, which Colima's image does not install. Loading Ze's profile there lets the probe answer; it does not make the check pass.

./le job run label kernel-host command go test -tags integration -count=1 \
  -run '^TestDockerKernelCheckOnThisHost$' ./internal/le/interoplab/

The report prints only after the last scenario, so Suite.Run also writes progress lines while it runs: one as image preparation starts, one when the images are ready with the time it took, and two for each scenario, interop: [i/n] name: running and then pass or FAIL with its elapsed time and error. They go to Suite.Progress, or to stderr when that is nil, which is the job log in CI. A suite that hangs then names the scenario that holds it. A failed progress write does not fail the suite.

A failure names the phase it happened in, after the elapsed time: interop: [i/n] name: FAIL (12s) in setup: <error>. The phase is setup (pre-clean, network, prepare, validation, images, start wrapping, a container run), readiness (a peer's declared readiness wait) or checker. Only a checker failure says anything about the feature: a helper that cannot start fails every scenario in setup or readiness. The scenario's result carries the same word in its phase field, which a pass omits, and the terminal report prints name: FAIL in <phase>: <error>. A pass line is unchanged, and the : FAIL ( marker and a checker's scenario <name> assertion N: text are both where the RFC recorder reads them. runScenario sets the phase as the run advances and copies it onto a failed result, so no error text is matched.

When three scenarios in a row fail in setup or readiness, the suite stops. A pass or a checker failure resets the count, so a run of feature reds always runs to the end. The report lists every scenario after the stop under not-run, counted neither passed nor failed, and stopped says why, with the fix: correct the setup cause the failures name, then run the lab again with the ./le action its catalog registers (Catalog.Action). The summary line reads interop: 0 passed, 5 failed, 2 not run: <failed names>.

Each local image build also creates a unique, run-owned tag. This keeps its image ID available when another build replaces the shared cache tag. The suite removes its tags after the scenarios, including after setup failure. Image cleanup errors appear in cleanup-errors and make the suite fail.

Scenario cleanup errors also make the suite exit unsuccessfully. They appear in progress, terminal reports, and the scenario's cleanup-errors field. Cleanup does not change the checker verdict, passed/failed counts, or failed scenario names.

A failed scenario reads every container it started before the teardown removes them, and its result carries one block per container in diagnostics, which the terminal report prints under the verdict: the state docker inspect reports (status, exit code, OOM kill, runtime error), its container log, and, while the container still runs, ip -d -o link show and ip -br address show from its namespace. A daemon that logs to a file rather than its stdout (accel-ppp's [log] log-file, xl2tpd's redirected stderr, a pppd logfile) leaves the container log empty, so the scenario that configures the file declares it on its peer (PeerConfig.LogFiles, a topology node's logFiles), and the block then carries tail -n 40 of each declared file. A checker's verdict names what it waited for; the blocks name why it never came, such as a daemon that exited at start or a link device that is missing. A read that fails is printed in place of its answer, and a read whose program the image lacks (docker exec exit 127, as ip in the pulled FreeRADIUS image) is printed as one not read: the image carries no <program> line. The container log is read whole (docker logs --tail all) and printed whole when it holds at most 80 lines; a longer one is printed as its first 40 lines, the error lines among the lines cut (a Ze level=ERROR record, the error: line ze prints before it exits, a Go panic:; at most 20, with their count), and its last 40 lines. A daemon that exits at startup prints its cause near the top and a cascade of follow-on failures after it, so a tail alone can hide the cause.

One ./le test integration <action> invocation returns that action's full report. Multiple action names return a combined summary.

The image-retention regression, TestDockerBuildRetainsImageAcrossRetag, carries the build tag docker, so ./le test integration docker-peers runs it against the host Docker daemon with ZE_REQUIRE_DOCKER=1. Run it alone through native admission:

ZE_REQUIRE_DOCKER=1 CGO_ENABLED=0 ./le --name image-retention job run label image-retention command \
  go test -tags docker -count=1 ./internal/le/interoplab \
  -run '^TestDockerBuildRetainsImageAcrossRetag$'

Neither it nor TestKilledLabRunIsCleanedByNextStart sets a bound of its own. Each build takes the wrapper's machine bound, and the test's context ends ten seconds before the deadline that go test -timeout sets. A one-minute build bound once failed the retention test under docker-peers, which runs every docker package in parallel.

internal/le/interoplab stays out of the QEMU guest population because of TestDockerKernelCheckOnThisHost alone: it judges the kernel of the daemon the host uses, which a guest cannot stand in for.

Daemons start conditionally: FRR if frr.conf exists, BIRD if bird.conf exists, GoBGP if gobgp.toml exists. This means each scenario only runs the daemons it needs. BIRD starts with IPv6 enabled, as FRR always does, only when rendering wrote a lab IPv6 address into its bird.conf: the base prefix fd00:1e:0:: replaced by the lab network's, or a link-local placeholder. The render records each file it wrote one into (scenarioRendering), and the decision reads that record rather than the rendered text, which names the lab's prefix and not the base one. A ze-peer.conf starts a second Ze daemon, the peer ze-peer, where no third-party daemon can serve the scenario (ospf-multiaf-v4-frr: FRR ospf6d has no IPv4-unicast address family). It starts the way the Ze under test does (zeDaemonPeer), with the CLI user the checkers log in as appended to its config, and takes FRR's host 3, so a scenario carrying both frr.conf and ze-peer.conf is refused.

A scenario directory is NAMED, never numbered (owner directive, 2026-08-24). Run order is lexical by scenario directory name and no scenario depends on it. The rule and its reasoning live in ai/rules/interop-and-goal-validation.md, which is always-on, so a spec author planning a scenario meets it without opening this page.

The exact grant-set check

Each lab's privilege test (TestL2TPLabPeersRunWithoutPrivilege and its PPPoE, IPsec and RSVP-TE siblings, the deployment proofs' container tests and the release-candidate container test) compares a container's whole grant set with the set its daemon needs, through internal/le/interoplab/grantcheck. A check that only looked for the wanted grants would pass a container that also held --cap-add ALL or --privileged; comparing the whole set fails on any extra.

grantcheck.Parse reads a docker run argv, or a lab peer's extra arguments, in every spelling Docker reads (--flag value, --flag=value, a short flag with its value joined or separate, combined short booleans). A flag that can widen what the container may do fills one list: capabilities, devices, security options, sysctls, host namespaces, networks, device cgroup rules, mounts, or other grants (--user, --volumes-from, --runtime and the like). --privileged counts in any spelling Docker reads as true. A flag classified as no grant (--name, --env, --rm...) is skipped. Every other flag is unclassified, and grantcheck.Problems reports each one whatever the wanted set, so a flag nobody classified fails the check rather than passing unread: classifying it is an edit to grantFlags or nonGrantFlags. A leading docker is skipped, reading stops at -- or at the image, and an argv that opens with any other word is refused, because reading would stop there before a single flag.

grantcheck.ParsePeer adds a lab peer's declared capabilities, its AppArmor profile and its mounts. A read-only mount of a file whose source, symlinks resolved, lies strictly inside the checkout or the system temporary directory (grantcheck.StagedRoots) is the lab staging its daemon's own config and is no grant. A writable mount, a directory, a unix socket, a root itself, a source whose symlinks do not resolve, or a source anywhere else is a mount grant the want must name. Every lab mounts files only, and a directory is refused because what it holds, a socket included, reaches the container unread.

Container Addresses

Daemon IP Container
Ze 172.30.0.2 ze-iop-ze-<pid>
FRR 172.30.0.3 ze-iop-frr-<pid>
BIRD 172.30.0.4 ze-iop-bird-<pid>
GoBGP 172.30.0.5 ze-iop-gobgp-<pid>
Raw injector 172.30.0.9 ze-iop-inject-<pid>
Compiled strict speaker 172.30.0.10 ze-iop-speaker-<pid>
Compiled strict speaker (2nd) 172.30.0.11 ze-iop-speaker2-<pid>
StayRTR 172.30.0.12 ze-iop-stayrtr-<pid>
pmacct pmbmpd 172.30.0.13 ze-iop-pmacct-<pid>
StayRTR standby cache 172.30.0.14 ze-iop-stayrtr-standby-<pid>
Second FRR (frr-sink.conf) 172.30.0.15 ze-iop-frr-sink-<pid>

Container names include the runner PID, which prevents name collisions. Runs that share a fixed subnet or staged binary paths must run serially.

The table gives the fixture addresses, which the runner rewrites onto its selected network. IPv6 scenarios also receive host addresses on a selected /64.

A scenario file may carry a link-local placeholder. Each one renders to an fe80:: address whose interface identifier holds the first six bytes of the selected IPv6 lab prefix and, in its last byte, the host number of the container that owns it:

Placeholder Owner Host
@ZE_LINK_LOCAL@ ze 2
@FRR_LINK_LOCAL@ frr 3
@FRR_SINK_LINK_LOCAL@ frr-sink 15

The placeholders, their hosts and the renderer are declared once, in internal/test/labconfig (LinkLocals, RenderLinkLocals), outside internal/le so the repository config sweep (TestRepositoryConfigsValidate) renders a scenario's ze.conf exactly as the lab does before validating it. The IPsec lab's %%PKI_B64:<file>%% placeholder renders there too (RenderPKIPlaceholders, reading the scenario's pki directory or the shared test/interop-ipsec/pki, FindPKIDir).

When any file of the scenario carries a placeholder, its owner's start is wrapped: /bin/sh assigns the address to eth0 with nodad, then execs the start it replaced (Ze's ze, or FRR's image entrypoint /sbin/tini with its command -- /usr/lib/frr/docker-start). The daemon's first interface snapshot therefore already holds the address. A placeholder also selects an IPv6 lab network, because the address is built from its prefix. So does a ze.conf that opens an address-family block inside its ospf block, in either spelling the parser accepts (address-family ipv6 { ... } or address-family { ipv6 { ... } }), because every family there is an OSPFv3 instance sending from an IPv6 link-local, and Docker disables IPv6 on eth0 of a network without it (zeRunsOSPFv3 reads block paths, so a comment selects nothing). The scenario's name decides nothing.

A scenario's prestart is Go, not a scenario file: its register file adds an ownerPrestarts entry, one shell fragment per link-local owner, to scenarioPrestarts under the scenario's name. The interop tree carries no interpreted helper (TestInteropTreeUsesOnlyCompiledHelpers), so the harness compiled into le holds it. The fragment's link-local placeholders render as they do in the scenario's files, and it runs inside that owner's wrapper, under sh -e, after the scenario's links (below) and the eth0 assignment, and before the image's own start. It is where a scenario builds what must exist before its daemon sends a packet: a second link, a link-local on it, an nft table. A prestart for an owner whose placeholder no scenario file carries would never run, so preparation refuses it by name rather than skipping it.

A scenario that needs an extra lab address on a peer's eth0 before the daemon starts declares it in scenarioHostAddresses, as a peer and a host number. addHostAddresses renders the address on the run's subnet and puts its ip address add at the head of that peer's prestart, and refuses a declaration that names a peer the scenario does not start. ldp-backoff-clear-frr declares ze's .250 transport address there.

bgp-rfc2545-linklocal-nexthop-frr uses IPv6 host 2 for Ze and host 3 for FRR: the speaker, its on-link global next hop and the recipient share ONE IPv6 subnet, not separate IPv4 and IPv6 connected prefixes. Its ze.conf carries @ZE_LINK_LOCAL@, so the advertised link-local next hop belongs to the same router as the global address. The checker requires FRR to decode the global-plus-link-local pair and install the on-link route via that owned link-local address. A second prefix uses an off-link global next hop and must decode as global-only; the IPv6 session must remain established after both assertions. These are two routes on one session, not identical UPDATEs differing only in their next-hop length.

bgp-rfc2545-third-party-linklocal-frr is the same sentence for a next hop that names another router. A second FRR, frr-sink (IPv6 host 15), runs no BGP and carries no link-local placeholder, so it owns exactly the one link-local its kernel gave eth0. The checker reads that address from the container's /proc/net/if_inet6 and its MAC from /sys/class/net/eth0/address, then pings frr-sink's global and link-local from Ze's container, so Ze's kernel learns both by Neighbor Discovery. It requires ip -j -6 neigh show dev eth0 in Ze's container to bind both addresses to that MAC, with no second link-local on it: any other table is a lab that cannot discriminate and fails there, not later. It then runs send bgp * update text ... nhop <frr-sink's global> through ze cli and requires FRR to decode frr-sink's global then frr-sink's link-local, and to install the route through that link-local. Ze's ze.conf sets no link-local leaf, so the second address can only come from Ze's neighbor table. The config is BGP-only, with no interface block, so the scenario also proves that Ze follows the table when no interface monitor runs: frr-sink resolves after the session is up, and only the reactor's neighbor table watch makes Ze read the table again. The announce is resent each round of the 60 second wait, because the watch signal that refreshes Ze's table read can race the first send; a resend is an implicit replace, and a Ze that never adds the router's link-local fails every round.

Typed checker queries and their expected JSON field values are rendered onto the same selected network. Rendering clones the expected-field map so concurrent or later runs cannot change the registered scenario's addresses. In inject.msg, literal UPDATE sends also receive the selected lab address in IPv4 NEXT_HOP and MP_REACH next-hop fields, including the address after a VPN RD. The renderer validates the envelope and attribute sequence before changing either field. RDs, IPv6 addresses, NLRI and unrelated attributes remain intact. Malformed framing and duplicate attribute codes remain unchanged, preserving negative protocol fixtures rather than repairing them.

The IPsec, L2TP, PPPoE and RADIUS labs mount only Ze's input configuration file read-only. Its parent /etc/ze belongs to the container and remains writable, so start /etc/ze/ze.conf creates its database tree there without writing to the scenario source directory. Each new container gets its own tree.

The deployment L2TP PPP proof uses a private /tmp directory owned by its invoking user and writes Ze's explicit configuration beneath ze/. VPP stages each input from the host scratch mount into /run/ze/<scenario-config>/ inside the container, where its explicit ze.conf and database tree live together. The root daemon cannot use a database directory beneath a mount owned by another user. A scenario's disable-and-restart check recopies the new input into the same private directory before each start.

After the native L2TP PPP proof has negotiated IPCP and carried traffic, it injects an LCP Configure-Request from the peer namespace into the existing kernel L2TP session. The packet uses that session's live tunnel/session IDs and UDP endpoints. RFC 2661 section 3.1 says, “Tunnel ID in each message is that of the intended recipient, not the sender,” and gives the same rule for Session ID. RFC 1661 section 3.4 says, “The receipt of the LCP Configure-Request causes a return to the Link Establishment phase from the Network-Layer Protocol phase or Authentication phase.” Its section 4.1 specifies tld,scr,sca/8 for an acceptable request in Opened. The independent xl2tpd/pppd peer then renegotiates LCP and IPCP; a credentialed input must also complete a fresh authentication exchange. ZE_L2TP_PPP_SCENARIO=no-auth is the default. Select chap-md5 to use the existing local credential handler and require CHAP-MD5 on both negotiations.

Each namespace must have exactly one kernel tunnel and one nonzero-ID data session. xl2tpd's kernel path also creates a tunnel-management PPPoL2TP socket: Linux pppol2tp_connect registers its local/peer session IDs as zero, without a PPP channel, and l2tp_nl_cmd_session_dump enumerates it alongside the data session. The proof validates this optional management record separately: both session IDs must be zero, both tunnel IDs must match the live tunnel, and no duplicate management record or second data session is accepted. Management presence is part of the before/after identity; its zero IDs never become injection targets.

The proof records the pppd log offset, Ze observation counts and both kernel transport identities before injection. Only later request/Ack exchanges, authentication when selected, route withdrawal/reinjection and address assignment can satisfy the restart assertion. Both namespaces must retain their original transport identities, recover their negotiated kernel addresses and deliver all three interface-bound ping replies in each direction. Finally, the peer leaves first and the proof requires another route withdrawal and ordinary kernel cleanup. Injection uses Python 3's standard-library raw IPv4 socket and requires CAP_NET_RAW in addition to the namespace/PPP privileges. The boundary JSON and peer log remain in the scratch directory on failure. Raw tunnel/session listings are also printed before the restart identity check returns, while both peers are still running. Later general diagnostics run after deferred peer/daemon cleanup; their empty kernel state or pppd SIGTERM must not be read as the state that caused an earlier assertion to fail.

The credentialed carrier then starts another xl2tpd/pppd peer with a wrong secret against the same daemon. It requires a matching CHAP Challenge/Response/Failure exchange, a local rejection, and session teardown. After the rejection boundary, neither the rejected peer's log nor new Ze observations can show network admission. PPP units, L2TP data sessions and subscriber routes must disappear before the proof stops the rejected peer. The control tunnel and its validated zero-ID management context may remain until that peer exits; they cannot stand in for a surviving PPP data session.

Every container sits on the lab bridge's untagged eth0, so a protocol that finds its neighbors by multicast (IS-IS, OSPF, PPPoE discovery) meets every daemon of the scenario there. A scenario that must keep peers apart declares links in the shared core, whatever suite it belongs to: ScenarioPlan.Links, one interoplab.Link per link with its name, its VLAN, an optional IPv4 prefix of at most /24, an optional IPv6 /64, and its members, which are the names of any planned peers, whatever image they run. A link with no prefix is a pure layer-2 segment, which a PPPoE session or a LAN that DHCP numbers needs. A member reaches its Nth link as eth<N>, N its ordinal among the links it joins in declaration order, at its host number on each prefix (the IPv6 address nodad), and the scenario's files name those devices and addresses. A checker reads them from CheckContext.Member rather than from a literal. Two fields cover an end the harness does not number. Devices gives a member, on that link only, the device name a file the harness does not write already names (a daemon configuration, a checker that takes the device down); the member's other links keep eth<N>, numbered with no gap. External names a planned peer that terminates the link in its own stack on no device the harness builds, as freeRouter does: rawInt.bin hands every frame of its container's eth0, tagged ones included, to the jar, whose own configuration holds the VLAN subinterface and its address. The external end joins no wrapper and holds no membership, so a link with one needs only one member.

runScenario builds one start wrapper per peer that joins a link or carries a PeerConfig.Prestart: the links first, then the prestart (the BGP suite's eth0 link-local and its scenario prestarts are such prestarts), then the image's own entrypoint and command, read once per image from the pinned image (docker image inspect), in exec form or shell form. The daemon's first hello therefore already goes out on its links. Before any container starts, the scenario is refused when a link cannot be built as declared: a VLAN outside 1 to 4094 or used twice, an IPv4 prefix longer than /24, an IPv6 prefix that is not a /64, fewer than two ends (an external end counts), a member named twice, a member or external end the scenario does not plan, an external end that is also a member, or a device name a member cannot hold (one for a peer that is not a member, anything but 1 to 15 lowercase letters and digits, eth<N>, lo, or a name the member already holds on another link), because that link would carry nothing and the checker would read a silent lab as the daemon's fault. Each refusal names the link, the fault and the fix. A member's image is probed once for sh and ip, and an image lacking either is refused naming its Dockerfile. A scenario with no links runs every daemon on eth0 alone.

isis-import-policy-frr declares two links: FRR-A (frr) meets Ze on VLAN 101 (10.101.0.0/24, Ze's eth1) and FRR-B (frr-sink) on VLAN 102 (10.102.0.0/24, Ze's eth2). FRR-A originates the hidden prefix 10.77.1.0/24 and the control 10.77.2.0/24; Ze's isis filter import rejects the first. The checker requires both adjacencies; the control in Ze's show rib and kernel FIB and the hidden prefix absent from both, read beside the control; both prefixes in Ze's show isis route, the SPF result the chain runs after (show isis database detail prints TLV values as hex, so it cannot be searched for a prefix); the JSON of show rib candidates prefix <p> with the hidden prefix's one IS-IS path hidden, hidden-by source, chain isis import, and the control's one path selected and not hidden; and last, FRR-B's show ip route <p> json with a selected IS-IS route for BOTH prefixes through Ze's 10.102.0.2. FRR-B's only neighbor is Ze, so that last assertion is the proof that Ze still floods the LSP whose prefix its own import chain hid.

ospf-import-policy-bird reads the same JSON per path: the tagged external 10.88.1.0/24 holds exactly an OSPF path at the distance 200 the modify set, not selected, and a selected static path at 150; the untagged 10.88.2.0/24 a selected OSPF path at 110 and a static path at 150. Every field is judged on its own path, so the static path's 150 cannot stand for the OSPF path's distance.

Each import scenario has an internal and a -forked system-RIB variant. The shared wrapper checks the daemon PID, configured owner's Running state and commands, then observes actual subprocess placement through procfs. The forked case requires one descendant running ze plugin rib; the internal case requires none. Both retain the original import, distance and flooding assertions.

Both OSPF variants also require BIRD's untagged 10.88.3.0/24 in Ze's SPF result and hidden centrally at distance 110. It must be absent from system and kernel routes beside the installed 10.88.2.0/24 control. The IS-IS wrapper reads both FRR versions and consumed image identities. The OSPF wrapper reads BIRD's version and image identity. Checker unit tests exercise these predicates, but do not establish a successful live peer run.

isis-import-policy-reload-frr and its -forked twin prove that a reload re-judges the IS-IS paths the Loc-RIB already holds. One FRR meets Ze on eth0 and originates the same two prefixes; ze.conf carries the same rejecting chain and ze-reload.conf is that file without it. The checker reads the hidden verdict as above, then a settled reading of Ze's show isis neighbor (each adjacency up) and show isis database (each LSP's level, ID, sequence number and checksum, lifetime left out), taken once three reads 10 s apart agree. It reloads into ze-reload.conf and waits for the hidden prefix's path to be selected and the prefix to reach show rib and the kernel FIB, then reloads back into ze.conf and waits for it to be hidden again at every layer. After each reload the settled reading must equal the first. FRR's config never changes and a sequence number only grows, so that equality proves no LSP was originated again and no adjacency went down: the change came from the reload's re-judging (locrib.PublishPolicy), not from a fresh advertisement. Both scenarios passed on 2026-11-14 and each went red that day under one removed line of dispatchReselected, on a rebuilt Ze image: the plain scenario at its relaxed reload with the re-admission dispatch removed (the prefix never reached show rib), the forked one at its tightened reload with the hide dispatch removed (the prefix stayed installed).

Run one scenario on Ze's Docker-host system with the native action below. Select each of isis-import-policy-frr, isis-import-policy-frr-forked, isis-import-policy-reload-frr, isis-import-policy-reload-frr-forked, ospf-import-policy-bird and ospf-import-policy-bird-forked separately. Retain isis-export-policy-frr and ospf-export-policy-frr as export controls.

./le test qemu docker-lab lab 'test integration interop' env 'INTEROP_SCENARIO=isis-import-policy-frr'

The action boots Ze's image, probes the guest Docker engine and runs the selected lab through its host runner. When validating uncommitted checker changes, use the launcher's --name form. Its inherited build identity keeps the child launcher on the same current-tree build. A registered scenario is a proof target until its live run and discrimination results are retained.

IGP export-policy scenarios

isis-export-policy-frr and ospf-export-policy-frr read an IGP export chain's verdicts and sets out of FRR, one JSON answer per assertion, so every absence sits beside the presences that prove the query saw Ze's routes.

In isis-export-policy-frr Ze redistributes three static routes into IS-IS under isis filter export { filters [ loopbacks tagged-loopback-metric ]; final accept; }: a prefix-list accepting only /32s, then a modify that sets metric 77 on tag 300. FRR's show ip route isis json must hold 10.66.0.1/32 through Ze at 87 (FRR's pinned link metric 10 plus 77), 10.66.0.2/32 at 110 (10 plus Ze's default 100), and no 10.66.1.0/24.

In ospf-export-policy-frr Ze redistributes three static routes into OSPFv2 under ospf filter export { filters [ ospf-export-refused ospf-export-tagged ]; final accept; }. FRR's show ip ospf database external json must hold exactly one LSA from Ze's router ID 10.255.0.2 for 10.67.1.0/24 at metric 50, E1, tag 200 (the modify's sets on the route tagged 100), one for 10.67.2.0/24 at the defaults (metric 20, E2, tag 0), and none for the rejected 10.67.3.0/24. The predicate collects every object carrying a linkStateId wherever FRR nests it, in a bounded iterative walk.

Scenario Structure

Each scenario is a directory under test/interop/scenarios/:

scenarios/bgp-ebgp-ipv4-frr/
  ze.conf        # Ze configuration (required)
  frr.conf       # FRR configuration (starts FRR container)

Every directory has one typed checker in internal/le/interoplab/bgp. The catalogue uses explicit operations for ordinary session, route, adjacency, log, and negative assertions, plus bespoke checkers for scenarios whose control flow cannot be represented as an ordered operation list.

Scenario-local register_*.go files can register bespoke checkers. The conditional-MED checker runs its four original baseline operations unchanged, then correlates collector events without adding an operation kind to the engine.

A bespoke checker is written in two halves. The body in check_rfc.go does the lab I/O and numbers each assertion, so a failure names the assertion that found it. The pure predicate in check_rfc_predicate.go takes the text or the decoded JSON a peer daemon produced, holds no lab handle, and decides. The split is what lets TestBespokeCheckerBranches drive both polarities of every predicate in seconds with no container.

A body MUST NOT call checkScenario (check_engine.go). That function answers scenario %s has no typed assertions for every name absent from scenarioOperations, and a bespoke name is absent from that table by design, so the call is an error that always fires and every line under it is unreachable.

Optional sidecars

A scenario directory may also carry files that start extra containers before Ze:

File Sidecar Purpose
inject.msg le test peer (raw injector, 172.30.0.9) Drive Ze with wire bytes no conforming daemon would emit. An optional inject-args file adds flags. Because the injector and Ze start before the peer daemons, an early route exercises Ze's replay-on-peer-up path.
speaker-args (and optional speaker2-args) le test interop-bgp speaker (172.30.0.10; second at 172.30.0.11) Dial Ze with an independent strict peer. The compiled speaker negotiates the requested families and ADD-PATH mode, frames BGP itself, applies the named native oracle, and writes a structured verdict to container logs. It catches wire output that Ze's own lenient decoder could accept.
vrps.json StayRTR (172.30.0.12:8282) Serve RPKI VRPs from a real third-party cache, so Ze is the RTR client of an implementation that is not its own. The typed checker asserts each per-prefix validation answer, not merely the RTR session.
vrps-standby.json A second StayRTR, stayrtr-standby (172.30.0.14:8282), same image Serve a different VRP set from a second cache, so the scenario can observe which cache Ze validates against as the preferred one stops and starts (rtr-stayrtr-preference).
dnsmasq.conf (and optional dnsmasq-b.conf) dnsmasq DHCPv4 server (172.30.0.20; second at 172.30.0.21). test/interop/Dockerfile.dnsmasq Serve Ze's DHCPv4 client from an implementation Ze did not write. The checkers of dhcpv4-renew-nak-dnsmasq and dhcpv4-rebind-two-servers-dnsmasq count the DHCP messages in dnsmasq's log-dhcp output and read Ze's eth0 addresses. The renew-NAK checker writes a reservation into the container's own /run/dhcp-hosts and reloads it with SIGHUP; the rebind checker stops the server that granted the lease.
knot.conf (with lab.example.zone) Knot DNS knot (172.30.0.40) and the LAN host dnsclient (172.30.0.42), both from dnsclient runs only sleep, holds NET_ADMIN for dhcpcd, and starts with net.ipv4.conf.default.promote_secondaries=1, because dhcpcd's Linux if_init writes that sysctl for eth0 unless it already reads 1 and /proc/sys is read-only in the container. test/interop/Dockerfile.knot Serve the lab zone lab.example. from an authoritative server Ze did not write, and query Ze's LAN DNS resolver from a separate host with Knot's kdig. The checkers (check_dns_resolver.go) parse kdig's default output (status, header flags, the answer RRset without TTLs) in check_dns_resolver_predicate.go. dns-resolver-dhcp-lan has dnsclient lease the one pool address 172.30.0.60 from Ze's DHCP server with dhcpcd, holds the dumped lease's domain_name_servers to exactly Ze's address, resolves through that server from the leased address, and requires REFUSED from the host's other address, outside allow-from
registered for dhcpv4-server-shared-segment-dhcpcd (register_dhcp_shared_segment.go) The LAN host dhcpclient (172.30.0.43) from sleep, with NET_ADMIN and promote_secondaries=1 as dnsclient; Ze's prestart adds to its eth0 beside the lab address. test/interop/Dockerfile.knot10.99.0.1/24 Serve two subnets of one shared network on one interface. The checker (check_dhcp_shared_segment.go) creates macvlan clients with fixed MACs on dhcpclient's eth0 and binds each with dhcpcd: the client reserved in the household subnet binds 172.30.0.20 and the unreserved client binds from the household pool, each naming Ze's lab address as server identifier; the printer first leases from the household pool under a stray MAC, then takes its own MAC back and reboots its lease, and must log a DHCPNAK and bind its reservation 10.99.0.20 from server identifier 10.99.0.1. dhcpcd's dhcp_start1 reboots from the saved lease of the interface name, and read_lease does not compare the lease's chaddr, which is what lets the stray lease be rebooted under the printer's MAC
unbound.conf Unbound unbound (172.30.0.41). test/interop/Dockerfile.unbound Put a real recursive resolver between Ze and Knot (stub-zone for lab.example.). dns-resolver-forward-unbound compares Ze's answer with Unbound's for an A RRset, a CNAME chain and a missing name; counts in Unbound's log-queries lines that Ze asked for www once across two client queries; requires the 30-record TXT RRset over TCP to equal Unbound's with Ze's upstream-queries truncated count at least one; and requires TC on Ze's own UDP reply to a 1232-octet buffer
kea-dhcp6.conf Kea DHCPv6 server kea-dhcp6 (IPv6 host 22 of the lab /64; the file's selects an IPv6 lab network). fd00:1e:0::test/interop/Dockerfile.kea Serve Ze's DHCPv6 client from an implementation Ze did not write. The checker of dhcpv6-client-renew-kea reads Kea's INFO log, one exchange per distinct transaction id (DHCP6_PACKET_RECEIVED, DHCP6_LEASE_ALLOC, DHCP6_LEASE_RENEW, and DHCP6_USING_SERVERID at each start), and samples Ze's eth0 every 2 s: the leased /128 must be present with a finite valid_lft of at most the 40 s lease at every sample. Over three lease periods Kea must see one Solicit and one Request, between 10 and 16 RENEW exchanges that each extended the lease, and no Rebind: a client that never renews, renews without extending its deadlines (it loses the address after one lease and solicits again) or renews at every chance fails one of those bounds. The pool is one address (host 0x50), so the lease is known before the run. The image deletes Kea's persisted server DUID on every start, while the memfile lease database stays in the container: the checker stops and starts Kea to change the server's identity under a live lease, so Ze's next RENEW names a server that no longer exists and only its REBIND is answered. Ze must hold the address for two lease periods after the stop, with no Solicit, and then renew with the restarted server. The image runs Kea on a copy of the mounted file (written at every start), because SIGHUP re-reads the file Kea started with. A scenario that also carries kea-dhcp6-moved.conf mounts it at: the checker of dhcpv6-client-confirm-kea copies it over the live file and sends SIGHUP to move Kea to another subnet under a running lease (DHCP6_DYNAMIC_RECONFIGURATION_SUCCESS marks the reload). That scenario's lease is long (T1 300 s), so no Renew falls inside the run, and the kea-dhcp6.packets logger runs at debug level 55, where DHCP6_RESPONSE_DATA prints each Reply's status code option. Phase one sets Ze's eth0 down and up: Kea must log one CONFIRM after the bind and answer it Success, with no second Solicit, no Renew and no Rebind, and Ze must hold the /128 on every 2 s sample for 20 s. Phase two reloads Kea onto and sets eth0 down and up again: Kea must answer the CONFIRM NotOnLink, then see one Solicit and one Request that leases, and within 60 s Ze's eth0 must hold that /128 and no longer hold the old address. A client that ignores the NotOnLink keeps the old address until T1, far past that bound. /run/kea/kea-dhcp6.conf/etc/kea/kea-dhcp6-moved.conffd00:1e:0:1::/64fd00:1e:0:1::50
pmbmpd.conf pmacct pmbmpd (172.30.0.13:1790) Read Ze's BMP stream with a collector Ze did not write. The file is also the selector: a scenario that carries it starts pmacct INSTEAD of Ze's own collector, because two collectors are two readings of one stream and only the third-party one is interop evidence. The typed checker greps pmacct's JSON msglog, so every needle is a field pmacct printed after decoding.

The ospf-te-frr and ospf-te-interas-frr scenarios keep their FRR adjacency, TED and flooded-LSA assertions and add a BGP-LS speaker sidecar. Ze exports its native OSPF database through bgp-ls-export; the collector records actual negotiated SAFI 71 UPDATEs, not injected substitutes. The checker reconstructs the final received inventory, including withdrawals, with the native UPDATE and BGP-LS decoders: the wire types and TLV codes come from the core codec internal/core/bgp/nlri/bgpls, and the attribute decode runs through the plugin's registered bgp-nlri-ls decoder. It requires exact Node and bidirectional Link auxiliary IPv4 Router-ID TLVs for ordinary TE, and local IPv4 plus remote IPv4/IPv6 IDs for the configured inter-AS link. It rejects missing, swapped, duplicate or invented local IPv6 IDs. The bounded capture must finish successfully and establish a BGP session; a historical matching UPDATE alone cannot pass.

The inter-AS source assertion reads structured show ospf database opaque-as output. It requires the configured remote AS and both ASBR IDs on one decoded link, no Link ID, and a matching live Type-11, opaque-type-6 header. FRR's AS-wide database must contain the same originator and LSA ID with matching checksum and length. Neither a display label nor an address elsewhere in the output proves that link. These source and flooding checks supplement, not replace, the collector's final BGP-LS link and auxiliary Router-ID assertions.

Both FRR fixtures explicitly configure capability opaque, separately from TE origination, so their DDs can negotiate the opaque LSAs needed by this proof. The native neighbor regression also keeps an O-clear peer: the production summary filter must still establish Full without advertising opaque headers to that peer. A capable interop fixture cannot replace that negative control or excuse an ExStart/SeqNumberMismatch loop.

The capture deadline bounds reads as well as the outer loop. Expiry while waiting for a new header is quiet completion; an unfinished header or body, EOF, notification or send failure cannot certify a complete capture. The same --duration deadline bounds the dial: while Ze refuses the connection or is unreachable, the speaker redials every 250 ms, and when the deadline passes it fails with the last dial error.

If either OSPF scenario fails, its checker captures both peers' interface and neighbor detail, Ze's packet/drop and NSM counters, and FRR's DD packet debug log before container teardown. The original Full-adjacency failure remains the result; diagnostic query failures are retained alongside it.

The RFC 7606 mixed-UPDATE scenario first observes 198.51.100.0/24 in FRR's BGP table and installed routing table. Only then does the checker send the injector's explicit release route. The injector withdraws that seed and announces 203.0.113.0/24 in one UPDATE, followed by a same-session fence route. FRR must remove the seed and install the announcement and fence through the injector's next hop. A byte-transparent relay independently checks the split withdrawal and announcement through that fence, including exactly one NEXT_HOP. FRR 10.7.1 discards a later duplicate attribute in bgp_attr_parse, so FRR installation alone cannot prove that sender-side wire property.

The typed-NLRI discard scenario requires the full requested speaker capture on a valid OPEN and Established session. Its independent wire assertion strictly decodes every reported UPDATE and requires exactly the injected type-2 MAC/IP Advertisement: RD 172.30.0.9:1, zero ESI and Ethernet tag, MAC 00:11:22:33:44:55, no IP address, and label 2. A different assigned type, altered retained route, withdrawal, malformed envelope, early EOF or missing OPEN cannot stand for successful discard of the unassigned type 99.

no-family-peer-eor-frr intentionally uses asymmetric OPENs: FRR advertises IPv4-unicast Multiprotocol capability, while Ze, configured with no family block, does not. The checker independently requires route arrival and that capability DTO, then matches FRR's complete native receive payload for Ze's exact address, IPv4 Unicast and VRF default. The default peer declaration is FRR 10.7.1 at commit f8c0b08dcb0c78f9e42b9b86ae70b049e4e617c1: bgp_update_receive recognizes the empty non-MP UPDATE and bgp_update_receive_eor emits the message. A table entry is not an EOR substitute.

The scenario's bgpd_options enables file logging before integrated configuration, so frr_run's native startup precedes the initial receive window. Only a decode after the sole startup is eligible, and that startup must name release 10.7.1, bare or with FRR's build suffix after an underscore: the lab image prints bgpd 10.7.1_git starting: . Another release, or a longer one such as 10.7.10, does not fence the log. The checker brackets the log read with bgpd PID and /proc start ticks, log device/inode, exact neighbor socket ports, Established state, one connection establishment and zero drops. Foreign peers, address substrings, other families/VRFs, send messages, pre-startup events and daemon/session/log replacement cannot certify this observation. An identity query the wait's end cuts short, before or after the log read, fails that probe and preserves the last complete observation for the EOR wait's verdict; an observed identity replacement still refuses immediately. The final Established check and identity comparison retain the original-session survival assertion.

These are checker/source contracts, not a live result. Current native GREEN, the actual Ze EOR-source cut producing RED despite route arrival, and restored GREEN remain separately recorded runtime obligations. They do not recover or replace missing historical originals.

A scenario that needs a test seam no shipped build carries holds a ze-le-daemon file. Its presence is the declaration: the ze container then starts /usr/local/lib/ze-le/ze, the daemon built with ze_le, under tini, instead of the shipped ze every other scenario runs, so no other verdict comes from a binary that differs from what ships. The checker's ze CLI commands still run the shipped ze. The file cannot be combined with ze-reload.conf, whose start runs the shipped ze through a shell, and the preparation refuses a scenario holding both (scenarioPeers). bgp-panic-cease is the one user: the ze_le reactor (session_panic_inject_le.go) panics on the next UPDATE from a peer once /run/ze-inject/bgp-read-panic-<peer address> exists, and removes the file first, so one trigger raises one panic. The checker arms it for FRR and has FRR redistribute a static route. It then requires that FRR's own neighbor JSON records a RECEIVED Cease with subcode 0 (lastResetDueTo, lastNotificationReason), that FRR counts two establishments and one drop, and that ze still counts one establishment and no drop toward BIRD, which still reports Established. In a shipped build injectReadPanic is an empty function (session_panic_inject.go, !ze_le).

bgp-manual-eor-requested-frr proves which End-of-RIB markers Ze sends a peer configured behavior { manual-eor true; }, against FRR 10.7.1. Its external process (le test interop-bgp process bgp-manual-eor-requested-frr manual-eor-requester) acts from inside its handler for the peer-UP event, which the engine awaits before it starts the initial sync (deliverPeerState, internal/component/bgp/server/events.go), and answers within 4 s, under the engine's 5 s delivery timeout. On the first session it announces 10.84.1.0/24 and asks for nothing: FRR must decode the route and, through a 10 s hold, no End-of-RIB. The checker then runs clear bgp <ze>. On the second session the process announces 10.84.2.0/24 and asks for the IPv4 unicast End-of-RIB with the session's queueing gate still closed: in the part of FRR's log written after the clear, FRR must decode the route and then exactly one End-of-RIB, held for 10 s, and its table must hold the route.

The checker reads FRR's own receive log rather than show bgp neighbor json. FRR's bgp_update_receive_eor (bgpd/bgp_packet.c at frr-10.7.1) logs rcvd End-of-RIB for IPv4 Unicast from <peer> in vrf default at informational level for every marker, before it tests PEER_STATUS_EOR_RECEIVED, so a second marker is logged too. bgp_update logs <peer> rcvd <prefix> under debug bgp updates in, and the order of the lines is the order FRR handled one connection's UPDATEs. The gracefulRestartInfo.endOfRibRecv flag is one bit, printed only for a peer whose Graceful Restart capability FRR received, so it can show neither a count nor an order, and the scenario negotiates no Graceful Restart.

It proves two things: an explicit request asked inside the sync is written, after the route, and the sync writes no marker on a manual-eor peer. It does not exercise a route-server replay's marker, because no replaying plugin is attached.

A scenario carrying frr.conf may also carry its own daemons file. That file names which FRR daemons run and what each one is started with, so a scenario needing a bgpd module (-M bmp for one that drives Ze's BMP receiver) carries its own copy instead of adding the module to every scenario in the suite. Without one, the shared test/interop/daemons is mounted.

It may also carry an frr-image file holding one image reference. Its FRR containers then start from that release instead of the suite's FRR (and FRR_IMAGE does not replace it), and the suite pulls each pinned release once. A scenario needs this when the behavior under test exists only in a later FRR than the suite's, or when the scenario must not follow an FRR_IMAGE override: bgp-linklocal-only-multihop-withdraw-frr needs FRR 10.4.1 or later, the first release here that negotiates the Link-Local Next Hop capability (77), because without it a different gate withholds the route and the scenario cannot go red. The five scenarios that carry one pin the same release as the suite, interoplab.FRRImage (every lab peer runs its latest stable release), so a run with FRR_IMAGE naming an older FRR still starts them from 10.7.1. A file naming no image, or two, is an error.

A scenario may carry a ze-sysctls file: whitespace-separated key=value assignments, each passed as --sysctl to Ze's container, so the runtime writes it into the container's own network namespace before ze starts. Docker mounts /proc/sys read-only, so a value ze would otherwise write itself is refused there. ospf-sr-anycast-nexthop-frr and ospf-sr-frr use it for net.mpls.platform_labels, which fib-kernel needs non-zero before it can install a label route. Only net.* keys are accepted; any other key, a token without = or an empty value refuses the scenario before a container starts.

bgp-linklocal-only-multihop-withdraw-frr exercises the off-connected-subnet gate on one lab network: FRR's session address is on its loopback (10.255.0.3), outside every subnet Ze is attached to, and the checker gives Ze a host route via FRR's adjacent interface. There is no intervening router. This proves the gate and FRR's received withdrawal, not a physically multihop path. test/plugin/linklocal-only-multihop-withdraw.ci supplies the separate routed IPv6-hop proof through a transit namespace. Ze's link scope reads the interface table, not the configured BGP TTL.

A scenario whose ze.conf runs the BMP sender (bgp bmp sender) and carries no pmbmpd.conf starts le test interop-bgp bmp-collector. The lab reads that from the parsed config, so a commented-out bmp block starts no collector, a receiver-only environment bmp starts none either, and a ze.conf that cannot be read or parsed fails the scenario's preparation, naming the file. Announcement and observer process plugins use le test interop-bgp process <scenario> <plugin>. These personalities are compiled into le; no interpreter or source mount is present in the Ze image.

The pinned StayRTR v0.6.4 image runs with -protocol 1 -enforce.version=true. Its default connection state advertises version 0 when rejecting Ze's initial version-2 query. Enforcement sets the connection version to 1 before that error, so Ze can reconnect at the advertised supported version. -refresh 10 is how often StayRTR re-reads its VRP file. The RTR timers it sends Ze in End of Data are -rtr.refresh 10 and -rtr.retry 10 (StayRTR's defaults are 3600 and 600 seconds), so Ze polls a cache every 10 seconds and asks a failed cache again after 10. Those 10 seconds apply only after a cache's first End of Data. Before it, RFC 8210 Section 6 has Ze retry a cache it never reached after the default Retry Interval of 600 seconds, as BIRD and rtrlib do, so a first query that fails costs that cache ten minutes, longer than any checker waits. Two lab defects made that first query fail. The image patches the first: unpatched v0.6.4 refuses Ze's version-2 query by queueing its Error Report and cancelling the connection at once, its send loop picks between the two at random, and Ze sometimes reads a bare close ("read header: EOF") with no version to retry at. The readiness probe (stayRTRReadyCommand) handles the second: StayRTR answers /rpki.json with status 200 and "roas":null before it reads its VRP file and binds the RTR port, so the probe also requires a prefix in the export and an open RTR port.

The rtr-stayrtr checker then requires two IPv4 and two IPv6 VRPs and checks the per-prefix validation answers. HTTP readiness alone proves no RTR transfer. A changed cache command requires an image rebuild; NO_BUILD=1 retains the old image. For a current result, rebuild the image and run INTEROP_SCENARIO=rtr-stayrtr ./le test integration interop. Read that run's VRP and validation checks; an archived result does not establish the current outcome.

Live RPKI policy reload

The rtr-stayrtr scenario also runs FRR as a source and BIRD as an export sink. It changes Invalid policy from reject to accept, then restores reject through SIGHUP. Each reload must advance the applied configuration generation once. The same Invalid route must change eligibility, selection, and BIRD export without a source UPDATE or route refresh. A Valid control route remains usable. All phases require the same complete four-VRP set from StayRTR.

Continuity uses FRR's connectionsEstablished counter while its peer is Established, plus Ze's connection, OPEN, UPDATE, and refresh counters. The FRR counter identifies a session within the configured peer object's lifetime, not across daemon restart or peer recreation. The checker does not use FRR's derived wall-clock epoch, which can change without a reconnect. BIRD's session identity, TCP endpoints, and Ze-side lifetime counters must also remain unchanged. RTR reconnection is permitted; route reannouncement is not. This carrier does not test disabling validation or rejecting a reload transaction.

RTR cache preference, failover and fail-back

The rtr-stayrtr-preference scenario gives Ze two StayRTR caches serving different VRP sets: the preferred cache at 172.30.0.12 (preference 10, from vrps.json) and a standby at 172.30.0.14 (preference 20, from vrps-standby.json). ze.conf lists the standby first, so a client that used the configuration order instead of the preference would sync the standby. FRR announces 9.59.0.0/24 and 10.59.0.0/16 from AS 65001, and Invalid routes are rejected: 9.59.0.0/24 is Valid only under the preferred set, and the standby set makes it Invalid. 10.59.0.0/16 is Valid under both and is the control.

The checker waits through three phases, each bounded at 90 seconds: the preferred cache in use, then the standby in use after it stops the preferred container, then the preferred cache again after it starts it. A phase holds when show bgp rpki status lists both caches with exactly one synced, the expected one; when show bgp rpki roa equals exactly that cache's set, so a union of both sets fails; and when BIRD holds 10.59.0.0/16 with path 65000 65001 and holds 9.59.0.0/24 under the preferred cache, or does not hold it under the standby. The predicate's accepted and refused shapes are unit-tested in TestRPKIPreferencePhaseError.

Results on 2026-10-27, INTEROP_SCENARIO=rtr-stayrtr-preference ./le test integration interop, each break built with GOFLAGS=-overlay and the image rebuilt by the harness:

Build Result
unmodified, before the StayRTR patch and readiness probe 3 passes, 2 failures in phase 1. Cause: StayRTR v0.6.4 sometimes dropped the Error Report that refuses Ze's version 2 query, so Ze read a bare close and waited the default 600-second Retry Interval, and the readiness probe answered before StayRTR served RTR (the StayRTR paragraph above)
unmodified, with the patched StayRTR image 10 passes in 10 runs, two loops in parallel
the preference sort reversed in startSessions red in phase 1: "show bgp rpki status names 172.30.0.14 as the cache in use, want 172.30.0.12". Rerun with the patched image: the same red, and Ze's log holds no transport error ("read header: EOF") and no refusal, so the red is the sort's
the preference sort made a no-op, before the config-order fix pass, because the cache list then came from ranging over the config's cache-server map, in random order. parseRPKIConfig now reads the operator's order, so the no-op break would follow ze.conf, which lists the standby first
cacheGroup.load refusing any cache more preferred than the holder (never fails back) phases 1 and 2 pass, red in phase 3: "fail-back to the preferred cache ... names 172.30.0.14 as the cache in use, want 172.30.0.12"

Live BFD profile edit

The bgp-bfd-profile-edit-live-frr scenario is the bfd-frr lab plus a reload. Ze's BGP peer opts into BFD under profile fast at 300ms and detect multiplier 3; ze-reload.conf edits that profile to 200ms and multiplier 5 and changes nothing else. Once FRR's BFD session is up at 300ms, the checker reloads Ze through SIGHUP (applyZeReload, the same reload rtr-stayrtr drives) while that session is Up. FRR must then report remote-transmit-interval 200, remote-receive-interval 200 and remote-detect-multiplier 5 in show bfd peer <ze> json, which it reads from Ze's packets. RFC 5880 Section 6.8.3 makes that change a Poll Sequence on an Up session.

Continuity compares four values read before and after the reload. Both BFD discriminators (id, and remote-id, Ze's discriminator) must stay the same, so the session was not rebuilt. The session-down counter of show bfd peer <ze> counters json must not move, so no flap went unseen between two samples. FRR's BGP connectionsEstablished must not move either. FRR's own intervals stay at the bfdd defaults throughout, so the remote fields are the only ones the reload can change.

bfd-link-local-frr runs no client protocol. Ze's bfd root pins one single-hop session to FRR's link-local on eth0, with Ze's own link-local as the source; FRR's bfd block names peer <Ze's link-local> local-address <FRR's link-local> interface eth0. Without local-address, bfdd 10.3.1 sources its packets from eth0's global address, so neither side matches the other's packets to its session and FRR stays down with remote-id 0. Both files carry both placeholders, so both containers own their address before the daemons start. The checker waits for FRR's show bfd peer <Ze's link-local> interface eth0 json to read up, then for Ze's show bfd sessions to hold an up session to FRR's address on eth0, and requires the discriminators to have crossed: FRR's remote-id equals Ze's local-discriminator, and Ze's remote-discriminator equals FRR's id. Each value exists only on the side that chose it until a packet carries it over, so neither clause holds on one direction of traffic.

The scenario is acceptance evidence. RFC 5881 Section 3 makes both sides Active, so once FRR answers, the session also converges through Your Discriminator, and the zone Ze reconciles on a first packet is not what the scenario can tell apart. The send side cannot fail here either: every session of the loop names eth0, so the loop's socket is bound to that device (resolveLoopDevices) and the kernel puts the packet on eth0 with no zone and no ifindex. Both reverts still passed the harness run on the Linux host (2026-10-20). The discrimination is test/bfd/bfd-first-packet-link-local.ci in the QEMU guest, and bfd-link-local-two-links-frr against FRR.

bfd-link-local-two-links-frr puts two links between Ze and FRR and ONE address pair on both. Each container's prestart builds eth1 (VLAN 101) and eth2 (VLAN 102) over eth0, and gives the container's link-local to both, so the address cannot tell the two sessions apart and only the link can. Ze names one session per link; FRR one peer per link. Two things make it discriminate:

  • Send. The sessions name two interfaces, so the loop's socket is bound to no device, and a packet reaches its link only through the zone sendZone puts on the destination or the ifindex pinsToLink asks for. Without either, the kernel refuses the write (sendmsg: invalid argument) and FRR hears nothing.
  • Receive. Ze's prestart installs an nft output rule in table inet bfdfirst that drops every BFD Control packet Ze sends with Your Discriminator zero. FRR then learns Ze's discriminator only from a packet Ze sends after selecting FRR's first packet by address and link, so without the zone reconciliation (reconcileZone) neither side leaves Down. Ze logs each dropped send as bfd send failed ... operation not permitted, which is the rule working.

The checker reads the links from Ze's own show bfd sessions (the interfaces of its sessions to FRR's link-local, at least two), then runs the one-link check on each: FRR's peer on that link Up, Ze's session on that link Up, and the discriminators crossed. On the Linux host (2026-10-20) the harness run passed with the code as it is and failed at assertion 2, FRR's peer on eth1 still down with remote-id 0, under each break: reconcileZone returning its key unchanged, and sendZone returning "" with pinsToLink returning false (Ze then logs sendmsg: invalid argument for every send).

Live attach and BFD opt-in without a session reset

Three scenarios change a running FRR peer's Ze config by SIGHUP and require FRR's BGP connectionsEstablished to stay where it was before the first reload, through every reload except the BFD opt-in removal.

Scenario Reloads What FRR must show
bgp-attach-receive-only-live-frr ze-reload.conf attaches adj-rib-in to the peer, receive-only exactly one more messageStats.routeRefreshRecv (one negotiated family, RFC 2918 Section 4), and Ze's show bgp adj-rib-in then holds the route FRR announced before the reload
bgp-bfd-enable-live-frr ze-reload.conf adds the peer's non-strict BFD opt-in, ze-reload-edit.conf edits its profile to 200ms and multiplier 5, ze.conf removes the opt-in BFD Up at 300ms after the add; the same discriminators and no session-down at 200ms after the edit; BFD Down with remote-diagnostic administratively down after the removal
bgp-bfd-dynamic-enable-live-frr ze-reload.conf adds the opt-in to the template of the group FRR joins as a dynamic peer, ze.conf removes it BFD Up at 300ms after the add, Down with remote-diagnostic administratively down after the removal

The attach checker first requires that adj-rib-in holds nothing for FRR, so the route it finds afterwards can only have come from the catch-up refresh. The BFD configs keep the bfd root in every step, because an opt-in names a profile of the bfd plugin.

The removal step asserts what Ze sends and stops there. Ze releases the BFD session with AdminDown, diagnostic 7 (RFC 5880 Section 6.8.16), and FRR 10.3.1 then resets BGP with Cease/BFD Down, although RFC 5882 Section 4.2 says "clients SHOULD NOT take any control protocol action" on a remote AdminDown. That reset is a recorded peer deviation, not a Ze failure, so the checker does not require BGP to survive the removal.

A scenario that reloads more than once names each further config ze-reload-<step>.conf. The harness mounts every such file in the ze container at /etc/ze/<file name>, beside ze-reload.conf at /etc/ze/reload.conf, and ze.conf stays at /etc/ze/bgp.conf, so a checker reloads back to the start with applyZeReload on that path.

RPKI route retention

INTEROP_SCENARIO=rpki-frr ./le test integration interop runs FRR as the BGP source for 9.43.0.0/24, 10.43.0.0/24, and 11.43.0.0/24. The checker requires an established session and all source routes in FRR. Ze must retain all received routes in Adj-RIB-In: Valid (1) and NotFound (2) are eligible; Invalid (3) has ineligible: true. Missing routes or missing eligibility fields fail the observation. The process observer and native checker use the same structural predicate and save the observed route objects in /tmp/rpki-check.json inside the Ze container.

This scenario exercises FRR-to-Ze BGP reception with RPKI eligibility decisions. It uses the same-repository le mock rpki RTR cache mock and does not establish independent RTR interoperability. The rtr-stayrtr scenario uses StayRTR for that purpose. The retention observation does not measure downstream export: FRR originates these routes, and there is no separate receiving peer.

The functional rpki-revalidate-late-sync carrier uses an observer-controlled RTR barrier. The mock waits for rpki-sync.release in the test's private working directory. Only after the observer reads a retained, eligible NotFound route and an empty VRP set does it create that file. The observer then requires the same received attribute, next-hop, and NLRI bytes with Invalid/ineligible state. Scheduling delays cannot let the cache synchronize before the initial observation.

The same private-directory barrier controls the RR and RS bgp-{rr,rs}-rpki-pending-withdraw-relay and bgp-{rr,rs}-rpki-revalidate-relay functional carriers. Each has its own wire expectation: pending-withdraw cases permit only the revalidated fence route; revalidation cases require the route on the destination's second connection, after its End of RIB. The RR expectation includes reflection attributes and the RS expectation has its own AS path. A successful RTR transfer alone proves none of these downstream outcomes. Each carrier needs its own live run and a discriminating route-behavior break, separately from the shared cwd break.

Prove a scenario discriminates

An interop scenario is evidence only if it goes RED when the behaviour it tests is broken. Before you rely on a new scenario, revert the fix, run the scenario, and confirm that it fails. Then restore the fix and confirm that it passes.

Let the harness build. Do NOT docker build -t ze-interop by hand and then run with NO_BUILD=1. A tag is shared by every run on the host. A build in another session rebinds that tag between yours and your container start. Your mutation run then measures a daemon you did not build, and that inverted a proof twice in one review on 2026-08-05. Quiet Docker.Build calls read the image ID from docker build -q. A caller supplying ImageBuild.Progress streams Docker's stdout and stderr, then obtains the immutable ID by inspecting its unique run-owned tag, never the shared cache tag. The suite pins every container of the run to that immutable ID. Quote that ID beside the result, because it names the binary the run measured.

The suite prints interop: image <name>: reference <ID> after each successful local build, before a peer starts. The default progress stream keeps that ID beside the scenario verdict in native job logs. Pulled images and NO_BUILD=1 do not print this build identity.

The GoBGP export-reload successor also records the actual image ID of each registered live peer container before reading versions or configuration. Its proof.json carries image-<peer> identities, including pulled peers whose suite build result still names a tag. CheckerLab.PeerImage, which the real Lab answers, inspects that container's consumed image and refuses missing or malformed IDs; it never substitutes the current shared tag. This records the current run's provenance and does not recover an earlier run's missing image identity. The same identification step requires the applied patch bytes inside that image to equal this checkout's patch, and validates its named SHA256 before reading configuration files. Neither an old image nor a version string supplies the patched producer's identity.

A scenario that passes either way (common when the peer must accept both the old and the new wire form) proves acceptance, not correctness. Say which one it proves in the spec's Goal Validation, and move the discrimination to a unit or mutation test that CAN fail.

A scenario added to ALREADY-WORKING code never had a red phase, so its discrimination is unproven until you force one. That is not TDD's red-then-green: a regression test and a scenario for existing behaviour both start green.

Six traps make a scenario pass whatever the code does. Check each by its tell before you call the scenario evidence:

Vacuity trap Why it passes anyway The tell
A scenario for a sender-side wire change whose receiver is obliged to accept any form (RFC 7606 Section 5.1: receivers accept any field combination) A conforming peer accepts the old and the new wire equally Reverting the sender change leaves the peer's routing table identical
A test asserting the ABSENCE of something (no log line, no allocation, no route) Deleting the mechanism leaves the same absence Ask what would still be absent if the code were removed
A test whose fixture is at an extreme (all fields set, maximum value) An off-by-one or a partial break still handles the extreme Boundary the fixture: test one below and one above
A test whose data reaches the peer by a DIFFERENT path than the one changed The unchanged path still delivers Trace which code path actually produces the asserted bytes. A scenario proving two peers are NOT merged is this trap when they are already kept apart by another fact: local-as-replace-as-partition passed with the group key's prepend zeroed until BIRD was made to advertise the RFC 8654 capability FRR advertises by default, because the two sessions differed in announceFacts.extended. Log each peer's key in the red build
An assertion whose clauses are all satisfied by ONE stimulus Each clause reads as an independent observation, and they are one observation written twice Name the single event that satisfies every clause. Then ask which clause a peer that did nothing would still satisfy
A counterfactual that fails at a step BEFORE the assertion it is meant to prove The run is red, so the proof looks discriminating, yet the assertion under test never ran Read which step the red run names. The Compose proof's break-volume failed at import, so its second-login checks had never been seen red until break-commit failed at second-login. internal/le/test/deployment/compose.go

The fifth trap is what the IPsec suite carried until 2026-09-04, and it is worth reading in full because the shape recurs. verifyTunnelTraffic (internal/le/interoplab/ipsec/helpers.go) pinged from Ze to strongSwan and passed when each peer's ESP byte counters advanced. Both clauses were satisfied by that ONE ping: Ze encrypting the echo request advances Ze's outbound SA, and strongSwan decrypting the same request advances strongSwan's inbound SA. Nothing required strongSwan to have encrypted anything toward Ze.

RFC 4301 Section 4.1 says why the aggregate could not discriminate:

An SA is a simplex "connection" that affords security services to the traffic carried by it.

A protected bidirectional flow is two SAs, and the RECEIVER chooses the SPI, so both peers name one direction by the same SPI value. Measured in psk-site-to-site: Ze holds src 172.28.0.2 dst 172.28.0.3 spi 0xc12fa7e3 and src 172.28.0.3 dst 172.28.0.2 spi 0xf008af63, and strongSwan holds those same two SPIs. A counter map keyed by SPI alone therefore folds the two peers' views of one direction into one entry. verifyESPDirections now reads each simplex SA by its own src/dst header and takes the set of directions its caller can claim, and it also refuses a ping whose % packet loss summary is missing or non-zero.

Which layer proves what

Each layer answers one question, and a question goes to the cheapest layer that can decide it.

Layer What it proves Where
Producer unit test Ze's product behaviour: the encoder, the FSM, the filter that makes the bytes The producing package's _test.go
Replay of a recording The checker's logic: what it concludes from the answers a real lab run gave it interoplab.ReplayScenario, from a test in the lab's package
One lab pass per scenario The whole path end to end, against the real peer daemon ./le test integration <lab>
Start probe A staged helper starts and answers ("The start probe" above) The lab's preflight

A lab pass that is red, and a checker change meant to turn it green, are not judged by rerunning the lab until it passes. Record the run, then judge the checker over the recording. A change to a checker whose scenario has a committed recording carries a replay test over it (ai/rules/points/testing/directives/replay-a-checker-change-over-its-recording.md). Replay never replaces the lab pass for product behaviour: a scenario still owes one green lab run end to end.

Recording and replaying a checker

Suite.Run hands every checker a recorder in front of the real Lab. The recorder forwards each CheckerLab call and writes it, in call order, as one JSON line to <session scratch>/interop-recordings/<lab>/<scenario>-<run id>.jsonl. The first line is a header: the format version, the lab, the scenario, and the network, links and memberships the CheckContext carried. Each call line holds the method, the peer, the argv, the NAMES of the environment variables the checker passed (never their values), the answer, the error text with its typed kind, and the exit code. The last line is the lab's verdict. The lab is the registered catalog whose scenario root holds the scenario, and the run id is the Docker client's.

The recorder and the replay lab implement CheckerLab and nothing more, so every lab capability a checker uses is a CheckerLab method, PeerImage included (recorded as peer-image, the image ID as its answer). A checker never type-asserts its Lab for a method outside that interface: the recorder in front of the real Lab does not carry it, and the checker would lose the capability in every lab run.

Each answer field is bounded at 1 MiB (recordingAnswerOctetsMax). A longer answer keeps its first MiB and is marked truncated, and an answer that is not UTF-8 text is marked not-text, because JSON cannot carry its octets unchanged.

A full run, with no selector, keeps only the failed scenarios' recordings. A run that selects fewer scenarios than its catalog holds keeps every one. A failed scenario names its recording in its diagnostics (interop: <scenario>: checker recording: <path>), and a kept passing recording is named on the progress stream. A run that cannot resolve a recording directory (no checkout, no session, no registered catalog) still runs, and says once interop: checker recordings are off: <reason>.

interoplab.ReplayScenario(ctx, checkout, path) opens a recording, finds the scenario's checker through its lab's registered catalog, and runs the checker against a lab that answers each call from the recording, in order, matched on the method, the peer, the argv, the environment names and the call's own parameter. It answers the lab's verdict beside the replayed one. It refuses, with a *ReplayRefusal naming the expected and the actual call, a call the recording does not hold at that position, a call past its end, a truncated or not-text answer, a recording of another version or with no verdict line, and a checker that returns before the recording's last call. The refusal says the checker must be judged by a lab run, and names the command that runs the scenario. A replay never guesses an answer, so a checker never passes on data the lab did not produce.

What replay can and cannot reproduce:

  • A checker that reaches the lab only through CheckerLab replays whole. The BGP relay checkers read their capture through CheckerLab.Query, so the recording already holds the bytes they judged. A checker that reads a file from its scenario directory or the checkout reads it again from the checkout.
  • A poll (interoplab.Wait) that ended on a ready answer replays the same probes in the same order, and each unready probe still waits its real interval. A poll that ended on its deadline made as many probes as its timeout allowed in the lab; replay answers at once, may start one more, and then refuses that call, or, when the checker's next call has the same argv, answers it with that call's answer. Judge a checker whose recording ends in a timed-out poll by a lab run.
  • Calls a checker makes from several goroutines are recorded in the order they finished, and a replay that issues them in another order refuses.

To commit a recording as a fixture, copy it from the session scratch into internal/le/interoplab/<lab>/testdata/<scenario>/, and review it before the commit like any fixture: it carries environment names only, but an answer can still print a secret, and such a recording is never committed. The BGP lab's TestLocalPrefStripReplaysTheLabRecording replays the recording of bgp-local-pref-strip-gobgp, and TestLDPBackoffClearReplaysTheLabRecording that of ldp-backoff-clear-frr.

IPsec diagnostic environment

The IPsec lab sets ZE_LOG=debug, the registered base logger key ze.log, on its Ze container. ZE_LOG_LEVEL names no effective base level and cannot enable the diagnostic records a checker observes. Scenario ze-env files remain additive: a more specific key such as ZE_LOG_IKE=info selects that subsystem's level without changing the shared default.

IPsec injected datagrams

out-of-sa-request-notified-over-sa proves RFC 7296 Section 2.21.4 against strongSwan. Once the PSK tunnel is up, strongSwan sends Ze, from its own address and port 4500, a header-only IKE datagram whose SPI pair names no IKE SA. rawprobe -p udp -x (test/interop-ipsec/rawprobe.c) writes it through a raw IP socket, as craftedProbe writes its probes. charon logs a parsed INFORMATIONAL request line only for a message it decrypted under an IKE SA, so a line carrying N(INVAL_IKE_SPI) shows that Ze's notice arrived protected over the existing SA. A copy marked as a response goes first and must draw no such line within five seconds, and the SA and its tunnel must survive both.

ipsec-auth-corrupt-response-strongswan proves RFC 7296 Section 2.4 with Ze as the initiator: forged IKE_AUTH responses that fail their integrity check do not end Ze's handshake. After a control establishment, strongSwan holds its own IKE_AUTH responses to Ze in an nftables table on its output hook, matching UDP source port 4500 and the Exchange Type octet (@th,240,8, 35). It then terminates the IKE SA, and Ze reconnects. Ze sends IKE_AUTH from port 4500 because it offers MOBIKE (mobikeAuthOffer), and charon answers along the path the request came in (build_response), so IKE_SA_INIT still passes on port 500. charon logs received retransmit of request with ID 1, retransmitting response for each retransmission Ze sends into the hold (process_message and is_retransmit in task_manager_v2.c, strongSwan 6.1.0). The checker then injects twenty IKE_AUTH responses under the live SPI pair, at Message ID 1, each with an Encrypted payload of filler. rawprobe's raw socket passes the same output hook as charon's socket, so the hold rule exempts the forgeries' UDP length. That socket also takes the XFRM output path, and charon has already installed its Child SA (selectors 0.0.0.0/0) when it answers into the hold, so without help the forgeries would leave as ESP under an SA Ze never installed and die in Ze's kernel (XfrmInNoStates). The checker adds an outbound XFRM policy with no template on strongSwan, for UDP toward Ze at priority 1, and removes it once the counter has risen and before the hold lifts. The selector names no port: rawprobe opens its raw socket without IP_HDRINCL, and raw_sendmsg (net/ipv4/raw.c) looks the flow up with the socket's protocol and both ports zero, so a selector on port 4500 never matches it. A failing run of this checker carries strongSwan's XFRM tables and nftables counters, Ze's XFRM and UDP counters, sockets and ze_ipsec series, and rawprobe's own report. ze_ipsec_error_notify_suppressed_total{reason="handshake-unauthenticated"} must rise by exactly twenty. The hold must lift before Ze's last retransmission, 32 seconds after its request (retransmitBase, retransmitBackoff and maxRetransmissions in fsm.go), and the checker refuses a run that took longer. Ze must then establish the IKE SA whose initiator SPI the forgeries named, and pass traffic over it. With the initiator fix reverted, the first forgery ends the handshake, so the counter stays still and Ze establishes a new SA under a new initiator SPI.

The outer DF bit and ICMP errors on a Child SA (RFC 4301)

tunnel-df-treatment proves RFC 4301 Section 8.1 on the wire. Ze initiates with df-bit clear (ze.conf), then the checker reloads it with df-bit copy (ze-copy.conf, the same file but for that leaf) and waits for the replacement Child SA. In each phase Ze pings strongSwan's inner address three times with -M do (inner DF set) and three times with -M dont (inner DF clear). strongSwan counts every ESP packet from Ze in an nftables table on its input hook, one rule per value of the outer header's DF bit (ip frag-off & 0x4000), so the bit is read as the peer receives it. Under clear all six packets must arrive with DF clear. Under copy each must carry its inner bit. Each burst must move exactly one counter by exactly three. show vpn ipsec dataplane sa must also report the phase's df-bit on Ze's outbound SA, with options-known true. iptables cannot read the bit on Ze's kernel: its only match for an arbitrary header bit is xt_u32, which the kernel does not build. So the strongSwan image carries nftables. df-bit set is not a phase, because every dataplane refuses it at commit.

icmp-error-payload-check proves RFC 4301 Section 6.2 for an ICMP error addressed to Ze itself. The checker turns forwarding on in strongSwan's container (the scenario sets writesPeerSysctl). It routes Ze's inner prefix there with the preferred source 10.9.9.9, which no selector covers. Ze then sends echo requests with TTL 1, and strongSwan answers each with Time Exceeded from 10.9.9.9. The consistent error embeds Ze's echo to 10.1.0.77, which crossed the SA. charon carries it over the SA because of icmp = yes (XFRM_POLICY_ICMP), and Ze must deliver it: Icmp.InTimeExcds moves in Ze's /proc/net/snmp, and ping prints the error. The inconsistent error embeds Ze's echo to 10.7.0.5, which left in the clear through ze.conf's outbound bypass. For that step the checker adds a strongSwan out policy for 10.9.9.9, bound to the SA's reqid. Ze must drop the error: Icmp.InTimeExcds stays still. In both steps strongSwan's Icmp.OutTimeExcds must move and Ze's inbound ESP byte counter must advance. Without that, a dropped error could be one that never reached Ze over the SA. A forwarded error is out of scope (LL-XFRM-ICMP-TRANSIT, docs/architecture/linux-limitations.md).

Each scenario owes a red run before it counts as evidence ("Prove a scenario discriminates"). Each break below is one line of Ze, and the harness rebuilds the Ze image so the break takes effect:

Scenario Break Expected red
tunnel-df-treatment xfrmDFFlag answers false for DFClear the clear phase's DF-set burst arrives with DF set
tunnel-df-treatment xfrmDFFlag answers true for DFCopy the copy phase's DF-set burst arrives with DF clear
icmp-error-payload-check xfrmStateFromParams no longer sets state.ICMP the consistent error arrives over the SA and is not delivered
icmp-error-payload-check xfrmPolicyFromParams widens the inbound protect policy's source selector to. 0.0.0.0/0 the inconsistent error is delivered

The ESP form fast path under a firewall rule

esp-form-firewall-drop proves that Ze's ESP form fast path never carries a peer's bare ESP past a firewall rule that drops ESP on the input path. The peers disagree about the ESP form as in esp-form-change: strongSwan initiates, floats IKE to UDP 4500 with no NAT, and still sends bare ESP, because build_i (src/libcharon/sa/ikev2/tasks/child_create.c, strongSwan 6.1.0) sets child.encap from COND_NAT_ANY. Ze's ze.conf holds one inet table whose input chain accepts by policy and drops protocol esp in the drop-esp term, so IKE on UDP 500 and 4500 passes. The fast path rewrites at tc ingress, ahead of every netfilter hook, so with that rule in the kernel it keeps its gate shut and the rule judges the bare ESP.

The checker first waits for show doctor | json to answer doctor-ipsec-esp-form-firewall naming ze_espdrop and drop-esp. A ping from Ze then proves the Child SA carries ESP toward strongSwan. strongSwan's ping to Ze must advance strongSwan's outbound SA, lose every echo request, leave Ze's inbound SA still, raise the drop-esp counter of show firewall ruleset espdrop | json by at least the echo requests, and leave Ze's XfrmInStateMismatch still, so no bare ESP reached XFRM through the complete path either.

Scenario Break Expected red
esp-form-firewall-drop writeGateLocked writes espFormFastPathGateOpen whatever it is given strongSwan's echo requests are rewritten past the rule and answered, and the drop-esp counter stays still

IPsec address movement

mobike-initiator moves Ze; mobike-responder moves strongSwan. Each native checker adds a new outer address, selects it as the route's source, then removes the old address without reloading either daemon or initiating another IKE SA.

The checker requires the same IKE and Child SPIs before and after movement, the new UDP 4500 endpoints, unchanged inner selectors, and exactly the original two directed XFRM states on each peer. It sends lossless pings in both directions and requires all four directed ESP byte and packet counters to advance. A replacement tunnel, a stale direction, or a one-way success does not satisfy the scenario.

The Docker host must support XFRM_MSG_MIGRATE_STATE. A kernel without atomic live ESP migration cannot run this scenario: Ze does not negotiate MOBIKE there. The checker does not convert that missing prerequisite into a pass, and the suite does not reach it on such a kernel: the Docker host kernel check refuses the run first, naming ipsec-mobike (CONFIG_XFRM_MIGRATE).

The IPsec NAT box

A scenario that carries nat.conf starts a third container, nat, at 172.28.0.5. It is keyed on that file exactly as strongSwan is keyed on swanctl.conf and FRR on frr.conf, and it is started FIRST so its addresses answer ARP before either daemon sends a datagram.

It exists because RFC 7296 Section 2.23.1 is only reachable across a REAL translation. natt-transport-inner-checksum and natt-tunnel-inner-checksum reach the UDP-encapsulated path through strongSwan's encap = yes, which fakes the NAT_DETECTION_SOURCE_IP hash with no middlebox on the path. The pre-NAT address and the observed address are therefore equal in those two, every Section 2.23.1 substitution is the identity, and its absence cannot show. That is the fourth vacuity trap above: the asserted bytes reach the peer by a path the mechanism under test does not touch.

nat.conf is one translation per line, <real> <public>. The box gives itself each public address as a SECONDARY address on eth0, then installs one DNAT rule in PREROUTING and one SNAT rule in POSTROUTING per line. Three consequences are load-bearing:

  • No peer needs a route. Both public addresses sit inside the lab bridge's own prefix, so a peer resolves them by ARP and the box answers. A NAT on its own segment would need a second Docker network, and interoplab.ScenarioPlan carries one NetworkSpec that the BGP suite shares.
  • Both addresses of a crossing datagram are rewritten, which is the two-NAT figure of Section 2.23.1 drawn with one box, and it is what makes all four NAT_DETECTION comparisons mismatch.
  • The rules carry no port and no protocol. IKE floats from UDP 500 to UDP 4500 mid exchange, so a port-scoped rule would translate the handshake and drop the ESP that follows.

Three of the scenarios that use it differ in one variable each. The two transport scenarios differ in ROLE, because the substitution has two producers and a single scenario cannot fail on one of them. real-nat-tunnel-control differs from them in MODE, and it does two jobs: it proves the substitution stays out of tunnel mode, and it makes a red transport scenario readable, because a broken topology reds it too. Its selectors are INNER addresses the box never sees, which is the one selector pair in this lab a translation cannot move.

Two more transport scenarios rekey across the box after an IKE SA rekey, one per role. real-nat-transport-rekey-after-ike-rekey has strongSwan rekey the IKE SA and then the transport Child SA, so Ze's responder judges a Child SA rekey on the successor IKE SA exactly as one before it. real-nat-transport-ze-initiator-rekey-after-ike-rekey has strongSwan rekey the IKE SA and Ze's own Child SA lifetime rekey the transport Child SA; it proves the working path only, because strongSwan's answer already names Ze's own addresses and the scenario passes with the fix broken. Its discriminating proof is two unit tests in internal/component/ike/engine: TestTransportChildRekeyAfterIKERekeySubstitutesBehindNAT/ze_rekeys at the rekey producers, and TestTransportChildRekeyAfterIKERekeyThroughSession/ze_rekeys through the session entry point.

The port column and the rebind step

A row may carry a third column, <real> <public> <port>: the public port the box maps that peer's UDP 4500 to. 0, 500 and 4500 are refused, because a mapping to any of them could not tell an encapsulation port taken from the IKE SA from a constant (RFC 3948 Section 2.1: "the Source Port and Destination Port MUST be the same as that used by IKE traffic,"). For such a row natSetupScript emits a port SNAT for UDP source port 4500 AHEAD of the row's address SNAT, and no DNAT: the peer initiates and conntrack reverses the replies. The other peer keeps its own address, so it is not behind a NAT. That is the precondition of the endpoint adoption the rebind step drives, and it is why the peer behind the port translation reaches the other one by a host route via its own public address, which the checker installs before it initiates (natPortEstablish).

The rebind step (natRebind) replaces the port rule with one to a second public port and deletes the box's UDP conntrack entries with conntrack -D -p udp (conntrack-tools in Dockerfile.nat). The flush is needed because a NAT rule applies to a flow's first packet only, and the peer's keepalives and DPD keep the old entry alive. The step fails when conntrack prints no deleted count, and when conntrack -L afterwards still holds a flow from strongSwan's UDP 4500 mapped to the first public port (conntrackDeletedCount, conntrackStaleMapping). A flush that did nothing then fails at once, rather than as a 90 s wait for a move that cannot come. The peer's next DPD then reaches Ze from the new port, Ze adopts it, and the checker waits for exactly one of Ze's four rebind log lines.

The two rebind scenarios set mobike = no in swanctl.conf. strongSwan offers MOBIKE by default. When MOBIKE is negotiated, Ze correctly refuses to adopt the new source of an authenticated IKE packet, because RFC 4555 Section 3.8 says "The host not behind a NAT MUST NOT use these dynamic updates for IKEv2 packets". The peer must send UPDATE_SA_ADDRESSES instead. Ze logs that refusal at Debug, so a scenario that loses the setting shows the reason in Ze's log.

Scenario Proves
real-nat-port-translated-ze-responder tunnel mode: outbound encap dport 14500, traffic both ways; after the remap the row 1 "migrated" line, dport 14501 on the SAME SPI, traffic both ways
real-nat-port-translated-transport-ze-responder transport mode: after the remap the row 2 "rekeyed" line with reason="transport mode", dport 14501 on a NEW SPI, traffic both ways
real-nat-port-translated-eap-ze-responder the EAP-TLS responder's first Child SA encapsulates to dport 14500
responder-eap-tls12 (no NAT) a strongSwan EAP-TLS client pinned to TLS 1.2 completes the TLS handshake through ze's Finished, ze refuses to export the RFC 5216 Section 2.3 MSK because the session has no RFC 7627 extended master secret, charon receives EAP-Failure, and neither end installs an XFRM state (checkResponderEAPTLS12)

None of the open-source IKEv2 implementations surveyed sends RFC 7627 from an EAP-TLS initiator on TLS 1.2: strongSwan through 6.1.0 defines the extension constant and never sends it, Libreswan has no EAP-TLS initiator, and OpenIKED has no EAP-TLS. So responder-eap-tls12 asserts the refusal, and the extended master secret stays required (owner decision, 2026-10-21).

All four have run in the nightly ipsec-interop suite. In the 2026-10-17 run the two rebind scenarios and responder-eap-tls12 were red, and real-nat-port-translated-eap-ze-responder passed, possibly vacuously (see the next sentence). On 2026-10-17 the two rebind scenarios got mobike = no. In the same change, waitZeOutboundEncap stopped accepting an outbound state at any port: it had read a drained textbuf.Buffer inside its poll and so matched "". No run since these fixes was recorded until 2026-10-21, when all four ran green in the Docker lab. The tunnel and transport rebind scenarios went red under their reverted fixes. responder-eap-tls12 went red when crypto/tls was patched to export keying material without RFC 7627: charon then established the IKE SA, and assertion 3 (EAP-Failure) timed out. The EAP port scenario sets mobike = no: with MOBIKE negotiated, Ze records the observed endpoint at the first IKE_AUTH (RFC 4555 Section 3.3, handleAuthRequest), so the EAP responder's own adoption never set the ports and removing it stayed green. With MOBIKE off, removing the adoption line in handleResponderEAP goes red: Ze refuses the IKE_AUTH Child SA because the peer has no authenticated IKE endpoint yet.

Two more scenarios reuse the tunnel-control topology to drive show mtu across a real NAT. mtu-tunnel-sizing-strongswan binds the tunnel to an xfrm interface, clamps the route the box forwards over to 1400 (ip route replace ... mtu 1400, which Linux honors when it forwards), and requires the measured path, the derived ceiling over the negotiated transform behind UDP encapsulation, the recommended interface MTU and its command; an iputils ping -M do at the old size MUST lose every packet and one at the recommended size MUST lose none. mtu-nat-installed-endpoint requires the peer measurement and the tunnel row to target the box's face, the address the Child SA carries ESP to, and never the peer's real address, where no ESP arrives. The ze lab image installs iputils for -M do, so every IPsec scenario's ping is the iputils one.

ike-padded-probe-strongswan reuses the sizing topology to drive the padded IKE path probe (docs/architecture/diagnostics/path-mtu.md, "The IKE prober") where ICMP cannot see the clamp. The checker keeps the 1400 clamp for UDP and ESP, routes ICMP past it through a policy-routing table (exemptICMPFromClamp, so an echo crosses at 1500), and drops every Fragmentation Needed the box would send Ze (dropTooBigToward). The ICMP search then measures 1500, the padded INFORMATIONAL asks 1500 on UDP/4500, its DF copy vanishes, its DF-clear retransmission crosses the box fragmented and strongSwan answers it, and the descent settles on 1400 with prober: ike. Each show mtu runs DETACHED (zeShowMTUDetached): a probe the peer never answers holds the request window for 30 s before the SA is deemed failed, longer than one docker exec is given, so the shell writes the document to a file and the checker polls for the marker. A second run races swanctl --rekey, and a third cuts strongSwan's reassembly memory (dropFragmentsAtPeer, net.ipv4.ipfrag_high_thresh) so no fragmented datagram is ever reassembled: an iptables -f rule matches nothing on either container, because conntrack defragments before any chain runs. The SA then fails exactly once and the row names sa-failed and the size. Before each lossless ping the checker waits until Ze lists the Child SA with its selectors (waitZeSelectors): after the SA is re-established Ze lists the IKE SA before it installs the Child SA, so a ping sent on the IKE state alone can meet no XFRM state.

The IPsec name server

A scenario that carries dns.conf starts a name server container, dns, at 172.28.0.8, keyed on that file as the NAT box is keyed on nat.conf. It exists so Ze's remote-address can be a DNS name: RFC 7296 Section 2.23 NAT detection must then hash over the address the name resolved to, which no scenario with a literal reaches.

dns.conf is one record per line, <name> <ipv4>. The lab refuses an empty file, a malformed line, a name given twice, an address outside the lab network and the server's own address, before any container starts (readDNSConfig). The image is Alpine with dnsmasq (Dockerfile.dns), built only when a planned scenario carries a dns.conf.

Two choices are load-bearing:

  • The container starts with no server running. A checker calls startDNS when its scenario wants the names to answer, which runs dnsmasq with one --host-record per line, no upstream and no hosts file, then waits until nslookup over the lab network returns every record. So a scenario can boot Ze while DNS is down and bring it up later, the home CPE's "boot before DNS" case.
  • Ze asks this server and nothing else. The lab mounts a resolv.conf naming 172.28.0.8 alone (options timeout:1 attempts:1) over Ze's /etc/resolv.conf, which replaces Docker's own file and its embedded resolver. While dnsmasq is down a query is refused at once, so the DNS-down phase measures Ze's backoff rather than a resolver timeout.

Ze's SPD catch-all DISCARD (vpn ipsec unmatched discard, the default) drops Ze's own DNS query like any other clear traffic, so each scenario's ze.conf carries a policy lab-dns bypass to 172.28.0.8 at order 1000, ahead of the tunnel's 2000. An operator with a DNS-named peer writes the same entry for their resolver (docs/guide/ipsec.md, "The catch-all entry").

Scenario Proves
fqdn-remote-dynamic-local Ze names no interface and no local-address (the route-chosen source) and dials vpn.example.net. With DNS down, Ze logs ike: connect cycle cannot start naming the name and "does not resolve", logs it again (the retry), and charon parses no IKE_SA_INIT. Once startDNS runs, the SA establishes; charon's parsed IKE_SA_INIT request and its generated response both carry N(NATD_S_IP) and N(NATD_D_IP); charon logs no NAT condition and Ze reports nat-detected false; charon received the request from 172.28.0.2 port 500, and the IKE SA sits on Ze's port 4500 (MOBIKE, RFC 4555); both ends hold the Child SA between 172.28.0.2 and 172.28.0.3, and traffic crosses it
fqdn-remote-real-nat real-nat-tunnel-control with remote-address vpn.example.net, answered with strongSwan's public 172.28.0.7. charon logs "local host is behind NAT" and "remote host is behind NAT", Ze reports the NAT verdict, the IKE SA sits on Ze's public 172.28.0.6 port 4500, Ze's inbound state carries encap type espinudp, and ESP crosses the box in all four directions

charon answers NAT detection only when the request carried both payloads (strongSwan 6.1.0 src/libcharon/sa/ikev2/tasks/ike_natd.c build_r: "if (this->src_seen && this->dst_seen)"), and it compares each received hash with the addresses it observed (process_payloads), setting COND_NAT_HERE on a destination mismatch and COND_NAT_THERE on a source mismatch, which ike_sa.c set_condition logs as the two "behind NAT" lines. Before spec-home-cpe-4, Ze built the payloads and checked the reply only when remote-address parsed as an IP, so a named remote sent none: the first scenario goes red on the parsed request and the second on charon's NAT lines and Ze's verdict.

IPv6 inside an IPv4 Child SA on an xfrm interface

vti-ipv6-inner-ipv4-outer is the home CPE's management path. Ze initiates over the plain IPv4 lab pair with two traffic-selector entries, 0.0.0.0/0 and then ::/0 on both sides, bound to vti0 (if-id 7), with fd00:db8:1::1/128 on vti0 and a static route for fd00:db8:1::/112 through it. strongSwan's child config names ::/0 alone, so it narrows the answer to ::/0 === ::/0: strongSwan 6.1.0's traffic_selector_list.c select_ intersects each configured selector with each received one, and traffic_selector.c get_subset answers NULL when the two types differ. Ze's unconfigured wildcard over an IPv4 outer pair would be 0.0.0.0/0 alone, which this child config cannot accept, so the established ::/0 child is itself proof that the IPv6 entry reached the wire. Ze's esp-group sets pfs enable, so strongSwan's esp_proposals name MODP_2048: the IKE_AUTH child carries no key exchange either way, but the rekey below does, because strongSwan 6.1.0's child_create.c build_i takes the child config's first key exchange method for a CREATE_CHILD_SA, and Ze's respondChildRekey refuses an offer with no group while PFS is on. With aes256gcm16 alone the rekey drew N(NO_PROP). The checker asserts:

Step State read Fails when
Selectors swanctl --list-sas, and the child of show vpn ipsec sa | json, each waited for, because Ze lists its Child SA only once the XFRM install is done (waitChildSelectors, waitZeSelectors) Either end holds anything but ::/0 on both halves, strongSwan holds an IPv4 selector, or Ze's child does not report if-id 7
Policies Ze's ip xfrm policy, the records on if_id 0x7 other than vti0's own final entry (action block at priority 4294967295, both families, isInterfaceSPDCatchAll) They are not exactly one ::/0 policy for each of out, in and fwd; any other IPv4 policy on the if_id, a BLOCK at another priority included, is refused
States Ze's ip xfrm state, the records on if_id 0x7 Fewer than two, or one without flag af-unspec (child 20's XFRM_STATE_AF_UNSPEC)
Datapath ip -6 route get fd00:db8:1::2 on Ze, a ping from, and the four directed ESP counters. fd00:db8:1::1 The route does not leave through vti0, the ping loses anything, or any simplex SA did not count bytes
Rekey The four rows above again after swanctl --rekey --child ze-child and a new SPI Any of them fails

The ping cannot pass outside the tunnel: xfrmi_xmit2 (net/xfrm/xfrm_interface_core.c) refuses a packet whose lookup returns no xfrm state, so vti0 has no clear exit. strongSwan's lab interface has IPv6 disabled on the IPv4-only network, and its policy-based tunnel decapsulates onto that interface, so the checker enables IPv6 there (writesPeerSysctl) and puts fd00:db8:1::2 on its loopback with a route through eth0. When the route or the ping fails, the error carries both peers' kernel state (vtiIPv6Evidence): the ping repeated once with its stderr, every XFRM policy rather than the if_id 0x7 records alone, ip -s xfrm state, /proc/net/xfrm_stat, IPv6 routes, rules and addresses, and disable_ipv6. The full policy dump matters because the first output lookup (xfrm_lookup_route, called from ip6_dst_lookup_flow) runs at if_id 0, before the packet reaches vti0, so a policy bound to no interface decides first. The red walk reverts child 20's state.AFUnspec = true in xfrmStateFromParams, rebuilds the Ze image, and must fail at the state row and the ping.

vti-unmatched-discard-interface-spd proves RFC 4301 Section 4.4.1's SPD selection function and each XFRM interface SPD's own final entry under unmatched discard. Ze binds one Child SA carrying 10.255.1.0/24 to 10.255.2.0/24 and fd00:db8:1::/64 to fd00:db8:2::/64 to vti0 (if_id 7); strongSwan's child names the same pairs, so select_ (traffic_selector_list.c) answers them unchanged. The checker requires IPv4 and IPv6 to cross the tunnel both ways, each Ze-initiated round trip counted by all four simplex SAs: Linux looks Ze's packet up at if_id 0 first, where the DISCARD catch-all would drop it, so only the selection entries hand it to vti0's SPD. It requires vti0's SPD to hold a BLOCK at priority 4294967295 in, out and fwd for both families. Last it pings 10.255.9.1 and fd00:db8:9::1, which ze.conf routes into vti0 outside every selector: every echo must be lost, vti0's tx_errors must rise (xfrmi_xmit counts the failed lookup), and XfrmOutPolBlock must rise, so the drop happened inside vti0's SPD rather than at if_id 0. The red walk rebuilds the Ze image without the exemptions (xfrmInterfaceExemptions answering nothing) and must fail at the first ping.

vti-ipv6-tunnel-source-no-cleartext-leak proves the tunnel-source guard from the WAN side. IPv6 is on for both lab interfaces because the scenario's lab network carries the IPv6 /64 fd00:db8:a::/64 beside its IPv4 subnet (networkIPv6 on its scenarioCheckers entry, leakWANLink), the route the BGP lab takes for OSPFv3. Docker disables IPv6 on eth0 of a container on an IPv4-only network, and a start-time --sysctl net.ipv6.conf.all.disable_ipv6=0 did not undo it, so Ze keeps a read-only /proc/sys and takes no --sysctl; Docker numbers the two interfaces fd00:db8:a::2 and fd00:db8:a::3. strongSwan's checker still writes its sysctls (writesPeerSysctl) for the tunnel side, and the checker reads disable_ipv6 back for Ze's eth0 before counting. The checker gives strongSwan the WAN-only destination 2001:db8:99::1 on lo, and routes it on Ze out of eth0 through a static neighbour with vti0's address fd00:db8:1::1 as the preferred source. Its observer is an ip6tables INPUT counter in strongSwan's namespace per source. Before counting, it requires the leak path to exist: the if_id 0 source BYPASS for fd00:db8:1::1, the guard's ip -6 rule from it, and the kernel choosing eth0 and fd00:db8:1::1 for the destination. A positive control, ICMPv6 from eth0's own address passed by the wan-control bypass, must reach the counter. Then echoes from the tunnel address, socket-chosen (ping -I, steered by the rule) and kernel-chosen (no source, dropped by the postrouting term), must leave the counter at zero. The red walk rebuilds the Ze image with the source exemption installed whether or not its guard holds (the !guarded[source] test in installXFRMInterfaceEntries answering false) and with no guard installed, and must fail at the zero count. Removing the guard alone is not that break: the source exemption waits for its guard, so the leak path itself would be missing and the run would stop at the leak-path check instead.

IPsec virtual-IP leases

Two scenarios prove Ze as the IRAS of RFC 7296 Section 2.19: a strongSwan client sends a CFG_REQUEST, and Ze answers with a lease from the remote-access pool its peer's virtual-ip-pool names. Ze's inner address is 10.20.0.1 in both, and the checker puts it on Ze's loopback.

Scenario Path Proves
virtual-ip-assignment EAP-MSCHAPv2, no NAT, vips = 0.0.0.0 charon reports an address inside 10.10.0.0/24, installed it, and narrowed its child to it. Ze's show vpn ipsec sa reports the same virtual-ip, and its Child SA's ts-remote is that address as a /32. Ze's kernel holds the /32 policy and no policy for the pool. charon wrote the pool's DNS server to. A ping from each end, from its inner address, moves ESP on all four simplex SAs. /etc/resolv.conf
virtual-ip-uniqueness PSK, two clients behind the NAT box, both vips = 10.1.2.3 RFC 3948 Section 5.1: the two leases differ, one of them is 10.1.2.3, and both sit in 10.1.2.0/24. Each Ze peer reports its own client's lease as virtual-ip and as ts-remote. Ze's kernel holds both /32 policies and no policy for the pool. A ping to each lease moves that client's four SAs and leaves the other client's outbound SA at Ze still

The uniqueness scenario needs two client sources in one strongSwan container. The checker adds 172.28.0.8/32 to strongSwan's eth0 before connection ze-b initiates. The scenario's nat.conf maps that address to 172.28.0.9 beside the two usual rows, so it has three translations, which is the most readNATConfig accepts (natTranslationsMax). Ze sees the clients at 172.28.0.7 and 172.28.0.9, one respond peer each, both bound to one pool.

Both scenarios went red on 2026-10-26 with the lease substitution removed (substituteVirtualIP in internal/component/ike/engine/cp.go returning its input, applied through a GOFLAGS=-overlay= build so the shared tree stayed untouched), and green again without it. In virtual-ip-assignment strongSwan still established, held 10.10.0.1 and narrowed its own side to it, so every client-side read passed. The failure was Ze's ts-remote, which named 10.10.0.0/24 where the lease /32 was wanted. In virtual-ip-uniqueness the second client never got a Child SA, because Ze could not install a second policy for 10.1.2.0/24 while swan-a held one. That refusal is the Section 5.1 hazard itself.

IPsec inner-address uniqueness without a pool

unbound-inner-address-uniqueness is the same two-laptop figure of RFC 3948 Section 5.1 with no pool to make the addresses unique. Ze's two respond peers bind no virtual-ip-pool and carry no traffic-selector, so each client chooses its own inner selector. Both strongSwan connections propose local_ts = 10.1.2.3/32 with no vips; ze-a asks for remote_ts = 10.20.0.1/32 and ze-b for remote_ts = 10.20.0.0/24. The two pairs overlap without being equal, so the dataplane's per-selector owner check, which keys on the exact selector, cannot refuse the second, and only the cross-peer guard can. The NAT box and the second client address are those of virtual-ip-uniqueness. The checker puts 10.1.2.3 on strongSwan's loopback and 10.20.0.1 on Ze's, each with a route to the other.

Step Proves
ze-a initiates strongSwan's Child SA is INSTALLED, and Ze's kernel holds exactly one outbound ESP policy to 10.1.2.3/32, tunneled to 172.28.0.7
ze-b initiates Ze's kernel still holds exactly the one outbound policy to 10.1.2.3/32, the first client's
strongSwan's answer strongSwan logs TS_UNACCEPTABLE and keeps IKE SA ze-b ESTABLISHED with no Child SA (RFC 7296 Sections 2.9 and 2.21.2)
Ze's view show vpn ipsec sa reports swan-a's Child SA with ts-remote 10.1.2.3/32 and swan-b established with no child-sa
Traffic a ping from 10.1.2.3 to 10.20.0.1 moves ESP on all four simplex SAs of the first client

The policy count reads ip xfrm policy one outbound policy at a time (innerOutPolicyTunnels), because the selector-pair set the other checkers use collapses two policies to one inner address into one entry.

The red run, on 2026-10-26: with the cross-peer guard reverted through a Go overlay and the Ze image rebuilt, the scenario FAILS at the ze-b step with "ze's kernel holds 2 outbound ESP policies to 10.1.2.3/32 (tunnels [172.28.0.9 172.28.0.7]), want exactly one". Ze accepts the second client's overlapping pair and installs it beside the first, so a packet to 10.1.2.3 could leave through either tunnel: the RFC 3948 Section 5.1 confusion itself. The kernel is judged before the notify, so that is the failure the run reports.

The strongSwan lab drop-in

Every scenario that starts a strongSwan peer mounts test/interop-ipsec/strongswan-lab.conf read-only at /etc/strongswan.d/98-lab.conf. It carries the charon settings the whole lab needs, and today that is two: charon.plugins.bypass-lan.load = no, and a stderr filelog with flush_line = yes at charon's default level.

charon loads bypass-lan by default, and that plugin installs a PASS shunt for every locally attached subnet. This lab puts both containers on 172.28.0.0/24, so the shunt covers the peer. Measured on 2026-08-30 in psk-site-to-site, the shunt sits at priority 175423 against the Child SA policy's 399999, the lower number wins, and every packet strongSwan sends to Ze leaves in the clear. A ping still succeeds under that shunt, which is exactly why a lossless ping is necessary and not sufficient evidence that a tunnel carried anything.

A scenario's own strongswan.conf still mounts at /etc/strongswan.d/99-interop.conf and composes with the lab file. A scenario MUST NOT set bypass-lan itself: TestNoScenarioCarriesItsOwnBypassLanOverride (test/interop-ipsec/parity_test.go) refuses a second copy, because two files setting one value is a disagreement with nothing to arbitrate it.

The logger exists because charon, with no logger configured, logs through a default stdout logger opened without line flushing (strongSwan 6.1.0 load_loggers, src/libcharon/daemon.c), and stdio fully buffers stdout on the pipe docker logs reads. A checker waiting for a charon line then waits on the buffer rather than on the exchange. Each scenario that read charon's log used to carry a private copy of this logger; mtu-negotiated-transform, which had none, timed out in evidence-nightly run 41298258262 waiting for a selected proposal: ESP: line the Child SA had already used. A scenario MUST NOT set flush_line itself (TestEveryStrongSwanPeerFlushesEachLogLine); it MAY add options to the same stderr logger, as child-rekey-refused-keeps-the-sa adds its timestamps. The native MOBIKE lab composes the same two files, the scenario's only when it exists (writeSwanConfig).

AES CCM on the strongSwan peer

ike-aes-ccm16 gives the IKE SA's Encrypted payload the transform RFC 5282 Section 7.2 numbers 16, AES CCM with a 16-octet ICV, and takes the Child SA to AES GCM because Ze refuses AES CCM for ESP at config parse (ipsec.EncryptionImplementedESP). The checker reads charon's own selected proposal: IKE:AES_CCM_16_256/ line, then Ze's encryption field, then ESP in both directions, so the agreement on the Transform ID and the verification of the encrypted ICV are separate observations.

The lab image carries that transform although /usr/lib/ipsec/plugins holds no libstrongswan-ccm.so. test/interop-ipsec/Dockerfile.strongswan builds strongSwan 6.1.0 from source with no --enable-ccm, so the standalone ccm plugin is absent, and two other plugins register the algorithm instead: on an x86-64 host swanctl --list-algs in the image answers AES_CCM_16[aesni], AES_CCM_12[aesni] and AES_CCM_8[aesni], and with aesni not loaded it answers AES_CCM_16[openssl], AES_CCM_12[openssl] and AES_CCM_8[openssl] (measured 2026-10-22). The aesni plugin is built only on x86, because its sources need -maes, -mpclmul and -mssse3, which gcc refuses on arm64; there openssl is the only provider. The absent FILE is not an absent capability, and a source build of the ccm plugin would add a second provider of one algorithm and nothing else.

Integrity-only ESP on the strongSwan peer

esp-integrity-only-null negotiates the Child SA as ENCR_NULL with HMAC-SHA2-256-128, the integrity-only ESP that RFC 4303 Section 1 requires to be negotiable, over an ordinary AES GCM IKE SA (RFC 7296 Section 5 keeps ENCR_NULL out of the IKE SA). Ze initiates and charon's esp_proposals = null-sha256 accepts nothing else. The checker reads charon's selected proposal: ESP:NULL/HMAC_SHA2_256_128/ line, then Ze's Child SA esp-encryption field (null), then both kernels' SAs and ESP in both directions. The ESP datapath is the Linux kernel's ecb(cipher_null) on both ends, which needs CONFIG_CRYPTO_NULL on the Docker host kernel.

The lab-wide strongswan-lab.conf sends charon's log to stderr with flush_line = yes, at the unchanged default level. StrongSwan's default stdout logger can retain the selected ESP proposal in its buffer after the Child SA is installed. The flushed logger makes that exact proposal observable without changing the negotiation, kernel-state or received-traffic assertions.

RFC 4739 multiple authentication on the strongSwan peer

multiple-auth-ze-initiator and multiple-auth-ze-responder run a second authentication round in each direction inside one IKE_AUTH phase. strongSwan takes its rounds from local-2 and remote-2, and charon announces the extension by default (ike_auth.c multiple_auth_enabled). Ze takes them from authentication second-round local and remote.

Scenario Ze role Round 1 Round 2
multiple-auth-ze-initiator initiator X.509 both ways (client.pem, server.pem) pre-shared key both ways, ze-user.example.com and swan-user.example.com
multiple-auth-ze-responder responder X.509 both ways (server.pem, client.pem) pre-shared key both ways, swan-user.example.com and ze-user.example.com

A tunnel coming up does not show that a second round ran, so each checker also reads both sides' logs. charon logs authentication of '<id>' with pre-shared key successful (psk_authenticator.c) for each of Ze's rounds it verifies, and Ze logs ike: second authentication round verified with the peer and id attributes on one line. Every round-2 identity is one that round 1 does not use, so the identity says which round a line belongs to.

The round-2 identities are FQDNs. Ze sends any identity that is not an address literal as ID_FQDN (encodeIKEID), while charon reads user@domain as ID_RFC822_ADDR, and the two types never match. charon matches a pre-shared key against an unordered list of owner identities (mem_cred.c), so the scenarios use one round-2 secret for both directions.

Two charon policies shape the responder scenario. charon's initiator runs the strict check of every configured remote round as soon as its current local round is a pre-shared key (ike_auth.c require_strict), so a strongSwan initiator whose round 1 is a PSK refuses a responder's first IKE_AUTH response with "insufficient authentication rounds" before the responder's second round can come; strongSwan's round 1 is therefore a certificate. And charon counts a configured round as done when a completed round complies with it (do_another_auth), so a local-2 identical to local is never sent, which is why strongSwan's second round is a pre-shared key rather than server.pem again. Ze's x509 second round is proven Ze to Ze (TestRFC4739ZeToZeTwoRoundsEachWay).

Each scenario was walked red: with multipleAuthRequestNotifies, settleAnotherAuth and moreAuthRounds reverted through an overlay, charon refuses the Ze initiator's single round ("insufficient authentication rounds"), and the Ze responder establishes after one round while charon still owes its second, so the strongSwan SA never comes up. Both scenarios set send_cert = always, and both flush charon's log to stderr the way esp-integrity-only-null does.

Legacy signature methods against a strongSwan without RFC 7427

The five x509-legacy-* scenarios set charon.signature_authentication = no, so strongSwan 6.1.0 neither sends SIGNATURE_HASH_ALGORITHMS nor reads Ze's (ike_init.c). Ze then receives no notify, as initiator or as responder, RFC 7427 Section 3 leaves it no method 14, and it must sign with its key's legacy method (RFC 8247 Section 3.1). charon signs its own AUTH the same way (pubkey_authenticator.c build).

Scenario Keys on both sides Method each side sends PKI
x509-legacy-rsa-signature RSA 2048 1, RSASSA-PKCS1-v1_5 over SHA-1 the scenario's pki/
x509-legacy-ecdsa-p256-signature ECDSA P-256 9, ECDSA-256 the shared lab PKI
x509-legacy-ecdsa-p384-signature ECDSA P-384 10, ECDSA-384 the scenario's pki/
x509-legacy-ecdsa-p521-signature ECDSA P-521 11, ECDSA-521 the scenario's pki/
x509-legacy-rsa-signature-ze-responder RSA 2048, strongSwan initiates 1, RSASSA-PKCS1-v1_5 over SHA-1 the scenario's pki/

Ze initiates in the first four and responds in the last. Each scenario's own pki/ holds leaf certificates issued by the shared lab CA. The checker requires the IKE and Child SAs, both kernels' ESP states and traffic both ways, then charon's two log lines: authentication of 'ze-test-client' with <method> successful for Ze's AUTH, and authentication of '172.28.0.3' (myself) with <method> successful for its own, which Ze verified before installing the Child SA. charon names a method 14 AUTH by signature scheme rather than by method, so a Ze that sent method 14 fails the first line.

Each scenario was walked red on 2026-10-31: with computeX509Auth reverted to refusing when no notify arrived, the rebuilt Ze logs ike: build IKE_AUTH failed and the strongSwan SA never comes up. The RSA scenario was also walked red with verifyLegacyRSAAuth reverted to SHA-256 only: Ze then refuses charon's SHA-1 method 1 AUTH.

Ed25519 signatures against strongSwan (RFC 8420)

Two scenarios run with RFC 7427 left on, so both sides send SIGNATURE_HASH_ALGORITHMS. strongSwan 6.1.0 announces Identity (5) because its openssl and curve25519 plugins each register an Identity hasher (ike_init.c send_supported_hash_algorithms), and Ze announces it in every IKE_SA_INIT. Each scenario's pki/ holds one Ed25519 leaf issued by the shared lab P-256 CA, beside copies of the shared CA and of the other side's P-256 leaf.

Scenario Ze's key strongSwan's key Who initiates charon line the checker requires
x509-ed25519-ze-initiator Ed25519 (client.pem) P-256 (server.pem) Ze authentication of 'ze-test-client' with ED25519 successful
x509-ed25519-ze-responder P-256 (client.pem) Ed25519 (server.pem) strongSwan authentication of '172.28.0.3' (myself) with ED25519 successful

Both checkers first require the IKE and Child SAs, both kernels' ESP states and traffic both ways. charon names a method 14 AUTH by its signature scheme (pubkey_authenticator.c: process logs the scheme it verified, build the scheme it signed with). In the initiator scenario the line is charon's proof that it verified Ze's Ed25519 signature. In the responder scenario it proves charon sent an Ed25519 AUTH, which it can only do when Ze announced Identity, and Ze installs the Child SA only after verifying that AUTH.

Not yet walked red. The break for the initiator scenario makes chooseEd25519Signature refuse as if the peer had not announced Identity: Ze then fails to build IKE_AUTH and no SA comes up. The break for the responder scenario makes verifyEd25519 refuse every signature: Ze answers AUTHENTICATION_FAILED and no Child SA is installed.

Ed448 signatures against strongSwan (RFC 8420)

The Ed448 twins of the two Ed25519 scenarios. strongSwan 6.1.0's openssl plugin provides Ed448 private and public keys, PRIVKEY_SIGN and PUBKEY_VERIFY for SIGN_ED448, and the Identity hasher (openssl_plugin.c, under OpenSSL 1.1.1 or later), and the lab image builds with --enable-openssl. The Ed448 leaves were minted with openssl genpkey -algorithm ed448 and issued by the shared lab P-256 CA. crypto/x509 parses such a leaf without reading its key, and Ze reads the key from the raw SubjectPublicKeyInfo.

Scenario Ze's key strongSwan's key Who initiates What the checker requires
x509-ed448-ze-initiator Ed448 (client.pem) P-256 (server.pem) Ze authentication of 'ze-test-client' with ED448 successful
x509-ed448-ze-responder P-256 (client.pem) Ed448 (server.pem), then Ed448 (server-untrusted.pem) strongSwan authentication of '172.28.0.3' (myself) with ED448 successful for ze; for untrusted, charon's received AUTHENTICATION_FAILED notify error, Ze's ike: initiator AUTH verification failed with peer=swan and certificate signed by unknown authority, and no new ESP state on Ze

The responder scenario's second connection, untrusted, carries the same identity with an Ed448 leaf from untrusted-ca.example.net (pki/untrusted-ca.pem), a CA Ze does not hold. The lab mounts server-untrusted.pem and server-untrusted-key.pem into strongSwan when a scenario's pki/ carries them. The checker initiates untrusted only after the trusted tunnel carries traffic.

Not yet walked red. The break for the initiator scenario makes chooseEd448Signature refuse as if the peer had not announced Identity. The break for the responder scenario makes verifyEd448 refuse every signature, which takes the trusted tunnel down. The break for the untrusted half makes getRemoteCert skip its cert.Verify chain check: untrusted then establishes, and Ze logs no refusal.

Peer certificate revocation and purpose (RFC 5280, RFC 4945)

Four scenarios, two pairs. Ze initiates with the lab P-256 leaf (client.pem) and checks the certificate strongSwan presents with send_cert = always. The two scenarios of a pair differ by one file in pki/, so that file is the only possible cause when one is refused and the other establishes.

Scenario The file that differs Expected
x509-crl-revoked-peer-ze-initiator crl of Ze's pki ca test-ca, lists the serial number of strongSwan's server.pem. pki/ca-crl.pem refused, was revoked by
x509-crl-unrevoked-peer-ze-initiator the same list, revoking nothing established
x509-eku-server-auth-only-refused-ze-initiator carries extendedKeyUsage serverAuth alone. pki/server.pem refused, extended key usage
x509-eku-ipsec-ike-ze-initiator carries extendedKeyUsage id-kp-ipsecIKE. pki/server.pem established

The two server.pem files of the purpose pair have the key, subject and subjectAltName of the lab server.pem, no keyUsage, and the lab CA as issuer. openssl issued them: strongSwan's pki --issue cannot write id-kp-ipsecIKE, because x509_generate_eku_extension (x509_cert.c) knows only serverAuth, clientAuth, iKEIntermediate, ocspSigning and msSmartcardLogon. charon does not judge the purpose of its own certificate: pubkey_authenticator.c checks keyUsage only in process, for the peer's AUTH (is_compliant_cert).

A refusal leaves strongSwan's SA in place. Ze sends no notification for a responder AUTH it refuses on the first round, and charon's responder holds an ESTABLISHED IKE SA from the moment it answered. So the refusing checker asserts nothing about strongSwan's SA list or kernel. It requires Ze's ike: remote AUTH verification failed line for peer=swan carrying the cause, charon's two pubkey_authenticator.c lines that show it verified Ze's AUTH and signed its own with the certificate under test, and no ESP state on Ze. The accepting checker requires Ze's ESP state and traffic both ways, which only an accepting Ze produces: the strongSwan SA alone would also be there after a refusal.

The red run, on 2026-11-10, after fbdd9a82e7: with an early return nil at the top of checkIKEPurpose and of checkPeerChainRevocation (internal/component/ike/engine/remote_cert_checks.go) and the Ze image rebuilt, both refusing scenarios FAIL with "ze did not refuse strongSwan's certificate: wait for ze log "ike: remote AUTH verification failed" timed out before the peer became ready": Ze established with the revoked certificate and with the serverAuth-only one. With the file restored byte-exact both pass again, and both accepting twins pass throughout.

ESP transform kernel proof

esp-aes-cbc-sha256, esp-chacha20poly1305 and esp-3des-sha1 offer one ESP combination to strongSwan. checkESPKernel requires the negotiated encryption and both kernels' installed transforms in both directions, then lossless traffic and increasing receiver counters. The IKE SA stays on AES-GCM/SHA256; the legacy 3DES/SHA1 choice applies only to ESP.

test deployment vpn-kernel selects these scenarios alongside the existing GCM and NULL scenarios, then runs the WireGuard kernel proof. test qemu vpn-kernel supplies the disposable Ze-system guest. The external scenarios exercise Ze-initiator establishment and bidirectional traffic, not both roles' rekey exchanges. Cleanup is part of the verdict: a scenario or image cleanup failure makes the VPN proof fail even when traffic passed. The report keeps the original Docker cleanup error beside that failure, rather than reporting traffic as a clean pass.

L2TP and PPPoE lifecycle fault scenarios

l2tp-unknown-mandatory-avp-xl2tpd relays xl2tpd's exchange without originating messages. An unknown optional AVP on SCCCN must permit PPP traffic. The relay then appends the same AVP with M=1 to a real HELLO. The checker requires a matching StopCCN with Result Code 2 and Error Code 8, route withdrawal and removal of both peers' PPP and kernel tunnel state. The native log retains the original and forwarded control datagrams. The native scenario passed, failed when a private build omitted HELLO AVP validation (no matching StopCCN 2/8), then passed after a normal rebuild.

ppp-code-reject-xl2tpd puts the same relay between xl2tpd and Ze, and it is the one scenario where the relay originates frames. No lab peer sends a PPP Code it does not know, so the relay injects them on the live L2TP session, and only after the checker creates /run/code-reject once PPP traffic has flowed. The next data message from the LAC is the trigger: the relay forwards it, then sends Ze four PPP packets behind a copy of its L2TP header, so they carry the live tunnel and session IDs. Two are LCP packets with Code 99 and two are IPCP packets with Code 12, all four with Identifier 0x42 and 204 octets long, against the MRU of 128 that pppd requests (mru 128). The relay refuses a data header carrying Ns and Nr, because a frame it injected would break the sequence. From the marker on, it logs every data datagram in both directions, unfiltered, and the checker refuses one it cannot read, so a frame the log holds is never passed over. The checker decodes the probes and Ze's answers from that log. It requires two LCP Code-Rejects and two IPCP Code-Rejects, the two of each protocol with different Identifiers (RFC 1661 Section 5.6: "The Identifier field MUST be changed for each Code-Reject sent."). The IPCP probes reach Ze on its PPP unit descriptor, not the channel: once NCP is up Ze connects the PPP channel to its unit, and Linux's ppp_input then queues an IPCP frame on the unit, so the IPCP answers prove Ze reads that descriptor. Each Code-Reject must have an Information field of at most 128 octets, and its Rejected-Packet must be the start of the probe, with the probe's Code, Identifier and Length unchanged (RFC 1661 Section 5.6: "The Rejected-Packet MUST be truncated to comply with the peer's established MRU."). The session must survive: a ping still crosses it, both PPP interfaces are the ones from before the probes, and neither peer sends an LCP or IPCP Configure or Terminate packet afterwards. Last, the checker creates /run/ipcp-terminate, the relay sends one IPCP Terminate-Request with Identifier 0x5a on the next LAC datagram, and Ze must answer with an IPCP Terminate-Ack carrying 0x5a (RFC 1661 Section 5.5: "Upon reception of a Terminate-Request, a Terminate-Ack MUST be transmitted."). That ends IPCP, so it runs after every other assertion.

pppoe-chap-rechallenge-ze-client observes two accel-ppp reauthentication exchanges after working IPCP traffic, checks each CHAP Response's Identifier and digest, and requires the session and connectivity to survive. pppoe-peer-terminate-ze-ac lets pppd originate LCP termination while filtering its outgoing PADT. Ze must send the correlated Terminate-Ack and its own delayed PADT, remove the session and release its kernel PPP state. Both scenarios passed, failed under isolated producer breaks, then passed after normal rebuilds. Removing network-phase CHAP handling left repeated Challenges unanswered; removing the Stopped lower-layer notification left no Ze PADT despite the peer's correlated Terminate-Request and Terminate-Ack. Shared product sources were not changed by these private build overlays.

The FreeRADIUS admin-login suite

internal/le/interoplab/radius/ runs ze's operator login against a real FreeRADIUS server at a pinned tag, pulled through ImageBuild{Pull: true}. It exists because every other RADIUS proof ze holds runs against a mock ze wrote: test/plugin/aaa-radius-admin.ci drives internal/test/mock/radius/radius.go, and the L2TP lab's peer is internal/le/interoplab/l2tp/radiusmock/, which is ze's own Go program in a container. A mock built beside ze's encoder agrees with ze by construction, and ze now computes a CHAP digest a server must reproduce from its own stored password. Only a server ze did not write can disagree.

It is its own suite rather than four more L2TP scenarios. The L2TP lab probes for the l2tp_ppp or pppol2tp kernel module and refuses to run without it, which is correct for a suite that carries PPP sessions. Admin login is ze's SSH listener, a UDP socket and a RADIUS server, so this lab declares no preflight beyond its own ze cross-compile, mounts no module tree, asks for no capability and runs nothing privileged. TestSuiteNeedsNoKernelModule holds that. The Docker host kernel check still applies: it is a rule about the host, so a kernel without PPPoL2TP refuses this suite as it refuses every other.

Every checker reads BOTH sides. Ze's log saying source=radius is not enough on its own, because a login the local bcrypt backend satisfied produces a line of the same shape and no server traffic at all. The lab therefore mounts a linelog module at /etc/raddb/mods-enabled/ze_request_log that writes one line per answered request to /var/log/freeradius/ze-request.log, carrying the verdict, the User-Name, the PRESENCE of a User-Password, of a CHAP-Password and of an EAP-Message, and the NAS-Identifier. Presence and not value: a fixture must not put a password or a digest in a log file. parseServerRecord refuses a line missing any of the six fields, so a truncated or reformatted line is never read as a partial verdict.

An EAP login is several requests and FreeRADIUS runs no post-auth section for an Access-Challenge, so the reply that asked the question records nothing. A second module, ze_state_echo_log, is called from authorize on a request carrying both an EAP-Message and a State, and writes verdict=state-echo. That is the server's own evidence that ze returned the State unmodified, which RFC 2865 Section 5.24 requires, and that the login was a conversation rather than one request.

Scenario Ze's side The server's side
radius-admin-pap-freeradius An operator logs in over ze's real SSH listener, ze's log says source=radius, the Filter-Id profile denies show bgp, and the local account's own password is refused verdict=accept with a User-Password present, a CHAP-Password absent and ze's NAS-Identifier, then verdict=reject for the wrong password
radius-admin-chap-freeradius The same, with auth-method chap against a Cleartext-Password entry verdict=accept with a CHAP-Password present and NO User-Password beside it, which is what RFC 2865 Section 4.1 demands of an Access-Request
radius-admin-chap-hashed-freeradius The CHAP login is REFUSED, and ze authenticates the user through no backend at all A radclient probe first proves the same entry accepts the same password over PAP, then verdict=reject for the CHAP request
radius-admin-eap-freeradius The same, with auth-method eap-mschapv2. Ze answers the EAP conversation itself from the operator's password, Naks the server's MD5-Challenge toward MSCHAPv2, and returns the State on every later round verdict=state-echo for at least one round carrying the State the server issued, then verdict=accept, both with an EAP-Message present and NEITHER password attribute beside it
radius-admin-eap-unsigned-freeradius On a server row with require-message-authenticator false, so the BlastRADIUS presence check stays out of the way and the RFC 3579 check is the one that runs: the EAP login through the relay twice. With the relay passing the answers, the login completes and ze_radius_unsigned_eap_discards_total reads 0. With the relay removing the Message-Authenticator, the login fails, ze logs radius: discarding the answer with the RFC 3579 Section 3.1 reason, and the counter is above 0 verdict=state-echo and verdict=accept for the passed login. The relay's own log records each answer it carried: an Access-Accept with the signature present, then an Access-Challenge with an EAP-Message and the signature stripped
radius-admin-pap-unsigned-freeradius BlastRADIUS (CVE-2024-3596) on a default server row: the PAP login through the relay twice. With the relay passing the signed Access-Accept, the login completes. With the relay removing its Message-Authenticator, the login fails, ze logs radius: discarding an answer without Message-Authenticator, ze_radius_unsigned_response_discards_total is above 0, and the RADIUS login count does not move verdict=accept for both requests: the server accepted, and only ze refused. The relay records an Access-Accept with no EAP-Message and the signature present, then the same with the signature stripped
radius-admin-pap-unsigned-legacy-freeradius The same stripped Access-Accept on a server row with require-message-authenticator false: the login completes with source=radius, and ze logs no BlastRADIUS discard verdict=accept, and the relay records the Access-Accept it carried with the signature stripped, so the login cannot have ridden a signed answer

Both lab clients.conf files set require_message_authenticator = yes, so FreeRADIUS discards any Access-Request without a valid Message-Authenticator. Ze signs every Access-Request (draft-ietf-radext-deprecating-radius-10 Section 4.2), so every scenario in this suite and in the subscriber suite also proves that FreeRADIUS verifies the signature ze computed. The red phase was run on 2026-10-28: with the insertMessageAuthenticatorFirst call in encodeRequest skipped and the lab binary rebuilt, radius-admin-pap-freeradius failed at its first assertion because FreeRADIUS dropped the unsigned Access-Request before its request log recorded anything; with the call restored it passed.

The unsigned scenarios need an answer FreeRADIUS will not send. RFC 3579 Section 3.1:

Access-Challenge, Access-Accept, or Access-Reject packets including EAP-Message attribute(s) without a Message-Authenticator attribute SHOULD be silently discarded by the NAS.

FreeRADIUS 3.2.10 writes a Message-Authenticator as the first attribute of every Access-Accept, Access-Reject and Access-Challenge it sends over UDP: rad_encode in src/lib/radius.c decides it from its code2ma table, which no configuration item reaches, so neither an unlang update reply nor a proxy server removes it. The lab therefore mounts test/interop-radius/unsigned-relay.pl into the server's container, and the checker starts it there on port 1912, the port the scenario's ze.conf names. The relay forwards each Access-Request unchanged to FreeRADIUS on 1812 and each answer back to ze. In strip mode it removes the Message-Authenticator and nothing else; in both modes it recomputes the Response Authenticator (RFC 2865 Section 3). The mode is a file the checker writes, and a missing or unknown mode forwards nothing. The image carries Perl and its Digest::MD5, and no other program that speaks UDP and MD5, which is why the relay is Perl.

The pass login is the control for the three ways the refusal could be read wrongly. It proves the relay carries the conversation, that its recomputed Response Authenticator is right (a wrong one would fail it, since pass mode recomputes too), and that the counter starts at 0. The strip login then differs from it by the missing attribute alone. The counter is read from ze's Prometheus endpoint on loopback, and a series the exposition does not carry is an error, never a zero.

The scenario passed on 2026-10-23 and discriminates: with verifyResponseMessageAuthenticator answering responseIntegrityAccepted for an unsigned EAP answer, as it did before the discard, and ze rebuilt, it fails with "assertion 6: ze completed an EAP login for radiusop through answers whose Message-Authenticator the relay removed".

radius-admin-pap-unsigned-freeradius passed on 2026-10-28 and discriminates: with the presence check in dispatchResponse disabled (if w.requireMessageAuthenticator { made if false && ...) and ze rebuilt, it fails with "assertion 6: ze completed a PAP login for radiusop on an Access-Accept whose Message-Authenticator the relay removed", and passes again once the check is restored.

The third scenario is the one that proves docs/guide/radius.md is telling the truth. RFC 2865 Section 2.2:

For example, CHAP requires that the user's password be available in cleartext to the server so that it can encrypt the CHAP challenge and compare that to the CHAP response. If the password is not available in cleartext to the RADIUS server then the server MUST send an Access-Reject to the client.

A rejection on its own would also follow from a typo in the user file, which is the fourth vacuity trap wearing another face: the asserted result would be produced by a different cause than the one under test. The radclient PAP probe removes it, because the storage form is then the only thing left to explain the CHAP rejection.

Two credentials share one username on purpose. radiusop exists in the server's user file AND in ze's local account list with a DIFFERENT password, so the scenario can send the local password while RADIUS rejects it: a chain that fell through to local bcrypt would accept that login. localop exists only locally and the server answers nothing for it, which is the positive control that proves the SSH listener and the local backend are both live before any refusal is read as evidence.

The FreeRADIUS subscriber suite

./le test integration interop-radius-subscriber puts FreeRADIUS 3.2.10, the release the admin-login suite pins, in front of ze's subscriber RADIUS surface: the accountant and the CoA and Disconnect listener of internal/component/l2tp/plugins/authradius. Both act only on a kernel L2TP session, which no fixture can stand in for, so the lab carries its own LAC (xl2tpd and pppd), and ze and the LAC hold NET_ADMIN and /dev/ppp. That is why it is a suite of its own: the admin-login suite promises no capability at all, and the L2TP lab's RADIUS peer is a mock ze wrote. FreeRADIUS writes every accounting record it accepts to its ze_detail file, and its radclient is the Dynamic Authorization Client.

Scenario What it proves
radius-coa-duplicate-freeradius RFC 5176 Section 2.3 duplicate detection. With ze paused, radclient sends one Disconnect-Request and retransmits it on its own socket: ze answers the copies from the cache, radclient gets a Disconnect-ACK, and ze logs one teardown. On a second session, radclient sends one Disconnect-Request and gets a Disconnect-ACK, then a bash process in the FreeRADIUS container sends the very octets ze received, read from the capture, out of a socket of its own, so from a new port: ze processes it afresh and answers Disconnect-NAK, where a key blind to the port would answer the cached ACK. A tcpdump capture in ze's container confirms the retransmitted copies share port, Identifier and Authenticator, and the two clients' requests are the same octets from two ports
radius-acct-restart-freeradius RFC 2866 Section 5.5 across a real restart. The Accounting-Start before ze restarts and the one after carry ids of the form tunnel-session-tag-counter, the two boot tags differ, and each Stop carries its Start's id, all read from FreeRADIUS's detail file
radius-coa-disconnect-freeradius Requires both kernel shaping rates to change after CoA-ACK; a mixed valid/unsupported request must NAK without changing either rate. A valid Disconnect must ACK and produce the matching Accounting-Stop
l2tp-radius-framed-protocol-reject Changes only FreeRADIUS's Framed-Protocol for one user: PPP must establish traffic, SLIP must produce authorization refusal and CHAP failure without Accounting-Start, then PPP must work again
radius-subscriber-dns RFC 2548 Sections 2.7.6 and 2.7.7 through the pool (spec-home-cpe-23b AC-9). The scenario's own users file has FreeRADIUS answer alice with MS-Primary-DNS-Server 198.51.100.53 and MS-Secondary-DNS-Server 198.51.100.54 (dictionary.microsoft, Microsoft attributes 28 and 29, moved into the reply by rlm_files' file_common), and its own pppd options add usepeerdns. The pool writes no dns-source and has its own dns-primary 203.0.113.53, so its built-in list (radius first) must offer the RADIUS pair: pppd's create_resolv writes exactly those two nameserver lines to, and the pool's address must not appear. Not yet run. /etc/ppp/resolv.conf

The server listens as a stock deployment does, type = auth on 1812 and type = acct on 1813 (site-default), and ze sends accounting to a server entry's accounting-port, 1813 by default, as RFC 2866 Section 3 asks. The pinned radclient has no -i, which is why the two-client half replays radclient's own octets rather than running a second radclient. The LAC dials once (redial = no in xl2tpd.conf): each new session comes from the checker restarting the LAC, so the Accounting-Start a checker reads is the session it asked for. Ze's container sets ze.log.l2tp.auth.radius=debug, because the teardown count reads the plugin's Info line and the default level is WARN.

The duplicate and restart scenarios passed on 2026-10-20, and each discriminates. With replayKey writing port 0 into the key, so blind to the source port, the two-client half fails: "ze answered Code 41, want a fresh Disconnect-NAK (42), not the first client's cached answer". With genSessionID writing the old tunnel-session-counter form, the restart scenario fails: "Acct-Session-Id "1702-33224-1" before the restart carries no boot tag".

The two additional scenarios passed on native Ze KVM guests on 2026-10-27. Their forced-red and restored-green proofs remain owed. The framing checker uses radclient's Access-Accept as its reload barrier and retries HUP no sooner than six seconds after the last signal: FreeRADIUS ignores HUP within five seconds of the preceding reload. After copying its initial users file, the framing peer starts through the image's entrypoint. The pinned Debian image resolves radiusd to freeradius there; a shell executing radiusd directly bypasses that translation. The LAC sends pppd's diagnostic messages to descriptor 2: xl2tpd closes stdout before starting pppd in kernel mode, while its foreground stderr reaches Docker logs. This keeps the checker's CHAP-refusal observation connected to the peer.

Independent TFTP duplicate-ACK recovery

The Linux, Docker-tagged TestTFTPCurlDuplicateACKInterop uses digest-pinned curl 8.23.0 through labtest.Run against Ze's production TFTP handlers. A bounded UDP relay drops DATA 2 and replays curl's previous ACK 1. It requires no immediate DATA response, timeout recovery and an exact 1,500-byte download. Linux receive timestamps distinguish timeout recovery from an immediate duplicate that sat unread in a socket queue; missing timestamps or an ACK replay outside the fault window refuse the run. The relay neither constructs client requests nor generates acknowledgments. Results on 2026-11-14 (arm64, Ze's kernel 7.3.0-ze in the QEMU docker-lab guest, curl 8.23.0 aarch64 pinned by digest): the real build passes, the relay reporting "replayed native ACK 1 once; dropped first DATA 2; kernel arrival interval 5.007927169s; forwarded final ACK 3; curl matched all 1500 bytes (8 relay datagrams)". With windowSender.awaitACK returning on an ACK that is not for the outstanding block, where it now keeps waiting (the behavior before 519d9f4f2a), FAIL: the relay reports "DATA in answer to duplicate ACK after 36.333µs, before five-second timeout", and curl exits 28 at its 35-second limit with 512 bytes received. The real build run again right after passes (interval 5.004289378s).

On a Mac that run used the Docker daemon of Ze's own docker-host image. The test binary is cross-built for linux/arm64 with go test -c under the feature tags plus docker. ./le test qemu docker-lab with lab words boots the image under HVF, forwards the guest's Docker socket to the host, and runs the host le with those words and DOCKER_HOST naming the forwarded socket. The words run the binary in a docker:cli container on the guest's host network with the guest's Docker socket mounted, because the relay, Ze's server and the curl container must share the guest's loopback, and the test needs Linux receive timestamps. The guest mounts the checkout at its host path, so the container bind-mounts the binary from the path it was built at. The docker-lab command mode cannot run the test, because that mode starts no Docker daemon. No ./le action selects one docker-tagged package test for this route yet: ./le test integration docker-peers runs every docker-tagged package, against the Docker daemon the host reaches.

Ze's TFTP server against tftp-hpa and curl options

Two Linux, Docker-tagged tests in tftp_hpa_docker_test.go drive Ze's production TFTP handlers (ServeConn) with independent clients, through a relay that copies every datagram both ways and records the client's RRQ options, the server's OACK and the longest run of DATA blocks between two ACKs. The relay constructs nothing.

  • TestTFTPHPAWindowsizeInterop builds the tftp-hpa 7.2 client from its release tag (test/interop-tftp/Dockerfile.tftp-hpa, which refuses any other commit) and downloads a 300-block file twice: with the client's default request (windowsize 64) and with -W 1000. Each must be granted windowsize 64, be sent in windows (64 DATA blocks between ACKs), and match the file's SHA-256. With grantWindowsize forced to one block the OACK carries no windowsize and the test goes red.
  • TestTFTPCurlTimeoutInterop runs curl 8.23.0 with its default options, whose RRQ carries timeout 5 for --max-time 25. The OACK must echo that timeout and curl must deliver the exact file. With the timeout left out of the OACK the test goes red.

Both run on a Mac the way the dnsmasq test below does: the test binary is cross-built with -tags docker and run in a docker:cli container on the ze Colima VM's host network.

Ze's TFTP client against dnsmasq

The Linux, Docker-tagged TestTFTPClientDnsmasqInterop builds dnsmasq 2.91 from test/interop-tftp/Dockerfile.dnsmasq through labtest.Start and runs it on the host network, because dnsmasq answers each transfer from a fresh port (its TID) that a published port does not carry. Ze's client fetches a 116-block file, a file of exactly three 1468-octet blocks and an empty file, and must negotiate blksize 1468 and tsize equal to the file size, fall back to a window of one block (dnsmasq acknowledges no windowsize), and deliver the exact bytes; the last fetch runs through the scheme table with a SHA-256. With tftp.ParseOACK broken to grant nothing, the client refuses dnsmasq's 1468-octet block 1 with ERROR 4 and the test goes red. On a Mac, where Colima's port forwarder does not reach the container's transfer port, the test binary is cross-built for linux/arm64 with go test -c -tags docker and run in a docker:cli container on the VM's host network with the Docker socket mounted.

The freeRouter RSVP-TE suite

internal/le/interoplab/rsvpte/ puts up to four nodes on one Docker segment, 172.29.81.0/24, in four roles: ingress (host 2), transit (3), egress (4) and relay (5). A scenario directory decides which implementation fills each role by the files it carries: <role>.conf makes the role a Ze node answering on the container address .<host>, and <role>-hw.txt with <role>-sw.txt makes it a freeRouter node answering on its own address .<10+host>. A role with neither file is absent. Nodes start downstream first, so the first PATH meets nodes that already listen. The addresses are fixed because the configurations in each scenario directory name them. Two shapes are used: freeRouter, Ze, freeRouter, where freeRouter originates and Ze relays; and Ze, freeRouter, Ze (and Ze, freeRouter, Ze, Ze), where Ze originates PATH, ResvErr, ResvTear, PathErr and strict hops and freeRouter is the independent implementation that parses, relays and re-encodes them. A Ze head-end tunnel in this suite requests fast-reroute, because freeRouter's packRsvp.parseDatPatReq refuses a PATH without SESSION_ATTRIBUTE and Ze emits that object only for a protected tunnel and a bypass. A third shape puts Ze at the point of local repair: Ze ingress, Ze PLR, freeRouter relay on the bypass, freeRouter egress as the merge point. Each freeRouter owns its own IPv4 stack and MAC: its container command (freeRtrScript) makes the container's eth0 promiscuous and joins it to the jar through the upstream rawInt.bin, so the suite needs Docker alone, never host root, a TAP device or a network namespace, and no container runs privileged: a freeRouter holds NET_ADMIN and NET_RAW. A Ze node holds NET_ADMIN to program MPLS labels and VLANs. A VLAN is a link of the shared core ("Separate links between daemons"): labLinks holds each scenario's interoplab.Link declarations, and the core's start wrapper builds a member's devices before its container command runs. The one user is plr-backup-path-to-egress-merge-point, whose protected link is VLAN 100 on 10.0.14.0/24 with the Ze PLR (transit, host 3, so 10.0.14.3) as its one member under the device name prot0 (Devices), because transit.conf and the checker name it, and the freeRouter egress as its External end: the egress's rtr-sw.txt holds interface ethernet1.100 at 10.0.14.14, reached over rawInt.bin, so its container gets no device. The node's container command (zeNodeScript) adds the routes the role's optional <role>-net.txt declares, one route <prefix> via <gateway> per line, and a line of any other shape, a vlan line included, refuses the run. The command writes its MPLS sysctls at run time, input on eth0 and on each link device the wrapper built (linkDevices, from interoplab.LinkMemberships), so it runs under ze-lab-net-sysctl with /proc/sys writable. No scenario carries a shell script: the commands are built in Go from typed declarations. The preflight reads /proc/sys/net/mpls/platform_labels with no grant, loads no module, and refuses a host kernel without MPLS routing with the host command that loads it.

Every assertion reads what a peer received. Every container, Ze or freeRouter, runs tcpdump -vvv on its eth0 into /run/fr/rsvp.txt, RSVP and every MPLS-labelled frame, because a PATH carried through a bypass LSP is labelled. The checker parses that text: a message Ze logs as sent counts for nothing until the peer's capture holds it.

Scenario What the peers observe
transit-loose-ero-expansion freeRouter ingress, Ze transit, freeRouter egress. The ingress names Ze and the egress loopback as loose hops and prepends its own next hop, Ze, as a strict subobject (ipFwdTab.fillRsvpFrst), so it signals [Ze strict, Ze loose, egress loopback loose]. Ze's native route to the loopback runs through .14, which no subobject names, so the PATH the egress captures carries .14 ahead of the still-loose loopback (RFC 3209 Section 4.3.4.1 steps 5 and 6). The ingress captures Ze's RESV with a label and an LSP_TUNNEL_IPv4 FILTER_SPEC naming the sender (RFC 3209 Section 4.6.3.1, class 10 C-Type 7, not a class 11 SENDER_TEMPLATE), and Ze's MPLS table holds a swap via .14
plr-backup-path-to-egress-merge-point Ze ingress, Ze PLR, freeRouter relay .15, freeRouter egress .14. The ingress asks facility protection for a tunnel whose protected hop is the PLR's prot0, VLAN 100 to the egress at 10.0.14.14; the PLR's configured bypass runs untagged through the relay and merges at that egress. Before the fault the egress holds no backup PATH. The checker takes prot0 down, and the egress captures, through the bypass, a PATH whose sender and RSVP_HOP are the PLR and whose ERO is 10.0.14.14 alone; the PLR captures at least two labelled RESVs from the egress naming its own sender and answers none with a ResvErr (RFC 4090 Sections 6.1 and 6.4). The egress answers from its eth0 address .14, not from 10.0.14.14, and Ze accepts an alternate merge-point source only when a live native IGP database attributes it to the same node, so the PLR and the egress run OSPF on eth0 (freeRouter router-id 10.0.14.14) and the checker waits for a Full adjacency, read through show ospf neighbor on the PLR, and then for the egress's router-LSA to carry its point-to-point link (48 octets or more in show ospf database router), before it takes prot0 down: the egress re-originates that LSA after the adjacency forms, and until it arrives nothing ties .14 to 10.0.14.14. After the cut the ingress keeps refreshing its PATH, whose ERO still names 10.0.14.14 strict across the failed link; the PLR relays it through the bypass, and the checker fails on a Routing Problem PathErr for the protected LSP-ID 1 within three refreshes (RFC 4090 Section 6.4.3). The Notify Tunnel locally repaired for LSP-ID 1 and a Routing Problem for a new LSP-ID the ingress signals over the same ERO are correct and do not count. Because a PLR that dropped the refresh in silence would send no PathErr either, the checker then waits until the protected PSB's lifetime has passed since the cut (10.5 seconds, RFC 2205 Section 3.7 with K = 3 and the ingress's 2-second refresh) and requires a further backup PATH at the egress and a further labelled RESV from the merge point at the PLR. freeRouter is the merge point only as an egress: it keys the backup PATH by sender and LSP-ID as a new LSP, and has no transit merge
transit-strict-hop-forwarded Ze ingress, freeRouter relay .15, Ze transit, freeRouter egress .14. The Ze ingress names every hop strict. The PATH the egress captures from Ze carries the ERO shortened to strict .14 alone, the Ze ingress captures a labelled RESV relayed by freeRouter, and the Ze transit holds a swap via .14
transit-strict-hop-outside-refused Ze ingress, freeRouter relay, Ze transit. The last strict hop's native route at the transit runs through a node outside both abstract nodes, so the transit sends PathErr Routing Problem / Bad strict node (24/2) and never forwards the PATH (RFC 3209 Section 4.3.3.1). The Ze ingress captures that PathErr as freeRouter relays it, naming the transit as the error node
ingress-resv-error-relayed Ze ingress, freeRouter relay, Ze egress. The ingress interface reserves less than the tunnel asks, so the ingress refuses the RESV freeRouter relays and sends a ResvErr naming itself, Error Code 1 Admission Control failure, value 2. The Ze egress captures that ResvErr from freeRouter with the error node, code and value intact (RFC 2205 Sections 2.5 and 3.1.8)
transit-resv-tear-relayed Ze ingress, freeRouter relay, Ze transit, Ze egress. Once the LSP is up the egress is frozen with SIGSTOP, so the transit's reservation times out while its path state is refreshed; the transit sends a ResvTear upstream, and the Ze ingress captures it as freeRouter relays it (RFC 2205 Section 3.1.6)
transit-resv-increase-refused-in-place freeRouter ingress, Ze transit, patched freeRouter egress. The ingress tunnel signals 10 Mbit/s (bandwidth 10000) and the transit's eth0 reserves 100 Mbit/s. The egress's second RESV, answering the ingress's first PATH refresh (freeRouter refreshes every 120 s), raises its reservation in place to 1 Gbit/s, same session, sender and LSP-ID. The transit captures that RESV, and the egress captures Ze's ResvErr naming the transit, Error Code 1 value 2, ERROR_SPEC flags 0x01 (InPlace). The transit's swap stays installed and the ingress never receives the raised rate (RFC 2205 Section 3.1.8)
transit-ff-resv-unknown-sender freeRouter ingress, Ze transit, patched freeRouter egress. Every RESV the egress sends is fixed-filter, with a second flow descriptor naming LSP-ID+1 of the same sender, which never signalled a PATH. The egress captures one ResvErr from the transit naming only that LSP-ID, Error Code 4 No sender information, and no ResvErr naming the real sender. The ingress receives a labelled RESV naming only the real sender, and the transit holds its swap (RFC 2205 Section 3.1.8)
egress-implicit-null-penultimate-pop freeRouter ingress, Ze transit, patched freeRouter egress. Every RESV the egress sends distributes Implicit NULL, label 3, so Ze is the penultimate hop. The transit captures that RESV, the ingress receives Ze's RESV carrying a label Ze allocated, not 3, and Ze's MPLS table pops that label toward .14 with no outgoing label: no as to (RFC 3032 Section 2.1)
transit-resv-state-expires freeRouter ingress, Ze transit, freeRouter egress. Once the LSP is up and Ze holds its swap via .14, the checker freezes the egress's Java process with SIGSTOP, so no RESV refreshes Ze's reservation while the ingress keeps refreshing the path. The ingress captures a ResvTear from Ze, none of which existed before the freeze, and Ze's MPLS table drops the swap via .14 (RFC 2205 Sections 2.3 and 3.1.6). The freeRouter egress advertises its fixed 120-second refresh period in the RESV's TIME_VALUES, so Ze, with refresh-multiplier 3, times the reservation out about six minutes after the freeze, and the checker waits that long

The checker compares the ERROR_SPEC code and value as the numbers tcpdump prints in parentheses, because tcpdump names no code for every value: it prints Admission Control failure as unknown (1).

The pinned freeRouter originates only PATH, PathTear and RESV. It relays PathErr, ResvErr and ResvTear but never originates them. It encodes its configured ERO hops as loose (clntMplsTeP2p.workDoer, ipFwdTab.fillRsvpPack) and prepends one strict subobject for its own next hop unless the first hop is already strict (ipFwdTab.fillRsvpFrst), and it routes each hop without reading the strict bit (rtrRsvpIface.getHop), so it does not enforce strict hops. It signals one fixed bandwidth for the life of an LSP, and sends only shared-explicit RESVs naming its own sender. In every scenario where Ze originates a message, freeRouter is the independent implementation that parses it, keeps the state it needs to relay it, and re-encodes it toward the next Ze node: the evidence is that freeRouter accepts and relays what Ze originates, not that freeRouter enforces strict hops or originates ResvErr, ResvTear or PathErr itself.

The image is built with one patch, test/interop-rsvpte/freertr/ze-interop-resv.patch, because the pinned jar can neither raise a reservation in place, name a sender it has no path for, nor distribute Implicit NULL as an egress (rtrRsvpIface always sends the label it allocated). The patch adds three knobs to the RESV a freeRouter egress originates, each off unless its environment variable is set, so every scenario that sets neither runs the upstream behavior. A scenario sets them in <role>-env.txt, one NAME=value per line, which the lab passes to that freeRouter container and refuses when a line is not NAME=value. FREERTR_ZE_RESV_RATE_AT=n:rate sends only the n-th RESV with its FLOWSPEC rate (bytes per second) raised; FREERTR_ZE_RESV_FF_EXTRA_SENDER=1 makes every RESV fixed-filter and appends a second flow descriptor (FLOWSPEC, FILTER_SPEC with LSP-ID+1, LABEL); FREERTR_ZE_RESV_IMPLICIT_NULL=1 puts label 3 in the LABEL object of every RESV in place of the allocated label.

Two scenarios depend on objects RFC 2205 makes optional and freeRouter's parser requires. packRsvp.parseDatPatErr refuses a PathErr without an ADSPEC, which transit-strict-hop-outside-refused relies on, and packRsvp.parseDatResTer refuses a ResvTear without a FLOWSPEC, which transit-resv-tear-relayed relies on. The PathErr sender descriptor is "<SENDER_TEMPLATE> <SENDER_TSPEC> [ ]" (Section 3.1.3), and "FLOWSPEC objects in the flow descriptor list of a ResvTear message will be ignored and may be omitted" (Section 3.1.6). Ze sends both objects (buildPathErr, buildReservationControl), so both scenarios pass against the unpatched image. Before Ze carried them, freeRouter's capture held Ze's message and freeRouter never relayed it.

The topology suite

The topology suite (internal/le/interoplab/topology) runs the scenarios no single protocol suite owns: several Ze nodes, each with its own name and its own configuration file, beside peers whose images other suites declare, on named links (see "Separate links between daemons"). A scenario directory under test/interop-topology/scenarios/ carries only the nodes' configuration files. Its Go register file declares the nodes in start order (name, registered image, host number, configuration file), the links and the checker, and Discover joins the directory with that declaration as it does for every suite, so a directory with no declaration is an error.

Scenario Nodes and links What the checker proves
ze-static-link-chain Ze nodes ze-a, ze-b, ze-c on links a-b (VLAN 181) and b-c (VLAN 182), B forwarding; outside on eth0 alone, running only tcpdump on ip net 10.181.1.0/24 or arp net 10.181.1.0/24. A ping from outside to a link address proves nothing: Linux answers ARP for any local address on any interface (arp_ignore 0), so ze-b answers 10.181.1.3 from its own eth0. The checker flushes ze-a's neighbours on the link so its ARP for ze-b, a tagged broadcast, reaches outside's port. Each node's file carries interface { backend netlink; }: with no interface block the interface component loads no backend, and show interface brief answers iface: no backend loaded. 10.181.1.0/2410.181.2.0/24 each node answers show interface brief by name; A pings C through B; outside's capture (tcpdump -e) holds a frame of link a-b with its 802.1Q tag, and none without it
two-ze-pppoe-link a Ze PPPoE client and a Ze PPPoE server on a link with no prefix; an observer on eth0 alone the client holds an address from the server's pool on a ppp device (assigned only once IPCP is Opened), the server's kernel PPPoE table holds the session on the link device, and the observer received no untagged PPPoE discovery frame. Its capture filter also matches the link's own tagged frames, because veth hands the tag to the packet socket out of band, so the checker reads the capture with tcpdump -e and counts only lines without an 802.1Q header
pppoe-mss-pmtu Ze CPE ze-cpe dialing accel-ppp accel on link access (VLAN 184, no prefix) with MTU 1492; lan-host behind Ze on link lan (VLAN 185); server beyond accel-ppp on link core (VLAN 186), capturing SYNs. 10.185.1.0/2410.186.1.0/24 Ze's pppoe0 holds an IPCP address and reports MTU 1492, its table lists tcp option maxseg size set rt mtu, and the server captures the SYN lan-host offered with MSS 1460 as 1452 and the one offered with 1200 as 1200
access-up the access chain (below): Ze CPE ze-cpe, running the home-CPE lab's converted configuration with one lab addition (subnet br0, with no dns-server and no dns-source, reserving 10.77.0.50 for lan-res-c), on link access (VLAN 191, no prefix) and on link lan (VLAN 197, no prefix) with lan-client and lan-res-c (dhcpcd), to Ze LAC ze-lac, which relays the empty Service-Name over link core (VLAN 192) to Ze LNS ze-lns, which checks CHAP with freeradius on the same link. 10.77.0.0/24192.0.2.0/24 the LAC's access capture holds a PADO with a zero-length Service-Name and a PADS with a non-zero session; the LAC holds one tunnel to the LNS with one session, the kernel PPPoE session on its access device, and no addressed PPP device; FreeRADIUS logged an Accept for cpe@lab.example.net with a CHAP-Password, no User-Password, NAS-Identifier ze-lns and NAS-IP-Address and source 192.0.2.3; the LNS session names the user and holds 203.0.113.1, its detail reports mru 1492 and auth-method chap-md5, its core capture holds its LCP Configure-Request with MRU 1492; the CPE's pppoe0 holds 203.0.113.1 with MTU 1492, its show interface pppoe-client reports pppoe0 up with dns-servers 192.0.2.53 then 192.0.2.54, the two servers the LNS offered in IPCP, while its resolv.conf holds only the converted configuration's static system name-server 192.0.2.53, because a static server wins over every learned one; lan-res-c then binds 10.77.0.50 and its dhcpcd --dumplease holds domain_name_servers exactly 192.0.2.53 192.0.2.54, the IPCP pair in order, which only the subnet default reaching its received class hands out (the host fallback alone gives 192.0.2.53); the CPE's eth0 holds no address, its route to 100.64.0.1 leaves on pppoe0, it pings that address with no loss, and the LNS's PPP device toward the subscriber received at least the pings sent
access-pool the access chain, with FreeRADIUS answering cpe@lab.example.net with no Framed-IP-Address the CPE's pppoe0 holds an address in, the LNS session holds the same one, and show l2tp pool reports one address allocated, to that session. 100.64.0.0/24
access-accounting the access chain, with the LNS's acct-interval 60 FreeRADIUS's ze_detail holds the session's Start, an Interim-Update and, once the CPE is stopped (its PPPoE client sends PADT), the Stop, all under one Acct-Session-Id and carrying 203.0.113.1, the Stop counting at least the 20 560 octets a 20-packet ping of 1000-octet payloads carried each way
access-wholesale the access chain with the scenario's own LAC, which relays the realm lab.example.net (relay-realm) and so hands the LNS the proxy LCP AVPs of RFC 2661 Section 4.4.5, and an LNS set proxy-lcp renegotiate the CPE's pppoe0 holds 203.0.113.1; FreeRADIUS logged the LNS's CHAP Accept for cpe@lab.example.net; the LNS session detail reports mru 1492, auth-method chap-md5 and proxy-lcp renegotiated; the LNS's core capture holds its own LCP Configure-Request with MRU 1492, which an LNS that accepted the proxy never sends; the subscriber pings across
l2tp-lns-lcp-renegotiate-pppd access-wholesale with pppd 2.5.5 (rp-pppoe plugin, the PPPoE suite's ze-pppoe-client image) as subscriber pppd@lab.example.net pppd's ppp0 holds 203.0.113.1; the LNS session reports proxy-lcp renegotiated and chap-md5; the LNS's core capture holds its LCP Configure-Request; pppd's debug log holds two received LCP Configure-Requests and two sent Configure-Acks (the LAC's negotiation and the LNS's renegotiation, RFC 1661 Section 4.3), a CHAP success and the IPCP address; pppd pings 192.0.2.3 over the default route defaultroute puts on ppp0 (its eth0 is flushed, so nothing else reaches the core)
conversion the home-CPE lab (below) with lan-client on the CPE's bridge conversion-record.md names every line of vyos.conf with its own text and a Ze form or a dropped reason; every scenario whose ze-cpe node runs mounts the rendered conversion (only pppoe-mss-pmtu is exempt); ze config validate passes on the file the CPE mounted; the CPE's pppoe0 holds 203.0.113.1 and the kernel holds br0, wg0 and vti0
lan-services the home-CPE lab with lan-client, lan-res-a and lan-res-b (dhcpcd), lan-host (static 192.168.50.10) and web lan-client binds inside 192.168.50.100-199; lan-res-a binds 192.168.50.30 and lan-res-b 10.99.0.40 (the later subnet); web captures lan-host's fetch from 203.0.113.1; SYNs offered with MSS 1460 and 1300 arrive as 1400 and 1300; web gets no answer from while the CPE's default-drop counter advances, and lan-host completes TCP to; an active-mode FTP download completes. 203.0.113.1:22192.168.50.1:22
road-warrior the home-CPE lab with lan-res-a, web and wg-client the CPE's wg show wg0 dump lists port 51820 and the laptop's key with a preshared key and wg-client holds a handshake, pings lan-res-a and fetches from web, which captures the SYN from 203.0.113.1. 10.88.0.2/32
mgmt-tunnel the home-CPE lab with lan-client, strongswan (vpn.example.net) and a dns gated shut ✕
mgmt-tunnel-ed25519 mgmt-tunnel with the lab issuing the CPE an Ed25519 key (PKCS #8) under the same P-256 CA every mgmt-tunnel clause, and strongSwan's log holds authentication of 'cpe.lab.example.net' with ED25519 successful, the line pubkey_authenticator.c process() writes only after the signature verified, so Ze signed its IKE AUTH with Ed25519 under RFC 8420
access-wholesale-accel-lns the access chain with access-wholesale's realm-relaying LAC, accel-ppp 1.14.0 as the LNS on link core at 192.0.2.3 in place of ze-lns, checking CHAP from its own chap-secrets, and no FreeRADIUS the CPE's pppoe0 holds 203.0.113.1, the address accel-ppp's chap-secrets names; accel-ppp's verbose log holds an LCP Configure-Request and a Configure-Ack it received (the CPE, Opened with the LAC, renegotiated when accel-ppp, which skips the proxy LCP AVPs in l2tp_recv_ICCN, sent its own request), a CHAP Response naming cpe@lab.example.net and its CHAP Success; the CPE pings 100.64.0.1 across and accel-ppp's PPP device received at least the pings sent
l2tp-lac-realm-relay-accel-ppp spec-home-cpe-14's realm path: pppd 2.5.5 subscribers pppd (pppd@lab.example.net) and pppd-stray (pppd@other.example.net) on link access (VLAN 191, no prefix) to Ze LAC ze-lac, which relays the realm lab.example.net (relay-realm) over link core (VLAN 192) to accel-ppp 1.14.0 at 192.0.2.3, whose chap-secrets holds pppd only; no FreeRADIUS. 192.0.2.0/24 pppd's ppp0 holds 203.0.113.1, the address accel-ppp's chap-secrets names; the LAC holds one L2TP session, reporting relay-realm lab.example.net, relay-remote lns, proxy-lcp-sent and proxy-auth-sent true; accel-ppp's verbose log holds an LCP Configure-Request and a Configure-Ack it received, a CHAP Response naming pppd@lab.example.net and its CHAP Success (it skips the proxy AVPs in l2tp_recv_ICCN and renegotiates); pppd pings 100.64.0.1 across; pppd-stray's log holds the LAC's own CHAP Failure no users configured (the local auth handler, with no user configured) and no CHAP success, and accel-ppp's log never names it
l2tp-lac-realm-relay-xl2tpd l2tp-lac-realm-relay-accel-ppp with xl2tpd 1.3.20 and pppd 2.5.5 as the LNS (xl2tpd-lns, the RADIUS subscriber suite's ze-radius-subscriber-lac image), answering the LAC's tunnel challenge from l2tp-secrets and checking CHAP from its pppd's chap-secrets, which holds pppd only pppd's ppp0 holds 203.0.113.1, handed by xl2tpd's ip range; the LAC session reports the same four keys; xl2tpd logs Call established with 192.0.2.2 (control_finish, on the ICCN, whose proxy AVPs it ignores); the LNS pppd's debug log holds a received LCP Configure-Request and Configure-Ack, a CHAP Response naming pppd@lab.example.net, the CHAP Success it sent and remote IP address 203.0.113.1; pppd pings 100.64.0.1; pppd-stray is refused by the LAC as above and the LNS pppd's log never names it
pppoe-dhcpv6-pd-ze-client a Ze CPE with its own file (pppoe0, ipv6 with accept-ra 2, autoconf, forwarding and dhcpv6 pd length 56; ethernet lan0 unit 0 with forwarding, delegated-prefix pppoe0 subnet-id 1 and router-advertisement at intervals 3 to 4 seconds and router-lifetime 1800) dialing accel-ppp 1.14.0 on link access (VLAN 198, no prefix), accel-ppp running IPv6CP, ipv6_nd and ipv6_dhcp with a /60 delegate pool and on its loopback; slaac-host (the Knot image's dhcpcd 10.5.2, kernel Router Discovery and SLAAC off) on link delegated-lan (VLAN 199, no prefix), which the CPE reaches as lan0; the CPE records every RA it sends on lan0 from its prestart. 2001:db8:ffff::1 spec-home-cpe-8. Phases 1 to 3: pppoe0 holds one fe80::/64 from a non-zero identifier and no pppN holds one; pppoe0's accept_ra is 2, forwarding and autoconf 1; ::/0 goes via (accel-ppp's ipv6-intf-id) on pppoe0; pppoe0 holds a SLAAC /64 from; accel-ppp's log holds a Solicit carrying IA-PD with the hint ::/56, then either a Request for a /60 from and the Reply delegating it, or, when the Solicit carried Rapid Commit, accel-ppp's Rapid Commit Reply delegating a /60 from that pool and a later Renew or Rebind from the CPE naming that exact prefix (RFC 7084 WPD-3). The two-message exchange sends no Request, so the CPE's Renew is its word that it kept the prefix: RFC 9915 Section 18.2.4 forbids a client to include a prefix it did not obtain, and Ze's client renews with the ::/56 hint when it holds none. accel-ppp's pref-lifetime 60 puts T1 at 30 seconds, inside the checker's wait. Phases 4 and 5 (checkDelegatedLAN): lan0 holds the delegated /60's subnet 1 with, its lifetimes finite and no longer than accel-ppp's 120 and 60 seconds; the CPE's own forwarding units hold net.ipv6.conf.all.forwarding at 1 (Linux forwards no IPv6 packet without it, and the checker writes no sysctl, so a Ze that leaves it off fails the ping), and the checker starts dhcpcd, which holds a SLAAC address in that /64, ::/0 via lan0's link-local and a route to the /60 from the RIO, and pings across the CPE. Every captured RA carrying the /64 is judged against the accel-ppp Reply before it: the PIO's valid and preferred lifetimes and the RIO's lifetime are no longer than what remained of the lease, a second and a half of clock rounding allowed, the RA carries the RIO and Router Lifetime 1800, and at least one such RA left ten seconds or more after its Reply, so lifetimes that never count down fail (red walk: the RFC 9915 Section 6.3 bound removed). The checker then sets accel-ppp's PADO-delay to -1, so the CPE cannot dial again, and runs accel-cmd terminate username alice: the first RA with Router Lifetime 0 leaves within five seconds of the termination, no later RA carries more (red walk: the W input forced true), and dhcpcd drops ::/0 via lan0 and logs no longer a default router (lifetime = 0) (RFC 7084 G-5). The client writes the three sysctls on each new session device at run time, through the sysctl component, so the CPE runs as a run-time net sysctl writer (netSysctlWriter: /proc/sys writable under the net-only AppArmor profile) and the checker polls each value. fe80::12001:db8:198::/482001:db8:2000::/40::12001:db8:ffff::1
access-reconnect the home-CPE lab with lan-host, web, strongswan and dns, FreeRADIUS answering from a writable copy of its users file after FreeRADIUS is changed to answer 203.0.113.2, restarted, and the LNS clears the session, the CPE holds 203.0.113.2, strongSwan's SA sits on it, ESP advances both ways, and web captures lan-host's fetch from it

The home-CPE lab (spec-home-cpe-11) runs the CPE every home-CPE scenario shares, as a run-time net sysctl writer the way a deployed CPE writes its devices' sysctls (netSysctlWriter): test/interop-topology/cpe/ze-cpe.conf, the model's conversion of the invented vyos.conf with conversion-record.md beside it. The file holds placeholders, never values: <<SECRET:name>> (filled from lab-values) and <<LAB:name>> (issued by the run: a fresh P-256 CA with path length 0, the vpn.example.net certificate with a P-256 key, the cpe.lab.example.net certificate with the key algorithm the scenario declares, P-256 in SEC 1 or Ed25519 in PKCS #8 for mgmt-tunnel-ed25519, and the laptop's X25519 public key derived from its lab private key). The scenario's prepare step renders it into the session scratch before any container starts, and the CPE mounts the rendered file, so no scenario carries a CPE file of its own: the access chain's scenarios below run it too. A scenario that needs a case the VyOS source cannot state inserts lab-only text into the rendered file at an anchor the file holds once (cpeAddition, home_lab_render.go), so ze-cpe.conf and its record stay the conversion: access-up adds a subnet left on the dns-source default, because a VyOS subnet with no option name-server converts to source configured. The lab adds a lan link (VLAN 194, no prefix: the CPE numbers it, br0 on eth2) and an internet link (VLAN 196, 198.51.100.0/24) the LNS routes, beside the access chain's access (VLAN 193) and core (VLAN 195); dns (Knot, 192.0.2.53) answers vpn.example.net. The prepare step also writes a resolv.conf naming the converted configuration's static system name-server, which the CPE mounts at /etc/resolv.conf: on the appliance that path links to /tmp/resolv.conf, the file Ze writes, so Go's resolver (the one IKE uses for the remote-address name) asks the configured server, while Docker's own file names only 127.0.0.11, and Ze's write by rename cannot be followed by a bind mount. access-up proves Ze's own file names the same single server. The converted configuration sets vpn ipsec unmatched bypass, because VyOS installs no SPD catch-all and Ze's defaults to discard, which would drop the CPE's own DNS and ping in the clear. It names no application-override, so the default deny attaches Ze's setsockopt guard, and the CPE holds BPF beside NET_ADMIN (capabilityBPF, internal/le/interoplab/topology/topology.go): without it the guard's map cannot be created and ze exits at startup. It also holds the host's /dev/net/tun beside /dev/ppp (convertedCPE, register_home_lab.go): a Child SA that receives UDP-encapsulated ESP makes ze create the ze-espform TUN device, whose sysctls it writes under the same run-time net sysctl grant. A lan-client dhcpcd node starts with net.ipv4.conf.default.promote_secondaries=1, because dhcpcd's if_init writes that sysctl for its interface when it reads anything else, and Docker's /proc/sys is read-only. Every node beyond the LNS flushes its eth0, so the links are its only path. The web and FTP server and lan-host run ze-topology-webftp (Debian, busybox httpd, vsftpd active-only, curl, bash), registered from test/interop-topology/home-lab/Dockerfile.webftp; wg-client runs ze-topology-wireguard, registered from the deployment proof's Dockerfile.wireguard.

The config corpus renders the converted CPE through the same issuer and placeholder substitution as the topology lab. Both readers use the lab values and fresh certificate material. The corpus validates that rendered config, rather than accepting or excluding its unresolved placeholders.

The access chain is the first end-to-end proof that a subscriber's PPP session crosses Ze's LAC (relay "") and terminates on Ze's LNS: the LAC terminates nothing, so LCP, CHAP and IPCP run between the CPE and the LNS across the LAC's kernel channel bridge. Every value is invented: core 192.0.2.0/24, subscriber 203.0.113.1/32, pool 100.64.0.0/24, realm lab.example.net. The three scenarios share one declaration (accessChain) and the files under test/interop-topology/access-chain/: the LAC configuration, which no scenario changes, and FreeRADIUS's clients.conf, a site-default that adds a ze_auth_log line per answered login (mods-ze-auth-log: the presence of a User-Password and a CHAP-Password, the NAS-Identifier, the NAS-IP-Address and the request's source), and the subscriber lab's ze_detail module mounted unchanged. Each scenario directory carries what differs: the LNS configuration and FreeRADIUS's users. The CPE is the home-CPE lab's (convertedCPE, with the same prepare step), and since its bridge holds eth2 it also joins link lan (VLAN 197) with lan-client, a link needing two members; l2tp-lns-lcp-renegotiate-pppd dials with pppd instead and has no lan, and the two l2tp-lac-realm-relay-* scenarios dial with two pppd subscribers to a third-party LNS and run no FreeRADIUS. FreeRADIUS joins the core link, so it runs ze-topology-freeradius, the pinned release built with iproute2 for the link's start wrapper. Each clause reads a different producer (vacuity trap 5), and the CPE's prestart flushes its eth0, so the session is its only path to the LNS (trap 4). The LAC and the LNS start their captures in their prestart, before their daemon, and the LNS's and FreeRADIUS's readiness gate the next node's start. The interim waits the full 60 seconds: RFC 2869 Section 5.16 says of Acct-Interim-Interval "The value MUST NOT be smaller than 60.", Ze clamps a received value to that floor and the acct-interval leaf's range starts there. The scenarios need a Docker host running Ze's kernel, with PPPoE, L2TP and PPPoL2TP (spec-docker-hosts-run-the-ze-kernel); the core's kernel check refuses another.

./le test integration interop-topology runs every scenario; ./le test integration interop-topology scenario <name>, or TOPOLOGY_INTEROP_SCENARIO=<name>, runs one. A checker that asks a Ze node's CLI does so through interoplab.ZeCLICommand, the same argv the IPsec lab sends, and that node's configuration serves the SSH account on 127.0.0.1 port 2222. A node whose configuration is not the lab's own names its account with interoplab.ZeCLICommandAs: the home CPE's converted configuration serves user admin on port 22, its password read from test/interop-topology/cpe/lab-values, and access-up asks it show interface pppoe-client that way. The CPE's input ruleset admits SSH only from its LAN, so that login dials the CPE's LAN gateway 192.168.50.1 rather than loopback, and the builder sets ze.ssh.insecure for a non-loopback host: the SSH client trusts an unknown host key only on loopback, the lab seeds none, and the session still never leaves the container. The access chain's checkers read the LAC and the LNS through their REST surface instead (POST /api/v1/execute on 127.0.0.1 port 17012, the api-server block its configurations carry), which answers the show l2tp commands as JSON.

Images are registered by name in the core. Each suite registers the images it builds from init() in its register_images.go (interoplab.RegisterImage), under a name no other suite uses (the image's tag), and derives its own list from the registry (SuiteImages, or ResolveImages where it also runs a core registration such as interoplab.FreeRADIUSImageName). A topology node names any registered image, and the suite resolves the names its plans use once each, so an image another suite declares is built from that one declaration. A name nobody registered is refused before any build, naming the registered images; two registrations of one name panic at init, naming both owners. FRR is not registered: its reference comes from each run's environment.

Typed checker operations

checkers.go is the complete scenario catalogue. Each operation identifies the peer, exact command, required or forbidden evidence, proof for negative assertions, and a bound. check_engine.go executes those operations through the shared CheckerLab interface. FRR, BIRD, GoBGP, Ze, speaker logs, and kernel state are all queried through explicit typed branches.

An absent value never proves a negative assertion by itself. The operation must also name positive evidence that the query mechanism ran. Failed and empty queries remain errors rather than becoming plausible empty protocol state.

That rule decides which command a negative assertion sends. opBIRDRouteAbsent reads the whole table with birdc show route, and never show route for <prefix>: BIRD answers a lookup for a network it does not hold with "Network not found", and birdc exits 1. A lookup therefore fails in the exact state the assertion exists to observe, and the failure is not absence. Ask a question the peer answers in both states, then read the absence out of the answer.

A checker that reads a BGP attribute from BIRD reads BIRD 3's spelling. birdc show route ... all prints each attribute on its own line as a tab, the attribute's name, a colon, a space and the value (ea_show, nest/rt-attr.c at v3.4.0), and the names are proto/bgp/attrs.c's bgp_origin, bgp_path, bgp_next_hop, bgp_med, bgp_local_pref, bgp_community, bgp_large_community, bgp_originator_id and bgp_cluster_list. BIRD 2's BGP.as_path family and its Type: BGP univ line are gone; BIRD 3 prints preference:, from: and source: lines instead, and closes each route with Internal route handling values:. An AS path is space-separated ASNs and a community list is (65001,100) (65001,200), as in BIRD 2. Captured from the lab image on 2026-10-23, for an iBGP route reflected once:

10.77.0.0/24         unicast [b 17:15:20.935 from 10.99.0.3] * (100) [AS65020i]
	via 10.99.0.2 on eth0
	preference: 100
	from: 10.99.0.3
	source: BGP
	bgp_origin: IGP
	bgp_path: 65010 65020
	bgp_next_hop: 10.99.0.2
	bgp_med: 77
	bgp_local_pref: 250
	bgp_community: (65001,100) (65001,200)
	bgp_originator_id: 10.99.0.2
	bgp_cluster_list: 10.99.0.3
	bgp_large_community: (65001, 0, 1) (65001, 2, 3)
	Internal route handling values: 0L 2G 0S id 1

Scenarios with non-linear behavior register a bespoke checker in specialCheckers (check_special.go). Each one owes a named subtest in TestBespokeCheckerBranches that drives its predicate in BOTH polarities: the true case, and a false case written against one stated wrong reading, such as two tokens matched across two log lines, a peer-originated event passing as a received one, or an absence with no proof that the query ran.

Raw frame injection from inside a peer container

A checker that must put a hand-built Ethernet frame on a peer's wire, as that peer, sends it with interoplab.SendFrameInContainer. The helper writes the frame as a one-record pcap inside the named container and runs tcpreplay there through docker exec, so the image MUST carry tcpreplay. Ze's BGP lab image does, for the isis-purge-reorig-frr own-LSP purge, and so does the PPPoE client image, for pppoe-padr-replay, pppoe-pap-ze-ac and the IPv6CP replays.

The send runs in the container, never from the checker process. On a rootful Docker daemon a container's network namespace belongs to root: an unprivileged checker cannot open /proc/<pid>/ns/net, because the kernel's ptrace access check refuses another user's process, and could not setns into it without CAP_SYS_ADMIN. docker exec already runs inside the namespace and needs neither.

The octets FRR received from Ze

When FRR prints nothing for the attribute a check is about, read the bytes instead. A scenario's frr.conf names dump bgp updates <file>, and FRR then writes every UPDATE it receives to that file as MRT, octet for octet as Ze sent it. The check runs od -An -v -tx1 <file> | tr -d ' \n' in the FRR container and asserts on the hex. bgp-prefix-sid-malformed-frr does this for the Originator SRGB TLV, which FRR parses and never shows. Its absence assertion names two proofs: the NLRI of the UPDATE it judges, so that UPDATE is in the capture, and a TLV Ze relays unchanged in another UPDATE, so the capture holds attribute bytes at all.

Querying Ze

Ze.cli(command) is the only way to ask the Ze daemon anything. It runs ze cli -c <command> --user ... --format json inside the container.

Two properties of that line are load-bearing and neither is obvious.

ze cli -c rather than the verb form ze show bgp rib status: --user and --format are flags of ze cli, and the verb form has no slot for either.

The daemon starts an SSH listener only when its config asks for one (infraSetup, cmd/ze/hub/infra_setup.go), and ze cli reaches the daemon over SSH. No scenario ze.conf asks. The harness appends ZE_CLI_CONFIG -- the listener plus the account it authenticates against -- to the RENDERED copy of every ze.conf (renderScenario), so no scenario carries the boilerplate and none can forget it. The native IPsec plan appends the same blocks in renderZeConfig (internal/le/interoplab/ipsec/ipsec.go).

Rendered configurations stay under tmp/interop-rendered in the checkout. Docker must share the checkout with its Linux VM. An unshared system-temp path gives the container a directory instead of the configuration file. IPsec and L2TP create private directories there even without an AI session ID.

A Ze helper never converts a failed query into a plausible number. Ze.rib_count raises when the command fails or answers without a routes-in field, because 0 is a legitimate RIB size and a failed query is not (ai/rules/evidence.md). It returned 0 on failure until 2026-08-07 and three separate faults hid behind that one number for three days (spec-fixit-test-harness-fail-open-guards, guard 3). Write new Ze helpers the same way.

A checker asserts on the STRUCTURED answer, never on the rendered text. The IPsec suite asks with show vpn ipsec sa | json and decodes the list of SA records (zeIKESAs, internal/le/interoplab/ipsec/helpers.go). The text rendering is a table, and its column order follows the field names, so a line-anchored <field> <value> regex over that table matches by accident. On 2026-09-06 one new field, behind-nat, took first place from child-sa in that order. Every nested Child SA key moved off the line start, and child-rekey-narrowing, peer-reload-narrowing and initiator-rekey-answer-narrows went red with no daemon behavior changed.

The VPN withdrawal checker receives its phase from private named constants, not from FRR output. An unknown internal phase is a BUG assertion; malformed or unsupported external table data remains an ordinary rejected observation.

A number two readers print in two bases is normalized to one TYPE, and the decode is typed. The dataplane-readback scenario joins the SPI set show vpn ipsec dataplane sa | json answers against the set ip xfrm state prints in the same container. iproute2 writes an SPI as 0xc1a2b3c4 and Ze answers a JSON number, so a string comparison is false for every SPI. The checker decodes the Ze answer into a struct whose spi field is a uint32, and parses the printed form to the same type. Decoding into map[string]any would route the number through float64, which holds a uint32 today and says nothing about the uint64 byte counter beside it.

An agreement between two readers asserts NON-EMPTY before it asserts EQUAL. Two empty sets are equal, so a read-only comparison passes over a kernel that holds nothing, which is what the dump answers with its whole body deleted. requireSameSPISet refuses an empty side first and names which side was empty. requireSPISetChanged refuses any old SPI that remains after rekey. The checker polls through the bounded old-SA cleanup window until Ze and iproute2 agree on a nonempty replacement set, and checks strongSwan's kernel against that set too. RFC 7296 Section 2.8 permits overlap during rekey; overlap after cleanup fails.

The same scenario reads all four Child SA byte and packet counters as uint64 values and compares them with the directed lifetime-current records from ip -s xfrm state. It samples before and after bidirectional traffic and requires each counter to advance. A bounded one-way probe makes the directions differ, so swapping inbound and outbound counters cannot pass on a symmetric ping.

Dataplane observation also crosses the live operator surfaces. With the Prometheus exporter enabled, the checker scrapes the installed-SA count and clean drift gauge, deletes one kernel SA, and requires CLI drift, degraded show health, and the corresponding gauge transition. It restores the kernel state and waits for recovery. Removing the configured peer then removes both the peer and interface-label series.

An unreadable observation is induced inside the Ze scenario container by temporarily setting only the daemon's soft RLIMIT_NOFILE to zero. The netlink package's default handle opens a socket for each XFRM dump. A Python helper finds the daemon as the one ze process whose parent pid in /proc/<pid>/stat is tini, because /proc/<pid>/task/<tid>/children exists only with CONFIG_PROC_CHILDREN, which Ze's runtime kernel does not set. The helper opens and warms an HTTP connection before changing the limit, scrapes that same connection until both dataplane metric families have no series, and restores the original limit in finally. Reconnection is refused during the fault. During the fault, live health must identify an unknown or unreadable dataplane; the preceding known-drift diagnosis cannot satisfy that assertion. Restoring the limit must restore both the drift diagnosis and the published drift gauges. The checker requires published series before the fault and their return after restoration, so an exporter that never publishes cannot satisfy the test. This helper needs Python in the lab image; it changes no production backend.

All session waiters use explicit bounds (default 90 seconds, override via SESSION_TIMEOUT). The harness passes that value into the Ze container so a compiled process helper can size its barriers against the same budget.

A compiled process helper that fails

Scenario process personalities use the Go plugin SDK. A helper failure writes the ZE-OBSERVER-FAIL sentinel to stderr and requests daemon shutdown. The checker failure path reads the last 2,000 Ze log lines and appends that measured cause when the sentinel is present.

An unreadable log is not a plugin verdict. checkerFailure retains the original scenario assertion when the log read fails, so a Docker diagnostic cannot replace the protocol failure that triggered it.

Ordered-checker failures include the selected IPv4 and IPv6 networks. A JSON wait retains its last answer, and a pmacct failure captures the collector log plus the last 80 msglog rows before container teardown. These diagnostics retain the original assertion failure; they do not substitute a guessed route state.

The external SDK carrier srv6-service-export-control announces two VPN services with different route targets and SRv6 SIDs. A destination's export policy withholds RT 10 while permitting RT 20; the unfiltered control peer must receive both SIDs. Both peers remain connected while the observer checks the socket-write counters, and the filtered peer continues rejecting RT 10 after its initial End-of-RIB. These are BGP export observations, not SRv6 dataplane reachability evidence.

The BMP Route Monitoring collector also waits past initial End-of-RIB messages. It completes only after a received-direction UPDATE carries the test prefix in its announced IPv4 NLRI. A matching byte sequence in attributes, a withdrawal, or an Adj-RIB-Out report cannot establish reception of that route.

Conditional MED observed by a foreign collector

bgp-addpath-best-path-pmacct retains its two-identifier baseline and adds a separate prefix, 10.99.77.0/24, with three sources. FRR sends A from AS65004 with MED0 and Router ID198.51.100.3. GoBGP sends B from AS65005 with MED50 and Router ID198.51.100.2. The existing injector sends C from AS65004 with MED100 and Router ID198.51.100.1. All three are external, have one AS in their path, and carry ORIGIN IGP. The checker reads the exact received attributes from pmacct's latest Adj-RIB-In route, including the absence of AIGP and received LOCAL_PREF; Ze explicitly configures their common preference base. Router ID is carried by Peer Up, not Route Monitoring. The checker joins the route to its preceding Peer Up using the BMP router, connection port, peer address, RIB view and increasing collector sequence, and requires the expected ASN in both rows. A Peer Down, a new Peer Up or a BMP stream restart invalidates the old route. An earlier correct identifier cannot satisfy a route from a different epoch.

A removes C at MED, then B defeats A at Router ID. The checker requires B, withdraws non-best A through FRR, requires C, restores A, and requires B again. Before each change it checkpoints the collector's append-only log. The selection query keeps pre-checkpoint Peer Up context and every epoch boundary, rejecting snapshots beyond its 256-line bound rather than silently truncating them. Only the newest route in the current Loc-RIB epoch may satisfy the assertion, and its absolute file position must follow the checkpoint. A matching Peer Down, stream reset or replacement Peer Up invalidates an earlier winner. AS_PATH, NEXT_HOP and ORIGIN identify the source because Loc-RIB reports omit MED. All four original baseline assertions run before and after the cycle.

DelayOpen timing against native FRR

bgp-delayopen-frr keeps Ze and both FRR peers administratively held until a bounded packet capture is active inside Ze's container. Native administrative commands then release the peers; no relay or payload rewrite sits between them. The fixture pairs a shorter Ze delay with a longer FRR delay to observe timer expiry, and a longer Ze delay with a shorter FRR delay to observe early-OPEN release. Both speakers must report Established.

The checker measures OPEN timing from each completed TCP handshake and checks every observed connection, including losers. Each peer must have one surviving connection with OPEN followed by KEEPALIVE in both directions. Missing bytes, incomplete handshakes, truncation, capture drops or an unfinished capture cannot prove that no premature OPEN was sent. Retained wire.pcap and proof.json carry the wire history, configurations and native answers; retain the suite's image-identity JSON alongside them. Opposite-initiator overlap is reported only as a collision opportunity, not proof of an internal arbitration order. The real-Peer tests retain that detailed ordering responsibility.

Timing uses each byte's first captured arrival. ACK and FIN boundaries must already have complete payload coverage at their capture time; a later retransmission or FIN cannot repair earlier evidence. Every packet needs a nonzero capture timestamp, and timestamps must be monotone across both directions of each complete address-and-port tuple. Independent connections may have records with out-of-order timestamps because packets can be timestamped and queued on different CPUs, as described in the libpcap timestamp manual. The checker retains the original packet order and timestamps. Missing times or a reversal within one connection refuse the timing proof, not the peer's protocol behavior.

Scenario Inventory

The suite has grown to over 100 scenario directories in test/interop/scenarios/. The table below lists the core BGP scenarios (01-37); beyond these, the suite also covers route reflection, policy import/export, RPKI origin validation, BMP monitoring (bmp-locrib-pmacct puts Ze's RFC 9069 Loc-RIB feed in front of pmacct and requires pmacct to print the configured Peer AS, the configured Peer BGP ID, the global VRF/Table Name TLV and reason code 6 on the Peer Down; bmp-locrib-receiver-frr turns the direction around, so FRR's bmpd drives Ze's BMP receiver and show bmp peers must report the third party's Loc-RIB peer and its address family; bgp-addpath-best-path-pmacct has the raw injector announce one prefix under two ADD-PATH identifiers, MED 10 then MED 50, and requires pmacct's reading of the Loc-RIB stream to carry the MED 10 best and never the MED 50 path, told apart by their AS_PATHs since Ze's Loc-RIB Route Monitoring carries no MED, because Ze advertises no best path to a BGP peer and the Loc-RIB feed is where a foreign implementation can observe its election), an RFC 8277 labelled withdrawal (bgp-labeled-withdraw-compatibility-frr has the raw injector announce 10.10.0.0/24 under label 100 and 10.11.0.0/24 under label 101 in ipv4/mpls-label, then withdraw 10.10.0.0/24 alone with the Compatibility value 0x800000 in the label field. FRR, fed through bgp-rs, must hold both routes, then lose 10.10.0.0/24 with exactly one logged withdrawal while the injector session stays Established. The checker then stops the injector, FRR must lose 10.11.0.0/24 through bgp-rs's peer-down withdrawals, and its log must still hold one withdrawal of 10.10.0.0/24. bgp-rs relays the withdrawal's bytes unchanged, so FRR losing the route alone cannot tell whether Ze read it: Ze's reading decides what bgp-rs keeps in its route inventory, and a misread withdrawal leaves 10.10.0.0/24 there to be withdrawn a second time at peer down), PATHS-LIMIT, max-prefix cease, a reload of the global router-id (bgp-reload-global-router-id starts Ze with 10.255.0.1, reloads it to 10.255.0.2, and requires BIRD, a static peer, and FRR, which Ze adds after the reload with create bgp peer, each to report 10.255.0.2 as Ze's BGP Identifier), GTSM (bgp-gtsm-frr for the session, and gtsm-related-icmp-ttl for RFC 5082 Section 3's related ICMP messages: FRR's kernel reads the hop limit of an ICMPv6 error ze's kernel generates about the session, and its TCPMinTTLDrop counter is what goes red when ze's host-route metric is absent), AS112, the RFC 7454 Section 9 transit leak (bgp-path-asn-leak-frr gives FRR two prefixes that differ only in their AS_PATH, and requires ze to drop the one reached through a listed transit ASN, keep the other, and keep the session), an import prefix-list over multiprotocol UPDATEs (bgp-policy-modify-mp-frr: the raw injector sends each case as one UPDATE, an IPv6 MP_REACH_NLRI with one prefix kept and one refused, IPv4 unicast inside MP_REACH_NLRI, an all-refused MP_REACH_NLRI, and thirty kept /64s with one refused so the rebuilt attribute is 291 octets, then an MP_UNREACH_NLRI for the kept /48; FRR, behind the OPEN-only relay at .11, must hold exactly the kept prefixes and lose the withdrawn one, and the relay's capture of Ze's frames must show the extended-length MP_REACH_NLRI and no MP_REACH_NLRI without NLRI), next-hop self under local ip auto (bgp-nexthop-self-local-auto-frr configures no local address toward FRR, and requires FRR to hold both ze's own route and a route relayed from a raw injector with NEXT_HOP 172.30.0.2, ze's connected endpoint, never the injector's 172.30.0.9), ADD-PATH re-advertisement (bgp-addpath-readvertise-collision-frr proves a receiver keeps two paths whose sources both chose one Path Identifier, and bgp-addpath-rail-agreement-speaker proves the live forward and the peer-up replay emit the same bytes for one path), the RFC 6793 mixed-width relay (as-path-mixed-width-relay-frr gives ze a route from a two-octet injector whose AS_PATH carries AS_TRANS and whose AS4_PATH carries the real four-octet AS number, and requires FRR to report that AS number and never 23456; as-path-prepend-two-octet-peer turns the direction around, so ze's own non-mappable AS is prepended toward an FRR that refused the four-octet AS capability), the RFC 9774 AS_SET deprecation (bgp-rfc9774-as-set-frr has the raw injector announce a route whose AS_PATH holds an AS_SET and then a clean control, and Ze originate a static route with a configured AS_SET and then its clean twin; FRR, running no bgp reject-as-sets so it would keep a set, must hold both clean prefixes and neither set prefix, and Ze's own show bgp rib must hold the control and not the injected set route, because the send guard alone already keeps a relayed set off FRR and only Ze's own RIB can show the receive half), the RFC 6368 ATTR_SET under RFC 9774 (bgp-attr-set-included-as-set-frr-bird has the raw injector announce a route whose ATTR_SET carries an included AS_PATH holding an AS_SET, then one whose included AS_PATH holds AS 0, then a clean control; RFC 9774 binds only the UPDATE's own paths, and FRR's bgp_attr_unknown and BIRD's bgp_decode_unknown keep code 128 opaque, so FRR and BIRD must both hold the set route, FRR's MRT capture of what Ze sent must hold the included AS_SET segment, and neither peer nor Ze's RIB may hold the AS 0 route, which Ze treats as withdrawn), the RFC 9774 Section 3 operator override (bgp-as-set-accept-frr has FRR and frr-sink each build a summary-only as-set aggregate and send it to Ze, FRR to a peer with session as-set accept and frr-sink to a twin peer without it; each router's own table must show the set, Ze's RIB must hold both control routes and the accepted aggregate with its set member, and must not hold the twin's aggregate, which the receive verdict line shows Ze treated as withdrawn; with the accept path forced off the run fails at the RIB wait, the accepted aggregate never arriving), the RFC 8669 Section 6 repeated Prefix-SID TLV (bgp-prefix-sid-duplicate-tlv-frr gives ze an SRv6 L3VPN route whose Prefix-SID carries the SRv6 L3 Service TLV twice, and requires FRR, whose own parser refuses a repeated type-5 TLV, to hold the route with the first SID only), the Software Version capability (frr-software-version holds two sessions to one FRR: peer legacy sends the length-prefixed form over IPv4 and must reach Established with FRR showing ze's version, while peer draft sends the draft's bare form over IPv6 and must be refused, with ze recording FRR's NOTIFICATION OPEN Message Error/Unspecific, because FRR 10.3.1 reads the first octet as a length, and with ze's log carrying the one WARN notification received line that names it), the RFC 9072 Section 2 boundary (bgp-open-extended-params-boundary-frr pads ze's capabilities to 254 octets with a 200-octet hostname, so the Optional Parameters field is 256 octets and must take the extended form, and requires FRR to reach Established and report a prefix of that hostname), and full IS-IS (auth, convergence, dual-stack, LAN DIS, P2P, redistribution) and OSPFv2/OSPFv3 (auth, BFD, TE, LFA/TI-LFA, graceful restart, segment routing, opaque LSAs, stub/NSSA, virtual links, and more) interop families.

ospf-stub-nssa-frr observes Ze's first NSSA exchange, not a replacement adjacency. Its startup wrapper waits for tcpdump's activated eth0 packet socket before executing Ze, so even a startup-only Type 5 flood is in scope. The original configuration still omits default-originate. Native FRR must reach Full and hold the exact selected, installed OSPF default through Ze. The complete Ethernet capture must independently contain both peers' NSSA Hellos and initial database descriptions, Ze's matching border-router LSA, and its non-MaxAge Type 7 default. Every outbound update is checked for Type 5; FRR's empty external database alone cannot prove no flood, because FRR discards incoming Type 5 LSAs in an NSSA.

The observation window is intentionally bounded at 120 seconds. The lab image explicitly installs GNU coreutils 9.11-r0; its fixed /usr/bin/timeout owns and joins tcpdump after SIGINT. Only the normal timeout completion, complete packet counts, zero kernel drops, valid checksums and matching capture timestamps, interface, MACs, addresses, router IDs and area support absence. Truncation, forced termination and missing positive traffic refuse proof. Retained packet/configuration/binary hashes and native FRR answers attribute this current session; they do not replace missing historical originals or establish kernel, VM or conformance results.

ospf-virtual-link-frr and ospfv3-vlink-frr put a transit FRR between Ze and the far virtual-link endpoint. VLAN 100 connects Ze eth1 to transit eth1; VLAN 200 connects transit eth2 to the far endpoint's eth2. Docker eth0 remains the management link and carries no OSPF adjacency. The routed data links use 10.200.0.0/30 and 10.200.0.4/30, with IPv6 2001:db8:100::/64 and 2001:db8:200::/64. The OSPFv2 endpoint is FRR. OSPFv3 uses the existing BIRD peer image because FRR 10.3.1 does not implement the OSPFv3 virtual-link configuration command. Ze advertises 192.0.2.1/32 and 2001:db8:10::1/128 from the dedicated passive interface backbone0, so the IPv4 host prefix is the interface's primary address rather than an unadvertised secondary beside 127.0.0.1 on lo.

The checker requires the far endpoint's address in Ze's Full backbone neighbor record and an OSPF route at the independent endpoint for Ze's backbone host prefix. Changing the transit egress cost from 10 to 30 must change the virtual cost from 20 to 40 without changing the database-exchange identity or persistent adjacency-reset counters. Taking that data interface down must withdraw the route and virtual reachability; bringing it up must restore both through a new database exchange.

Ze's existing RIB and kernel FIB plugins install the transit route. The checker reads the actual kernel route to the far data subnet, requires the intermediate router as next hop on eth1, and requires withdrawal and restoration across the same down/up cycle. IPv6 uses the transit router's observed link-local address. Neither endpoint receives a preinstalled transit or backbone route; FRR uses its normal zebra forwarding and BIRD exports OSPF routes through its kernel protocol.

ospf-self-lsa-flush-frr proves RFC 2328 Section 13.4 for an LSA Ze no longer wishes to originate. Two reloads leave FRR holding Ze's default AS-external-LSA at sequence 0x80000003 or later. FRR's eth0 then goes down, Ze is stopped and started (opStop, then opStart), so it originates the default again at 0x80000001, and ze-reload-withdraw.conf withdraws it. With no neighbour to acknowledge the flush, Ze keeps its own copy at MaxAge, 0x80000002, and the checker reads that copy out of show ospf database external before FRR comes back. In the database exchange that follows, FRR hands Ze its newer instance. Ze must flush it at MaxAge with FRR's own sequence number, so the LSA must leave FRR's database and the default must leave show ip route ospf json, whose answer is {} once FRR's OSPF table is empty. The text form prints nothing then, and the lab never reads an empty answer as an absence. A Ze that re-originates the withdrawn body at age 0 leaves both in place, and the absences time out. opStop returns only once Docker reports the container stopped, so the start that follows cannot race a container that is still shutting down. Results on 2026-11-14 (arm64, QEMU docker-lab guest, FRR 10.7.1): with the if local.Age.IsMaxAge() branch removed from (*LSDB).handleSelfReceived, FAIL at assertion 21, the withdrawn LSA never leaving FRR's database; the real build passes (2m24s).

ospf-gr-frr and ospf-gr-helper-broadcast-frr prepare FRR 10.7.1 with graceful-restart prepare ip ospf, save ospfd's PID, and send that process SIGTERM. Preparation alone does not restart the daemon. Watchfrr owns the restart; the checker neither starts a competing ospfd nor stops zebra, the container, or its network. The existing post-restart Full wait also requires a different ospfd PID. The broadcast helper still requires no prior helper exit, then reason="flushed" within 100 seconds, shorter than the 120-second grace period, and Full on both sides. The helper fixture enables Prometheus so those assertions read the daemon's actual counters; Netdata collectors are disabled because they are not involved. Without the registry, show metrics values refuses instead of proving that no earlier helper exit occurred.

FRR's ospf_gr_prepare and ospf_gr_restart_exit originate and flush the Grace-LSA; ospf_finish_final preserves LSAs and routes during prepared shutdown, and daemon_down, wakeup_down, and try_restart drive the supervised restart.

ospf-sr-frr and ospfv3-sr-frr read the kernel label table with the installed iproute2 command ip -f mpls route show 16100, not the absent mpls -ls utility. The destination filter selects label 16100; the existing content assertion still requires that label, so an empty successful query fails. Their 60-second label deadlines and adjacency assertions are unchanged. In ospf-sr-frr, FRR's Prefix-SID prefix 10.255.255.3/32 sits on FRR's loopback: FRR's sr_prefix_sid looks the prefix up on an interface and, when none holds it, defers the Prefix-SID without an error, so no label 16100 would reach Ze.

ospf-sr-anycast-nexthop-frr proves RFC 8665 Section 5's rule that a router "MUST take into account ... the E-, NP-, and M-Flags advertised by the next-hop router if that router advertised the SID for the prefix", for a next hop that is not the binding's Originator. Ze and two FRRs share one broadcast LAN with SR on. Each peer enables segment-routing on inside router ospf, before configuring its SRGB and Prefix-SID. FRR 10.7.1 registers the enable command only in that context; placing it at top level leaves SR disabled even when adjacency forms. Both FRRs bind index 100 to the anycast 10.255.255.100/32 on lo with PHP requested: frr (router-id 192.0.2.3) at stub cost 1, so it is Ze's only next hop, and frr-sink (192.0.2.1) at cost 50. Ze reads Extended Prefix LSAs in LSDB key order, and both FRRs use the same Opaque ID, so 192.0.2.1 becomes the Originator and the next hop is the second advertiser. The checker first requires that order from Ze's show ospf database opaque-area (a lab where the next hop became the Originator passes either way, so it fails instead). It then reads ip -f mpls route show in Ze's container until in-label 16100 is popped toward the next hop, holds that pop for 15 seconds (three times OSPF's 5-second SPF hold ceiling), failing on the first read that is not the pop, and re-reads the LSDB order at the end. A single read cannot discriminate: the first pop can be the route Ze installed before the second advertiser's LSA reached its SR table, when the next hop was the binding's only advertiser, and a Ze applying only the Originator's flags pops then too. Such a Ze later swaps 16100 to 17100, the next hop's SRGB base plus the index. Results on 2026-11-14 (arm64, QEMU docker-lab guest, FRR 10.7.1): with the body of (*srInstaller).forwarding restored to its form before 19de202493, FAIL at assertion 3, "the pop held for 5.046s, then sample 5 found in-label 16100 is swapped, not popped: 16100 as to 17100"; the real build passes (1m49s).

ospf-ext-prefix-link-frr is the Extended Prefix/Link scenario: it proves both directions of the RFC 7684 exchange. FRR enables capability opaque and segment-routing on inside router ospf and holds its Prefix-SID prefix on a loopback, so it originates Extended Prefix and Extended Link LSAs. Without the loopback FRR defers the Prefix-SID and originates no Extended Prefix LSA. Ze originates both types too, so a substring match on Ze's show ospf database opaque-area passes on Ze's own LSAs alone. The checker decodes that answer and binds each decoded row to its advertising-router: it requires an Extended Prefix row and an Extended Link row from FRR's router-id, and one of each from Ze's. On FRR's side it reads show ip ospf database opaque-area json and requires, under areaLocalOpaqueLsa.areas, an LSA whose Link State ID has opaque type 7 and one with opaque type 8, each with advertisingRouter equal to Ze's router id. A match on Ze's address alone is not enough, because FRR's own Extended Link LSA names Ze as its Link ID. The Ze read waits rather than reading once at Full, because FRR schedules each opaque type min_ls_interval after the previous once an opaque-capable neighbor appears (ospf_opaque_lsa_originate_schedule). Results on 2026-11-14 (arm64, QEMU docker-lab guest, FRR 10.7.1): with FRR's Prefix-SID on no loopback, FAIL at assertion 2, "no decoded Extended Prefix LSA advertised by 172.30.0.3"; with the fixture corrected, pass (1m13s).

bgp-graceful-restart-frr retains its original session, route, GR and EOR checks and adds two strict input speakers with FRR as the independent receiver. One source omits IPv4 from GR while advertising a 40-second IPv4 LLST; its IPv6 family has a 20-second GR period followed by the same LLST. The other source advertises zero LLST. After fencing initial receipt and bilateral FRR LLGR negotiation, the checker sends each native source USR1 through Lab.Signal. Only this oracle handles that signal: it closes its BGP socket without NOTIFICATION, stops KEEPALIVEs, and keeps PID 1, the container and its next-hop interfaces alive until its existing bounded lifetime ends or lab cleanup. Unexpected peer closure, NOTIFICATION, or lifetime expiry before control remains a failure. The main loss timestamp precedes the signal; the DOWN poll and three-second observation margin are unchanged. The checker requires immediate removal for zero LLST, immediate IPv4 LLGR_STALE, and IPv6's conventional-to-LLGR transition. IPv4 must expire at its original DOWN+40 deadline while IPv6 survives until DOWN+60. FRR's established/dropped connection counters must remain unchanged; its recomputed wall-clock epoch is not a session identity. Complete observation cycles, including final expiry probes, must stay within the sampling bound. An explicit FRR-origin received-route check preserves the original reverse direction proof despite the added native sources. This does not cover NOTIFICATION, reconnect F-bit or received-stale-policy behavior.

The receipt query is show bgp rib received | json; its CLI output is a top-level array of route rows, not an object containing a routes field. An empty array is a measured RIB with no matching route; null or an object is not a valid CLI observation.

The received-route fence retains the last completed CLI result (stdout, stderr, exit status and parse error), the last successfully measured RIB, and probe counts when it fails. A final cancelled Docker exec cannot overwrite this earlier evidence. Capturing it adds no queries or time to the original 30-second receipt fence and changes none of the 40/60-second expiry windows. FRR route-observation errors retain the queried prefix and raw JSON response, so an invalid route can be distinguished from an unexpected response shape.

bgp-nexthop-self-local-auto-frr retains both original self-next-hop checks and adds AIGP on the general forwarding path. The received metric is 100; source-link cost 7 and destination-link cost 43 must produce 107 and Ze's next hop at FRR. GoBGP is the unchanged-next-hop control and must retain 100. The fixture's ASes are in one administrative domain. FRR 10.3.1 requires both neighbor ... aigp and neighbor ... oad to retain received AIGP on eBGP; aigp alone leaves the route installed but discards its AIGP attribute. oad keeps the session eBGP, including its AS-path and next-hop assertions. See the pinned FRR receive gate.

A third-party next hop initially has no distance: the rewritten route must be absent while the unchanged control holds it. An SDK route installation with metric 11 restores 111, then metric 7 produces 107; replacing that distance with zero withdraws the whole rewritten route, and restoring 7 recovers it again. Source UPDATE, EOR and session counters fence the recovery against a new advertisement. FRR's established/dropped counters and local/foreign socket ports are also captured before the initial withheld inventory and checked after every metric transition and the final baseline checks. Recovery through a replacement recipient session cannot satisfy that fence.

A separate iBGP wire recipient uses the existing speaker-args container. Its passive Ze peer binds the explicit lab address; local ip auto supplies no address for a passive listener. The original FRR local-auto checks remain unchanged. The wire recipient records one connection's original received frames, never reconnects, and must observe exact AIGP TLVs 107 and 111, Ze's next-hop bytes, and a complete withdrawal of 10.10.3.0/24. Its history also checks the subsequent 107/withdrawal/107 transitions. Both the FRR readiness prefix and a checker release route precede injection; the release follows all recipients' initial EORs. This carries the former aigp-source-cost-recovery draft's wire contract on a legal source topology: 127/8 NEXT_HOPs are rejected before AIGP, even when unassigned to a host interface. Run it with INTEROP_SCENARIO=bgp-nexthop-self-local-auto-frr ./le test integration interop.

bgp-labeled-withdraw-compatibility-frr also exercises VPNv4 and VPNv6, with each prefix announced under three distinct RDs. Compatibility values 800000 and 123456 withdraw two RDs while the third survives. FRR's RD-keyed table and exact withdrawal log identify each route. After the injector goes down, only the surviving RD may receive another withdrawal; the previously removed routes must not be withdrawn twice. Receiver session generation and a subsequent received KEEPALIVE fence the final observation.

bgp-vpn-withheld-withdraw-frr covers the sending half of RFC 8277 Section 2.4 on the forward rail. The injector announces two VPNv4 routes at one RD and two labelled-unicast routes, each under one label, then re-announces one of each under three labels with NO_EXPORT, which ze withholds from FRR (an external peer) and converts into withdrawals. FRR must lose exactly those two routes, keep both siblings, decode one withdrawal of each, and keep its session generation, having decoded only the one-label VPN announcement. A VPN withdrawal that copied the three-label stack would leave FRR a 72-bit IPv4 prefix and be refused. The labelled-unicast leg does not discriminate: FRR walks a copied stack to its bottom-of-stack bit.

bgp-flowspec-sctp-gobgp supplies both a covering unicast route and a FlowSpec from GoBGP, checks the installed kernel rule, then withdraws and restores only the unicast route. Its checker requires the existing FlowSpec filter to disappear and return without another FlowSpec announcement. Finally it withdraws the FlowSpec itself. Empty-ruleset checks require nft's JSON response, not a table that should have been removed.

bgp-flowspec-gobgp retains its originated-rule and OR-of-AND checks and adds a received rule with a nonzero next hop. A byte-preserving relay captures the UPDATE delivered to GoBGP: its MP_REACH next-hop length must be zero, and GoBGP must decode the rule and traffic-rate action. Withdrawing and restoring only the covering route drives stored replay without a new source FlowSpec UPDATE. The checker requires the same rule and action after recovery. This independent-peer scenario covers IPv4 FlowSpec; the four-family wire matrix remains a separate internal proof.

The Linux integration-tagged TestSelectedFlowSpecKernelPacketSemantics uses an isolated network namespace and actual UDP packets to distinguish ordered terminal and continuing actions, deferred DSCP marking, withdrawal, and excess-rate drops. It requires CAP_SYS_ADMIN, CAP_NET_ADMIN, and nftables kernel support. A capability skip is not packet-forwarding evidence.

# Scenario Daemons What It Tests
01 bgp-ebgp-ipv4-frr Ze, FRR Basic eBGP session establishment
02 bgp-ebgp-ipv4-bird Ze, BIRD Basic eBGP session with BIRD
03 bgp-ibgp-frr Ze, FRR iBGP session (same AS)
04 bgp-4byte-asn-frr Ze, FRR 4-byte ASN negotiation (RFC 6793)
05 bgp-routes-from-frr Ze, FRR Ze receives routes originated by FRR
06 bgp-routes-from-bird Ze, BIRD Ze receives routes originated by BIRD
07 bgp-routes-to-frr Ze, FRR FRR receives routes originated by Ze
08 bgp-triangle Ze, FRR, BIRD Three-way topology, multi-peer stability
09 bgp-route-withdrawal-frr Ze, FRR Route withdrawal propagation
10 bgp-ipv6-ebgp-frr Ze, FRR IPv6 eBGP session and route exchange
11 bgp-addpath-frr Ze, FRR ADD-PATH capability (RFC 7911)
12 bgp-route-refresh-frr Ze, FRR Route Refresh (RFC 2918)
13 bgp-graceful-restart-frr Ze, FRR Graceful Restart negotiation (RFC 4724)
14 bgp-route-server-frr Ze, FRR, BIRD Route server: forwards without inserting own ASN
15 bgp-community-frr Ze, FRR Standard community propagation
16 bgp-extended-community-frr Ze, FRR Extended community propagation
17 bgp-md5-auth-frr Ze, FRR TCP MD5 authentication (RFC 2385)
18 bgp-ebgp-gobgp Ze, GoBGP eBGP session with GoBGP
19 bgp-routes-gobgp Ze, GoBGP Route exchange with GoBGP
20 bgp-role-frr Ze, FRR RFC 9234 Role capability negotiation
21 bgp-role-gobgp Ze, GoBGP RFC 9234 Role capability negotiation
22 bgp-evpn-frr Ze, FRR EVPN Type-2 route exchange; FRR accepts an IP Prefix route (Type 5) whose Overlay Index is the ESI. FRR 10.3.1 prints no Type 5 ESI in any view, so bgp-evpn-gobgp asserts its octets
23 bgp-vpn-frr Ze, FRR, BIRD VPN-IPv4 route exchange with FRR; RFC 4659 Section 3.2.1.1 VPN-IPv6 next hops, read at BIRD (the 48-octet field's link-local, which FRR 10.3.1 does not display) and at FRR (the off-link route's sole global)
24 bgp-flowspec-frr Ze, FRR FlowSpec rule exchange
25 bgp-ipv6-ebgp-bird Ze, BIRD IPv6 eBGP route exchange
26 bgp-ipv6-ebgp-gobgp Ze, GoBGP IPv6 eBGP route exchange
27 bgp-multihop-ebgp-frr Ze, FRR Multi-hop eBGP with outgoing-ttl
28 bgp-evpn-gobgp Ze, GoBGP EVPN Type-2 route exchange, Ethernet A-D ESI Labels, and the IP Prefix route (Type 5) sent with overlay-index esi: GoBGP reads the non-zero ESI and a zero GW IP from the NLRI (RFC 9136 Section 3.1)
29 bgp-vpn-gobgp Ze, GoBGP VPN (L3VPN) route exchange
30 bgp-flowspec-gobgp Ze, GoBGP FlowSpec rule exchange
31 bgp-multihop-ebgp-bird Ze, BIRD Multi-hop eBGP with outgoing-ttl
32 bgp-multihop-ebgp-gobgp Ze, GoBGP Multi-hop eBGP with outgoing-ttl
33 bfd-frr Ze, FRR BFD opt-in and BFD-triggered BGP teardown
34 bgp-ecmp-frr Ze, FRR, GoBGP FRR ECMP selection for the same prefix from Ze and GoBGP
35 bgp-srv6-frr Ze, FRR SRv6 VPNv6 route exchange and Prefix-SID handling
36 bgp-remove-private-as-frr Ze, FRR, GoBGP remove-private-as export policy to FRR
37 bgp-remove-private-as-as4path-frr Ze, FRR, BIRD remove-private-as handling for AS4_PATH private ASNs
38 bgp-vpn-addpath-frr Ze, FRR ipv4/mpls-vpn over ADD-PATH with no path-information: FRR holds the exact RD, prefix and label, so ze wrote Path Identifier 0 (RFC 7911 Section 3)
39 ldp-keepalive-expiry-frr Ze, FRR ldpd ze's LDP KeepAlive Timer expires after an ldplab nft table in ze's container rewrites the TCP window ze advertises on the session to zero, so FRR can send ze no PDU while its kernel still acknowledges ze's segments (a drop of FRR's segments would also drop the ACKs they carry and stall ze's own stream); FRR logs msg[in]: notification: ... status KeepAlive Timer Expired (fatal error) (RFC 5036 Section 3.5.1.2.3) and leaves OPERATIONAL. Red with the send reverted: FRR logs only connection closed by remote end. Ships its own daemons (ldpd=yes) and gives ze transport address 172.30.0.250 so ze is the active LSR
53 ldp-init-unknown-tlv-frr Ze, FRR ldpd RFC 5036 Section 3.3 on ze's receive side: "if U is clear (=0), a notification MUST be returned to the message originator and the entire message MUST be ignored". FRR cannot send an unknown TLV, so an ldplab nft rule in ze's container rewrites the type of the first Capability TLV in FRR's real Initialization (0x8506, the Dynamic Capability Announcement ldpd's send_init writes at LDP offset 36) to 0x3f00, U and F clear, and counts each match. FRR, the originator, logs msg[in]: notification: ... status Unknown TLV, and stays short of OPERATIONAL for 10 s because ze never answers the ignored Initialization with a KeepAlive. With the rule removed the same peers reach OPERATIONAL, the positive control. FRR proves the Notification reached the originator and decodes as Unknown TLV; the rule's counter proves ze was shown the TLV. Red with skipTLV returning nil for every TLV: FRR logs no received status Unknown TLV Notification. Shares the daemons and the 172.30.0.250 transport address of ldp-keepalive-expiry-frr
66 ldp-backoff-clear-frr Ze, FRR ldpd A session that reached OPERATIONAL leaves no setup backoff when it ends (RFC 5036 Section 2.5.3, "After such a configuration action, there is no further need to throttle subsequent session establishment attempts"). Once FRR has reported the session OPERATIONAL for 3 s, FRR runs clear mpls ldp neighbor <ze's transport address>: ldpd's nbr_clear_ctl calls session_shutdown with S_SHUTDOWN, so FRR sends a Shutdown Notification and closes the TCP connection while the adjacency stays. FRR holds the passive role and ze dials on each Hello FRR sends (processDiscoveryPacket, startSessionForAdj), every 1 s by discovery hello interval 1. FRR's show mpls ldp neighbor json must report ze OPERATIONAL again with an upTime no older than the clear (nbr_to_ctl restarts it on each entry to OPERATIONAL) within 8 s of the clear: the released backoff lands near 2 s, a recorded one forbids the dial for 15 s. ze's own log, read at info (environment { log { ldp info } }), must carry a new fatal Notification and a new session end, the proof that the clear reached ze and that info lines are read, and no new ldp: session setup failed, next attempt delayed. Red with the operational arm of startSessionForAdj's session-end callback recording a setup failure (one line changed in the working tree for the run, because the ze image builds from the tree; image sha256:a218f6cd97d5 and sha256:c58b3f2734ff, 2026-11-13): ze logged "ldp: session setup failed, next attempt delayed" 1 time(s) after FRR cleared a session that had reached OPERATIONAL: its end recorded a setup backoff, which delays the redial by 15 s or more (the probes ended 15.454s after the clear). Shares the daemons of ldp-keepalive-expiry-frr. The lab adds ze's 172.30.0.250 transport address in ze's prestart (scenarioHostAddresses), because ze dials from it on FRR's first Hello: added by the checker, the address could arrive after that dial, and each dial that fails to bind costs a backoff step of 15 s or more. sha256:3c4dee38ac2e
40 bgp-attr-flags-low-nibble-gobgp Ze, GoBGP, raw injector RFC 4271 Section 4.3: attribute 241 sent to Ze at flags 0xCF reaches GoBGP at 0xE0 (low nibble cleared, Partial stamped by Section 5), never 0xEF. Results on 2026-11-14 (arm64, QEMU docker-lab guest): with attrs[pos] &^= attrFlagsUnusedMask deleted from normalizeTransmittedUpdate, FAIL after 2m41s at assertion 3, the route never reaching GoBGP; the real build passes (1m5s)
41 bgp-large-community-no-repeat-gobgp Ze, GoBGP, raw injector, native speaker RFC 8092 Section 3: an export policy adding values the route already carries leaves GoBGP with each Large Community once, and the native speaker's no-repeated-large-community oracle finds no value sent twice on the wire. GoBGP alone cannot go red: it removes repeats on receipt, as the same section requires of a receiver (v4.10.0 decodes every value, then ValidateAttribute in drops each repeat). Its ze.conf lists bgp-filter-community:egress in the export chain, which Ze requires since a965ee38fb. Results on 2026-11-14 (arm64, QEMU docker-lab guest): with the AttrLargeCommunity branch of normalizeTransmittedUpdate disabled, FAIL after 3m8s at assertion 6, the oracle reporting "LARGE_COMMUNITY value 0000fde80000000100000001 transmitted twice in one UPDATE" and the same for 65000:2:2, while GoBGP's own view (assertion 4) still passed; the real build passes (1m42s). pkg/packet/bgp/validate.go
42 bgp-mup-malformed-t1st-withdraw-gobgp Ze, GoBGP, raw injector draft-ietf-bess-mup-safi Section 3.1.3.1: a malformed T1ST withdraws the earlier generation at GoBGP, a well-formed sibling stays, the session stays up. Results on 2026-11-14 (arm64, QEMU docker-lab guest): with recognizerFor returning nlritype.Keep where it returned Withdraw, FAIL after 3m4s at assertion 4, the wait for the valid route sharing the UPDATE timing out; the real build passes (2m2s)
43 bgp-role-rs-client-learned-not-upward-frr Ze, FRR 10.7.1, raw injector RFC 9234 Section 3.1: a route learned on a session where Ze is the Route Server never reaches FRR, Ze's Provider; Ze's own control route does. Results on 2026-11-14 (arm64, QEMU docker-lab guest): with || srcRole == roleRS removed from OTCEgressFilter, FAIL after 1m22s at assertion 4, FRR holding 10.10.0.0/24, which must be absent; the real build passes (48s)
44 bgp-rfc7999-blackhole-readvertise-frr Ze, BIRD, FRR 10.7.1 RFC 7999 Section 3.1: a BLACKHOLE-tagged route from BIRD is never re-advertised to FRR, an eBGP neighbor with no agreement; the untagged control is. Results on 2026-11-14 (arm64, QEMU docker-lab guest): with blackholeReadvertiseRefused returning false, FAIL after 2m2s at assertion 5, FRR holding 10.200.0.1/32, which must be absent; the real build passes (1m34s)
45 bgp-sr-policy-gobgp Ze, GoBGP RFC 9830: GoBGP's RIB, read through gobgp-paths (Dockerfile.gobgp, a gRPC ListPath reader, because no gobgp CLI release names an SR Policy family), holds Ze's configured IPv4 and IPv6 SR Policies with their distinguisher, color and endpoint, Binding SID label 1048575 and Type A segment labels 16 and 1048575, and a third IPv4 policy (color 300) whose SRv6 BSID 2001:db8::b5 rides the type-13 Binding SID sub-TLV at Length 18 (RFC 9830 Section 2.4.2), which GoBGP prints through its type-13 decoder with no B-Flag. Every policy must also carry the configured Route Target as GoBGP types it, (RFC 9830 Section 1, RFC 4360 Section 4). Red with parseConfigRoute writing sub-type 3 (Route Origin) into the community it appends: GoBGP still holds every policy and String still prints. Removing the append is not a red for this assertion, because the config parser then refuses the configured extended community and Ze never starts. Goal validation for origination: in-range labels never reach the 20-bit range check, so no revert turns the label checks red. {"type":1,"subtype":2,"value":"172.30.0.5:0"}[172.30.0.5:0]
46 bgp-sr-policy-malformed-receive-gobgp Ze, GoBGP, raw injector RFC 9830 Section 5: after policies A, B and control C reach GoBGP, A beside a 13-octet NLRI and then B without its Tunnel Encapsulation attribute are each treat-as-withdraw, so GoBGP loses A, then B, keeps C, and its session with Ze stays up: Ze's peer detail for GoBGP shows one connection established and none dropped (GoBGP resets a session that carries an SR Policy NLRI whose length is neither 96 nor 192 bits)
47 bfd-link-local-frr Ze, FRR bfdd RFC 5881 Section 3: a BFD-plugin-pinned single-hop session to FRR's IPv6 link-local on eth0 reaches Up, each side holding the other's discriminator. Acceptance, not discrimination (see "BFD over link-local")
48 bfd-link-local-two-links-frr Ze, FRR bfdd RFC 5881 Section 3: two sessions to ONE link-local address, one each on eth1 and eth2, reach Up on their own link with their own discriminators crossed, while Ze's Your-Discriminator-zero sends are dropped. Discriminates the receive-side zone reconciliation and the send-side link pinning (see "BFD over link-local")
49 bgp-exabgp-bridge-sr-policy-gobgp Ze (exabgp bridge), GoBGP RFC 9830: an ExaBGP API script writes announce ipv4 sr-policy ... preference 300 with two Type A segment lists and a Route Target; the bridge hands it to update text, whose sr-policy section reaches the registered config-route parser, and GoBGP's RIB (through gobgp-paths) holds the NLRI (color 300, endpoint 10.0.0.3), the Preference sub-TLV, labels 16001 and 16002, and the Route Target typed as sub-type 2. Red with the sr-policy registration's NLRISectionCarriesAttributes set false, which stops the hand-off. The Route Target on this path comes from update text's own attribute block, not from parseConfigRoute: red with the extended-community case of update_text.go writing sub-type 3, and green with the same change made in parseConfigRoute
50 bgp-exabgp-bridge-rtc-gobgp Ze (exabgp bridge), GoBGP RFC 4684 Section 4: an ExaBGP API script writes announce ipv4 rtc origin-as 65001 route-target 65001:100 next-hop. ; the bridge hands it to update text. mp ipv4/rtc add, and GoBGP's RIB (through gobgp-paths 1 132) holds membership. The 0-bit default membership is not sent: GoBGP v3.31.0 panicked on it (nil Route Target in BgpServer.propagateUpdate); v4.10.0 handles a nil Route Target (processRTCMembership), and the scenario still sends the specific membership only. Red with the bridge's rtc row of bridgeSAFI removed, which refuses the line. 65001:65001:100pkg/server/server.go
51 bgp-exabgp-bridge-vpnv4-frr Ze (exabgp bridge), FRR RFC 4364: an ExaBGP API script writes ExaBGP 6.0's announce ipv4 mpls-vpn 10.0.0.0/24 next-hop ... rd 65001:100 label 100; the bridge hands it to update text ... mp ipv4/mpls-vpn add, and FRR holds the route under RD 65001:100 with remote label 100, its session staying up. Red with the bridge's mpls-vpn row of bridgeSAFI removed, which refuses the line
52 bgp-flowspec-redirect-check-frr Ze (route server, bgp-rib), FRR, raw injector draft-simpson-idr-flowspec-redirect-02 Section 3: the injector (AS 65004) sends a cover for 192.0.2.0/24 originated in AS 65009, then a FlowSpec rule for that prefix with the 0x0800 redirect-to-IP community and next hop 192.0.2.77. The longest match for the next hop is not originated in the peer's AS, so FRR must hold the rule (show bgp ipv4 flowspec detail json) with no FS:redirect in its ecomlist. No lab daemon originates a 0x0800 rule, hence the raw frames; both reach FRR through Ze's peer-up replay. The plain show bgp ipv4 flowspec json is FRR's generic route printer and carries no extended communities, so a check on it passes either way. Red with the RIB's refusal disabled (flowSpecRedirectRefused answering false, by Go overlay): FRR holds "ecomlist":"FS:redirect IP 0x0"
54 bgp-route-server-replacement-frr Ze (route server, bgp-rib), FRR, GoBGP, BIRD RFC 4271 Section 9.2 and RFC 7947 at a client that negotiated no ADD-PATH: BIRD holds one route for 10.98.0.0/24, and Ze decides whose. The checker first asserts from BIRD's show protocols all ze_peer that no ADD-PATH capability was exchanged. GoBGP announces the prefix with one prepended AS, and BIRD holds AS_PATH 65004 65004. FRR announces it, better and later, and BIRD moves to 65002. GoBGP re-announces it, still worse, with community 65004:2; Ze's show bgp rib received community 65004:2 must list the prefix, the proof the revision reached the route server, and BIRD must keep 65002 through a 20 s window read every second (assertion 12). FRR withdraws it with no network; BIRD must reach 65004 65004 with no reading in between that lacks the prefix (assertion 14), then keep it through the window. The two sources are attached receive-only: in a run where Ze also relayed to them, FRR re-advertised GoBGP's route back as 65002 65004 65004 (a route server adds no AS, so nothing stops the loop), and FRR's no network became an implicit replacement that never exercised the withdrawal half. Red on 2026-10-26, each from a git archive export of HEAD with the Ze image rebuilt: oweAnnouncement answering false fails assertion 12 at reading 1, BIRD holding 65004 65004; owesWithdrawal answering false fails assertion 14 at reading 1, BIRD answering the prefix lookup with no route
55 bgp-route-server-control-community-frr Ze (route server), FRR, BIRD RFC 7947 Section 2.2.2.1 and the route server's large control communities: FRR (AS 65002) announces 10.99.0.0/24 with, asking Ze (AS 65001) to prepend once toward BIRD (AS 65003), and 10.98.0.0/24 with, naming a client that is not BIRD; both carry FRR's own. BIRD must decode 65001 65002 for the first and 65002 for the second, and its bgp_large_community must hold (65002, 1, 1) and no value under 65001. Red on 2026-10-27, working tree with the Ze image rebuilt by the harness: RSPrepend returning its input fails assertion 5, BIRD decoding 65002 for 10.99.0.0/24; StripLargeControlCommunities returning nil fails assertion 6, BIRD holding (65001, 101, 65003) (65002, 1, 1). 65001:101:6500365001:101:6509965002:1:1
56 bgp-paths-limit-receive-bird Ze, BIRD 3.4.0 draft-abraitis-idr-addpath-paths-limit Section 3, the receive ceiling against a sender that ignores it: BIRD has no capability 76 and, with add paths tx, exports every route its table holds, so four static protocols give 10.70.0.0/24 four paths. Ze advertises limit { receive 2; } and must hold exactly two in show bgp rib received, count two in ze_bgp_paths_limit_refused_total, install 10.71.0.0/24 and keep the session. The wait ends once every offered path is held or refused, so the held count is read after the last offer. Red on 2026-10-27, processValidatedMessage skipping receiveCeiling (a GOFLAGS=-overlay build, image rebuilt by the harness): assertion 3, "ze holds 4 paths of 10.70.0.0/24, want the advertised ceiling 2 (refused 0)", which is also the reading that BIRD offered four
57 bgp-paths-limit-best-path-frr Ze (route server, bgp-rib), FRR 10.7.1, BIRD 3.4.0 draft-abraitis-idr-addpath-paths-limit Section 1, best paths under a peer's limit: FRR advertises PATHS-LIMIT 2 for ipv4/unicast, and BIRD (add paths tx, allow bgp_med yes) gives Ze four paths of 10.72.0.0/24 one at a time, worst first, by enabling static protocols MED 300, 200, 100, then 50. After each step FRR's show bgp ipv4 unicast 10.72.0.0/24 json must hold exactly the two best MEDs from Ze; more than two fails at once. Disabling MED 50 must bring MED 200 back under the path identifier FRR held it with before its demotion. Without allow bgp_med, BIRD strips a MED on an EBGP export that its own export filter did not set, and every path arrives with none. Red on 2026-10-27, owePathsLimitElections returning before it owes an election (a GOFLAGS=-overlay build, image rebuilt by the harness): assertion 6, FRR holding MED 300 and MED 200 after MED 100 arrived
58 bgp-session-protection-frr Ze (GTSM ttl { max 1; }, control-plane-protection bgp), FRR 10.7.1 RFC 5082 Section 2.2 and Section 3, and RFC 7454 Section 4, read at the sender: FRR runs ttl-security hops 1, and the checker opens python3 connects to Ze's port 179 inside FRR's container, because FRR's image has no other tool that sets a socket's TTL. A SYN from FRR's address at TTL 254 must time out (only the ze_gtsm listen-port term can drop it: FRR is a CoPP neighbor); the same address at TTL 255 must complete the handshake, the control that proves the probe can see a SYN-ACK; a SYN at TTL 255 from 172.30.0.40, an address the checker adds to FRR's interface and Ze does not know, must time out, never reset (only CoPP's unknown-source-drop can drop it: GTSM leaves an unknown source alone), and that term's counter in nft list table inet ze_copp must rise. FRR's session must keep its established and dropped counts and both ports across the probes. Red on 2026-10-27, each producer broken by a GOFLAGS=-overlay build with the image rebuilt by the harness: peerTerms without the listen-port term, assertion 4, "a SYN from 172.30.0.3 at TTL 254 got a completed handshake from ze"; translatePolicy without unknown-source-drop, assertion 6, "a SYN from the unconfigured 172.30.0.40 got a completed handshake from ze"
59 rtr-stayrtr-preference Ze (two RTR caches, preference 10 and 20), StayRTR v0.6.4 twice, FRR 10.7.1, BIRD 3.4.0 RFC 8210 Section 10, cache preference, failover and fail-back observed at BIRD's export: the preferred cache's set is in use first, the standby's after the checker stops the preferred container, the preferred cache's again after it starts it ("RTR cache preference, failover and fail-back" above). Red on 2026-10-27 in phase 1 with the preference sort reversed, and in phase 3 with fail-back disabled. Green in 10 of 10 runs once the StayRTR image carried its Error Report patch and a readiness probe that waits for RTR
60 bgp-panic-cease Ze (the ze_le daemon), FRR 10.7.1, BIRD 3.4.0 RFC 4271 Section 6.7 and Section 4.5, a panic on one session's read path contained to that session: the scenario's ze-le-daemon file starts the ze_le build, the checker creates /run/ze-inject/bgp-read-panic-<FRR address> in ze's container and makes FRR redistribute 10.49.1.0/24, and the injected panic on that UPDATE must reach FRR as a received NOTIFICATION Cease / Unspecific (lastResetDueTo naming a received notification, lastNotificationReason a Cease with no named subcode). FRR's session must come back (two establishments, one drop), while ze counts one establishment and no drop toward BIRD and BIRD stays Established. Green on 2026-10-27, twice. Red on 2026-10-27, ceaseAfterPanic without its tryNotificationWithin send (a GOFLAGS=-overlay build, image rebuilt by the harness): assertion 5, FRR's lastResetDueTo "Peer closed the session" after ze logged the contained panic. The shipped ze-linux holds neither injectReadPanic nor the trigger path; only ze-le-linux does
61 isis-import-policy-frr Ze, FRR 10.7.1 twice (FRR-A, FRR-B on separate links) isis filter import decides what Ze installs and nothing else: the hidden prefix is out of Ze's RIB and FIB, listed by show rib candidates as hidden by the isis import chain, and still flooded on, so FRR-B learns it through Ze ("Separate links between daemons")
62 ospf-import-policy-bird Ze, BIRD 3.4.0 ospf filter import sets distance 200 on BIRD's tagged external, which then loses to a static path at 150, while the untagged one keeps 110 and wins, judged per path in show rib candidates
Paired import isis-import-policy-frr-forked Ze with external system RIB, FRR 10.7.1 twice Same source-hidden/control and onward-flooding assertions as the internal scenario, with exact subprocess placement and consumed peer identities. Live paired validation remains required.
Paired import ospf-import-policy-bird-forked Ze with external system RIB, BIRD 3.4.0 Same tagged-distance/control assertions as the internal scenario, plus the central-only reject on 10.88.3.0/24 in both placements. Live paired validation remains required.
63 bgp-rfc8212-default-reject-frr Ze (route server, bgp-rib), FRR 10.7.1 twice (FRR at host 3 with no Ze policy, the FRR sink at host 15 with an explicit Ze policy) RFC 8212 Section 3, the EBGP default reject in both directions, against an FRR that keeps its own bgp ebgp-requires-policy and holds explicit route-maps toward Ze: with no policy on Ze's FRR session, Ze's show bgp rib best must hold the sink's 10.81.15.0/24 and not FRR's 10.81.1.0/24 (after FRR's advertised-routes names it and Ze counts its UPDATE and End-of-RIB), FRR must hold neither Ze's config-originated 10.81.0.0/24 nor the sink's route, and the sink must receive 10.81.0.0/24 from the same config update path. A reload adding a bgp-level final accept in both directions must then deliver every route across the FRR session, with FRR and the sink each counting one establishment and no drop. Results on 2026-10-31, each image built by the harness: unmodified, assertions 1 to 15 pass, so both default rejects hold and the reload's export recovery reaches FRR with no reset, and assertion 16 fails: FRR's 10.81.1.0/24 never reaches Ze's RIB after the reload, because the import policy rising did not yet re-learn the routes the absent policy discarded. Ze's FRR peer now negotiates Route Refresh (route-refresh enable), since a rising import on a peer without it is refused (spec D-1). On 2026-10-31, after the import-rise refresh (e9385df360 through 16744a9e7e), the unmodified build passes all assertions in 25s: the reload sends ROUTE-REFRESH to FRR and FRR's 10.81.1.0/24 reaches Ze's RIB with no session reset. Policy.AcceptsAtEnd answering true for an unconfigured policy (a GOFLAGS=-overlay build): red at assertion 7, Ze's RIB holds 10.81.1.0/24. An export-only break, the PolicyFilterChain final verdict skipped for export and egressFinalAccepts, forward_rs.go and egress_inject_filter.go admitting (overlay): red at assertion 8, FRR holds 10.81.0.0/24. The same break without egressFinalAccepts left assertion 8 green, so the session write gate is the guard Ze's config-originated route meets. sha256:16bb31efdbb8sha256:7a6490c76e9bsha256:41f076b8fae9
64 isis-export-policy-frr Ze, FRR 10.7.1 isis filter export decides what enters Ze's LSP and at what metric: FRR installs the tagged /32 at its link metric plus the 77 the modify set, the untagged /32 at the link metric plus Ze's default 100, and no route for the /24 the prefix-list rejects ("IGP export-policy scenarios"). Results on 2026-10-31 at 697dc6a618, each image built by the harness: unmodified, pass; exportVerdict answering accepted at DefaultRedistMetric, assertion 2 red with "FRR's IS-IS route for 10.66.0.1/32 has metric 110, want 87"; restored, pass. sha256:310feaf0c24f
65 ospf-export-policy-frr Ze, FRR 10.7.1 ospf filter export decides which AS-External-LSAs Ze originates and what they carry: FRR holds the tagged route's LSA at metric 50, E1, tag 200, the untagged one at metric 20, E2, tag 0, and none for the rejected route ("IGP export-policy scenarios"). Results on 2026-10-31 at 697dc6a618, each image built by the harness: unmodified, pass; applyExportSets returning its input, assertion 2 red with "FRR's AS-External-LSA for 10.67.1.0/24 has metric 20, want 50"; restored, pass. The first run found FRR 10.7.1 writing "E2 (Larger than any link state path)" for type 2, which the checker now reads by its first word (697dc6a618). sha256:7459d9f4decasha256:49ebad4b55ccsha256:8cfc88af0ea1
67 bgp-rapid-reconnect-initial-sync-frr Ze, FRR 10.7.1 A session's initial sync ends with its session: FRR runs clear bgp <ze> three times, each on the first probe (every 100 ms) that shows the next session Established by its connectionsEstablished count, so each clear lands inside ze's initial sync. On the fourth session FRR's show bgp neighbor <ze> json must count four establishments and three drops, report gracefulRestartInfo.endOfRibRecv for ipv4Unicast and ipv6Unicast and no other family, and, in that same answer, an acceptedPrefixCounter equal to the prefixes ze.conf originates in each family (three IPv4, two IPv6). show bgp <afi> unicast neighbors <ze> routes json must then hold each of those prefixes with exactly one path from ze and nothing else, and a last read of the neighbor must still pass. ze advertises Graceful Restart because FRR prints End-of-RIB received only for a peer whose Graceful Restart capability it received (bgp_show_peer_gr_extra_info), and FRR clears those flags when a session stops (bgp_stop), so the flags read belong to the fourth session. Claimed for the positive outcome only (spec R-9). Results, 2026-11-14, QEMU docker-lab on arm64: pass with ze image. Revert run with the join of the initial sync removed from the runOnce session-end defer through a build overlay, ze image: also pass, so the scenario does not discriminate the fix; the unit test TestPredecessorSyncWritesNothingOnSuccessor does. sha256:5f80bd3c10b4sha256:484c9683e046
68 peer-stops-reading-bird Ze, BIRD 3.4.0 RFC 4271 Section 8.2.2, Event 10, when the peer has stopped reading and ze still has routes to write to it: the hold timer must drop the TCP connection behind the blocked writer. A frozen BIRD and an idle ze cannot show that, because with nothing to write no writer holds ze's write path and the close succeeds on any build (bgp-holdtime-deadpeer-frr is that case). So the checker first requires the session up with two routes held on each side, freezes BIRD's container, and reloads ze with ze-reload.conf, which adds 131072 routes to the peer (two /8 split into /24, one UPDATE of 451 octets each, about 56 MiB). The reload delivers them on the running session and is not waited for: it is the writer that blocks. Ze must log session closed for BIRD (Peer.sessionDown, logged once Session.Run has returned) between 5 s and 20.5 s after the freeze. The hold time is 9 s and BIRD's last KEEPALIVE is at most 3 s old at the freeze, so the timer expires 6 s to 9 s after it and the close may take two more; a close that waits for the blocked write instead comes no sooner than the 30 s write budget (ze.fwd.write.deadline, which the lab leaves at its default), and 20.5 s is halfway between the two. Two readings then prove the writer was blocked, because without one the timing proves nothing: /proc/net/tcp in ze's container showed at least 16 KiB unsent toward BIRD, and ze logged no delivered Hold Timer Expired NOTIFICATION, which it can write only on a free write path. The checker thaws BIRD on every exit path. After the thaw BIRD must log ze_peer: State changed to up once more, ze must report the peer Established, BIRD must hold exactly 131074 routes from ze (show route protocol ze_peer count), and a last count of both logs must show exactly one session ended and one returned. The scenario's ze.conf logs bgp.reactor.peer at info for the two lines the checker counts
69 bgp-manual-eor-requested-frr Ze, FRR 10.7.1 On a behavior { manual-eor true; } peer, Ze sends no End-of-RIB nobody asked for, and sends the one a process asks for from inside the initial sync after the route that process announced first. Session 1: the process announces inside its peer-UP handler and asks for nothing; FRR's log must hold the route decode and no bgp_update_receive_eor line through a 10 s hold. After clear bgp <ze>, session 2: the process announces and asks for the IPv4 unicast End-of-RIB inside its peer-UP handler; the log written after the clear must hold the route decode, then exactly one marker through a 10 s hold. FRR's connectionsEstablished must read 1, then 2, and the process's records must show each session's requests answered within 4 s. Recorded 2026-11-14 in the Docker lab, each run with the Ze image rebuilt from the tree under test: green on the fix (ze image); red with the requested form suppressed during the sync again in announceEOR (assertion 7: FRR never logged the End-of-RIB after the second route; image); red with the sync's marker forced on the manual-eor peer in sendInitialRoutes (assertion 3: FRR decoded 1 End-of-RIB on the first session, where nothing asked for one; image sha256:fe2dd16c5cfc. ). 10.84.1.0/2410.84.2.0/24sha256:4da4a47314f2...sha256:883e8bd4bf3b...

Running

./le test integration interop
INTEROP_SCENARIO=bgp-ebgp-ipv4-frr ./le test integration interop
VERBOSE=1 ./le test integration interop
NO_BUILD=1 ./le test integration interop
FRR_IMAGE=quay.io/frrouting/frr:10.3 ./le test integration interop
BUILD_TIMEOUT=7200 ./le test integration interop

Interop tests require Docker and are not part of the offline precommit gate. They are a separate protocol-validation action.

The first run cross-compiles the lab binaries on the host, then builds the Docker images. No Dockerfile.ze carries a Go compiler: each one is an Alpine (alpine:3.24.3 in the BGP, IPsec, L2TP, PPPoE, RADIUS and RSVP-TE labs) base, one apk add, and a COPY of a binary the suite's preflight has already written into the build context (internal/le/interoplab/zebuild.go, StageBinaries). Each lab declares the binaries it needs beside the images it needs, and the bgp lab declares three. Its scenarios also run the harness from inside the container: a linux le built by internal/le/linuxle (linuxle.Base), staged at test/interop/le-linux and copied to /usr/local/bin/le. The third is the daemon built with ze_le added to DaemonBase, staged at test/interop/ze-le-linux and copied to /usr/local/lib/ze-le/ze, outside PATH: only a scenario holding a ze-le-daemon file runs it (see "Optional sidecars"). Harness peers start with entrypoint le and a command that begins test (le test interop-bgp speaker ...). The image holds no program under a retired harness name, so a scenario ze.conf that still runs one fails to start it.

Measured on 2026-09-06 on a 32-core workstation: 6.1s and 4.8s for the two cross-compiles against a warm cache/go-cache, at a peak resident set of 1.03 GiB and 0.98 GiB, then 54.7s for the docker build, which is now context transfer rather than compilation.

Measured again on 2026-10-09 on the same workstation, from a git archive export of 1cfc44afd7 with an empty cache/go-cache, while other sessions ran labs: INTEROP_SCENARIO=as-path-prepend-two-octet-peer ./le test integration interop reported 1 passed, 0 failed in 9m27s of wall time, which includes building the launcher itself from cold. The staged binaries were written about 18s and 9s apart. The largest process of the whole run peaked at 1524736 KiB (1.45 GiB) under /usr/bin/time -v. A separate docker build of test/interop/Dockerfile.ze over those staged binaries took 199.4s at a load average of 65, with the client peaking at 48.5 MiB, so the time is the contended daemon rather than any compile. Inside that image ze --version and le test both ran, and /etc/alpine-release read 3.21.7.

The shape before 2026-09-06 is what those numbers are read against. Dockerfile.ze copied the whole tree and compiled ze twice with no cache mount. One colima VM of 2 CPUs and 2 GB built it in 2m48s on 2026-09-04, and the same VM took 40m39s for it earlier that day, when the host disk was full and the guest was thrashing. On 2026-09-06 the kernel killed that build three times on an idle 31 GiB workstation, once with 23 GiB free and nothing else running, so the compiler in the container was the thing that did not fit rather than the machine being busy.

A consequence: docker build -f test/interop/Dockerfile.ze . on a clean checkout now fails at the COPY until ./le test integration interop has run its preflight. Each converted Dockerfile's header names the action that writes its binary.

Docker.Build never hands docker the context as a directory. Ze walks it, applies its ignore file, writes it as a tar stream, and runs docker build -f <Dockerfile inside the context> - with that stream on stdin. The reason is the macOS host. There the QEMU guest sees the checkout over 9p, and QEMU's darwin 9p backend answers llistxattr and lgetxattr on a symlink with ELOOP. BuildKit's directory walker accepts only ENOTSUP from those calls, so a guest build whose context held a symlink (the checkout's cache, the migrated tmp/ children) failed with too many levels of symbolic links, even with cache/ in .dockerignore. Ze's walk reads each entry with lstat and readlink and sends a symlink as a link entry, never following it, and it reads no extended attribute. The ignore file is the one Docker would read: <Dockerfile>.dockerignore beside the Dockerfile wins over the context's .dockerignore. An excluded directory is not entered unless a ! pattern could match inside it, and the Dockerfile and the ignore file are always sent. Owners are sent as root, as the docker CLI sends them. A socket is skipped, also as the docker CLI does, while a device or a named pipe fails the build with its path. A Dockerfile outside the context is refused, because the stream carries the context alone. A context named through a symlink to its directory (a migrated tmp/ child) is resolved before the walk, as the docker CLI resolves it, so its contents are sent rather than an empty archive. A walk that fails part way fails the build with its cause. Docker may still build an image from the part it read, so that failure also releases the build's own tag, and the image is never answered as built. A named context (--build-context) is still passed as a path.

Each build is bounded at 90 minutes, and BUILD_TIMEOUT sets that bound in whole seconds for a machine slower or faster than that one. The bound stops a wedged Docker daemon and is not a budget for the build, so a build that finishes returns at once and a generous bound costs nothing. A value that does not parse, or that is not positive, keeps the 90 minutes.

An image that needs more than the machine bound declares its own ImageBuild.Timeout. The build runs under the larger of that field and the machine bound, so the field only ever lengthens the bound and a value below it changes nothing; BUILD_TIMEOUT stays the one knob that lowers it. No suite declares one. The PPPoE suite did until 2026-09-04: 10 minutes for its ze image, 15 for accel-ppp and 10 for the client, each written when the shipped default was 10 minutes and each a cap once the default became 90, because the field then replaced the machine bound instead of raising it.

Subsequent runs with NO_BUILD=1 skip rebuilds. Once the images exist, the full suite takes roughly 5-10 minutes depending on session establishment times.

Source-built peers

Owner decision (ai/META-DECISIONS.md): every third-party lab peer runs its latest stable upstream release. A peer that publishes no stable release is built from source, and only from source less than one month old. Two do today: tac_plus-ng (test/interop-tacacs/Dockerfile.tacplusng, used by internal/component/tacacs/tacplusng_docker_test.go) and freeRouter (test/interop-rsvpte/Dockerfile.freertr for the RSVP-TE suite, and test/interop/Dockerfile.freertr for the perf DUT).

The Dockerfile is the declaration, and no list of peers exists beside it. A source-built peer's Dockerfile carries Docker build arguments that its own build also reads:

Build argument Meaning
ZE_SOURCE_REPOSITORY The upstream GitHub repository, owner/name
ZE_SOURCE_COMMIT The pinned upstream commit, the full hex id the codeload URL fetches
ZE_SOURCE_DATE That commit's committer date, RFC 3339
ZE_SOURCE_CHECKOUT Instead of a pin: the named build context that holds a local checkout the build follows

Docker.Build is where every lab image is built (a suite, a labtest Spec.Dockerfile, the perf DUTs), and it calls RequireFreshSource before docker build runs:

  • A pinned commit more than one calendar month old refuses the build. The refusal names the Dockerfile, the commit, its date and age, and the command that moves it: ./le lab pin update dockerfile <Dockerfile>. Bump asks the GitHub API for the newest commit of the default branch and rewrites ZE_SOURCE_COMMIT and ZE_SOURCE_DATE. The next image build is the proof that the new source still builds and that a Ze patch, such as freertr/ze-interop-resv.patch, still applies.
  • A followed checkout (the perf DUT's freertr context, FREERTR_DIR, default ../rtr) whose HEAD commit is more than one month old is updated first: git fetch, then git merge --ff-only @{upstream}. A checkout holding uncommitted changes is never touched: the build is refused, naming the changes and git -C <checkout> pull --ff-only as the fix. A fetch or a fast-forward that fails refuses the build the same way.
  • A Dockerfile that declares ZE_SOURCE_REPOSITORY with neither a complete pin nor a checkout is refused, so a half-declared pin never builds unjudged.

Lab pins against upstream

./le lab pin check lists every lab peer pin the tree holds and compares each with upstream's latest stable release. It is a report over the network, never a commit gate. The pins are derived from the files that carry them, under test/ and internal/ (vendor and testdata trees skipped):

Pin Read from Upstream asked
Image tag FROM in a Dockerfile (not a stage name, not scratch), an image string literal in Go (an official image with no repository path, such as, only with a dotted version tag on a line whose code names an image). Comments are never read: a comment that quotes an image, or an example of what is not one, pins nothing. Also a scenario's *-image file. alpine:3.24.3 The registry's OCI tag list, with the anonymous token it offers (Docker Hub, ghcr.io, quay.io)
Go module go install <path>@<version> in a Dockerfile, or go mod download <path>@<version> where the build patches the release source before compiling it (StayRTR) proxy.golang.org, including the next major version paths, since @latest of a /v3 path never names v4
Git tag git clone --branch <tag> https://github.com/..., a codeload or archive tag URL The GitHub repository's latest release
Source commit ZE_SOURCE_COMMIT The newest commit of the default branch

An image tag is compared only with tags of its own shape (the same prefix, number count and suffix), so golang:1.25-alpine is judged against 1.27-alpine, never 1.27.1-trixie, and a pre-release never wins. A moving tag such as latest is reported floating with the newest plain release beside it. An upstream that does not answer leaves its row unresolved with the cause.

A package test's own server container (labtest)

A package _test.go that needs one real server, such as stayrtr for the RPKI client, starts it with labtest.Start(t, spec) from internal/le/interoplab/labtest, never with its own exec.Command("docker"). Start pulls the image unless it is digest-pinned (name:tag@sha256:...) and the daemon already holds it, since the digest makes the local copy exactly the named image and a host holding it then runs offline; a tag can move, so a tag reference is always pulled. It then starts one labeled container through Docker.RunDetached (so a sweep removes it when the test binary is killed), answers the host address the server listens on, and returns only once Spec.Ready accepts that address. Spec.Port names the container port with its transport, interoplab.TCPPort(3323) or interoplab.UDPPort(123), and a port with no transport is refused before any docker command runs. The container either publishes that port on the host loopback at a port the daemon picks (Docker.PublishedPort reads which, under the same transport), or runs on the host network (Spec.HostNetwork) where it listens on 127.0.0.1:Port and reaches a listener the test opened. Spec.Entrypoint replaces the image's entry point, for an image whose entry point writes its own configuration and ignores the command. t.Cleanup writes the log tail of a failed test, then removes the container.

Machine state What the test does
✕ skips, naming the install route; fails under ZE_REQUIRE_DOCKER=1
docker info fails fails, naming the probe error
The image cannot be pulled fails; under the live tag it skips an ad hoc run and fails under ZE_REQUIRE_DOCKER=1 (owner decision Q-6, pull_live.go)
A Spec.Dockerfile image does not build fails, naming the image, the Dockerfile and the docker build command that shows the failing step
The container does not start, or Ready never accepts fails, then the log tail follows

Two package tests start their server this way. The RPKI live tests start stayrtr over TCP (internal/component/bgp/plugins/rpki/rpki_live_test.go, tag live). The NTP client tests start chrony (dockurr/chrony:4.9) over UDP (internal/plugins/ntp/ntp_chrony_docker_test.go, tag docker) and run syncWorker.doSync with the real query: a server on its local reference at stratum 7 is accepted and reported at stratum 7 with a near-zero offset, and a server with no synchronised source (leap indicator 3, stratum 0 on the wire) is rejected with no clock step. The container reads the host's own kernel clock, so the offset is bounded, never a configured value.

A server with no published image is built from its Dockerfile: Spec.Dockerfile names it from the module root, and Start then builds Spec.Image through Docker.Build, with the Dockerfile's directory as the context, instead of pulling it, and releases the tag the run owns after the container is removed; a build that fails fails the test. A server no probe from the host can observe, such as a UDP listener behind a published port that accepts datagrams before the server binds, names Spec.ReadyInContainer instead of Spec.Ready: a command run in the container whose exit 0 is ready. A test that reads what the server wrote calls labtest.StartContainer, which answers the Container with its address and Container.Exec. The flow-export collector tests run this way (internal/plugins/flowexport/collectors_docker_test.go, tag docker): nfcapd/nfdump and sflowtool built from test/interop-flowexport/Dockerfile.collectors, ready once the collector's socket is in the container's UDP table, their decoded records read with docker exec (docs/architecture/flowexport/flow-export-1-counter-export.md, "Independent collector tests").

A package test whose real peer is a client, against a server the test runs in-process, runs it with labtest.Run(t, client). Run pulls the image as Start does, runs one labeled container to its exit through Docker.RunOneShot, and answers the client's stdout, stderr and exit code, a non-zero exit included, so a test can assert a refused request. Client.HostNetwork puts the client in the host's network namespace (OneShotContainer.Network), where it reaches the test's listener on 127.0.0.1, and Client.Mounts bind-mounts files the test wrote, such as a CA certificate (OneShotContainer.Mounts). The machine states in the table below are handled as Start handles them; a run the docker client did not finish (interoplab.OneShotDidNotExit), or an exit code docker run keeps for its own failure (125 to 127), fails the test. The gNMI server tests run gnmic this way (internal/component/gnmi/gnmic_docker_test.go, tag docker): Capabilities, Get, Subscribe ONCE and Set over TLS with a bearer token, decoded from gnmic's protojson output, and a request with no token or a wrong one refused Unauthenticated.

labtest runs no Docker host kernel check: Ze runs in the test process on the host, and that check guards a containerised Ze. A peer that needs a Ze-kernel feature belongs in an interoplab.Suite lab. Only a _test.go file, or tooling under internal/le, imports labtest (./le arch tier check, docs/architecture/module-tiers.md).

Debugging Failures

The shared orchestrator does not read container logs. Each suite's checker adds them to the failure it returns:

Suite What a failed checker carries
BGP (interop) the last 80 log lines of each peer the scenario started (checkerFailure), pmacct's message log, and for speaker and speaker2 a section headed extended relay files: the relay's -result.json verdict and the last 8 rows of its -ze.jsonl and -frr.jsonl captures under the extendedCaptureBase prefix, each row cut to 512 characters and the section to its last 8192 octets. A file that does not exist is printed missing, and a prefix with none of the three files says so in one line. The section is read even when the speaker logged nothing (extendedRelayDump). internal/le/interoplab/bgp/check_engine.gointernal/le/interoplab/bgp/extended_relay.go
IPsec (interop-ipsec) the last 60 log lines of Ze, strongSwan, the NAT box and FRR, for each one the scenario started, read after the checker's own deadline has passed, all of them within one 20 s bound (checkerFailure). internal/le/interoplab/ipsec/failure.go
RSVP-TE (interop-rsvpte) each peer's capture tail and the last 60 log lines (diagnosticError). internal/le/interoplab/rsvpte/checkers.go
A package test's own server (labtest.Start) the last Spec.LogLines lines (1 to 1000) of its container, written to the test log by t.Cleanup when the test failed, before the container is removed (host.release). internal/le/interoplab/labtest/labtest.go

A docker exec that runs past its 30 s bound names the container, the command it ran there (cut at its first line and at 160 bytes), and the bound (execOperation, internal/le/interoplab/docker.go).

For more detail:

  • VERBOSE=1 enables debug output (polling status, container commands, raw CLI output)
  • SESSION_TIMEOUT=120 increases the session establishment timeout (default 90s)
  • Single-scenario runs isolate the problem: INTEROP_SCENARIO=bgp-graceful-restart-frr ./le test integration interop

Writing a New Scenario

  1. Create a descriptively named directory under test/interop/scenarios/.
  2. Add ze.conf and whichever peer configs the scenario needs.
  3. Add the scenario's ordered assertions to scenarioOperations and scenarioExtras, or register a bespoke checker in specialCheckers when the control flow is non-linear.
  4. For a bespoke checker, put each decision in a pure predicate in check_rfc_predicate.go and add its both-polarity subtest to TestBespokeCheckerBranches.
  5. Run INTEROP_SCENARIO=<name> ./le test integration interop.

TestCheckerPopulationMatchesProducer compares every scenario directory with the package-local registry. TestEveryCheckerFailsClosedWithoutPeerEvidence rejects a checker that can pass without reading a peer. Each negative assertion must carry positive proof that its query mechanism ran.

A scenario that reads the RIB attaches the RIB plugin

plugin { internal rib { use bgp-rib; } } loads the plugin. It does NOT feed it. A peer delivers an event to a process only where both halves agree, and the peer's half is its attach block (Server.PeerScopedProcs, internal/component/plugin/server/delivery_graph.go). A peer with no attach process rib block grants nothing, so the plugin sees no peer at all and every RIB question answers empty:

	attach process rib {
		receive [ update state refresh ];
	}

The tell is "peers": 0 from show bgp rib status while show bgp peer list reports both sessions Established. Ze also logs it at startup: "the plugin declared events and no peer attaches it".

Explicit configuration is authoritative at every start

ze start <file> reads that file on every start, including a container restart. Persisted configuration from a previous run does not replace the explicit input. The labs mount individual configuration files, so replacing a source file by rename can leave an existing container's bind mount on the old inode. Recreating the container refreshes the mount; the harness creates each container once.

For Ze's configuration syntax, see docs/architecture/config/syntax.md. Copy an existing scenario's ze.conf as a starting point.

ExaBGP Wire Compatibility (test/exabgp-compat/)

A separate test suite validates that Ze's wire encoding matches the reviewed ExaBGP API 6.0.0 contract fixtures. The compiled Go server negotiates each BGP session and compares every received frame byte-for-byte.

What It Tests

The harness migrates each ExaBGP-derived configuration, runs Ze, and compares its wire bytes with the known-good fixture. The 42 .ci cases in test/exabgp-compat/encoding/ use option=file:, option=serial, 1:cmd:, 1:raw:, 1:signal:, and 1:json: records rather than the standard .ci format. option=serial marks process-driven fixtures that must not overlap other ExaBGP harness instances; the runner executes those after the parallel batch.

The mock replies with the client's OPEN capabilities, changing only its AS number and router ID; it does not infer capabilities from expected UPDATEs. Migration expands an omitted or empty ExaBGP family block to every family registered in the current binary. An explicit family block stays restricted to its entries. conf-vpn explicitly declares ipv4 mpls-vpn, which migrates to session > family ipv4/mpls-vpn, so both OPENs advertise AFI 1, SAFI 128. The negotiated-family guard remains active: RFC 4760 Section 8 requires both speakers to advertise the family for bidirectional exchange.

<prefix>:signal:<NAME> marks the point in a connection's script where the runner reloads Ze. It divides the script: every raw frame written before it must match before the reload happens, and the frames after it are matched only once it has. Each signal step consumes the NEXT option=file: config the case names, so a case owes one config more than it has signals, and the runner refuses a case where those two counts disagree. test/exabgp-compat/api/api-reload.ci is the case that drives this: its withdrawal of 2.0.0.0/24 is produced BY the reload.

The fixtures name ExaBGP's reload signal, SIGUSR1. Ze reloads on SIGHUP, so the runner translates the name (deliverExaBGPReloads), writes the next migrated config over the path Ze reads, and signals Ze's own pid rather than the process group, which also holds the bridge's scripts.

Coverage includes:

Category Examples
Address families IPv4/IPv6 unicast, VPN, FlowSpec, FlowSpec VPN, EVPN, VPLS, MPLS labeled, MUP, MVPN
Path attributes ORIGIN, AS_PATH, NEXT_HOP, MED, LOCAL_PREF, communities (standard, extended, large), AGGREGATOR, ORIGINATOR_ID, PREFIX_SID, SRv6
Capabilities 4-byte ASN, ADD-PATH, link-local next-hop, software version, hostname
Edge cases Generic/unknown attributes, self-referencing routes, group limits, IPv4+IPv6 mixed configs, deferred announcement (watchdog)

Running

./le test functional exabgp

ExaBGP compatibility is part of the offline precommit gate.

Test Hierarchy

Workflow Includes Interop? Includes ExaBGP? Requires Docker?
Offline precommit gate ✕ ✓ ✕
Standard functional sweep ✕ ✓ ✕
./le test integration interop ✓ ✕ ✓
./le test functional exabgp ✕ ✓ ✕

Interop tests are intentionally separate from the pre-commit gate because they require Docker and take longer to run. ExaBGP wire compatibility tests run as part of the standard verification suite.

Current Scope

Interop scenarios cover core BGP: session establishment, route exchange, withdrawal, capabilities (4-byte ASN, ADD-PATH, GR, route refresh, PATHS-LIMIT), communities, MD5 auth, route server behavior, route reflection, policy import/export, RPKI origin validation, BMP monitoring, BFD failover, ECMP, SRv6 VPNv6, remove-private-as export policy, GTSM, AS112, and non-unicast address families (EVPN, VPN, FlowSpec). The suite also includes full IS-IS and OSPFv2/OSPFv3 interop families (adjacency, flooding, SPF, dual-stack, authentication, TE, LFA/TI-LFA, graceful restart, and segment routing). ExaBGP compat covers wire encoding for all supported address families.

Known Vendor Limitations

Vendor Limitation Affected Scenario Workaround
BIRD (observed at 2.15; unverified at 3.4.0 until bgp-ipv6-ebgp-bird runs) Enforces next-hop reachability for IPv6 routes. On IPv4-only Docker networks, IPv6 next-hops are unreachable and BIRD rejects routes as invalid (RFC 7606 treat-as-withdraw). bgp-ipv6-ebgp-bird Add multihop; to BIRD config to disable the directly-connected next-hop check.