Interoperability Testing
Ze validates protocol correctness against production BGP daemons in two complementary ways: live session interop tests (Docker containers running real daemons) and byte-level wire format validation against ExaBGP (Ze's predecessor, a BGP implementation in Python).
For BGP terminology used in this document, see BGP protocol.
ai/rules/interop-and-goal-validation.md states when an interop test is owed.
This page is the infrastructure it is owed against.
The suites, one per protocol area
| Protocol area | Peer implementation | Scenario directory | Native action |
|---|---|---|---|
| BGP (session, capability, NLRI, community, policy) | Docker: FRR, BIRD, GoBGP, StayRTR | test/interop/scenarios/ |
./le test integration interop |
| IPsec (IKEv2, EAP, MOBIKE) | Docker: strongSwan | test/interop-ipsec/ |
./le test integration interop-ipsec |
| L2TP | Docker | test/interop-l2tp/ |
./le test deployment l2tp, and ./le test deployment l2tp-ppp for the full PPP and NCP path |
| PPPoE (Ze as client) | Docker: accel-ppp | test/interop-pppoe/ |
./le test deployment docker-pppoe-accel |
| RADIUS (admin login: PAP, CHAP, EAP, Filter-Id) | Docker: FreeRADIUS | test/interop-radius/scenarios/ |
./le test integration interop-radius |
| RADIUS (subscriber: accounting, CoA and Disconnect) | Docker: FreeRADIUS, an xl2tpd + pppd LAC | test/interop-radius-subscriber/scenarios/ |
./le test integration interop-radius-subscriber |
| RSVP-TE (Ze as transit) | Docker: freeRouter, ingress and egress | test/interop-rsvpte/scenarios/ |
./le test integration interop-rsvpte |
| Multi-node topologies (several Ze nodes, any suite's images, named links) | Docker: any registered image | test/interop-topology/scenarios/ |
./le test integration interop-topology |
The L2TP lab's lcp-restart-timer-pppd scenario first proves a working PPP
data path, then loses pppd's LCP replies after a live restart. It measures Ze's
configured 500 ms retransmission, exact three-request budget and subsequent
CDN from the relay's monotonic wire observations. The lab page describes the
peer source and the final kernel cleanup assertion.
Every suite discovers its scenarios the same way. Discover
(internal/le/interoplab/discover.go) reads the scenario directory, keeps the
subdirectories, sorts the names lexically, and joins each one against the
owning checker registry. A scenario with no checker, or a nil checker, is an
ERROR rather than a skipped test, so a fixture and its registry cannot silently
disagree. Nothing depends on the order: each scenario gets its own setup, check
and teardown.
A reader outside a suite asks for its scenarios through the suite's catalog.
Each lab registers one in its register_catalog.go (interoplab.RegisterCatalog,
internal/le/interoplab/catalog.go), built on the same Discover call and
checker map its runner uses, so a name cited elsewhere is a name the runner
runs. ./le feature check resolves a feature's Interop entry
<suite>/<scenario> this way, and names the registered suites when it refuses
one it does not know. The catalog also declares RunScenario, which runs ONE
scenario through the lab's own RunAt with the name as its selector, winning
over the lab's selector variable. ./le feature run records an interop
green run through it, against the scenario directory's git tree id, so a
feature's Supported claim needs a current green run of each scenario it counts
(docs/contributing/feature-maturity.md, "Recorded runs").
The catalog is also where a lab declares where it lives and how the repository
runs it: Package (the checker directory), ScenarioRoot, SelectorVariable
and Action (<command> <area>/<verb>). The lab's own runner reads the same
scenarioDirectory and selectorVariable constants, so the declaration and
the run cannot disagree. ./le rfc derives its interop carriers from these
fields, one per registered suite, and ./le rfc record runs one
scenario through the declared action and selector.
A scenario directory carries only declarative inputs its runner reads: ze.conf
plus the peer configuration and argument files that topology needs. Assertions
never live there. They are typed Go checkers under internal/le/interoplab/:
BGP builds its registry from scenarioOperations in
internal/le/interoplab/bgp/checkers.go, and IPsec declares scenarioCheckers
in internal/le/interoplab/ipsec/checkers.go. A checker waits for readiness,
asserts the protocol behaviour, verifies stability where the scenario needs it,
and returns an error on failure.
A scenario directory is NAMED and carries no numeric prefix. The name is the
scenario's identity: Discover matches it exactly, ./le test integration takes it
as a scenario selector, and specs, journal rows and code comments cite it.
Role export policy on a roleless destination
bgp-role-export-roleless-destination-frr runs FRR as a Customer whose
Ze-side session deliberately declares no role. A raw Customer source permits
customer and its route must reach FRR; BIRD's source permits only
provider and its route must remain absent there. Ze's received RIB must
hold both source prefixes, and FRR must also hold Ze's independent control
route. The positive source detects failure to observe the destination's OPEN;
the negative source detects failure to insert egress on a roleless destination.
Before reading FRR's negative result, the checker waits for Ze's completed
export-set rejection naming FRR's address and advertised Customer role.
The source peers have receive-only forwarding bindings, making FRR the sole
send-granted target. The received RIB alone is not a forwarding fence.
BIRD's bgp_fire_tx drains queued UPDATEs before its initial End-of-RIB, so
the source worker's FIFO cannot produce that decision from an earlier EOR.
FRR 10.7.1's bgp_open_capability emits a configured local_role, and
bgp_role_violation rejects an absent remote Role only with strict mode.
Its bgp_otc_filter accepts an unmarked route as Customer and sets OTC
locally. BIRD 3.4.0's bgp_update_attrs adds OTC only for a configured
Provider, Peer or RS, so this unconfigured sender originates no OTC.
These peer-source contracts are in
FRR's OPEN code,
FRR's route code
and BIRD's attribute code
and packet sender.
Tested Daemons
| Daemon | Version | Image | Query Method | What It Validates |
|---|---|---|---|---|
| FRR | 10.7.1 | pinned by digest (interoplab.FRRImage, one declaration for the BGP, IPsec and L2TP labs and the perf runner). quay.io/frrouting/frr:10.7.1 |
vtysh | eBGP, iBGP, route exchange, GR, communities, MD5, route server |
| BIRD | 3.4.0 | Built from the upstream v3.4.0 tag on Debian trixie (which refuses a tag moved off the pinned commit). BIRD 3 line only; Alpine 3.24 packages 3.2.4, and 3.4.0 built on Alpine 3.24.3 dies with SIGILL in rte_announce_to on the first route it imports. test/interop/Dockerfile.birdai/META-DECISIONS.md |
birdc | eBGP, route exchange, triangle topologies |
| GoBGP | 4.11.0 with the export-reload source patch | Checksum-verified release and build-time source diff. golang:1.25-alpinetest/interop/Dockerfile.gobgp |
gobgp CLI, and gobgp-paths (a gRPC ListPath reader) for the families the CLI cannot name; it prints each attribute through GoBGP's String(), and an Extended Communities attribute also through its MarshalJSON, because String() writes an IPv4-address-specific Route Target and Route Origin alike |
eBGP, route injection and verification |
| StayRTR | 0.6.4, patched to write its queued Error Report before it closes | Go builder. golang:1.27-alpine3.24alpine:3.24.3test/interop/Dockerfile.stayrtr |
HTTP. /rpki.json |
RTR (RFC 8210) as the CACHE, so Ze is the client of an implementation that is not its own. Origin validation answers (RFC 6811) against VRPs a third party encoded |
pmacct pmbmpd |
1.8.0 | pmacct/pmbmpd:v1.8.0 |
JSON msglog file | BMP (RFC 7854, RFC 9069) as the COLLECTOR. pmacct decodes the per-peer header, the Peer Up Information TLVs and the Peer Down reason itself, so a scenario reads another implementation's reading of Ze's bytes rather than a string Ze emitted |
| Unbound | 1.26.2 | Built from the upstream release on (Alpine 3.24 packages 1.25.2). alpine:3.24.3test/interop/Dockerfile.unbound |
kdig from the dnsclient container; the query log on stderr (log-queries) |
Upstream recursive resolver of Ze's LAN DNS resolver (dns-resolver-forward-unbound): Ze's answer per query equals Unbound's own, a repeated query reaches Unbound once, and a truncated UDP reply is retried over TCP |
| Knot DNS | 3.6.2 | Built from the upstream release on (Alpine 3.24 packages 3.5.10). The same image runs the dnsclient LAN host, with Knot's kdig and Alpine's dhcpcd 10.5.2, the latest upstream release. alpine:3.24.3test/interop/Dockerfile.knot |
kdig |
Authoritative server of the lab zone lab.example. behind Unbound, and behind Ze directly in dns-resolver-dhcp-lan and in the L2TP lab's dns-resolver-l2tp-subscriber (172.29.0.7), where the xl2tpd + pppd LAC learns Ze's resolver through IPCP (usepeerdns) and resolves with busybox nslookup. docs/labs/l2tp-interop.md |
| dhcpcd | 10.5.2 | Alpine package in. test/interop/Dockerfile.knot |
dhcpcd --dumplease |
DHCPv4 client of Ze's DHCP server: the lease and its option 6 (dns-resolver-dhcp-lan, and the topology suite's access-up, where option 6 must hold the CPE's IPCP servers); the lease, its server identifier (dhcp_server_identifier) and the DHCPNAK it logs on a reboot (dhcpv4-server-shared-segment-dhcpcd) |
| ExaBGP | API 6.0.0 contract fixtures | Compiled Go wire server | Wire byte comparison | Byte-for-byte encoding across all address families |
The conditional-MED collector check requires a single AS number in the received AS_PATH. It uses the shared AS-number parser, so decimal and dotted spellings identify the same source without accepting a multi-AS path.
GoBGP export-reload successor proof
bgp-export-reload-gobgp and bgp-export-reload-secondary-gobgp are new
successor scenarios, not recovery of the historical four-peer experiment.
Their registered leaf checker uses GoBGP's actual SIGHUP configuration reload.
Both exact scenario identities also register their native peer preparation
with the shared preparation registry; the common dispatcher contains no
scenario-specific feature calls or rendered-directory-name heuristics.
The ordinary scenario uses the global RIB; the secondary scenario uses
route-server clients, with secondary selection enabled only toward FRR and
disabled toward the independent FRR sink. Ze and BIRD originate fixed routes.
Neither source changes its routes after the baseline fence.
Ordinary export-policy AS_PATH matching includes GoBGP's already-prepended
AS65005 (65005 65001 for the primary source); route-server matching sees the
unprepended source path. Their fallback expectations must not be interchanged.
The phase configurations exercise reject-to-accept, accept-to-reject, attribute-only unchanged-best changes, export-rejected best with an acceptable backup, import-driven replacement, last-candidate removal, simultaneous import removal and export rejection, repeated reload, and restoration. Exact FRR and sink route objects are the delivery oracle. GoBGP's computed Adj-RIB-Out is retained only as diagnostic policy state, never as delivery evidence.
Repeat retains a byte-identical configuration and the native
no-policy-difference reload path. Two separately armed, one-shot native
CLOSE_NOWRITE watches observe the actual running configuration file, one per
SIGHUP. Only the daemon reads that file while either watch is armed. The second
read proves that GoBGP's serial signal loop returned from the first
UpdateConfig and consumed the unchanged file again before restore overwrites
it. The first reload has completed; the second read does not claim its own
UpdateConfig completion or acknowledge a wire queue. Both receiver tables,
source counters and original-session checks remain required.
Native CLI capability collections are normalized once at their parser boundary
as canonical JSON multisets, retaining duplicate multiplicity and exact values.
GoBGP emits remote capabilities by iterating a map, so array order is not session
identity. The original OPEN bytes remain independently retained in the packet
capture; normalization does not replace that wire evidence.
The GoBGP container starts a bounded native packet capture before starting the patched v4.11.0 daemon against a writable container-local config copy. Only these two topologies start GoBGP first and await its native CLI readiness before starting the active source peers. This retains the original handshakes without collecting avoidable connection-refused attempts during daemon startup. The observation runtime is Alpine 3.24.3 with GNU coreutils 9.11-r0, tcpdump and inotify-tools 4.23.9.0-r0. The shared native-capture lifecycle owns and reaps the packet capture; each one-shot file observer also requires joined completion.
The daemon and CLI build from the checksum-verified v4.11.0 release with
gobgp-v4.11.0-export-reload.patch
applied before compilation. Policy reload performs retained-input processing
followed by export reconciliation. A second change withdraws a secondary
destination that was actually sent when import removes its last candidate and
the new export policy rejects every old candidate. Both changes use GoBGP's
existing outgoing queue and sent-route bookkeeping.
The image retains the exact patch input that its builder applied. Before reading
configuration files, the checker reads those bytes and their SHA256 from the
running GoBGP peer. They must match this checkout's declared patch and the named
runtime artifact. Missing, empty, different or inconsistent evidence refuses the
proof; an upstream version string cannot replace the applied source identity.
The observed bytes and digest remain in proof.json as gobgp-patch and
gobgp-patch-sha256, beside the actual image and binary identities.
The checker retains original OPENs, the complete captured interval, exact source Adj-RIB-In attributes, receiver tables, daemon identities and counter fences. Unknown route documents, failed queries, missing capture completion, capture loss, a reconnect, or source UPDATE/ROUTE-REFRESH after the baseline cannot establish a reload proof. Never-sent routes must remain absent without spurious withdrawals. All phases remain proof debt until the native run actually reaches them; a first-phase failure proves no later phase.
The scope is stable-session, non-ADD-PATH IPv4 unicast only. Original four-peer configurations, OPENs and captures remain missing; ADD-PATH IDs and send-max, VRFs, deferred families and preexisting orphan advertisements remain unverified. These successors neither replace that evidence nor discharge the fifteen historical live-checker obligations.
bgp-local-pref-strip-gobgp reuses the bounded single-session relay below in
transparent mode: it changes neither OPEN nor UPDATE nor KEEPALIVE bytes.
The existing GoBGP destination remains AS 65003 with external peer AS 65001;
only its transport neighbor moves to the relay at .10. A RIB cannot prove
LOCAL_PREF stripping: GoBGP v4.11.0 ProcessMessage
removes the attribute before storage for an external, non-route-server-client
peer, including this fixture's peer.
The wire proof covers a declared interval on that original connection, not its
future traffic. Ze's direction must contain the original OPEN and KEEPALIVE,
all three injected subjects (the third follows EOR and arrives in a later
UPDATE than the first two), and the 10.54.255.0/24 FIFO fence after them.
RFC 4271 Section 4.3: "An UPDATE message is used to advertise feasible routes
that share common path attributes to a peer, or to withdraw multiple
unfeasible routes from service (see 3.1)." Routes in one UPDATE share one
attribute set and arrive together, so the fence is accepted in the UPDATE that
completes the three subjects, in either NLRI order inside it. The whole UPDATE
is validated before its subjects and fence count. A fence in an UPDATE that
leaves a subject outstanding, a second fence, and a subject in an UPDATE after
the fence's are refused, and the attribute checks apply to every NLRI an
UPDATE carries. Only after forwarding and recording that full fence
can the reverse worker begin reading its subsequent KEEPALIVE barrier. Each
closed capture ends with its actual source/target endpoints, exact frame count
and declared boundary. Timeout, EOF, truncation or the 256-frame limit cannot
produce a completion marker. The checker revalidates both intervals, exact
speaker identities, AS 65004/injector next hop and every LOCAL_PREF code 5,
while retaining all three arrivals and both surviving native sessions. The
workers keep forwarding later bytes, which are explicitly outside the proof.
When the relay session ends on a transport error, the relay records it as
transport-error in its -result.json file and also prints
extended relay transport error: <cause> to stdout, so the speaker's docker
logs carry it. A failed checker's dump reads both ("Debugging Failures").
The Extended Message FRR scenarios use a bounded, single-session relay that changes capability 6 in OPEN only. It records Ze's original OPEN separately, leaves UPDATE and KEEPALIVE bytes unchanged, caps each direction at 256 frames, and joins both socket workers before returning. FRR 10.3.1 gates extension bilaterally, so its facing OPEN always advertises capability 6 while Ze's facing advertisement varies independently. This isolates Ze's directional behavior without replacing FRR's UPDATE producer or consumer.
The producer is frr at host .3; the consumer is a separate frr-sink at
host .15, both in AS 65002, with distinct native router IDs matching those
addresses. Relays .10 and .11 connect them to Ze independently. The consumer
originates no test routes: its decoded path must come from .11 with AS_PATH
65001 65002, and allowas-in 1 admits the producer's AS once.
The old one-daemon topology presented the same (AS, router ID) on two configured
Ze peers. Ze's default identifier-claim policy rejected the second OPEN with 2/3
through Peer.validateOpen, routerIDClaims.claim, and
Session.runOpenValidator. This was not RFC 4271 Section 6.8 TCP collision
resolution, which closes the losing connection with Cease.
RFC 6286 Section 2.1
says an identifier "should be unique within an AS"; Section 2.2 mandates Bad BGP
Identifier for zero or a self-identifier from an internal peer, not this case.
Section 3 explicitly mentions parallel sessions between the same speakers.
The fixture therefore uses distinct native speakers compatible with Ze's default
policy; it does not disable that policy or rewrite identifiers in transit.
FRR's bgp_open_make
writes each peer's native local_id into its OPEN.
No test route is originated from the producer's startup configuration. The checker
first requires both native FRR sessions to be Established and both Ze peers
to report established through show bgp peer list | json. These two-sided
observations share one 60-second readiness deadline. Only then does it originate
the baseline once through FRR's existing vtysh path and require its decoded path
at the consumer. Large and later control routes follow through the same origin
path. This is a first-origination barrier, not a route retry or session replacement.
The source relay starts before the sink relay. A startup network announcement
could therefore reach reactorForwardRSSection before the sink has live
forwardFacts, when that destination is correctly skipped. These live fast-path
fixtures do not attach the route-server/Adj-RIB-In event consumers for retained
peer-up replay, so waiting for the sink afterward cannot recover that earlier
announcement. FRR's Established state alone is insufficient: FRR can receive Ze's
initial KEEPALIVE before Ze processes FRR's KEEPALIVE. Ze's peer-list producer
handleBgpPeerList serializes reactorAPIAdapter.Peers's Peer.State(), not the
raw session FSM state. Peer.runOnce publishes negotiated encoding contexts and
forwarding facts before publishing PeerStateEstablished, so that peer state
is the local admission barrier. The checker reuses waitZePeerState for each
original relay peer and retains the independent FRR observations. It changes
neither AS-loop policy nor capability permissions.
The FRR-facing Extended Message relays start only after the native configuration probes pass.
Ze's CLI must list both relay peers. Each destination FRR daemon's last passive
neighbor must reach Active (passive), after peer_unshut_after_cfg clears the configuration hold.
A listening TCP socket or successful show version is not sufficient: FRR
bgp_accept can close a connection while configuration is still loading.
These barriers use the existing ordered ReadyProbe hooks, with probe-specific
environment and required stdout strings. Probes with no required strings retain
exit-only readiness; relays to other daemons and other speaker personalities retain their startup order.
No readiness probe opens a BGP connection, and the relay never reconnects a
connected socket.
Ze's own probe waits for a listener whenever Ze answers a peer it never dials.
The runner starts the peers in order, Ze before the third-party BGP daemons,
and waits on each probe before it starts the next. When the config a Ze starts from (ze.conf, or
ze-peer.conf for a second Ze) gives a peer connection remote connect false,
or holds a dynamic group, the probe runs ss -ltn and requires :<port> for
each such peer's listen port: 179 unless connection local port sets another.
The question is answered from the parsed config, so a comment that names
connect false decides nothing. Without the barrier, BIRD or FRR can dial
before Ze listens, and the refused dial waits the daemon's connect-retry (120
seconds by default), which outlasts the session budget. A Ze that dials every
peer keeps the probe that only runs true, because it redials the daemons
started after it on its own backoff. A relay barrier keeps the listener check:
one sh runs ss -ltn and then the peer list, and the probe requires both
sets of strings.
A GoBGP-facing relay, as in bgp-local-pref-strip-gobgp, is gated the same way.
GoBGP's InitialConfig calls StartBgp, which opens the listener, before it
adds its neighbors, and passConnToPeer closes a connection from an address it
has no peer for ("Can't find configuration for a new passive connection").
GoBGP's barrier is gobgp neighbor <relay address>, which must print
BGP neighbor is <relay address>: it fails with "not found neighbor" until
AddPeer has run. gobgp global -j is no barrier, because it answers as soon
as StartBgp has run. Ze's CLI must also list the relay peer, and the relay
starts after GoBGP.
That barrier admits the relay; it does not make GoBGP's session come up in
time for the injector. Under load GoBGP's session to the relay has reached
Established well over a minute after Ze started, while the injector used to
announce after four of Ze's KEEPALIVEs (about 20 s), so its burst reached
GoBGP, if at all, through the route server's peer-up replay rather than the
forward rail the wire capture judges. The injector therefore waits on a state:
its inject.msg declares option=await-file with its own 240 s bound, and
the checker writes that file into the injector's container only after FRR and
GoBGP report their sessions Established and Ze's show bgp peer list holds
both destination peers (.3 and the relay .10) established, the state the
forward rail reads. While it waits the injector answers Ze's KEEPALIVEs, so
the 15 s hold does not expire. After the burst and End-of-RIB it waits for two
more of Ze's KEEPALIVEs before the third subject, a full interval wherever the
release fell in Ze's cycle, so the third reaches GoBGP in a later UPDATE than
the first two.
Bounded readiness waits check cancellation before each probe, including the first. A simultaneously ready poll tick cannot admit another probe after cancellation is observed. Failure retains the last measured state or probe error and does not start dependent peers or the scenario checker.
A failed check names what the peer printed, not only what the checker
expected. A wait that measured the peer but timed out ends its error with
last output: and the last value its probe returned: text as the peer printed
it, any other value in Go's %v form. A wait whose value is a verdict, such as
a bool or a count, also ends with last answer: and the text its probe last
read. A one-shot assertion over an answer it read ends with searched output:
and that answer. Every quoted answer is trimmed and cut to its last 40 lines and
its last 4096 bytes, with a note on its first line saying how much was cut, so
a full routing table cannot flood the report. An empty answer is named as
empty.
Lab.Query refuses a command that exited 0 and printed nothing with an
EmptyAnswerError, kept apart from an exec that failed or was cut short. The
BGP suite's positive waits (waitContains, waitContainsFold,
waitContainsAny) read that refusal as a measured empty answer, so a route
that never arrives times out with last output: (the peer printed nothing).
FRR's vtysh prints nothing for an empty show ipv6 route ospf6, and before
this the red read "never measured peer state", which names a peer that never
answered. An empty answer still satisfies no positive wait. A negative wait
keeps the refusal, because there an empty answer would read as absence.
The shared FRR BGP route wait reads an exact-prefix show bgp <family> <prefix> json answer for presence: the top-level prefix must match and its
paths must be a nonempty array of objects. Absence instead reads show bgp <family> json and requires FRR's default-VRF table envelope with a routes
object. Only that table's missing exact-prefix key, or an empty array under
that key, proves absence. A prefix query's {} cannot: FRR 10.7.1 also emits
it when no BGP process exists. Errors, missing fields, unrelated JSON, and a
different prefix's per-prefix answer are not measured absence.
Installed-route assertions use FRR's native zebra JSON for BGP and OSPF,
including IPv6 link-local installation. One entry under the exact prefix key
must carry the matching prefix and expected protocol, with selected,
destSelected, and installed true. Its matching next-hop object must itself
carry both active and fib true. A protocol, state, address substring, or
next hop found in another entry cannot satisfy the proof. These fields come
from FRR 10.7.1
vty_show_ip_route
and
nexthop_json_helper.
The pure boundary regressions supplement, not replace, live scenario proof;
source identity alone does not establish the identity of a running image.
bgp-ospf-redistribute-default-frr follows FRR's real OSPF dummy subnet
10.82.0.0/24 through Ze's redistribution consumer and back to FRR over eBGP.
It requires the producer address 10.82.0.3/24, OSPF point-to-point network
type on ze-redist0, Full adjacency, Ze's exact protocol-250 kernel route
via FRR, and established BGP before judging the received path. FRR's native
BGP JSON must report the exact prefix, one valid best path from external
peer Ze, AS_PATH 65001, and exactly Ze's session-local address as next hop.
The connected subnet remains owned by FRR, so this assertion reads its
BGP RIB. It does not require zebra to replace that connected route. Ze's
final originated-route send guard withholds a next hop owned by the
destination peer. Reverting the consumer to explicit IGP-gateway origination
therefore meets that guard here; the useful path must use Ze's endpoint.
The counterfactual requires the real OSPF and BGP prerequisites alongside
the missing accepted path. Receiver absence alone does not establish where
the route stopped. FRR 10.7.1's bgp_update independently refuses a host-route
NLRI matching one of its local addresses, so an accepted path for its own
/32 is an invalid expectation. Its point-to-point OSPF producer advertises
the configured /24 subnet, which preserves the same next-hop discrimination.
The existing registered peer-preparation hook creates and raises ze-redist0
before FRR starts; zebra configures its address. FRR explicitly refuses
network-type changes on lo, so the producer is an ordinary dummy interface.
This scenario enables debug only for the consumer's bgp.redistribute and
the upstream redistribute.orchestrator logger at startup, so its actual
injection command survives. After failure, read-only diagnostics capture
FRR's BGP table, neighbor counters and routes, and Ze's log rings, peer state,
advertised RIB, OSPF routes, system-RIB candidates and metrics. The reads share
an independent 15-second deadline and retain query errors alongside the
original assertion failure. The recorder keeps full query answers; the error
report carries bounded excerpts.
The FRR readiness producers are
peer_unshut_after_cfg,
bgp_start, and
bgp_accept.
All five bgp-extended-message-*-frr fixtures build the native FRR attribute
through four EXTENDED route-map sequences of 100 distinct large communities.
The first sets the attribute; the next three use additive, with on-match next
on the first three sequences. FRR therefore accumulates all 400 values (4800
attribute-value octets) on one route without exceeding its 4096-byte VTY input
buffer. Repeating set large-community within one sequence is not equivalent:
FRR replaces the previous set rule of that kind.
The FRR producers are config_from_file,
route_map_add_set and route_map_apply_ext,
and route_set_lcommunity.
Before teardown, a failed Extended Message checker reads both relays' result JSON, original/delivered frame captures, TCP socket tables, and last 80 log lines. The diagnostic queries share a 15-second deadline and preserve read errors and the original assertion failure. They do not retry or replace the BGP session.
bgp-parsed-empty-mp-unreach-frr drives the parsed forwarding path from an
ASN4 source to a two-octet recipient, with real FRR behind the OPEN-only relay.
Both sessions negotiate IPv4 and IPv6 without ADD-PATH. The FRR fixture supplies
a local IPv6 address: IPv6-family negotiation still needs it on IPv4 transport.
After initial synchronization, the source seeds a route, withdraws it in an
UPDATE carrying an empty IPv6 MP_UNREACH, then sends a genuine standalone IPv6
EOR in a separate epoch. Distinct-MED announcements fence each epoch.
For the parsed-empty-mp-source speaker, explicit --result <path> selects the
JSONL capture path and places its readiness marker at <path>.ready. Both files
are created exclusively with mode 0600. Each run needs fresh paths: an existing
file or symlink causes an error rather than an overwrite.
Without --result, the defaults remain /tmp/parsed-empty-mp-source.jsonl for
the capture and /tmp/parsed-empty-mp-ready for readiness. The scenario checker
still reads these fixed defaults inside the isolated lab.
This capture-path meaning belongs to the parsed-empty-MP source. The generic
runOracleSpeaker uses --result for its textual verdict instead.
The checker requires FRR installation and removal, complete source and recipient
wire histories, the negotiated AS_PATH widths, and unchanged FRR daemon/session
identity. A mixed withdrawal must not manufacture an EOR; the later genuine
marker must survive. Before teardown, pass and failure both retain proof.json
under the native session scratch directory of the checkout the suite runs from,
including fixture inputs, captures, FRR table, PID, version and logs. The checker
resolves that directory against CheckContext.Checkout, never the working
directory, and refuses when no checkout is given. Required capture or write
failures fail the proof. Diagnostic collection has a cancellation-independent
fifteen-second bound.
Prerequisites
| Requirement | Used By | Notes |
|---|---|---|
| Docker | Interop tests | Containers for FRR, BIRD, GoBGP, Ze |
| A Docker daemon kernel with every feature Ze enrolls | Interop tests, docker-* deployment proofs, the l2tp, vpp and vpp-iface proofs, TestVPPSRv6ServiceRoute, the terminal-demo render and validate mode, the ./le perf ze DUT, ./le verify evidence release-candidate |
Checked before any image build or container start; see "The Docker host kernel check" |
| ~1.5 GB disk | Interop tests | Docker images (Go builder, FRR, Alpine) |
| Docker | ./le test integration clean, ./le test integration docker-peers |
✕ |
The interop test network uses 172.30.0.0/24. MD5 authentication scenarios require
NET_ADMIN capability (granted automatically by the orchestrator).
Real VPP SRv6 service-route proof
Run the focused service-route proof through native admission:
./le job run label vpp-srv6 command go test -tags integration -count=1 \
-timeout 30m ./internal/le/test/deployment \
-run '^TestVPPSRv6ServiceRoute$' -v
This test reuses the deployment VPP image, its container (NET_ADMIN,
/dev/net/tun and /dev/vhost-net, never --privileged), build recipes,
sockets, and staged Ze configuration. It does not replace the legacy
./le test deployment vpp population. Docker, Go, a VPP image with the
SRv6 API and AF_PACKET plugin, and a Docker kernel with veth support are required.
The managed phase also needs /usr/bin/vpp, writable /etc/vpp, and sufficient
already-reserved hugepage resources for its configured 64M heap, 1024 buffers,
and 16M stats segment. The harness never changes global hugepage reservations.
ZE_VPP_DOCKER_IMAGE, ZE_VPP_DOCKER_PLATFORM, and ZE_VPP_DOCKER_GOARCH select
the image and matching target architecture through the existing VPP settings.
The test prints the selected image, VPP version, and retained scratch directory.
These Docker-host integration tests are excluded from the QEMU guest package population: the guest has no Docker daemon. The native admission command above runs their real container and packet probes rather than substituting guest mocks.
The compiled peer sends IPv4/unicast MP_REACH with an IPv6 next hop and an RFC 9252 L3 Service TLV through a real BGP session. Only Ze installs service policies and steering. The probe reads VPP's policy and steering dumps and CLI output. It requires an encapsulation policy whose single segment is the received SID, a distinct local BSID, and matching prefix steering in table zero. Two prefixes share a policy. A normal Ze restart against the still-running external VPP must retain the confirmed BSID identity without duplicates and resume packet forwarding. Withdrawal of one, replacement of one SID, last-user withdrawal, and reinstallation each have state and packet assertions.
The same harness then switches to Ze-managed VPP. A test-only external plugin
uses the existing sdk.Plugin.EmitEvent ingress to publish best-change entries
for the same prefix in tenant tables 10 and 20. This is a separate producer
contract from the BGP/sysrib path above, not a claim of BGP VRF configuration.
Steering must use the tenant table while policy/outer lookup remains in backend
table zero. The probe kills the real managed VPP child, requires a different PID
and a production replay request, restores only interface/underlay prerequisites,
then answers that request through the ordinary event ingress. Restored forwarding,
repeated replay without duplicate BSIDs, replacement in one table, independent
withdrawal, shared-policy retention, and final removal are checked.
The runtime publisher also decodes the exact bytes it sends and requires their
replay marker to match the requested replay, rather than treating an ordinary
incremental event as a replay proof.
Before querying VPP, the probe waits for the binary API handshake's Connected
event using the production connector's bounded connection mechanism.
AllPluginsReady starts the plugin population; it does not prove the managed
VPP child has opened its API socket. The harness retains the complete managed
daemon/child stdout and stderr in srv6-managed-daemon.log and copies the
generated configuration and VPP logfile before container cleanup, including
failures before API readiness.
The packet oracle injects a unique Ethernet/IPv4 datagram through a veth and
captures VPP's emitted frame. It checks the IPv6 source and received-SID
destination, reduced encapsulation without an SRH, and preserved inner payload
with one TTL decrement. Withdrawal must stop forwarding the corresponding
datagram; adjacent positive probes prevent a dead capture from passing.
The managed phase uses VPP's built-in packet generator for real graph ingress
and Ethernet egress capture, with the same independent byte oracle. It requires
the generated-packet counter to reach one before a negative assertion. Input and
output pcaps and the external plugin's complete test transcript remain in the
scratch directory.
Prerequisite mutations must return their successful CLI response: vppctl exit
status zero alone is not acceptance. A failed prerequisite or positive packet
assertion records bounded VPP logging, interface, error, IPv6 FIB, and trace
diagnostics. Packet observations log at most eight frame prefixes per injection;
these diagnostics do not replace the forwarding assertion or extend its deadline.
The external-VPP configuration activates the real connected producer and
netlink interface monitor. A test-only SDK plugin waits for all plugins to be
ready, adds the Linux covering addresses through real netlink events, and queries
show rib until both next-hop and SID covering prefixes have protocol connected.
The BGP probe waits for that result before sending service announcements. After
Ze restarts, the SDK probe re-adds the addresses to the new monitor and publishes
a new process-generation readiness record; the old record cannot satisfy the
barrier. Its actual RIB result remains in srv6-connected.log.
The fixture additionally supplies the VPP IPv6 underlay route, neighbor, tenant tables, and encapsulation source. These prerequisites are not evidence that Ze added underlay provisioning, SID reachability validation, remote decapsulation, or external-VPP reconnect support.
Synthetic oracle tests check rejection of wrong policy bindings, tables, SIDs,
and packet fields. They are not dataplane evidence. A real proof requires an
executed, non-skipped TestVPPSRv6ServiceRoute result. For discrimination, omit
policy creation in (*govppSRv6Backend).acquirePolicy, rerun the command above
to rebuild the daemon, and require failure at the first installed-state assertion. Restore the source
and require the packet proof to pass before claiming closure.
Real VPP label pop proof (mpls-label-pop)
./le test deployment vpp runs nine scenarios in order against one VPP
container. mpls-label-pop is the fourth, after mpls-fib, and it proves
that an LDP egress pop reaches VPP's MPLS table 0 when the engine, not a
fib { } block, chooses the label writer.
Before Ze starts, the harness creates the packet-generator interface pg0, so
the VPP interface backend's first interface dump names it. Ze then runs with
interface { backend vpp; ethernet pg0 { unit default { mpls { enable true; } } } }
and ldp { lsr-id 10.0.0.1; transport-address 10.0.0.1; interfaces lo; }, and
no fib { } block. The container's Linux loopback offers LDP no advertisable
prefix, so the LSR-ID host route is the only FEC LDP originates. Every verdict
comes from VPP's own CLI:
| Check | VPP query | What passes |
|---|---|---|
installed |
show mpls fib table 0 |
Exactly one label of 16 or above holds an eos entry whose path is an ip4 deag and a neos entry whose path is an mpls deag |
mpls-input |
show mpls interface |
pg0 is followed by MPLS enabled |
popped |
show trace |
A frame carrying that label, S set, reaches mpls-lookup and then lookup-ip4-dst or ip4-lookup |
unprogrammed-drop |
show trace |
A frame carrying label 999999 reaches mpls-lookup, then mpls-drop, and no IPv4 lookup |
removed |
show mpls fib table 0 |
After the configuration loses ldp { } and the running daemon gets SIGHUP, the label has neither entry |
The parsers follow VPP's formatters in FDio/vpp v26.06, the release of the
ligato/vpp-base:26.06-release image. mpls_fib_show prints each entry with
format_fib_entry, whose first token is format_fib_prefix: for an MPLS
prefix, "%U:%U" of the label and of format_mpls_eos_bit, then "/%d".
format_mpls_eos_bit writes MPLS_EOS_BITS, "eos" or "neos", and a
reserved label prints by name, so only an allocated label matches.
format_fib_path writes "%U weight=%d " with format_dpo_proto (ip4,
mpls) and then fib_path_type_names, where a lookup path is deag.
show_mpls_interface_one prints the interface name, then " MPLS enabled".
A table 0 miss forwards through mpls_fib_drop_dpo_index, whose node is
mpls-drop. A frame on an interface without MPLS input meets the
mpls-not-enabled feature and never reaches mpls-lookup, so popped also
proves that Ze enforced mpls { enable true; } in VPP.
The frames go in through the packet generator from a pcap file in the scratch
mount, the pattern of vppSRv6PGPacket. show packet-generator must count one
generated frame before the trace is judged, so a stream that never ran cannot
pass the drop check.
The scenario is red when fib-vpp no longer consumes label entries, for example
with the (mpls-fib, entry) subscription in fib-vpp's register.go removed,
or when LDP names fib-kernel in Dependencies again: installed then fails.
The daemon binary is rebuilt by the run itself, so reverting the source and
rerunning ./le test deployment vpp is enough to show it.
Live Interop Tests (test/interop/)
Each scenario runs Ze and one or more peer daemons in Docker containers on a shared
network (172.30.0.0/24), establishes real BGP sessions, and asserts correct behavior
via each daemon's native CLI.
How It Works
The native internal/le/interoplab/bgp package discovers scenario directories in
test/interop/scenarios/. Before any scenario, the shared interoplab.Suite
engine reports what a killed earlier run left behind ("Crash-safe teardown"
below), runs the lab's preflight (which stages ze and le, and runs le's start
probe: "The start probe" below), checks the Docker daemon's kernel with that
ze ("The Docker host kernel check" below), and builds the images. For each
scenario it then:
- Creates an isolated Docker network, after refusing an existing container or network of the same name that this process did not create.
- Starts Ze and the peer daemons declared by the scenario's config files,
in the order
scenarioPeerslists them. - Waits for each peer's declared readiness probe before it starts the next.
- Runs the scenario's typed checker from the package-local BGP registry, recording every call it makes to the lab ("Recording and replaying a checker" below).
- Tears down the containers and the network this scenario created, including
after setup or checker failure and after a Ctrl-C or SIGTERM, which cancel
the run rather than end the process. A resource is recorded for teardown
only once its create succeeds, or for a container once
docker runwas issued and failed for a reason other than the daemon refusing the name as taken (ContainerMayExist): the daemon may have created it. A failure before docker ran (an address, a mount or an AppArmor option the wrapper could not render) records nothing. So a name the pre-clean refused, one another run took first, or one never created is never removed.Docker.CreateNetworkrefuses a network name that already exists rather than join it. A SIGKILL, a lost machine or a panic runs no teardown. The next lab start reports the leftovers. The explicit./le test integration cleancommand can remove them.
An exact BGP-lab scenario selector, whether passed explicitly or through
INTEROP_SCENARIO, changes image preparation to the selected peers' dependency
closure. After the network is created and dynamic configuration preparation has
returned its final peers, Suite.preparePeerImages matches their logical image
names against the suite's existing declarations. It builds Ze and pulls the
actual FRR reference those peers use, including a scenario-specific version pin,
without preparing unrelated BIRD, Kea or other peer images. An undeclared image
or a failed dependency fails the scenario before any peer starts; an optional
image failure affects only scenarios that depend on it. Successful builds are
shared within the run and their owned retention tags are released once, including
after setup failure or cancellation. The selected-image progress line follows
the scenario's running line. Without a selector, the lab retains the eager
full-suite image preparation described above; the kernel probe, binary staging
and deadlines are unchanged.
The start probe
A staged binary can build and still fail to start. On 2026-11-10 every le
process panicked at start on a help line over the 60-character limit (commit
c77a8dd2f3). The cross-compile passed, so the preflight passed, and every
helper container that runs le (speaker, relay, injector) then failed its
readiness wait. Four scenarios failed one after another, each after its image
build, before anyone saw that no scenario had reached its checker.
A LabBinary may declare a StartProbe: the arguments the staged binary is
started with once, right after its build. StageBinaries runs it in a
one-shot alpine:3.24.3 container through RunOneShot, with the binary
mounted read-only at /lab/<name>. The file keeps its declared name because
cmd/ze selects its personality from that name. The container is built the
same way as the kernel probe's (stagedBinaryContainer), so a darwin host
runs the linux binary, and a probe that is cut short is removed by name.
Every lab that stages le declares linuxle.StartProbe, which is le help.
le runs its registration checks before it reads its first word
(leroot.DispatchWithStartup), so help reaches the check that panicked,
and help needs no checkout. A binary that declares no probe is not run: a
staged daemon needs a config to start, and the kernel check already runs the
staged ze.
A probe that exits non-zero, or that Docker cannot run to an exit, stops the
preflight before any image build. The refusal names the binary, its output
path, the probe argv, the exit status and the first twelve lines of its
stderr, then the fix: correct the start failure the stderr names, and run
./le test integration <lab> again. Each probe prints one progress line with
its verdict and elapsed time:
interop: start probe le (`le help`): pass (<elapsed>)
Crash-safe teardown
Every container, one-shot container (the kernel probe included), network and
built image the wrapper (interoplab.Docker) creates carries owner labels, all
from one helper, ownerIdentity.labelArguments:
| Label | Value |
|---|---|
ze.lab |
1, the marker required by every cleanup verdict |
ze.lab.run |
a random id unique to the creating process |
ze.lab.launch |
an optional inherited id from the parent that launched the creating process |
ze.lab.pid |
the creating process's pid |
ze.lab.pid-start |
that pid's start token, which changes when the pid is reused (procscan) |
ze.lab.boot-id |
the client machine's boot id |
ze.lab.machine-id |
the client machine's id (/etc/machine-id, or the Mac's IOPlatformUUID) |
ze.lab.started |
when the run started, RFC 3339 |
ze.lab.purpose |
the creating command line, for the operator |
ze.lab.kind |
the object the create made: container, network or image |
ze.lab.uid |
the creating process's uid |
ze.lab.os |
the client's operating system, as Go names it (linux, darwin) |
ze.lab.pid-ns |
on Linux, the inode of the creating process's pid namespace (/proc/self/ns/pid). Absent on macOS, which has no pid namespace |
A container inherits the labels of its image, so one a person starts by hand
from a lab-built image carries the builder's owner labels with ze.lab.kind
image. The wrapper sets the kind on every create, overriding the inherited
one, and the cleanup judges a resource only when its kind label names the
object it is: such a container is kept and reported as carrying image labels
only, not created by a lab run.
Every owner fact is read on the client, the machine running le, never from
the daemon: on a Mac the daemon runs in a VM whose pids are not the Mac's.
Automatic startup only inspects leftovers and reports a dry cleanup preview. A same-name container or network is reclaimed only when every owner label equals this process's own labels. An exited foreign owner still causes a name refusal.
A parent that launches test children passes Docker.ChildEnvironment to each
child and uses the same client for Docker.CleanupOwnedOrphans after they exit.
The private bootstrap variable ZE_LAB_LAUNCH carries the parent's launch id.
Descendants keep that id but generate their own ze.lab.run and process labels.
The owned cleanup filters and rechecks ze.lab.launch before the ordinary owner
checks below. It retains resources with no launch label or another launch id.
This attributes cooperating children to their launcher. It does not protect against copied labels or isolate suites within one launch.
The explicit Docker.CleanupOrphans removes a labeled resource only when its owner is
provably gone. Every path to "gone" first requires the owner's machine id,
operating system, uid and pid namespace to equal this client's: a client in
another pid namespace (a devcontainer mounting the host's docker.sock) or
under another uid (a /proc mounted hidepid=2) sees a live owner's pid as
absent. Then one of three paths proves it: the same boot with the pid absent;
the same boot with the pid held by a process with another start token; or
another boot id with the owner started before this machine booted (btime in
/proc/stat, kern.boottime on macOS). A pid held by a process that has
exited and is not yet reaped (a zombie, Z or X in /proc/<pid>/stat or in
ps -o state=) is absent: the go test child a docker-peers Ctrl-C kills is
one until le reaps it, and it owns nothing. An owner under another boot id that
started after this boot is a cloned machine sharing the machine id, which may
still be running, so it is kept. It removes containers, then networks, each by
the id the labeled listing gave, then the run-owned ze-interop-retained: tag
by its name (Docker.ReleaseImage). Removing a tag by name untags only that
reference, never the image another tag holds, and only a tag in the
ze-interop-retained repository is a candidate, so the shared cache tag is
never named.
Everything else is kept and reported with its reason and the command that
resolves it: an owner still running, an owner on another machine (run the
cleanup there), in another pid namespace (run it there) or under another uid
(run it as that uid), labels it cannot read, a boot time it cannot read, or a
network another run's container is still attached to. An unlabeled resource, a volume and the shared cache tag
that NO_BUILD=1 reuses are never candidates.
A removal call that fails, or that outlasts the 30 s client bound while the
daemon completes it, is not reported failed on its own say. The wrapper
inspects that exact resource again under the same bound
(Docker.confirmRemoval): gone is "removed", still present is "failed" with
the removal's error, and an inspect that fails too is "unknown", naming both
errors.
A docker rm -f the daemon refuses with "removal of container --rm does for a one-shot whose bound ended. The wrapper inspects the
container at once, then every createSettleInterval until
createSettleWindow ends (Docker.awaitRemoval). Gone is "removed", an
inspect that fails is "unknown", and a container still present when the window
ends is reported by name as still present, with ./le test integration cleanup. Only a container removal gets that answer: it is the one "already in
progress" refusal in the dockerd 29.9 binary, so a network or image removal
keeps the single inspect above.
A one-shot container (Docker.RunOneShot: the kernel probe, a host
preflight, a terminal-demo render) runs in the foreground with --rm and a
ze-oneshot-<random> name. Killing the docker client stops nothing the
daemon runs, so when the caller's context is canceled or the one-shot's bound
ends, the wrapper removes the container by that name on a context the
cancellation does not reach, and reports the removal's outcome with the
run's error. Its free-form OneShotContainer.Arguments, like a lab peer's
PeerConfig.Arguments, are refused before docker runs when they carry
--name (which would replace the name the removal finds it by), a ze.lab
label in any spelling (which would override the owner labels), or
--label-file, with a BUG: error naming the field. Any spelling includes
-l at the end of a short-flag cluster such as -dl <v>, -dl=<v> or
-dl<v>, read the way docker's flag parser reads one: past the value-free
flags of docker run (-d -i -P -q -t), stopping at the first flag that takes
a value. --name has no short form.
A caller outside a lab Suite that needs a long-running container, such as the
deployment proofs (./le test deployment l2tp, vpp-iface,
vpp), starts it with Docker.RunDetached. Its typed spec
(DetachedContainer: platform, network and address, capabilities, devices,
mounts, environment, working directory, entrypoint, command) carries no raw
docker flag, and the container carries the same owner labels, so a run killed
before its own removal is reaped like a lab's. Commands inside it run through
Docker.ExecAttached, with attached streams and their own bound. Those verbs
root on SIGINT and SIGTERM, remove their container on a context the signal
does not reach, and name a failed removal in the report's
container-removal-error, exiting 1. The removal is owed from the moment
docker run -d was issued, under the same ContainerMayExist test a lab uses:
the daemon creates the container before it starts it, so a start it refuses
(a device the host lacks, such as /dev/vhost-net) or a Ctrl-C during the
start still removes the container. A run that returned an error (a step it
could not perform, or an interrupt) reached no verdict, so it prints none: the
error, then a removal that failed with the command that finishes it, and it
exits 1.
Killing the docker client does not cancel a create the daemon already
accepted, so after a Ctrl-C during docker run the container can appear after
the first docker rm -f answered that no container has that name. Docker
29.9 gives that answer as exit 0 with no output, and prints the name only for a
container it removed; older releases refuse with "No such container", and both
answers count as absent. When a start failed or was canceled, Docker.RemoveContainer does not accept that
first answer. It runs docker rm -f again every createSettleInterval until
createSettleWindow ends, and removes the container if it appears. If the
container appears and cannot be removed, or the wait is stopped, the error
names the container and ./le test integration clean. This applies to every
start that marks its name: Docker.RunDetached (the deployment proofs, the
benchmark), a lab peer's run, and a one-shot container. A create that lands
after the window still carries the owner labels, so the cleanup removes it.
Removing a container whose start succeeded takes one docker rm -f. So does
any later removal of a name: the first removal after a failed start uses the
mark up, whatever its docker rm -f answered.
A docker command le stopped is named for what stopped it. One whose own bound
ended reads "did not finish within timeout(1) answers. One that a deadline of its caller's context ended first
reads "was cut short by its caller's deadline after Wait window that closed while its probe's exec ran is the
usual case, and the command did not hang. It carries exit 124 and still wraps
context.DeadlineExceeded, so a caller that branches on a deadline takes the
same branch; only the text differs. The runner tells the two apart by the
cause the command's own bound records, not by reading the caller's context
afterwards, which a deadline passing just after the bound fired would fool. A
lab cross-compile its caller cut short is refused the same way, without the
slow-build reasons. One whose context was canceled, by an interrupt reaching
the verb's signal root or by the run stopping, reads "was canceled before it
finished" and carries no exit code, because the code would be le's own kill.
Every docker command runs in a process group of its own, so a terminal Ctrl-C
reaches le and not the docker client: the signal root cancels the run, and
the removals it then issues on a context the cancel does not reach are not
killed by the same signal, or by a second Ctrl-C. A command handed a terminal
as one of its streams stays in the group of le, where reading the terminal
raises no SIGTTIN ("How a suite's child processes end" in
runner-architecture.md).
The multi-DUT benchmark (./le perf run, ./le perf record evidence) is the
other such caller. Its network goes through Docker.CreateNetwork and
Docker.RemoveNetwork with the owner labels, each DUT and the sender through
Docker.RunDetached, the readiness wait through Docker.ContainerRunning,
and each image through Docker.Build (a named build context, such as
FreeRtr's source checkout, is a typed NamedBuildContext) followed by
Docker.ReleaseImage, which drops the run-owned tag and keeps the DUT's own.
Only BuildKit accepts --build-context, so a build that carries a named
context first asks docker buildx version. When the docker CLI has no buildx
plugin, docker build would run the legacy builder and answer with its usage
text, so Docker.Build refuses before docker build runs: the error names
BuildKit and the missing plugin, and names ./le setup install as the fix.
That run's docker-buildx step installs the plugin on macOS and names the
package to install as root on Linux (docs/guide/developer-setup.md, "Docker
labs"). The error then gives the same install by hand: brew install docker-buildx and the cliPluginsExtraDirs entry Homebrew prints on macOS,
added to the config file this docker client reads ($DOCKER_CONFIG/config.json
from the client's environment, else ~/.docker/config.json, resolved by
setup.DockerConfigPath, the helper the setup step edits it through), and the
distribution's buildx package on Linux. An environment that sets
DOCKER_BUILDKIT to a false value forces the legacy builder even with buildx
installed, so that build is refused too, naming the variable and
unset DOCKER_BUILDKIT; Ze never overrides the setting. The named context is
never dropped to make the build run.
Docker.ContainerRunning answers an inspect it cannot read as an error, never
as "not running". It removes only a DUT or sender name it recorded, and records
one only when interoplab.ContainerMayExist says its RunDetached may have
left a container: never for a name the daemon refused as another container's.
Suite.Run inspects resources after the Docker probe and before Preflight.
It prints each proposed removal or skip, then interop: cleanup: N to remove, N kept, N failed, N unknown. An empty inspection prints interop: cleanup: no Ze lab resource on this Docker daemon. A failed listing stops setup and
names the failing Docker command to diagnose before another run.
Before a scenario creates resources, its pre-clean reclaims a same-name resource
only when all labels equal this process's labels (ownerIdentity.created).
This includes an earlier scenario's shared network whose removal outlasted its
bound. Every other owner causes a refusal, including an exited owner. The
refusal names the lab's suffix variable, such as ZE_L2TP_INTEROP_SUFFIX, when
available, or another action that avoids the conflicting name. The scenario's
teardown removes only resources the scenario itself created.
./le test integration clean runs the cleanup by hand and exits 1 when a
removal failed, naming the failing command. ./le test integration clean dry lists what would go and what stays, removes nothing, and also lists
ze-interop-retained: tags left from before the labels existed, which no mode
removes. Both render through | json.
The Docker host kernel check
A container runs on the Docker daemon's kernel, so a kernel feature Ze needs and
that kernel lacks fails scenarios for a fact about the host. mobike-initiator
and mobike-responder showed the cost: on a 6.8 kernel they spent fifteen
minutes and reported two protocol failures that were the missing
XFRM_MSG_MIGRATE_STATE. Every Docker run that runs Ze therefore checks the
daemon's kernel first, and refuses a kernel that lacks any feature. The owner's
ruling is that a wrong kernel "should not be possible - fail": there is no skip,
no warning mode and no exemption.
The required set is not listed in le. Suite.Run starts the ze the lab staged
for the daemon's architecture (Suite.StagedZe, from StagedZePath) in a
throwaway container and reads its answer:
docker run --rm --cap-add NET_ADMIN --cap-add SYS_ADMIN --network none \
-v <staged ze>:/ze:ro alpine:3.24.3 /ze doctor --json kernel-capabilities
ze doctor kernel-capabilities probes every capability a package enrolls in
internal/component/kernelcap, with configuration ignored
(docs/architecture/doctor-and-health-checks.md). A capability enrolled later
is required of every Docker host with no le change. NET_ADMIN lets the
netlink probes ask; SYS_ADMIN lets the xfrm-interface probe unshare a network
namespace. Without it that row reads unknown, which also refuses.
On a daemon whose docker info security options list name=apparmor, the
probe container runs under Ze's AppArmor profile instead of docker-default
(--security-opt apparmor=ze-kernel-probe): docker-default denies the MPLS
transit MTU probe's mount and its /proc/sys write, and that row would read
denied (owner D-7: the host is set up for Ze, never unconfined). On a Linux
client whose /sys/kernel/security/apparmor/profiles lacks the profile, the
check refuses before any container starts and names
./le setup docker kernel apparmor confirm ze-kernel-probe. An unreadable list
(apparmorfs refuses it to a reader without policy-view privilege, such as a
non-root user) is never read as "loaded": the probe runs, and
when runc answers "unable to apply apparmor profile" the check refuses with the
same command rather than reporting an unreadable answer. Both refusals also name
the daemon's kernel release and the kernel route below, because the profile lets
the probe answer and adds no kernel feature: a stock kernel may still lack some
(MOBIKE's xfrm migrate, for example), and the operator reads the whole path in
one run. On a Mac both refusals, and a denied row's fix, name the same
command and say that there it loads the profile into the Colima VM behind the
Docker endpoint, and that the labs' ze profile boots Ze's kernel, which
applies no AppArmor, so a daemon applying it is one DOCKER_HOST names
(appArmorColimaLoad); then the macOS route, ./le setup docker host start.
Colima's Ubuntu default VM applies AppArmor and holds only the profiles loaded inside it, so
runc's "write ... attr/apparmor/exec: no such file or directory" is the
kernel's answer for a profile it has not loaded (colima 29.5.2,
6.8.0-117-generic, 2026-10-21: docker-default applies, an unknown name
fails with the same text). On a client
that is neither Linux nor macOS neither refusal names the load command, which
refuses there (appArmorLoadPlatform): the refusal says so, still names the
daemon's kernel release, and names only the kernel route. The client system is the one the Docker wrapper
records in its owner labels (ze.lab.os); it is read from the client even when
the machine or boot identity is not, so a host with no /etc/machine-id still
gets the Linux text (readHostFacts).
A security options query that fails or prints no JSON refuses, because the argv depends on it. Ze's own kernel carries no AppArmor, so the QEMU guest's daemon lists none and the guest route is unchanged.
The probe is the first container to run under a Ze profile, and the same rule
holds for every lab container: one that needs more than docker-default grants
runs under a Ze profile that grants exactly that, never apparmor=unconfined.
Each profile registers in interoplab (RegisterAppArmorProfile, from a lab's
register_apparmor.go) with its name, what runs under it, what docker-default
denies it, its text and the command to run once it is loaded. A peer names its
profile in PeerConfig.AppArmorProfile. On a daemon that applies AppArmor the
container starts with --security-opt apparmor=<name>; on a Linux client whose
profile list lacks it, the lab refuses before the container starts, naming
./le setup docker kernel apparmor confirm <name>; where that list is
unreadable the container runs, and when runc answers "unable to apply apparmor
profile" the lab refuses naming the same command (labAppArmorUnapplied); on a
Mac both refusals name the command, which loads the profile into Colima's VM,
and then the macOS route, ./le setup docker host start; on any other client they name the route
instead of the command, which refuses there (appArmorLoadUnreachable); on a
daemon that applies no AppArmor the option is left out. ze-kernel-probe parses with apparmor_parser -Q
4.0.1 (Ubuntu 24.04).
A lab peer runs with no more than its daemon needs, never --privileged
(plan/release/validation/spec-lab-containers-least-privilege.md). A sysctl set once
before the daemon starts is a --sysctl argument, written by Docker into the
peer's own network namespace. A sysctl written while the peer runs takes
interoplab.NetSysctlWriteArguments (systempaths=unconfined) together with
the generic profile ze-lab-net-sysctl, which is docker-default with every
/proc/sys write denied outside net/ and every mount denied; any lab names it,
none defines its own copy (owner, 2026-10-10: one shared profile granting only
what the labs together need). The VRRP scenarios' ze (any scenario carrying
keepalived.conf) is such a peer: once it knows its interface VRRP writes the
per-device knobs under net/ipv4/conf/ and net/ipv6/conf/ and the
namespace-wide net/ipv4/conf/all/rp_filter and
net/ipv4/icmp_errors_use_inbound_ifaddr (applyDataplaneSysctls,
internal/plugins/vrrp/dataplane_linux.go), and
TestVRRPWritesOnlyLabGrantedSysctls records every write it makes and checks
the profile grants each. In the IPsec lab the NAT box, strongSwan and ze hold
NET_ADMIN, and ze also holds BPF, because vpn ipsec application-override deny (the default) loads and attaches its setsockopt guard and fails the apply
without it (capabilityBPF, internal/le/interoplab/ipsec/ipsec.go). In every
scenario ze also holds the host's /dev/net/tun (interoplab.TUNDevice,
--device) and runs under ze-lab-net-sysctl with /proc/sys writable: a
Child SA that receives UDP-encapsulated ESP makes ze create the ze-espform
TUN device and write its disable_policy and rp_filter (openESPFormTUN,
internal/component/ike/dataplane/espform_tun_linux.go), and the install is
refused when either fails. Whether an SA is encapsulated is decided at run
time (NAT detection, MOBIKE, which floats IKE to port 4500 before IKE_AUTH, or
a peer that came in on 4500), so no scenario is exempt. TUNSETIFF needs
CAP_NET_ADMIN in the container's namespace, which ze already holds. The NAT
box sets its three sysctls through --sysctl; strongSwan
runs under ze-lab-net-sysctl only in a scenario whose scenarioCheckers
entry sets writesPeerSysctl (internal/le/interoplab/ipsec/checkers.go),
because its checker writes strongSwan's sysctls at run time (it cuts the
reassembly marks through dropFragmentsAtPeer, or enables IPv6 on the lab
interface through vtiIPv6PeerInner), and TestIPsecLabPeersRunWithoutPrivilege
checks the profile grants every one of them;
TestOnlyCheckersThatWritePeerSysctlsAreFlagged reads the source and refuses
a checker that reaches one of those helpers without the flag, or the flag
without the write. In the L2TP lab ze and the xl2tpd LAC hold NET_ADMIN and
--device /dev/ppp, and the preflight probes with the same grants and loads no
module, so it passes only where the peers can run. The PPPoE lab grants ze,
accel-ppp and the pppd client the same two, its preflight probes the same way,
and no PPPoE image installs kmod or runs modprobe. The RADIUS subscriber lab
grants ze and its LAC the same two and FreeRADIUS neither.
The /dev/ppp half of those preflights is declared once, in
internal/le/interoplab/pppdevice.go, because the Docker host kernel check
below cannot see a character device. Each lab still runs one preflight
container: it puts PPPDeviceProbe (the line printing DEV_PPP=ok or
DEV_PPP=missing) first in its sh -c command and adds its own probes, parses
the output with ParsePreflight, takes the device entry of its refusal from
PPPDeviceMissing, and refuses a container that did not run through
PPPPreflightRunRefusal. That refusal always wraps Docker's error, keeping the
operation, exit code, stderr and cause, and adds sudo modprobe ppp_generic
only when Docker's stderr names /dev/ppp.
The TUN clone device every lab Ze running IPsec holds is declared once too,
as TUNDevice and TUNDeviceArguments in internal/le/interoplab/tundevice.go;
the IPsec lab's ze and the home-CPE lab's CPE take it from there. It has no
preflight: Ze's kernel builds tun in (CONFIG_TUN=y,
gokrazy/kernel/runtime.config), so a Docker host on Ze's kernel always has
the node, and on any other host Docker refuses the container start with an
error gathering device information line naming /dev/net/tun.
TestIPsecLabPeersRunWithoutPrivilege holds the IPsec lab's ze to exactly
NET_ADMIN, BPF, that device and the net sysctl grant.
The check runs after the preflight, because it needs the staged ze, and before the first image build, so a refused host costs no build and counts no scenario. It refuses when:
| Answer | Refusal names |
|---|---|
a row is unknown |
the probe failure, every non-present row by subsystem, CONFIG_ symbol, state and reason, and the daemon's kernel release. Unknown does not establish a missing kernel feature and never prescribes an install, kernel rebuild or reboot. Resolve the reported probe failure and rerun the same /ze doctor --json kernel-capabilities probe through ./le setup docker kernel check ze '<staged-ze>', with the actual supplied executable shell-quoted in the refusal so the retry does not cross-build a different capability population |
| ✕ | every such row by subsystem, CONFIG_ symbol, state and reason, the daemon's kernel release, and the next step for the platform (DockerKernelRoute, owner D-6), its command on a line of its own: on Linux a kernel with every feature or ./le setup docker kernel install and a reboot; on a daemon whose kernel release already ends in -ze (Ze's CONFIG_LOCALVERSION: the QEMU guest of ./le test qemu docker-lab, or a host after the install), on any client system but macOS, Ze's runtime kernel's own fix instead (dockerKernelRouteFor: add the options to runtime.require, then rebuild the kernel cache entry with ./le build host-driver and ./ze-host appliance kernel --target runtime --arch <arch>); on macOS the ze Colima profile, which boots Ze's docker-host image on Ze's runtime kernel: unset a DOCKER_HOST that names another daemon, or add the options to runtime.require and rebuild the image with ./le setup docker host build, then ./le setup docker host start; on any other system the refused run's own le words against the Docker daemon of the Ze-kernel QEMU guest, ./le test qemu docker-lab lab '<the words>' printed with the words filled in, followed by env '<NAME=value>' with each scenario selector variable the suite declared to ReadEnvironment and the run set (IPSEC_INTEROP_SCENARIO=mobike-responder, for example), so the guest reruns the one scenario rather than the whole suite; or ./le test qemu docker-lab (boot and check only) when the process is not le or a word or an assignment holds white space or a quote. A denied row adds, on a daemon applying AppArmor, ./le setup docker kernel apparmor confirm ze-kernel-probe; on any other daemon it says the policy the row names must grant the probe its steps, and under SELinux names sudo ausearch -m avc -ts recent to read the denial. gokrazy/kernel/runtime.configgokrazy/kernel/runtime.config |
| the container fails, or prints no JSON | the command it ran, its exit code, stdout and stderr |
| the answer holds no row | the command it ran |
every row is present but the probe exits non-zero or answers ready false |
the command it ran, its exit code and the ready it read: the producer's verdict and its rows disagree, so the answer is unread, never a pass |
| the suite names no staged ze | Suite.StagedZe |
The refusal is the suite's setup error: interop: setup: <reason>, exit 1. The
docker-l2tp-ppp and docker-pppoe-accel deployment proofs run the
L2TP and PPPoE suites, so they reach the same check. A suite with no kernel
module of its own, such as the FreeRADIUS admin-login suite, is checked too:
the owner's rule is about the host, not about the scenario.
The vpp-iface VLAN QoS scenario keeps one Ze daemon running through two
SIGHUP reloads. A successful empty-map preflight establishes a fresh pool.
VPP's own show qos egress map output must contain exactly one map, whose
external ID is the VLAN index and whose mark refers to the first pool entry.
Its output must first contain
the full VLAN row with priorities 3 and 5 mapped to themselves, then only
priority 3 after replacement. The next reload removes the egress map from
config, and show qos mark must report no mark on the same existing VLAN
interface. A failed CLI query, a changed interface index or an exited daemon
cannot satisfy the proof. The signal targets the exact daemon command in the
proof's private container, as the other VPP deployment scenarios do.
The l2tp, vpp and vpp-iface deployment proofs run no suite,
and neither does TestVPPSRv6ServiceRoute (build tag integration), which
prepares its daemon through the step vpp-iface uses. Each cross-compiles its own daemon to tmp/evidence/bin/ze-linux-<arch>, and
buildCheckedDaemon runs the same check with that binary right after the build,
before the proof inspects or pulls its peer image and before its first
container, so a refused host downloads and starts nothing. With
NO_BUILD=1 the proof skips the build and probes with the daemon already at
that path, and refuses by name when there is none.
The terminal-demo render (le site terminal-demo render and render all) and
the renderer's validate mode run Ze in validator and recorder containers, so
both make the same check, through one shared preflight, with the demo binary
(tmp/terminal-demos/bin/ze) after confirming that binary exists and before the
renderer image is inspected or any container starts.
./le perf run and ./le perf record evidence prepare the ze DUT's inputs
through the BGP lab's own staging whenever the selected build or test includes
Ze (every DUT, or ze named). That staging cross-compiles the same three
binaries test/interop/Dockerfile.ze consumes: ze-linux, le-linux and
ze-le-linux. A test then makes the same kernel check with the staged
test/interop/ze-linux before the benchmark starts. A build-only invocation
stages before image construction, but starts no Ze and does not probe the
runtime kernel. Selecting only other DUTs neither stages Ze nor probes for it.
Every route refuses a staged ze path that holds no file, or holds a directory,
naming the path, before any docker command runs. Docker answers a bind mount of
a missing path by creating a root-owned directory there, which a later staging
then cannot replace. A missing file names, on a line of its own, the run's own
le words again: env -u NO_BUILD ./le <words> when the run skipped its build,
./le <words> otherwise (perf stages without NO_BUILD).
./le verify evidence release-candidate runs the whole verify gate, Ze's
functional tests included, inside a container on the daemon's kernel. After the
clean-tree check and before the container starts, it cross-compiles a linux ze
for the daemon's architecture into tmp/verify-evidence/ze-linux, which git
ignores so the tree stays clean, and makes the same check with it. The
container never runs --privileged: it holds one --cap-add for each
capability the option=needs-linux:caps= vocabulary names and one --device
for each node a production path reached by a test opens (/dev/ppp, opened by
ppp.DevPPPSetup, and /dev/net/tun, opened by openESPFormTUN for an IPsec
Child SA that receives UDP-encapsulated ESP), read from the runner's own tables
(runner.ContainerCapsDockerNames, runner.ContainerDevicePaths), and keeps
Docker's default seccomp and AppArmor profiles. A test declares no device:
TestContainerDevicesCoverWhatTestedPathsOpen reads the L2TP and IPsec
dataplane sources for every /dev node they open, as a literal or a package
constant, and checks the table holds exactly those. A token added
to either table reaches the container with no other edit. sys-time is left out:
a container shares the Docker host's clock, so a verify run must not set it,
and the NTP restore test skips there through its own caps gate. A container
never loads a kernel module, so a Docker host without one of those nodes makes
Docker refuse the start; the run then answers an error naming the node and the
host command that creates it (sudo modprobe ppp_generic for /dev/ppp,
sudo modprobe tun for /dev/net/tun)
rather than a bare exit code.
The run is a release check only Thomas starts. The verb declares
leaction.Action.WholeTree, so before its body runs, and so before the
clean-tree check, the kernel probe or any docker command, the dispatcher
refuses it with exit 2 unless stdin and stdout are both terminals
(human.RequireTerminal). An agent's tool call reaches le through pipes, so it
is refused, and the refusal gives the exact line to hand Thomas and
./le verify changed for the agent's own evidence. The container is started
with docker run -t, which gives the inner ./le verify current, itself a
whole-tree verb, a pseudo-terminal on stdin and stdout: the host side already
proved a person started the run, and the terminal carries that fact in without
an override keyword another caller could type.
The container is a Docker.RunOneShot, so it carries the owner labels and a
name, and runs under a four-hour bound (owner D-1, 2026-10-19): the gate takes
25 to 53 minutes, so the bound never ends a normal run and only stops a wedged
daemon. SIGINT and SIGTERM cancel the action's root context; the wrapper then
removes the container by name, and the run answers the interruption as an
error, never as a failed gate. The three named cache volumes
(ze-gomod-cache, ze-gobuild-cache, ze-apt-cache) are mounted, never
removed: they are what keeps a second run minutes long.
On macOS the daemon in hand is the ze Colima profile's, which boots Ze's
docker-host image (ai/META-DECISIONS.md). With DOCKER_HOST unset or empty, every
docker command le runs for a lab, the kernel check included, carries
DOCKER_HOST=unix://<colima home>/ze/docker.sock, where the Colima home is
resolved as Colima resolves it ($HOME/.colima by default, COLIMA_HOME or
$XDG_CONFIG_HOME/colima when set; dockerhost.ColimaHome) (dockerhost.Environment,
through Docker.run, the one route to the process runner), whatever
/var/run/docker.sock or the current Docker context names. When that socket is
absent the first docker command refuses, before any image build or container,
naming the missing socket, that the labs on macOS run on the ze profile, and
./le setup docker host start on a line of its own; Colima's default profile
and the context are never used in its place. A DOCKER_HOST the operator set is
kept as it is, because it names a daemon on purpose (the QEMU docker-lab's
forwarded socket is one), and the kernel check stays the gate that judges that
daemon's kernel. ze appliance build, which every QEMU verb that builds an
appliance runs (./le test qemu installer and its siblings,
vpp-hugepages, mpls-boot, appliance-hostname, the two crash-capture-* verbs), runs
under the same selection. ffmpeg, which le site terminal-demo runs, is no Docker
client and runs under le's own environment.
./le setup docker kernel check asks the daemon in hand the same question. With
no ze it cross-builds tmp/qemu/linux-<arch>/ze for the daemon's architecture
(docker version's server arch) through BuildLinuxZe, the producer
./le test qemu docker-lab uses; ze <path> names another. Either way the file
must be an ELF executable for that architecture, or the check refuses before any
container runs, naming the path, what it is instead (a directory, not ELF,
another machine), and the build route. ./le setup docker kernel apparmor runs on
Linux, against this machine's kernel, and on macOS, against the Colima VM behind
the client's Docker endpoint (docker context inspect, dockerhost.ColimaProfile):
a docker.sock beside a colima.yaml names that directory's profile, under
any Colima home (~/.colima, $XDG_CONFIG_HOME/colima or COLIMA_HOME), and
the docker.sock at a home's root, which Colima forwards for the default
profile (its Lima instance _lima/colima), names default. It refuses any
other macOS endpoint, such as Docker Desktop's, naming the macOS route,
./le setup docker host start. On macOS
every step runs inside the VM through colima ssh --profile <profile> --: a VM
that does not answer true refuses naming colima start --profile <profile>, a
missing colima or docker names its brew install, and a question that
hangs for 30 s names colima status --profile <profile>. The profile is staged
under the checkout's tmp/, which Colima's home mount shows the VM at the same
path; a VM that cannot read it refuses before any root step, naming colima start --profile <profile> --edit. It refuses where
/sys/module/apparmor/parameters/enabled does not read Y or
apparmor_parser is absent on that machine, installs the registered profile named after
confirm as /etc/apparmor.d/<name> (so it loads again at boot) and loads it
with apparmor_parser -r -W; without confirm naming a registered profile it
prints the two sudo steps of every registered profile, runs none, and names
the command that loads each. ./le setup docker kernel install is the Linux
route: it installs the cached runtime kernel for the host's architecture under
/boot and /lib/modules, rebuilds the initramfs and the GRUB menu with the
Debian tools, and saves the new entry as GRUB's default by its title. It
refuses before any step unless GRUB_DEFAULT=saved, and unless the cached
release is the release the version internal/appliance/kernel.version pins
builds (7.3.0 for pin 7.3; see docs/architecture/appliance/kernel-profiles.md)
followed by the CONFIG_LOCALVERSION suffix its config declares (-ze, so never a bare
upstream release another kernel could own, nor an entry built before a version
bump), and unless the
cached kernel's config builds in, or as a module, every option this host
relies on its running kernel for: the PCI hotplug drivers when an interface in
/sys/class/net, or one netplan's generated
/run/systemd/network/10-netplan-<id>.network names, is a slot name (ens18,
which the kernel without them renames enp0s18); the option behind each
iptables table, match or target module in /proc/modules; and UFW's stock
matches and targets when /etc/ufw/ufw.conf exists. One table maps each
module to its option, and a loaded netfilter module it does not map is
refused by name. The refusal names each dependency, what the kernel would
break, the missing options, and the rebuild (hostdeps.go). After the steps
it reads /boot/grub/grub.cfg, through sudo cat when the host keeps it
root-only, to find the entry's title. Without
confirm <release> it prints the steps and runs none, because passwordless
sudo asks nothing: ./le setup docker kernel install confirm <release>, where
<release> is the version internal/appliance/kernel.version pins with the
-ze suffix. The cache entry is keyed by the backend that built the kernel, and ze-host answers
docker's entry whenever the docker binary is on PATH, so a kernel built with
QEMU on such a host is named with builder qemu
(./le setup docker kernel install builder qemu confirm <release>); the dry
run's confirming command and every rebuild it prints keep that backend. A
reinstall replaces: it removes /lib/modules/<release> and
/boot/initrd.img-<release> before writing them again, so cp -a never nests
the new modules inside the old tree. The copy drops the cache's ownership, so
the installed modules belong to root, not to the user who built them. It prints each sudo step before it runs
it, and never reboots. Off Linux it refuses: on macOS it names the ze Colima
profile, ./le setup docker host start; on any other system the guest route,
./le test qemu docker-lab (docs/architecture/testing/qemu-integration.md).
An interrupted or failed confirmed plan lists its completed command prefix and
unfinished work, including the GRUB default selection. A stop before its first
step says no install step started. A command that might have changed files
instead leaves an explicit do-not-reboot warning until installation finishes,
including the gap between removing and replacing modules. The refusal states
that no automatic reboot was requested and prints the exact command to finish,
with the same builder and confirmed release.
To ask a host by hand, stage ze with any lab once and run the command above.
TestDockerKernelCheckOnThisHost (build tag integration) does that against
the daemon this machine uses: it stages ze, starts the probe container and
nothing else, and judges the verdict against the rows it read. On colima's
6.8.0-117-generic (2026-10-10) the check refuses ipsec-mobike, l2tp,
l2tp-ppp and mpls as absent and mpls-transit-mtu as unknown, and an
unconfined probe on 2026-10-21 answered the same five rows: that kernel's
config leaves CONFIG_XFRM_MIGRATE unset, and its L2TP and MPLS routing
modules are in linux-modules-extra, which Colima's image does not install. Loading
Ze's profile there lets the probe answer; it does not make the check pass.
./le job run label kernel-host command go test -tags integration -count=1 \
-run '^TestDockerKernelCheckOnThisHost$' ./internal/le/interoplab/
The report prints only after the last scenario, so Suite.Run also writes
progress lines while it runs: one as image preparation starts, one when the
images are ready with the time it took, and two for each scenario, interop: [i/n] name: running and then pass or FAIL with its elapsed time and error. They go to
Suite.Progress, or to stderr when that is nil, which is the job log in CI. A
suite that hangs then names the scenario that holds it. A failed progress write
does not fail the suite.
A failure names the phase it happened in, after the elapsed time:
interop: [i/n] name: FAIL (12s) in setup: <error>. The phase is setup
(pre-clean, network, prepare, validation, images, start wrapping, a container
run), readiness (a peer's declared readiness wait) or checker. Only a
checker failure says anything about the feature: a helper that cannot start
fails every scenario in setup or readiness. The scenario's result carries the
same word in its phase field, which a pass omits, and the terminal report
prints name: FAIL in <phase>: <error>. A pass line is unchanged, and the
: FAIL ( marker and a checker's scenario <name> assertion N: text are both
where the RFC recorder reads them. runScenario sets the phase as the run
advances and copies it onto a failed result, so no error text is matched.
When three scenarios in a row fail in setup or readiness, the suite stops. A
pass or a checker failure resets the count, so a run of feature reds always
runs to the end. The report lists every scenario after the stop under
not-run, counted neither passed nor failed, and stopped says why, with the
fix: correct the setup cause the failures name, then run the lab again with the
./le action its catalog registers (Catalog.Action). The summary line reads
interop: 0 passed, 5 failed, 2 not run: <failed names>.
Each local image build also creates a unique, run-owned tag. This keeps its
image ID available when another build replaces the shared cache tag.
The suite removes its tags after the scenarios, including after setup failure.
Image cleanup errors appear in cleanup-errors and make the suite fail.
Scenario cleanup errors also make the suite exit unsuccessfully. They appear in
progress, terminal reports, and the scenario's cleanup-errors field. Cleanup
does not change the checker verdict, passed/failed counts, or failed scenario names.
A failed scenario reads every container it started before the teardown
removes them, and its result carries one block per container in
diagnostics, which the terminal report prints under the verdict: the
state docker inspect reports (status, exit code, OOM kill, runtime
error), its container log, and, while the container still runs,
ip -d -o link show and ip -br address show from its namespace. A
daemon that logs to a file rather than its stdout (accel-ppp's [log] log-file, xl2tpd's redirected stderr, a pppd logfile) leaves the
container log empty, so the scenario that configures the file declares it
on its peer (PeerConfig.LogFiles, a topology node's logFiles), and the
block then carries tail -n 40 of each declared file. A checker's verdict names what it waited for; the blocks name why it never
came, such as a daemon that exited at start or a link device that is
missing. A read that fails is printed in place of its answer, and a read
whose program the image lacks (docker exec exit 127, as ip in the pulled
FreeRADIUS image) is printed as one not read: the image carries no <program>
line. The container
log is read whole (docker logs --tail all) and printed whole when it holds
at most 80 lines; a longer one is printed as its first 40 lines, the error
lines among the lines cut (a Ze level=ERROR record, the error: line ze
prints before it exits, a Go panic:; at most 20, with their count), and its
last 40 lines. A daemon that exits at startup prints its cause near the top
and a cascade of follow-on failures after it, so a tail alone can hide the
cause.
One ./le test integration <action> invocation returns that action's full report.
Multiple action names return a combined summary.
The image-retention regression, TestDockerBuildRetainsImageAcrossRetag,
carries the build tag docker, so ./le test integration docker-peers runs it
against the host Docker daemon with ZE_REQUIRE_DOCKER=1. Run it alone through
native admission:
ZE_REQUIRE_DOCKER=1 CGO_ENABLED=0 ./le --name image-retention job run label image-retention command \
go test -tags docker -count=1 ./internal/le/interoplab \
-run '^TestDockerBuildRetainsImageAcrossRetag$'
Neither it nor TestKilledLabRunIsCleanedByNextStart sets a bound of its own.
Each build takes the wrapper's machine bound, and the test's context ends ten
seconds before the deadline that go test -timeout sets. A one-minute build
bound once failed the retention test under docker-peers, which runs every
docker package in parallel.
internal/le/interoplab stays out of the QEMU guest population because of
TestDockerKernelCheckOnThisHost alone: it judges the kernel of the daemon the
host uses, which a guest cannot stand in for.
Daemons start conditionally: FRR if frr.conf exists, BIRD if bird.conf exists,
GoBGP if gobgp.toml exists. This means each scenario only runs the daemons it needs.
BIRD starts with IPv6 enabled, as FRR always does, only when rendering wrote a
lab IPv6 address into its bird.conf: the base prefix fd00:1e:0:: replaced by
the lab network's, or a link-local placeholder. The render records each file it
wrote one into (scenarioRendering), and the decision reads that record rather
than the rendered text, which names the lab's prefix and not the base one.
A ze-peer.conf starts a second Ze daemon, the peer ze-peer, where no
third-party daemon can serve the scenario (ospf-multiaf-v4-frr: FRR ospf6d has
no IPv4-unicast address family). It starts the way the Ze under test does
(zeDaemonPeer), with the CLI user the checkers log in as appended to its
config, and takes FRR's host 3, so a scenario carrying both frr.conf and
ze-peer.conf is refused.
A scenario directory is NAMED, never numbered (owner directive, 2026-08-24).
Run order is lexical by scenario directory name and no scenario depends on it.
The rule and its reasoning live in ai/rules/interop-and-goal-validation.md,
which is always-on, so a spec author planning a scenario meets it without
opening this page.
The exact grant-set check
Each lab's privilege test (TestL2TPLabPeersRunWithoutPrivilege and its
PPPoE, IPsec and RSVP-TE siblings, the deployment proofs' container tests and
the release-candidate container test) compares a container's whole grant set
with the set its daemon needs, through internal/le/interoplab/grantcheck. A
check that only looked for the wanted grants would pass a container that also
held --cap-add ALL or --privileged; comparing the whole set fails on any
extra.
grantcheck.Parse reads a docker run argv, or a lab peer's extra arguments,
in every spelling Docker reads (--flag value, --flag=value, a short flag
with its value joined or separate, combined short booleans). A flag that can
widen what the container may do fills one list: capabilities, devices,
security options, sysctls, host namespaces, networks, device cgroup rules,
mounts, or other grants (--user, --volumes-from, --runtime and the like).
--privileged counts in any spelling Docker reads as true. A flag classified
as no grant (--name, --env, --rm...) is skipped. Every other flag is
unclassified, and grantcheck.Problems reports each one whatever the wanted
set, so a flag nobody classified fails the check rather than passing unread:
classifying it is an edit to grantFlags or nonGrantFlags. A leading
docker is skipped, reading stops at -- or at the image, and an argv that
opens with any other word is refused, because reading would stop there before
a single flag.
grantcheck.ParsePeer adds a lab peer's declared capabilities, its AppArmor
profile and its mounts. A read-only mount of a file whose source, symlinks
resolved, lies strictly inside the checkout or the system temporary directory
(grantcheck.StagedRoots) is the lab staging its daemon's own config and is no
grant. A writable mount, a directory, a unix socket, a root itself, a source
whose symlinks do not resolve, or a source anywhere else is a mount grant the
want must name. Every lab mounts files only, and a directory is refused because
what it holds, a socket included, reaches the container unread.
Container Addresses
| Daemon | IP | Container |
|---|---|---|
| Ze | 172.30.0.2 | ze-iop-ze-<pid> |
| FRR | 172.30.0.3 | ze-iop-frr-<pid> |
| BIRD | 172.30.0.4 | ze-iop-bird-<pid> |
| GoBGP | 172.30.0.5 | ze-iop-gobgp-<pid> |
| Raw injector | 172.30.0.9 | ze-iop-inject-<pid> |
| Compiled strict speaker | 172.30.0.10 | ze-iop-speaker-<pid> |
| Compiled strict speaker (2nd) | 172.30.0.11 | ze-iop-speaker2-<pid> |
| StayRTR | 172.30.0.12 | ze-iop-stayrtr-<pid> |
pmacct pmbmpd |
172.30.0.13 | ze-iop-pmacct-<pid> |
| StayRTR standby cache | 172.30.0.14 | ze-iop-stayrtr-standby-<pid> |
Second FRR (frr-sink.conf) |
172.30.0.15 | ze-iop-frr-sink-<pid> |
Container names include the runner PID, which prevents name collisions. Runs that share a fixed subnet or staged binary paths must run serially.
The table gives the fixture addresses, which the runner rewrites onto its selected network. IPv6 scenarios also receive host addresses on a selected /64.
A scenario file may carry a link-local placeholder. Each one renders to an
fe80:: address whose interface identifier holds the first six bytes of the
selected IPv6 lab prefix and, in its last byte, the host number of the container
that owns it:
| Placeholder | Owner | Host |
|---|---|---|
@ZE_LINK_LOCAL@ |
ze |
2 |
@FRR_LINK_LOCAL@ |
frr |
3 |
@FRR_SINK_LINK_LOCAL@ |
frr-sink |
15 |
The placeholders, their hosts and the renderer are declared once, in
internal/test/labconfig (LinkLocals, RenderLinkLocals), outside
internal/le so the repository config sweep (TestRepositoryConfigsValidate)
renders a scenario's ze.conf exactly as the lab does before validating it.
The IPsec lab's %%PKI_B64:<file>%% placeholder renders there too
(RenderPKIPlaceholders, reading the scenario's pki directory or the shared
test/interop-ipsec/pki, FindPKIDir).
When any file of the scenario carries a placeholder, its owner's start is
wrapped: /bin/sh assigns the address to eth0 with nodad, then execs the
start it replaced (Ze's ze, or FRR's image entrypoint /sbin/tini with its
command -- /usr/lib/frr/docker-start). The daemon's first interface snapshot
therefore already holds the address. A placeholder also selects an IPv6 lab
network, because the address is built from its prefix. So does a ze.conf
that opens an address-family block inside its ospf block, in either
spelling the parser accepts (address-family ipv6 { ... } or
address-family { ipv6 { ... } }), because every family there is an OSPFv3
instance sending from an IPv6 link-local, and Docker disables IPv6 on eth0
of a network without it (zeRunsOSPFv3 reads block paths, so a comment
selects nothing). The scenario's name decides nothing.
A scenario's prestart is Go, not a scenario file: its register file adds an
ownerPrestarts entry, one shell fragment per link-local owner, to
scenarioPrestarts under the scenario's name. The interop tree carries no
interpreted helper (TestInteropTreeUsesOnlyCompiledHelpers), so the harness
compiled into le holds it. The fragment's link-local placeholders render as
they do in the scenario's files, and it runs inside that owner's wrapper, under
sh -e, after the scenario's links (below) and the eth0 assignment, and before
the image's own start. It is where a
scenario builds what must exist before its daemon sends a packet: a second
link, a link-local on it, an nft table. A prestart for an owner whose
placeholder no scenario file carries would never run, so preparation refuses
it by name rather than skipping it.
A scenario that needs an extra lab address on a peer's eth0 before the daemon
starts declares it in scenarioHostAddresses, as a peer and a host number.
addHostAddresses renders the address on the run's subnet and puts its
ip address add at the head of that peer's prestart, and refuses a declaration
that names a peer the scenario does not start. ldp-backoff-clear-frr declares
ze's .250 transport address there.
bgp-rfc2545-linklocal-nexthop-frr uses IPv6 host 2 for Ze and host 3 for FRR:
the speaker, its on-link global next hop and the recipient share ONE IPv6 subnet,
not separate IPv4 and IPv6 connected prefixes. Its ze.conf carries
@ZE_LINK_LOCAL@, so the advertised link-local next hop belongs to the same
router as the global address.
The checker requires FRR to decode the global-plus-link-local pair and install
the on-link route via that owned link-local address. A second prefix uses an
off-link global next hop and must decode as global-only; the IPv6 session must
remain established after both assertions. These are two routes on one session,
not identical UPDATEs differing only in their next-hop length.
bgp-rfc2545-third-party-linklocal-frr is the same sentence for a next hop that
names another router. A second FRR, frr-sink (IPv6 host 15), runs no BGP and
carries no link-local placeholder, so it owns exactly the one link-local its
kernel gave eth0. The checker reads that address from the container's
/proc/net/if_inet6 and its MAC from /sys/class/net/eth0/address, then pings
frr-sink's global and link-local from Ze's container, so Ze's kernel learns
both by Neighbor Discovery. It requires ip -j -6 neigh show dev eth0 in Ze's
container to bind both addresses to that MAC, with no second link-local on it:
any other table is a lab that cannot discriminate and fails there, not later.
It then runs send bgp * update text ... nhop <frr-sink's global> through
ze cli and requires FRR to decode frr-sink's global then frr-sink's
link-local, and to install the route through that link-local. Ze's ze.conf
sets no link-local leaf, so the second address can only come from Ze's
neighbor table. The config is BGP-only, with no interface block, so the
scenario also proves that Ze follows the table when no interface monitor
runs: frr-sink resolves after the session is up, and only the reactor's
neighbor table watch makes Ze read the table again. The announce is resent
each round of the 60 second wait, because the watch signal that refreshes
Ze's table read can race the first send; a resend is an implicit replace, and
a Ze that never adds the router's link-local fails every round.
Typed checker queries and their expected JSON field values are rendered onto the
same selected network. Rendering clones the expected-field map so concurrent or
later runs cannot change the registered scenario's addresses. In inject.msg,
literal UPDATE sends also receive the selected lab address in IPv4 NEXT_HOP
and MP_REACH next-hop fields, including the address after a VPN RD. The
renderer validates the envelope and attribute sequence before changing either
field. RDs, IPv6 addresses, NLRI and unrelated attributes remain intact.
Malformed framing and duplicate attribute codes remain unchanged,
preserving negative protocol fixtures rather than repairing them.
The IPsec, L2TP, PPPoE and RADIUS labs mount only Ze's input configuration file
read-only. Its parent /etc/ze belongs to the container and remains writable,
so start /etc/ze/ze.conf creates its database tree there without writing to
the scenario source directory. Each new container gets its own tree.
The deployment L2TP PPP proof uses a private /tmp directory owned by its
invoking user and writes Ze's explicit configuration beneath ze/. VPP stages
each input from the host scratch mount into /run/ze/<scenario-config>/ inside
the container, where its explicit ze.conf and database tree live together.
The root daemon cannot use a database directory beneath a mount owned by another
user. A scenario's disable-and-restart check recopies the new input into the
same private directory before each start.
After the native L2TP PPP proof has negotiated IPCP and carried traffic, it
injects an LCP Configure-Request from the peer namespace into the existing
kernel L2TP session. The packet uses that session's live tunnel/session IDs and
UDP endpoints. RFC 2661 section 3.1
says, “Tunnel ID in each message is that of the intended recipient, not the
sender,” and gives the same rule for Session ID.
RFC 1661 section 3.4
says, “The receipt of the LCP Configure-Request causes a return to the Link
Establishment phase from the Network-Layer Protocol phase or Authentication
phase.” Its section 4.1 specifies tld,scr,sca/8 for an acceptable request in
Opened. The independent xl2tpd/pppd peer then renegotiates LCP and IPCP; a
credentialed input must also complete a fresh authentication exchange.
ZE_L2TP_PPP_SCENARIO=no-auth is the default. Select chap-md5 to use the
existing local credential handler and require CHAP-MD5 on both negotiations.
Each namespace must have exactly one kernel tunnel and one nonzero-ID data
session. xl2tpd's kernel path also creates a tunnel-management PPPoL2TP socket:
Linux pppol2tp_connect
registers its local/peer session IDs as zero, without a PPP channel, and
l2tp_nl_cmd_session_dump
enumerates it alongside the data session. The proof validates this optional
management record separately: both session IDs must be zero, both tunnel IDs
must match the live tunnel, and no duplicate management record or second data
session is accepted. Management presence is part of the before/after identity;
its zero IDs never become injection targets.
The proof records the pppd log offset, Ze observation counts and both kernel
transport identities before injection. Only later request/Ack exchanges,
authentication when selected, route withdrawal/reinjection and address
assignment can satisfy the restart assertion. Both namespaces must retain
their original transport identities, recover their negotiated kernel addresses
and deliver all three interface-bound ping replies in each direction. Finally,
the peer leaves first and the proof requires another route withdrawal and
ordinary kernel cleanup. Injection uses Python 3's standard-library raw IPv4
socket and requires CAP_NET_RAW in addition to the namespace/PPP privileges.
The boundary JSON and peer log remain in the scratch directory on failure.
Raw tunnel/session listings are also printed before the restart identity check
returns, while both peers are still running. Later general diagnostics run
after deferred peer/daemon cleanup; their empty kernel state or pppd SIGTERM
must not be read as the state that caused an earlier assertion to fail.
The credentialed carrier then starts another xl2tpd/pppd peer with a wrong secret against the same daemon. It requires a matching CHAP Challenge/Response/Failure exchange, a local rejection, and session teardown. After the rejection boundary, neither the rejected peer's log nor new Ze observations can show network admission. PPP units, L2TP data sessions and subscriber routes must disappear before the proof stops the rejected peer. The control tunnel and its validated zero-ID management context may remain until that peer exits; they cannot stand in for a surviving PPP data session.
Separate links between daemons
Every container sits on the lab bridge's untagged eth0, so a protocol that
finds its neighbors by multicast (IS-IS, OSPF, PPPoE discovery) meets every
daemon of the scenario there. A scenario that must keep peers apart declares
links in the shared core, whatever suite it belongs to: ScenarioPlan.Links,
one interoplab.Link per link with its name, its VLAN, an optional IPv4
prefix of at most /24, an optional IPv6 /64, and its members, which are the
names of any planned peers, whatever image they run. A link with no prefix is
a pure layer-2 segment, which a PPPoE session or a LAN that DHCP numbers
needs. A member reaches its Nth link as eth<N>, N its ordinal among the
links it joins in declaration order, at its host number on each prefix (the
IPv6 address nodad), and the scenario's files name those devices and
addresses. A checker reads them from CheckContext.Member rather than from a
literal. Two fields cover an end the harness does not number. Devices gives
a member, on that link only, the device name a file the harness does not
write already names (a daemon configuration, a checker that takes the device
down); the member's other links keep eth<N>, numbered with no gap.
External names a planned peer that terminates the link in its own stack on
no device the harness builds, as freeRouter does: rawInt.bin hands every
frame of its container's eth0, tagged ones included, to the jar, whose own
configuration holds the VLAN subinterface and its address. The external end
joins no wrapper and holds no membership, so a link with one needs only one
member.
runScenario builds one start wrapper per peer that joins a link or carries a
PeerConfig.Prestart: the links first, then the prestart (the BGP suite's
eth0 link-local and its scenario prestarts are such prestarts), then the
image's own entrypoint and command, read once per image from the pinned
image (docker image inspect), in exec form or shell form. The daemon's first
hello therefore already goes out on its links. Before any container starts,
the scenario is refused when a link cannot be built as declared: a VLAN
outside 1 to 4094 or used twice, an IPv4 prefix longer than /24, an IPv6
prefix that is not a /64, fewer than two ends (an external end counts), a
member named twice, a member or external end the scenario does not plan, an
external end that is also a member, or a device name a member cannot hold
(one for a peer that is not a member, anything but 1 to 15 lowercase letters
and digits, eth<N>, lo, or a name the member already holds on another
link), because that link would carry nothing and
the checker would read a silent lab as the daemon's fault. Each refusal names
the link, the fault and the fix. A member's image is probed once for sh and
ip, and an image lacking either is refused naming its Dockerfile. A scenario
with no links runs every daemon on eth0 alone.
isis-import-policy-frr declares two links: FRR-A (frr) meets Ze on VLAN 101
(10.101.0.0/24, Ze's eth1) and FRR-B (frr-sink) on VLAN 102
(10.102.0.0/24, Ze's eth2). FRR-A originates the hidden prefix
10.77.1.0/24 and the control 10.77.2.0/24; Ze's isis filter import rejects
the first. The checker requires both adjacencies; the control in Ze's
show rib and kernel FIB and the hidden prefix absent from both, read beside
the control; both prefixes in Ze's show isis route, the SPF result the
chain runs after (show isis database detail prints TLV values as hex, so it
cannot be searched for a prefix); the JSON of
show rib candidates prefix <p> with the hidden prefix's one IS-IS path
hidden, hidden-by source, chain isis import, and the control's one
path selected and not hidden; and last, FRR-B's show ip route <p> json with
a selected IS-IS route for BOTH prefixes through Ze's 10.102.0.2. FRR-B's
only neighbor is Ze, so that last assertion is the proof that Ze still floods
the LSP whose prefix its own import chain hid.
ospf-import-policy-bird reads the same JSON per path: the tagged external
10.88.1.0/24 holds exactly an OSPF path at the distance 200 the modify set,
not selected, and a selected static path at 150; the untagged 10.88.2.0/24 a
selected OSPF path at 110 and a static path at 150. Every field is judged on
its own path, so the static path's 150 cannot stand for the OSPF path's
distance.
Each import scenario has an internal and a -forked system-RIB variant.
The shared wrapper checks the daemon PID, configured owner's Running state
and commands, then observes actual subprocess placement through procfs. The
forked case requires one descendant running ze plugin rib; the internal
case requires none. Both retain the original import, distance and flooding
assertions.
Both OSPF variants also require BIRD's untagged 10.88.3.0/24 in Ze's SPF
result and hidden centrally at distance 110. It must be absent from system
and kernel routes beside the installed 10.88.2.0/24 control. The IS-IS
wrapper reads both FRR versions and consumed image identities. The OSPF
wrapper reads BIRD's version and image identity. Checker unit tests exercise
these predicates, but do not establish a successful live peer run.
isis-import-policy-reload-frr and its -forked twin prove that a reload
re-judges the IS-IS paths the Loc-RIB already holds. One FRR meets Ze on
eth0 and originates the same two prefixes; ze.conf carries the same
rejecting chain and ze-reload.conf is that file without it. The checker
reads the hidden verdict as above, then a settled reading of Ze's
show isis neighbor (each adjacency up) and show isis database (each LSP's
level, ID, sequence number and checksum, lifetime left out), taken once three
reads 10 s apart agree. It reloads into ze-reload.conf and waits for the
hidden prefix's path to be selected and the prefix to reach show rib and the
kernel FIB, then reloads back into ze.conf and waits for it to be hidden
again at every layer. After each reload the settled reading must equal the
first. FRR's config never changes and a sequence number only grows, so that
equality proves no LSP was originated again and no adjacency went down: the
change came from the reload's re-judging (locrib.PublishPolicy), not from a
fresh advertisement. Both scenarios passed on 2026-11-14 and each went red that
day under one removed line of dispatchReselected, on a rebuilt Ze image: the
plain scenario at its relaxed reload with the re-admission dispatch removed
(the prefix never reached show rib), the forked one at its tightened reload
with the hide dispatch removed (the prefix stayed installed).
Run one scenario on Ze's Docker-host system with the native action below.
Select each of isis-import-policy-frr, isis-import-policy-frr-forked,
isis-import-policy-reload-frr, isis-import-policy-reload-frr-forked,
ospf-import-policy-bird and ospf-import-policy-bird-forked separately.
Retain isis-export-policy-frr and ospf-export-policy-frr as export controls.
./le test qemu docker-lab lab 'test integration interop' env 'INTEROP_SCENARIO=isis-import-policy-frr'
The action boots Ze's image, probes the guest Docker engine and runs the
selected lab through its host runner. When validating uncommitted checker
changes, use the launcher's --name form. Its inherited build identity keeps
the child launcher on the same current-tree build. A registered scenario is
a proof target until its live run and discrimination results are retained.
IGP export-policy scenarios
isis-export-policy-frr and ospf-export-policy-frr read an IGP export
chain's verdicts and sets out of FRR, one JSON answer per assertion, so every
absence sits beside the presences that prove the query saw Ze's routes.
In isis-export-policy-frr Ze redistributes three static routes into IS-IS
under isis filter export { filters [ loopbacks tagged-loopback-metric ]; final accept; }: a prefix-list accepting only /32s, then a modify that sets metric 77
on tag 300. FRR's show ip route isis json must hold 10.66.0.1/32 through Ze
at 87 (FRR's pinned link metric 10 plus 77), 10.66.0.2/32 at 110 (10 plus Ze's
default 100), and no 10.66.1.0/24.
In ospf-export-policy-frr Ze redistributes three static routes into OSPFv2
under ospf filter export { filters [ ospf-export-refused ospf-export-tagged ]; final accept; }. FRR's show ip ospf database external json must hold exactly
one LSA from Ze's router ID 10.255.0.2 for 10.67.1.0/24 at metric 50, E1, tag
200 (the modify's sets on the route tagged 100), one for 10.67.2.0/24 at the
defaults (metric 20, E2, tag 0), and none for the rejected 10.67.3.0/24. The
predicate collects every object carrying a linkStateId wherever FRR nests it,
in a bounded iterative walk.
Scenario Structure
Each scenario is a directory under test/interop/scenarios/:
scenarios/bgp-ebgp-ipv4-frr/
ze.conf # Ze configuration (required)
frr.conf # FRR configuration (starts FRR container)
Every directory has one typed checker in internal/le/interoplab/bgp. The
catalogue uses explicit operations for ordinary session, route, adjacency, log,
and negative assertions, plus bespoke checkers for scenarios whose control flow
cannot be represented as an ordered operation list.
Scenario-local register_*.go files can register bespoke checkers. The
conditional-MED checker runs its four original baseline operations unchanged,
then correlates collector events without adding an operation kind to the engine.
A bespoke checker is written in two halves. The body in check_rfc.go does the
lab I/O and numbers each assertion, so a failure names the assertion that found
it. The pure predicate in check_rfc_predicate.go takes the text or the decoded
JSON a peer daemon produced, holds no lab handle, and decides. The split is what
lets TestBespokeCheckerBranches drive both polarities of every predicate in
seconds with no container.
A body MUST NOT call checkScenario (check_engine.go). That function answers
scenario %s has no typed assertions for every name absent from
scenarioOperations, and a bespoke name is absent from that table by design, so
the call is an error that always fires and every line under it is unreachable.
Optional sidecars
A scenario directory may also carry files that start extra containers before Ze:
| File | Sidecar | Purpose |
|---|---|---|
inject.msg |
le test peer (raw injector, 172.30.0.9) |
Drive Ze with wire bytes no conforming daemon would emit. An optional inject-args file adds flags. Because the injector and Ze start before the peer daemons, an early route exercises Ze's replay-on-peer-up path. |
speaker-args (and optional speaker2-args) |
le test interop-bgp speaker (172.30.0.10; second at 172.30.0.11) |
Dial Ze with an independent strict peer. The compiled speaker negotiates the requested families and ADD-PATH mode, frames BGP itself, applies the named native oracle, and writes a structured verdict to container logs. It catches wire output that Ze's own lenient decoder could accept. |
vrps.json |
StayRTR (172.30.0.12:8282) | Serve RPKI VRPs from a real third-party cache, so Ze is the RTR client of an implementation that is not its own. The typed checker asserts each per-prefix validation answer, not merely the RTR session. |
vrps-standby.json |
A second StayRTR, stayrtr-standby (172.30.0.14:8282), same image |
Serve a different VRP set from a second cache, so the scenario can observe which cache Ze validates against as the preferred one stops and starts (rtr-stayrtr-preference). |
dnsmasq.conf (and optional dnsmasq-b.conf) |
dnsmasq DHCPv4 server (172.30.0.20; second at 172.30.0.21). test/interop/Dockerfile.dnsmasq |
Serve Ze's DHCPv4 client from an implementation Ze did not write. The checkers of dhcpv4-renew-nak-dnsmasq and dhcpv4-rebind-two-servers-dnsmasq count the DHCP messages in dnsmasq's log-dhcp output and read Ze's eth0 addresses. The renew-NAK checker writes a reservation into the container's own /run/dhcp-hosts and reloads it with SIGHUP; the rebind checker stops the server that granted the lease. |
knot.conf (with lab.example.zone) |
Knot DNS knot (172.30.0.40) and the LAN host dnsclient (172.30.0.42), both from dnsclient runs only sleep, holds NET_ADMIN for dhcpcd, and starts with net.ipv4.conf.default.promote_secondaries=1, because dhcpcd's Linux if_init writes that sysctl for eth0 unless it already reads 1 and /proc/sys is read-only in the container. test/interop/Dockerfile.knot |
Serve the lab zone lab.example. from an authoritative server Ze did not write, and query Ze's LAN DNS resolver from a separate host with Knot's kdig. The checkers (check_dns_resolver.go) parse kdig's default output (status, header flags, the answer RRset without TTLs) in check_dns_resolver_predicate.go. dns-resolver-dhcp-lan has dnsclient lease the one pool address 172.30.0.60 from Ze's DHCP server with dhcpcd, holds the dumped lease's domain_name_servers to exactly Ze's address, resolves through that server from the leased address, and requires REFUSED from the host's other address, outside allow-from |
registered for dhcpv4-server-shared-segment-dhcpcd (register_dhcp_shared_segment.go) |
The LAN host dhcpclient (172.30.0.43) from sleep, with NET_ADMIN and promote_secondaries=1 as dnsclient; Ze's prestart adds to its eth0 beside the lab address. test/interop/Dockerfile.knot10.99.0.1/24 |
Serve two subnets of one shared network on one interface. The checker (check_dhcp_shared_segment.go) creates macvlan clients with fixed MACs on dhcpclient's eth0 and binds each with dhcpcd: the client reserved in the household subnet binds 172.30.0.20 and the unreserved client binds from the household pool, each naming Ze's lab address as server identifier; the printer first leases from the household pool under a stray MAC, then takes its own MAC back and reboots its lease, and must log a DHCPNAK and bind its reservation 10.99.0.20 from server identifier 10.99.0.1. dhcpcd's dhcp_start1 reboots from the saved lease of the interface name, and read_lease does not compare the lease's chaddr, which is what lets the stray lease be rebooted under the printer's MAC |
unbound.conf |
Unbound unbound (172.30.0.41). test/interop/Dockerfile.unbound |
Put a real recursive resolver between Ze and Knot (stub-zone for lab.example.). dns-resolver-forward-unbound compares Ze's answer with Unbound's for an A RRset, a CNAME chain and a missing name; counts in Unbound's log-queries lines that Ze asked for www once across two client queries; requires the 30-record TXT RRset over TCP to equal Unbound's with Ze's upstream-queries truncated count at least one; and requires TC on Ze's own UDP reply to a 1232-octet buffer |
kea-dhcp6.conf |
Kea DHCPv6 server kea-dhcp6 (IPv6 host 22 of the lab /64; the file's selects an IPv6 lab network). fd00:1e:0::test/interop/Dockerfile.kea |
Serve Ze's DHCPv6 client from an implementation Ze did not write. The checker of dhcpv6-client-renew-kea reads Kea's INFO log, one exchange per distinct transaction id (DHCP6_PACKET_RECEIVED, DHCP6_LEASE_ALLOC, DHCP6_LEASE_RENEW, and DHCP6_USING_SERVERID at each start), and samples Ze's eth0 every 2 s: the leased /128 must be present with a finite valid_lft of at most the 40 s lease at every sample. Over three lease periods Kea must see one Solicit and one Request, between 10 and 16 RENEW exchanges that each extended the lease, and no Rebind: a client that never renews, renews without extending its deadlines (it loses the address after one lease and solicits again) or renews at every chance fails one of those bounds. The pool is one address (host 0x50), so the lease is known before the run. The image deletes Kea's persisted server DUID on every start, while the memfile lease database stays in the container: the checker stops and starts Kea to change the server's identity under a live lease, so Ze's next RENEW names a server that no longer exists and only its REBIND is answered. Ze must hold the address for two lease periods after the stop, with no Solicit, and then renew with the restarted server. The image runs Kea on a copy of the mounted file (written at every start), because SIGHUP re-reads the file Kea started with. A scenario that also carries kea-dhcp6-moved.conf mounts it at: the checker of dhcpv6-client-confirm-kea copies it over the live file and sends SIGHUP to move Kea to another subnet under a running lease (DHCP6_DYNAMIC_RECONFIGURATION_SUCCESS marks the reload). That scenario's lease is long (T1 300 s), so no Renew falls inside the run, and the kea-dhcp6.packets logger runs at debug level 55, where DHCP6_RESPONSE_DATA prints each Reply's status code option. Phase one sets Ze's eth0 down and up: Kea must log one CONFIRM after the bind and answer it Success, with no second Solicit, no Renew and no Rebind, and Ze must hold the /128 on every 2 s sample for 20 s. Phase two reloads Kea onto and sets eth0 down and up again: Kea must answer the CONFIRM NotOnLink, then see one Solicit and one Request that leases, and within 60 s Ze's eth0 must hold that /128 and no longer hold the old address. A client that ignores the NotOnLink keeps the old address until T1, far past that bound. /run/kea/kea-dhcp6.conf/etc/kea/kea-dhcp6-moved.conffd00:1e:0:1::/64fd00:1e:0:1::50 |
pmbmpd.conf |
pmacct pmbmpd (172.30.0.13:1790) |
Read Ze's BMP stream with a collector Ze did not write. The file is also the selector: a scenario that carries it starts pmacct INSTEAD of Ze's own collector, because two collectors are two readings of one stream and only the third-party one is interop evidence. The typed checker greps pmacct's JSON msglog, so every needle is a field pmacct printed after decoding. |
The ospf-te-frr and ospf-te-interas-frr scenarios keep their FRR adjacency,
TED and flooded-LSA assertions and add a BGP-LS speaker sidecar. Ze exports its
native OSPF database through bgp-ls-export; the collector records actual
negotiated SAFI 71 UPDATEs, not injected substitutes. The checker reconstructs
the final received inventory, including withdrawals, with the native UPDATE
and BGP-LS decoders: the wire types and TLV codes come from the core codec
internal/core/bgp/nlri/bgpls, and the attribute decode runs through the
plugin's registered bgp-nlri-ls decoder. It requires exact Node and bidirectional Link auxiliary
IPv4 Router-ID TLVs for ordinary TE, and local IPv4 plus remote IPv4/IPv6 IDs
for the configured inter-AS link. It rejects missing, swapped, duplicate or
invented local IPv6 IDs. The bounded capture must finish successfully and
establish a BGP session; a historical matching UPDATE alone cannot pass.
The inter-AS source assertion reads structured show ospf database opaque-as
output. It requires the configured remote AS and both ASBR IDs on one decoded
link, no Link ID, and a matching live Type-11, opaque-type-6 header. FRR's
AS-wide database must contain the same originator and LSA ID with matching
checksum and length. Neither a display label nor an address elsewhere in the
output proves that link. These source and flooding checks supplement, not
replace, the collector's final BGP-LS link and auxiliary Router-ID assertions.
Both FRR fixtures explicitly configure capability opaque, separately from
TE origination, so their DDs can negotiate the opaque LSAs needed by this
proof. The native neighbor regression also keeps an O-clear peer: the
production summary filter must still establish Full without advertising
opaque headers to that peer. A capable interop fixture cannot replace that
negative control or excuse an ExStart/SeqNumberMismatch loop.
The capture deadline bounds reads as well as the outer loop. Expiry while
waiting for a new header is quiet completion; an unfinished header or body,
EOF, notification or send failure cannot certify a complete capture.
The same --duration deadline bounds the dial: while Ze refuses the connection
or is unreachable, the speaker redials every 250 ms, and when the deadline
passes it fails with the last dial error.
If either OSPF scenario fails, its checker captures both peers' interface and neighbor detail, Ze's packet/drop and NSM counters, and FRR's DD packet debug log before container teardown. The original Full-adjacency failure remains the result; diagnostic query failures are retained alongside it.
The RFC 7606 mixed-UPDATE scenario first observes 198.51.100.0/24 in
FRR's BGP table and installed routing table. Only then does the checker send
the injector's explicit release route. The injector withdraws that seed and
announces 203.0.113.0/24 in one UPDATE, followed by a same-session fence
route. FRR must remove the seed and install the announcement and fence through
the injector's next hop. A byte-transparent relay independently checks the
split withdrawal and announcement through that fence, including exactly one
NEXT_HOP. FRR 10.7.1 discards a later duplicate attribute in bgp_attr_parse,
so FRR installation alone cannot prove that sender-side wire property.
The typed-NLRI discard scenario requires the full requested speaker capture
on a valid OPEN and Established session. Its independent wire assertion
strictly decodes every reported UPDATE and requires exactly the injected
type-2 MAC/IP Advertisement: RD 172.30.0.9:1, zero ESI and Ethernet tag,
MAC 00:11:22:33:44:55, no IP address, and label 2. A different assigned
type, altered retained route, withdrawal, malformed envelope, early EOF or
missing OPEN cannot stand for successful discard of the unassigned type 99.
no-family-peer-eor-frr intentionally uses asymmetric OPENs: FRR advertises
IPv4-unicast Multiprotocol capability, while Ze, configured with no family
block, does not. The checker independently requires route arrival and that
capability DTO, then matches FRR's complete native receive payload for Ze's
exact address, IPv4 Unicast and VRF default. The default peer declaration is
FRR 10.7.1 at commit
f8c0b08dcb0c78f9e42b9b86ae70b049e4e617c1:
bgp_update_receive
recognizes the empty non-MP UPDATE and
bgp_update_receive_eor
emits the message. A table entry is not an EOR substitute.
The scenario's bgpd_options enables file logging before integrated
configuration, so frr_run's native startup precedes the initial receive
window. Only a decode after the sole startup is eligible, and that startup
must name release 10.7.1, bare or with FRR's build suffix after an underscore:
the lab image prints bgpd 10.7.1_git starting: . Another release, or a
longer one such as 10.7.10, does not fence the log. The checker brackets
the log read with bgpd PID and /proc start ticks, log device/inode, exact
neighbor socket ports, Established state, one connection establishment and
zero drops. Foreign peers, address substrings, other families/VRFs, send
messages, pre-startup events and daemon/session/log replacement cannot certify
this observation. An identity query the wait's end cuts short, before or
after the log read, fails that probe and preserves the last complete
observation for the EOR wait's verdict; an observed identity replacement
still refuses immediately. The final Established check and identity
comparison retain the original-session survival assertion.
These are checker/source contracts, not a live result. Current native GREEN, the actual Ze EOR-source cut producing RED despite route arrival, and restored GREEN remain separately recorded runtime obligations. They do not recover or replace missing historical originals.
A scenario that needs a test seam no shipped build carries holds a
ze-le-daemon file. Its presence is the declaration: the ze container then
starts /usr/local/lib/ze-le/ze, the daemon built with ze_le, under tini,
instead of the shipped ze every other scenario runs, so no other verdict
comes from a binary that differs from what ships. The checker's ze CLI
commands still run the shipped ze. The file cannot be combined with
ze-reload.conf, whose start runs the shipped ze through a shell, and the
preparation refuses a scenario holding both (scenarioPeers).
bgp-panic-cease is the one user: the ze_le reactor
(session_panic_inject_le.go) panics on the next UPDATE from a peer once
/run/ze-inject/bgp-read-panic-<peer address> exists, and removes the file
first, so one trigger raises one panic. The checker arms it for FRR and has FRR
redistribute a static route. It then requires that FRR's own neighbor JSON
records a RECEIVED Cease with subcode 0 (lastResetDueTo,
lastNotificationReason), that FRR counts two establishments and one drop,
and that ze still counts one establishment and no drop toward BIRD, which
still reports Established. In a shipped build injectReadPanic is an empty
function (session_panic_inject.go, !ze_le).
bgp-manual-eor-requested-frr proves which End-of-RIB markers Ze sends a peer
configured behavior { manual-eor true; }, against FRR 10.7.1. Its external
process (le test interop-bgp process bgp-manual-eor-requested-frr manual-eor-requester) acts from inside its handler for the peer-UP event,
which the engine awaits before it starts the initial sync (deliverPeerState,
internal/component/bgp/server/events.go), and answers within 4 s, under the
engine's 5 s delivery timeout. On the first session it announces
10.84.1.0/24 and asks for nothing: FRR must decode the route and, through a
10 s hold, no End-of-RIB. The checker then runs clear bgp <ze>. On the
second session the process announces 10.84.2.0/24 and asks for the IPv4
unicast End-of-RIB with the session's queueing gate still closed: in the part
of FRR's log written after the clear, FRR must decode the route and then
exactly one End-of-RIB, held for 10 s, and its table must hold the route.
The checker reads FRR's own receive log rather than show bgp neighbor json.
FRR's bgp_update_receive_eor (bgpd/bgp_packet.c at frr-10.7.1) logs
rcvd End-of-RIB for IPv4 Unicast from <peer> in vrf default at
informational level for every marker, before it tests
PEER_STATUS_EOR_RECEIVED, so a second marker is logged too. bgp_update
logs <peer> rcvd <prefix> under debug bgp updates in, and the order of the
lines is the order FRR handled one connection's UPDATEs. The
gracefulRestartInfo.endOfRibRecv flag is one bit, printed only for a peer
whose Graceful Restart capability FRR received, so it can show neither a
count nor an order, and the scenario negotiates no Graceful Restart.
It proves two things: an explicit request asked inside the sync is written,
after the route, and the sync writes no marker on a manual-eor peer. It does
not exercise a route-server replay's marker, because no replaying plugin is
attached.
A scenario carrying frr.conf may also carry its own daemons file. That file
names which FRR daemons run and what each one is started with, so a scenario
needing a bgpd module (-M bmp for one that drives Ze's BMP receiver) carries
its own copy instead of adding the module to every scenario in the suite. Without
one, the shared test/interop/daemons is mounted.
It may also carry an frr-image file holding one image reference. Its FRR
containers then start from that release instead of the suite's FRR (and
FRR_IMAGE does not replace it), and the suite pulls each pinned release once.
A scenario needs this when the behavior under test exists only in a later FRR
than the suite's, or when the scenario must not follow an FRR_IMAGE override:
bgp-linklocal-only-multihop-withdraw-frr needs FRR 10.4.1 or later, the first
release here that negotiates the Link-Local Next Hop capability (77), because
without it a different gate withholds the route and the scenario cannot go red.
The five scenarios that carry one pin the same release as the suite,
interoplab.FRRImage (every lab peer runs its latest stable release), so a run with FRR_IMAGE naming an
older FRR still starts them from 10.7.1. A file naming no image, or two, is an
error.
A scenario may carry a ze-sysctls file: whitespace-separated key=value
assignments, each passed as --sysctl to Ze's container, so the runtime writes it
into the container's own network namespace before ze starts. Docker mounts
/proc/sys read-only, so a value ze would otherwise write itself is refused
there. ospf-sr-anycast-nexthop-frr and ospf-sr-frr use it for
net.mpls.platform_labels, which fib-kernel needs non-zero before it can
install a label route. Only net.*
keys are accepted; any other key, a token without = or an empty value refuses
the scenario before a container starts.
bgp-linklocal-only-multihop-withdraw-frr exercises the off-connected-subnet
gate on one lab network: FRR's session address is on its loopback (10.255.0.3),
outside every subnet Ze is attached to, and the checker gives Ze a host route
via FRR's adjacent interface. There is no intervening router. This proves the
gate and FRR's received withdrawal, not a physically multihop path.
test/plugin/linklocal-only-multihop-withdraw.ci supplies the separate routed
IPv6-hop proof through a transit namespace. Ze's link scope reads the interface
table, not the configured BGP TTL.
A scenario whose ze.conf runs the BMP sender (bgp bmp sender) and carries
no pmbmpd.conf starts le test interop-bgp bmp-collector. The lab reads that
from the parsed config, so a commented-out bmp block starts no collector, a
receiver-only environment bmp starts none either, and a ze.conf that cannot
be read or parsed fails the scenario's preparation, naming the file. Announcement and
observer process plugins use le test interop-bgp process <scenario> <plugin>.
These personalities are compiled into le; no interpreter or source mount
is present in the Ze image.
The pinned StayRTR v0.6.4 image runs with -protocol 1 -enforce.version=true.
Its default connection state advertises version 0 when rejecting Ze's initial
version-2 query. Enforcement sets the connection version to 1 before that
error, so Ze can reconnect at the advertised supported version.
-refresh 10 is how often StayRTR re-reads its VRP file. The RTR timers it
sends Ze in End of Data are -rtr.refresh 10 and -rtr.retry 10 (StayRTR's
defaults are 3600 and 600 seconds), so Ze polls a cache every 10 seconds and
asks a failed cache again after 10.
Those 10 seconds apply only after a cache's first End of Data. Before it, RFC
8210 Section 6 has Ze retry a cache it never reached after the default Retry
Interval of 600 seconds, as BIRD and rtrlib do, so a first query that fails
costs that cache ten minutes, longer than any checker waits. Two lab defects
made that first query fail. The image patches the first: unpatched v0.6.4
refuses Ze's version-2 query by queueing its Error Report and cancelling the
connection at once, its send loop picks between the two at random, and Ze
sometimes reads a bare close ("read header: EOF") with no version to retry
at. The readiness probe (stayRTRReadyCommand) handles the second: StayRTR
answers /rpki.json with status 200 and "roas":null before it reads its VRP
file and binds the RTR port, so the probe also requires a prefix in the export
and an open RTR port.
The rtr-stayrtr checker then requires two IPv4 and two IPv6 VRPs and checks
the per-prefix validation answers. HTTP readiness alone proves no RTR transfer.
A changed cache command requires an image rebuild; NO_BUILD=1 retains the
old image. For a current result, rebuild the image and run
INTEROP_SCENARIO=rtr-stayrtr ./le test integration interop. Read that run's VRP
and validation checks; an archived result does not establish the current outcome.
Live RPKI policy reload
The rtr-stayrtr scenario also runs FRR as a source and BIRD as an export sink.
It changes Invalid policy from reject to accept, then restores reject through
SIGHUP. Each reload must advance the applied configuration generation once.
The same Invalid route must change eligibility, selection, and BIRD export
without a source UPDATE or route refresh. A Valid control route remains usable.
All phases require the same complete four-VRP set from StayRTR.
Continuity uses FRR's connectionsEstablished counter while its peer is
Established, plus Ze's connection, OPEN, UPDATE, and refresh counters.
The FRR counter identifies a session within the configured peer object's
lifetime, not across daemon restart or peer recreation. The checker does not
use FRR's derived wall-clock epoch, which can change without a reconnect.
BIRD's session identity, TCP endpoints, and Ze-side lifetime counters must also
remain unchanged. RTR reconnection is permitted; route reannouncement is not.
This carrier does not test disabling validation or rejecting a reload transaction.
RTR cache preference, failover and fail-back
The rtr-stayrtr-preference scenario gives Ze two StayRTR caches serving
different VRP sets: the preferred cache at 172.30.0.12 (preference 10, from
vrps.json) and a standby at 172.30.0.14 (preference 20, from
vrps-standby.json). ze.conf lists the standby first, so a client that used
the configuration order instead of the preference would sync the standby. FRR
announces 9.59.0.0/24 and 10.59.0.0/16 from AS 65001, and Invalid routes are
rejected: 9.59.0.0/24 is Valid only under the preferred set, and the standby
set makes it Invalid. 10.59.0.0/16 is Valid under both and is the control.
The checker waits through three phases, each bounded at 90 seconds: the
preferred cache in use, then the standby in use after it stops the preferred
container, then the preferred cache again after it starts it. A phase holds
when show bgp rpki status lists both caches with exactly one synced, the
expected one; when show bgp rpki roa equals exactly that cache's set, so a
union of both sets fails; and when BIRD holds 10.59.0.0/16 with path
65000 65001 and holds 9.59.0.0/24 under the preferred cache, or does not
hold it under the standby. The predicate's accepted and refused shapes are
unit-tested in TestRPKIPreferencePhaseError.
Results on 2026-10-27, INTEROP_SCENARIO=rtr-stayrtr-preference ./le test integration interop, each break built with GOFLAGS=-overlay and the image
rebuilt by the harness:
| Build | Result |
|---|---|
| unmodified, before the StayRTR patch and readiness probe | 3 passes, 2 failures in phase 1. Cause: StayRTR v0.6.4 sometimes dropped the Error Report that refuses Ze's version 2 query, so Ze read a bare close and waited the default 600-second Retry Interval, and the readiness probe answered before StayRTR served RTR (the StayRTR paragraph above) |
| unmodified, with the patched StayRTR image | 10 passes in 10 runs, two loops in parallel |
the preference sort reversed in startSessions |
red in phase 1: "show bgp rpki status names 172.30.0.14 as the cache in use, want 172.30.0.12". Rerun with the patched image: the same red, and Ze's log holds no transport error ("read header: EOF") and no refusal, so the red is the sort's |
| the preference sort made a no-op, before the config-order fix | pass, because the cache list then came from ranging over the config's cache-server map, in random order. parseRPKIConfig now reads the operator's order, so the no-op break would follow ze.conf, which lists the standby first |
cacheGroup.load refusing any cache more preferred than the holder (never fails back) |
phases 1 and 2 pass, red in phase 3: "fail-back to the preferred cache ... names 172.30.0.14 as the cache in use, want 172.30.0.12" |
Live BFD profile edit
The bgp-bfd-profile-edit-live-frr scenario is the bfd-frr lab plus a
reload. Ze's BGP peer opts into BFD under profile fast at 300ms and detect
multiplier 3; ze-reload.conf edits that profile to 200ms and multiplier 5 and
changes nothing else. Once FRR's BFD session is up at 300ms, the checker reloads
Ze through SIGHUP (applyZeReload, the same reload rtr-stayrtr drives) while
that session is Up. FRR must then report remote-transmit-interval 200,
remote-receive-interval 200 and remote-detect-multiplier 5 in
show bfd peer <ze> json, which it reads from Ze's packets. RFC 5880 Section
6.8.3 makes that change a Poll Sequence on an Up session.
Continuity compares four values read before and after the reload. Both BFD
discriminators (id, and remote-id, Ze's discriminator) must stay the same,
so the session was not rebuilt. The session-down counter of
show bfd peer <ze> counters json must not move, so no flap went unseen
between two samples. FRR's BGP connectionsEstablished must not move either.
FRR's own intervals stay at the bfdd defaults throughout, so the remote fields
are the only ones the reload can change.
BFD over link-local
bfd-link-local-frr runs no client protocol. Ze's bfd root pins one
single-hop session to FRR's link-local on eth0, with Ze's own link-local as
the source; FRR's bfd block names
peer <Ze's link-local> local-address <FRR's link-local> interface eth0.
Without local-address, bfdd 10.3.1 sources its packets from eth0's global
address, so neither side matches the other's packets to its session and FRR
stays down with remote-id 0.
Both files carry both placeholders, so both containers own their address
before the daemons start. The checker waits for FRR's
show bfd peer <Ze's link-local> interface eth0 json to read up, then for
Ze's show bfd sessions to hold an up session to FRR's address on eth0,
and requires the discriminators to have crossed: FRR's remote-id equals Ze's
local-discriminator, and Ze's remote-discriminator equals FRR's id.
Each value exists only on the side that chose it until a packet carries it
over, so neither clause holds on one direction of traffic.
The scenario is acceptance evidence. RFC 5881 Section 3 makes both sides
Active, so once FRR answers, the session also converges through Your
Discriminator, and the zone Ze reconciles on a first packet is not what the
scenario can tell apart. The send side cannot fail here either: every session
of the loop names eth0, so the loop's socket is bound to that device
(resolveLoopDevices) and the kernel puts the packet on eth0 with no zone
and no ifindex. Both reverts still passed the harness run on the Linux host
(2026-10-20). The discrimination is
test/bfd/bfd-first-packet-link-local.ci in the QEMU guest, and
bfd-link-local-two-links-frr against FRR.
bfd-link-local-two-links-frr puts two links between Ze and FRR and ONE
address pair on both. Each container's prestart builds eth1 (VLAN 101) and
eth2 (VLAN 102) over eth0, and gives the container's link-local to both, so
the address cannot tell the two sessions apart and only the link can. Ze names
one session per link; FRR one peer per link. Two things make it discriminate:
- Send. The sessions name two interfaces, so the loop's socket is bound to
no device, and a packet reaches its link only through the zone
sendZoneputs on the destination or the ifindexpinsToLinkasks for. Without either, the kernel refuses the write (sendmsg: invalid argument) and FRR hears nothing. - Receive. Ze's prestart installs an nft output rule in table
inet bfdfirstthat drops every BFD Control packet Ze sends with Your Discriminator zero. FRR then learns Ze's discriminator only from a packet Ze sends after selecting FRR's first packet by address and link, so without the zone reconciliation (reconcileZone) neither side leaves Down. Ze logs each dropped send asbfd send failed ... operation not permitted, which is the rule working.
The checker reads the links from Ze's own show bfd sessions (the interfaces of
its sessions to FRR's link-local, at least two), then runs the one-link check
on each: FRR's peer on that link Up, Ze's session on that link Up, and the
discriminators crossed. On the Linux host (2026-10-20) the harness run passed
with the code as it is and failed at assertion 2, FRR's peer on eth1 still
down with remote-id 0, under each break: reconcileZone returning its key
unchanged, and sendZone returning "" with pinsToLink returning false
(Ze then logs sendmsg: invalid argument for every send).
Live attach and BFD opt-in without a session reset
Three scenarios change a running FRR peer's Ze config by SIGHUP and require
FRR's BGP connectionsEstablished to stay where it was before the first reload,
through every reload except the BFD opt-in removal.
| Scenario | Reloads | What FRR must show |
|---|---|---|
bgp-attach-receive-only-live-frr |
ze-reload.conf attaches adj-rib-in to the peer, receive-only |
exactly one more messageStats.routeRefreshRecv (one negotiated family, RFC 2918 Section 4), and Ze's show bgp adj-rib-in then holds the route FRR announced before the reload |
bgp-bfd-enable-live-frr |
ze-reload.conf adds the peer's non-strict BFD opt-in, ze-reload-edit.conf edits its profile to 200ms and multiplier 5, ze.conf removes the opt-in |
BFD Up at 300ms after the add; the same discriminators and no session-down at 200ms after the edit; BFD Down with remote-diagnostic administratively down after the removal |
bgp-bfd-dynamic-enable-live-frr |
ze-reload.conf adds the opt-in to the template of the group FRR joins as a dynamic peer, ze.conf removes it |
BFD Up at 300ms after the add, Down with remote-diagnostic administratively down after the removal |
The attach checker first requires that adj-rib-in holds nothing for FRR, so
the route it finds afterwards can only have come from the catch-up refresh.
The BFD configs keep the bfd root in every step, because an opt-in names a
profile of the bfd plugin.
The removal step asserts what Ze sends and stops there. Ze releases the BFD session with AdminDown, diagnostic 7 (RFC 5880 Section 6.8.16), and FRR 10.3.1 then resets BGP with Cease/BFD Down, although RFC 5882 Section 4.2 says "clients SHOULD NOT take any control protocol action" on a remote AdminDown. That reset is a recorded peer deviation, not a Ze failure, so the checker does not require BGP to survive the removal.
A scenario that reloads more than once names each further config
ze-reload-<step>.conf. The harness mounts every such file in the ze container
at /etc/ze/<file name>, beside ze-reload.conf at /etc/ze/reload.conf, and
ze.conf stays at /etc/ze/bgp.conf, so a checker reloads back to the start
with applyZeReload on that path.
RPKI route retention
INTEROP_SCENARIO=rpki-frr ./le test integration interop runs FRR as the BGP
source for 9.43.0.0/24, 10.43.0.0/24, and 11.43.0.0/24. The checker
requires an established session and all source routes in FRR. Ze must retain
all received routes in Adj-RIB-In: Valid (1) and NotFound (2) are eligible;
Invalid (3) has ineligible: true. Missing routes or missing eligibility
fields fail the observation. The process observer and native checker use the
same structural predicate and save the observed route objects in
/tmp/rpki-check.json inside the Ze container.
This scenario exercises FRR-to-Ze BGP reception with RPKI eligibility decisions.
It uses the same-repository le mock rpki RTR cache mock and does not establish
independent RTR interoperability. The rtr-stayrtr scenario uses StayRTR for
that purpose. The retention observation does not measure downstream export:
FRR originates these routes, and there is no separate receiving peer.
The functional rpki-revalidate-late-sync carrier uses an observer-controlled
RTR barrier. The mock waits for rpki-sync.release in the test's private working
directory. Only after the observer reads a retained, eligible NotFound route and
an empty VRP set does it create that file. The observer then requires the same
received attribute, next-hop, and NLRI bytes with Invalid/ineligible state.
Scheduling delays cannot let the cache synchronize before the initial observation.
The same private-directory barrier controls the RR and RS
bgp-{rr,rs}-rpki-pending-withdraw-relay and
bgp-{rr,rs}-rpki-revalidate-relay functional carriers. Each has its own wire
expectation: pending-withdraw cases permit only the revalidated fence route;
revalidation cases require the route on the destination's second connection,
after its End of RIB. The RR expectation includes reflection attributes and the
RS expectation has its own AS path. A successful RTR transfer alone proves none
of these downstream outcomes. Each carrier needs its own live run and a
discriminating route-behavior break, separately from the shared cwd break.
Prove a scenario discriminates
An interop scenario is evidence only if it goes RED when the behaviour it tests is broken. Before you rely on a new scenario, revert the fix, run the scenario, and confirm that it fails. Then restore the fix and confirm that it passes.
Let the harness build. Do NOT docker build -t ze-interop by hand and then run
with NO_BUILD=1. A tag is shared by every run on the host. A build in another
session rebinds that tag between yours and your container start. Your mutation run
then measures a daemon you did not build, and that inverted a proof twice in one
review on 2026-08-05. Quiet Docker.Build calls read the image ID from
docker build -q. A caller supplying ImageBuild.Progress streams Docker's
stdout and stderr, then obtains the immutable ID by inspecting its unique
run-owned tag, never the shared cache tag. The suite pins every container of
the run to that immutable ID. Quote that ID beside the result, because it
names the binary the run measured.
The suite prints interop: image <name>: reference <ID> after each successful
local build, before a peer starts. The default progress stream keeps that ID
beside the scenario verdict in native job logs. Pulled images and NO_BUILD=1
do not print this build identity.
The GoBGP export-reload successor also records the actual image ID of each
registered live peer container before reading versions or configuration. Its
proof.json carries image-<peer> identities, including pulled peers whose
suite build result still names a tag. CheckerLab.PeerImage, which the real
Lab answers, inspects that container's
consumed image and refuses missing or malformed IDs; it never substitutes the
current shared tag. This records the current run's provenance and does not
recover an earlier run's missing image identity.
The same identification step requires the applied patch bytes inside that image
to equal this checkout's patch, and validates its named SHA256 before reading
configuration files. Neither an old image nor a version string supplies the
patched producer's identity.
A scenario that passes either way (common when the peer must accept both the old and the new wire form) proves acceptance, not correctness. Say which one it proves in the spec's Goal Validation, and move the discrimination to a unit or mutation test that CAN fail.
A scenario added to ALREADY-WORKING code never had a red phase, so its discrimination is unproven until you force one. That is not TDD's red-then-green: a regression test and a scenario for existing behaviour both start green.
Six traps make a scenario pass whatever the code does. Check each by its tell before you call the scenario evidence:
| Vacuity trap | Why it passes anyway | The tell |
|---|---|---|
| A scenario for a sender-side wire change whose receiver is obliged to accept any form (RFC 7606 Section 5.1: receivers accept any field combination) | A conforming peer accepts the old and the new wire equally | Reverting the sender change leaves the peer's routing table identical |
| A test asserting the ABSENCE of something (no log line, no allocation, no route) | Deleting the mechanism leaves the same absence | Ask what would still be absent if the code were removed |
| A test whose fixture is at an extreme (all fields set, maximum value) | An off-by-one or a partial break still handles the extreme | Boundary the fixture: test one below and one above |
| A test whose data reaches the peer by a DIFFERENT path than the one changed | The unchanged path still delivers | Trace which code path actually produces the asserted bytes. A scenario proving two peers are NOT merged is this trap when they are already kept apart by another fact: local-as-replace-as-partition passed with the group key's prepend zeroed until BIRD was made to advertise the RFC 8654 capability FRR advertises by default, because the two sessions differed in announceFacts.extended. Log each peer's key in the red build |
| An assertion whose clauses are all satisfied by ONE stimulus | Each clause reads as an independent observation, and they are one observation written twice | Name the single event that satisfies every clause. Then ask which clause a peer that did nothing would still satisfy |
| A counterfactual that fails at a step BEFORE the assertion it is meant to prove | The run is red, so the proof looks discriminating, yet the assertion under test never ran | Read which step the red run names. The Compose proof's break-volume failed at import, so its second-login checks had never been seen red until break-commit failed at second-login. internal/le/test/deployment/compose.go |
The fifth trap is what the IPsec suite carried until 2026-09-04, and it is worth
reading in full because the shape recurs. verifyTunnelTraffic
(internal/le/interoplab/ipsec/helpers.go) pinged from Ze to strongSwan and
passed when each peer's ESP byte counters advanced. Both clauses were satisfied by
that ONE ping: Ze encrypting the echo request advances Ze's outbound SA, and
strongSwan decrypting the same request advances strongSwan's inbound SA. Nothing
required strongSwan to have encrypted anything toward Ze.
RFC 4301 Section 4.1 says why the aggregate could not discriminate:
An SA is a simplex "connection" that affords security services to the traffic carried by it.
A protected bidirectional flow is two SAs, and the RECEIVER chooses the SPI, so
both peers name one direction by the same SPI value. Measured in
psk-site-to-site: Ze holds src 172.28.0.2 dst 172.28.0.3 spi 0xc12fa7e3 and
src 172.28.0.3 dst 172.28.0.2 spi 0xf008af63, and strongSwan holds those same
two SPIs. A counter map keyed by SPI alone therefore folds the two peers' views of
one direction into one entry. verifyESPDirections now reads each simplex SA by
its own src/dst header and takes the set of directions its caller can claim,
and it also refuses a ping whose % packet loss summary is missing or non-zero.
Which layer proves what
Each layer answers one question, and a question goes to the cheapest layer that can decide it.
| Layer | What it proves | Where |
|---|---|---|
| Producer unit test | Ze's product behaviour: the encoder, the FSM, the filter that makes the bytes | The producing package's _test.go |
| Replay of a recording | The checker's logic: what it concludes from the answers a real lab run gave it | interoplab.ReplayScenario, from a test in the lab's package |
| One lab pass per scenario | The whole path end to end, against the real peer daemon | ./le test integration <lab> |
| Start probe | A staged helper starts and answers ("The start probe" above) | The lab's preflight |
A lab pass that is red, and a checker change meant to turn it green, are not
judged by rerunning the lab until it passes. Record the run, then judge the
checker over the recording. A change to a checker whose scenario has a
committed recording carries a replay test over it
(ai/rules/points/testing/directives/replay-a-checker-change-over-its-recording.md).
Replay never replaces the lab pass for product behaviour: a scenario still owes
one green lab run end to end.
Recording and replaying a checker
Suite.Run hands every checker a recorder in front of the real Lab. The
recorder forwards each CheckerLab call and writes it, in call order, as one
JSON line to <session scratch>/interop-recordings/<lab>/<scenario>-<run id>.jsonl.
The first line is a header: the format version, the lab, the scenario, and the
network, links and memberships the CheckContext carried. Each call line holds
the method, the peer, the argv, the NAMES of the environment variables the
checker passed (never their values), the answer, the error text with its typed
kind, and the exit code. The last line is the lab's verdict. The lab is the
registered catalog whose scenario root holds the scenario, and the run id is
the Docker client's.
The recorder and the replay lab implement CheckerLab and nothing more, so
every lab capability a checker uses is a CheckerLab method, PeerImage
included (recorded as peer-image, the image ID as its answer). A checker
never type-asserts its Lab for a method outside that interface: the recorder
in front of the real Lab does not carry it, and the checker would lose the
capability in every lab run.
Each answer field is bounded at 1 MiB (recordingAnswerOctetsMax). A longer
answer keeps its first MiB and is marked truncated, and an answer that is not
UTF-8 text is marked not-text, because JSON cannot carry its octets unchanged.
A full run, with no selector, keeps only the failed scenarios' recordings. A
run that selects fewer scenarios than its catalog holds keeps every one. A
failed scenario names its recording in its diagnostics
(interop: <scenario>: checker recording: <path>), and a kept passing
recording is named on the progress stream. A run that cannot resolve a
recording directory (no checkout, no session, no registered catalog) still
runs, and says once interop: checker recordings are off: <reason>.
interoplab.ReplayScenario(ctx, checkout, path) opens a recording, finds the
scenario's checker through its lab's registered catalog, and runs the checker
against a lab that answers each call from the recording, in order, matched on
the method, the peer, the argv, the environment names and the call's own
parameter. It answers the lab's verdict beside the replayed one. It refuses,
with a *ReplayRefusal naming the expected and the actual call, a call the
recording does not hold at that position, a call past its end, a truncated or
not-text answer, a recording of another version or with no verdict line, and
a checker that returns before the recording's last call. The refusal says the
checker must be judged by a lab run, and names the command that runs the
scenario. A replay never guesses an answer, so a checker never passes on data
the lab did not produce.
What replay can and cannot reproduce:
- A checker that reaches the lab only through
CheckerLabreplays whole. The BGP relay checkers read their capture throughCheckerLab.Query, so the recording already holds the bytes they judged. A checker that reads a file from its scenario directory or the checkout reads it again from the checkout. - A poll (
interoplab.Wait) that ended on a ready answer replays the same probes in the same order, and each unready probe still waits its real interval. A poll that ended on its deadline made as many probes as its timeout allowed in the lab; replay answers at once, may start one more, and then refuses that call, or, when the checker's next call has the same argv, answers it with that call's answer. Judge a checker whose recording ends in a timed-out poll by a lab run. - Calls a checker makes from several goroutines are recorded in the order they finished, and a replay that issues them in another order refuses.
To commit a recording as a fixture, copy it from the session scratch into
internal/le/interoplab/<lab>/testdata/<scenario>/, and review it before the
commit like any fixture: it carries environment names only, but an answer can
still print a secret, and such a recording is never committed. The BGP lab's
TestLocalPrefStripReplaysTheLabRecording replays the recording of
bgp-local-pref-strip-gobgp, and TestLDPBackoffClearReplaysTheLabRecording
that of ldp-backoff-clear-frr.
IPsec diagnostic environment
The IPsec lab sets ZE_LOG=debug, the registered base logger key ze.log, on
its Ze container. ZE_LOG_LEVEL names no effective base level and cannot
enable the diagnostic records a checker observes. Scenario ze-env files
remain additive: a more specific key such as ZE_LOG_IKE=info selects that
subsystem's level without changing the shared default.
IPsec injected datagrams
out-of-sa-request-notified-over-sa proves RFC 7296 Section 2.21.4 against
strongSwan. Once the PSK tunnel is up, strongSwan sends Ze, from its own
address and port 4500, a header-only IKE datagram whose SPI pair names no IKE SA.
rawprobe -p udp -x (test/interop-ipsec/rawprobe.c) writes it through a raw IP
socket, as craftedProbe writes its probes.
charon logs a parsed INFORMATIONAL request line only for a message it
decrypted under an IKE SA, so a line carrying N(INVAL_IKE_SPI) shows that Ze's
notice arrived protected over the existing SA. A copy marked as a response goes
first and must draw no such line within five seconds, and the SA and its tunnel
must survive both.
ipsec-auth-corrupt-response-strongswan proves RFC 7296 Section 2.4 with Ze as
the initiator: forged IKE_AUTH responses that fail their integrity check do not
end Ze's handshake. After a control establishment, strongSwan holds its own
IKE_AUTH responses to Ze in an nftables table on its output hook, matching
UDP source port 4500 and the Exchange Type octet (@th,240,8, 35). It then
terminates the IKE SA, and Ze reconnects. Ze sends IKE_AUTH from port 4500
because it offers MOBIKE (mobikeAuthOffer), and charon answers along the path
the request came in (build_response), so IKE_SA_INIT still passes on port 500.
charon logs received retransmit of request with ID 1, retransmitting response
for each retransmission Ze sends into the hold (process_message and
is_retransmit in task_manager_v2.c, strongSwan 6.1.0). The checker then
injects twenty IKE_AUTH responses under the live SPI pair, at Message ID 1, each
with an Encrypted payload of filler. rawprobe's raw socket passes the same output
hook as charon's socket, so the hold rule exempts the forgeries' UDP length.
That socket also takes the XFRM output path, and charon has already installed
its Child SA (selectors 0.0.0.0/0) when it answers into the hold, so without
help the forgeries would leave as ESP under an SA Ze never installed and die in
Ze's kernel (XfrmInNoStates). The checker adds an outbound XFRM policy with
no template on strongSwan, for UDP toward Ze at priority 1, and removes it once
the counter has risen and before the hold lifts. The selector names no port:
rawprobe opens its raw socket without IP_HDRINCL, and raw_sendmsg
(net/ipv4/raw.c) looks the flow up with the socket's protocol and both ports
zero, so a selector on port 4500 never matches it. A failing run of this
checker carries strongSwan's XFRM tables and nftables counters, Ze's XFRM and
UDP counters, sockets and ze_ipsec series, and rawprobe's own report.
ze_ipsec_error_notify_suppressed_total{reason="handshake-unauthenticated"}
must rise by exactly twenty. The hold must lift before Ze's last retransmission,
32 seconds after its request (retransmitBase, retransmitBackoff and
maxRetransmissions in fsm.go), and the checker refuses a run that took
longer. Ze must then establish the IKE SA whose initiator SPI the forgeries
named, and pass traffic over it. With the initiator fix reverted, the first
forgery ends the handshake, so the counter stays still and Ze establishes a new
SA under a new initiator SPI.
The outer DF bit and ICMP errors on a Child SA (RFC 4301)
tunnel-df-treatment proves RFC 4301 Section 8.1 on the wire. Ze initiates with
df-bit clear (ze.conf), then the checker reloads it with df-bit copy
(ze-copy.conf, the same file but for that leaf) and waits for the replacement
Child SA. In each phase Ze pings strongSwan's inner address three times with
-M do (inner DF set) and three times with -M dont (inner DF clear).
strongSwan counts every ESP packet from Ze in an nftables table on its input
hook, one rule per value of the outer header's DF bit (ip frag-off & 0x4000),
so the bit is read as the peer receives it. Under clear all six packets must
arrive with DF clear. Under copy each must carry its inner bit. Each burst
must move exactly one counter by exactly three. show vpn ipsec dataplane sa
must also report the phase's df-bit on Ze's outbound SA, with
options-known true. iptables cannot read
the bit on Ze's kernel: its only match for an arbitrary header bit is xt_u32,
which the kernel does not build. So the strongSwan image carries nftables.
df-bit set is not a phase, because every dataplane refuses it at commit.
icmp-error-payload-check proves RFC 4301 Section 6.2 for an ICMP error
addressed to Ze itself. The checker turns forwarding on in strongSwan's
container (the scenario sets writesPeerSysctl). It routes Ze's inner prefix
there with the preferred source 10.9.9.9, which no selector covers. Ze then
sends echo requests with TTL 1, and strongSwan answers each with Time Exceeded
from 10.9.9.9. The consistent error embeds Ze's echo to 10.1.0.77, which
crossed the SA. charon carries it over the SA because of icmp = yes
(XFRM_POLICY_ICMP), and Ze must deliver it: Icmp.InTimeExcds moves in Ze's
/proc/net/snmp, and ping prints the error. The inconsistent error embeds Ze's
echo to 10.7.0.5, which left in the clear through ze.conf's outbound bypass.
For that step the checker adds a strongSwan out policy for 10.9.9.9, bound to
the SA's reqid. Ze must drop the error: Icmp.InTimeExcds stays still. In
both steps strongSwan's Icmp.OutTimeExcds must move and Ze's inbound ESP byte
counter must advance. Without that, a dropped error could be one that never
reached Ze over the SA. A forwarded error is out of scope
(LL-XFRM-ICMP-TRANSIT, docs/architecture/linux-limitations.md).
Each scenario owes a red run before it counts as evidence ("Prove a scenario discriminates"). Each break below is one line of Ze, and the harness rebuilds the Ze image so the break takes effect:
| Scenario | Break | Expected red |
|---|---|---|
tunnel-df-treatment |
xfrmDFFlag answers false for DFClear |
the clear phase's DF-set burst arrives with DF set |
tunnel-df-treatment |
xfrmDFFlag answers true for DFCopy |
the copy phase's DF-set burst arrives with DF clear |
icmp-error-payload-check |
xfrmStateFromParams no longer sets state.ICMP |
the consistent error arrives over the SA and is not delivered |
icmp-error-payload-check |
xfrmPolicyFromParams widens the inbound protect policy's source selector to. 0.0.0.0/0 |
the inconsistent error is delivered |
The ESP form fast path under a firewall rule
esp-form-firewall-drop proves that Ze's ESP form fast path never carries a
peer's bare ESP past a firewall rule that drops ESP on the input path. The
peers disagree about the ESP form as in esp-form-change: strongSwan
initiates, floats IKE to UDP 4500 with no NAT, and still sends bare ESP,
because build_i (src/libcharon/sa/ikev2/tasks/child_create.c, strongSwan
6.1.0) sets child.encap from COND_NAT_ANY. Ze's ze.conf holds one inet
table whose input chain accepts by policy and drops protocol esp in the
drop-esp term, so IKE on UDP 500 and 4500 passes. The fast path rewrites at
tc ingress, ahead of every netfilter hook, so with that rule in the kernel it
keeps its gate shut and the rule judges the bare ESP.
The checker first waits for show doctor | json to answer
doctor-ipsec-esp-form-firewall naming ze_espdrop and drop-esp. A ping
from Ze then proves the Child SA carries ESP toward strongSwan. strongSwan's
ping to Ze must advance strongSwan's outbound SA, lose every echo request,
leave Ze's inbound SA still, raise the drop-esp counter of
show firewall ruleset espdrop | json by at least the echo requests, and
leave Ze's XfrmInStateMismatch still, so no bare ESP reached XFRM through
the complete path either.
| Scenario | Break | Expected red |
|---|---|---|
esp-form-firewall-drop |
writeGateLocked writes espFormFastPathGateOpen whatever it is given |
strongSwan's echo requests are rewritten past the rule and answered, and the drop-esp counter stays still |
IPsec address movement
mobike-initiator moves Ze; mobike-responder moves strongSwan. Each native
checker adds a new outer address, selects it as the route's source, then removes
the old address without reloading either daemon or initiating another IKE SA.
The checker requires the same IKE and Child SPIs before and after movement, the new UDP 4500 endpoints, unchanged inner selectors, and exactly the original two directed XFRM states on each peer. It sends lossless pings in both directions and requires all four directed ESP byte and packet counters to advance. A replacement tunnel, a stale direction, or a one-way success does not satisfy the scenario.
The Docker host must support XFRM_MSG_MIGRATE_STATE. A kernel without atomic live
ESP migration cannot run this scenario: Ze does not negotiate MOBIKE there. The
checker does not convert that missing prerequisite into a pass, and the suite
does not reach it on such a kernel: the Docker host kernel check refuses the
run first, naming ipsec-mobike (CONFIG_XFRM_MIGRATE).
The IPsec NAT box
A scenario that carries nat.conf starts a third container, nat, at
172.28.0.5. It is keyed on that file exactly as strongSwan is keyed on
swanctl.conf and FRR on frr.conf, and it is started FIRST so its addresses
answer ARP before either daemon sends a datagram.
It exists because RFC 7296 Section 2.23.1 is only reachable across a REAL
translation. natt-transport-inner-checksum and natt-tunnel-inner-checksum
reach the UDP-encapsulated path through strongSwan's encap = yes, which fakes
the NAT_DETECTION_SOURCE_IP hash with no middlebox on the path. The pre-NAT
address and the observed address are therefore equal in those two, every
Section 2.23.1 substitution is the identity, and its absence cannot show. That is
the fourth vacuity trap above: the asserted bytes reach the peer by a path the
mechanism under test does not touch.
nat.conf is one translation per line, <real> <public>. The box gives itself
each public address as a SECONDARY address on eth0, then installs one DNAT rule
in PREROUTING and one SNAT rule in POSTROUTING per line. Three consequences are
load-bearing:
- No peer needs a route. Both public addresses sit inside the lab bridge's own
prefix, so a peer resolves them by ARP and the box answers. A NAT on its own
segment would need a second Docker network, and
interoplab.ScenarioPlancarries oneNetworkSpecthat the BGP suite shares. - Both addresses of a crossing datagram are rewritten, which is the two-NAT figure of Section 2.23.1 drawn with one box, and it is what makes all four NAT_DETECTION comparisons mismatch.
- The rules carry no port and no protocol. IKE floats from UDP 500 to UDP 4500 mid exchange, so a port-scoped rule would translate the handshake and drop the ESP that follows.
Three of the scenarios that use it differ in one variable each. The two transport
scenarios differ in ROLE, because the substitution has two producers and a single
scenario cannot fail on one of them. real-nat-tunnel-control differs from them in
MODE, and it does two jobs: it proves the substitution stays out of tunnel mode, and
it makes a red transport scenario readable, because a broken topology reds it too.
Its selectors are INNER addresses the box never sees, which is the one selector pair
in this lab a translation cannot move.
Two more transport scenarios rekey across the box after an IKE SA rekey, one per
role. real-nat-transport-rekey-after-ike-rekey has strongSwan rekey the IKE SA and
then the transport Child SA, so Ze's responder judges a Child SA rekey on the
successor IKE SA exactly as one before it. real-nat-transport-ze-initiator-rekey-after-ike-rekey
has strongSwan rekey the IKE SA and Ze's own Child SA lifetime rekey the transport
Child SA; it proves the working path only, because strongSwan's answer already names
Ze's own addresses and the scenario passes with the fix broken. Its discriminating
proof is two unit tests in internal/component/ike/engine:
TestTransportChildRekeyAfterIKERekeySubstitutesBehindNAT/ze_rekeys at the rekey
producers, and TestTransportChildRekeyAfterIKERekeyThroughSession/ze_rekeys through
the session entry point.
The port column and the rebind step
A row may carry a third column, <real> <public> <port>: the public port the
box maps that peer's UDP 4500 to. 0, 500 and 4500 are refused, because a mapping
to any of them could not tell an encapsulation port taken from the IKE SA from a
constant (RFC 3948 Section 2.1: "the Source Port and Destination Port MUST be the
same as that used by IKE traffic,"). For such a row natSetupScript emits a port
SNAT for UDP source port 4500 AHEAD of the row's address SNAT, and no DNAT: the
peer initiates and conntrack reverses the replies. The other peer keeps its own
address, so it is not behind a NAT. That is the precondition of the endpoint
adoption the rebind step drives, and it is why the peer behind the port
translation reaches the other one by a host route via its own public address,
which the checker installs before it initiates (natPortEstablish).
The rebind step (natRebind) replaces the port rule with one to a second public
port and deletes the box's UDP conntrack entries with conntrack -D -p udp
(conntrack-tools in Dockerfile.nat). The flush is needed because a NAT rule
applies to a flow's first packet only, and the peer's keepalives and DPD keep the
old entry alive. The step fails when conntrack prints no deleted count, and when
conntrack -L afterwards still holds a flow from strongSwan's UDP 4500 mapped to
the first public port (conntrackDeletedCount, conntrackStaleMapping). A flush
that did nothing then fails at once, rather than as a 90 s wait for a move that
cannot come. The peer's next DPD then reaches Ze from the new port, Ze adopts it,
and the checker waits for exactly one of Ze's four rebind log lines.
The two rebind scenarios set mobike = no in swanctl.conf. strongSwan offers
MOBIKE by default. When MOBIKE is negotiated, Ze correctly refuses to adopt the new
source of an authenticated IKE packet, because RFC 4555 Section 3.8 says "The host
not behind a NAT MUST NOT use these dynamic updates for IKEv2 packets". The peer
must send UPDATE_SA_ADDRESSES instead. Ze logs that refusal at Debug, so a
scenario that loses the setting shows the reason in Ze's log.
| Scenario | Proves |
|---|---|
real-nat-port-translated-ze-responder |
tunnel mode: outbound encap dport 14500, traffic both ways; after the remap the row 1 "migrated" line, dport 14501 on the SAME SPI, traffic both ways |
real-nat-port-translated-transport-ze-responder |
transport mode: after the remap the row 2 "rekeyed" line with reason="transport mode", dport 14501 on a NEW SPI, traffic both ways |
real-nat-port-translated-eap-ze-responder |
the EAP-TLS responder's first Child SA encapsulates to dport 14500 |
responder-eap-tls12 (no NAT) |
a strongSwan EAP-TLS client pinned to TLS 1.2 completes the TLS handshake through ze's Finished, ze refuses to export the RFC 5216 Section 2.3 MSK because the session has no RFC 7627 extended master secret, charon receives EAP-Failure, and neither end installs an XFRM state (checkResponderEAPTLS12) |
None of the open-source IKEv2 implementations surveyed sends RFC 7627 from an
EAP-TLS initiator on TLS 1.2: strongSwan through 6.1.0 defines the extension
constant and never sends it, Libreswan has no EAP-TLS initiator, and OpenIKED
has no EAP-TLS. So responder-eap-tls12 asserts the
refusal, and the extended master secret stays required (owner decision,
2026-10-21).
All four have run in the nightly ipsec-interop suite. In the 2026-10-17 run the
two rebind scenarios and responder-eap-tls12 were red, and
real-nat-port-translated-eap-ze-responder passed, possibly vacuously (see the
next sentence). On 2026-10-17 the two rebind scenarios got mobike = no. In the same change,
waitZeOutboundEncap stopped accepting an outbound state at any port: it had
read a drained textbuf.Buffer inside its poll and so matched "". No run since
these fixes was recorded until 2026-10-21, when all four ran green in the
Docker lab. The tunnel and transport rebind scenarios went red under their
reverted fixes. responder-eap-tls12 went red when crypto/tls was patched to
export keying material without RFC 7627: charon then established the IKE SA,
and assertion 3 (EAP-Failure) timed out. The EAP port scenario sets
mobike = no: with MOBIKE negotiated, Ze records the observed endpoint at the
first IKE_AUTH (RFC 4555 Section 3.3, handleAuthRequest), so the EAP
responder's own adoption never set the ports and removing it stayed green. With
MOBIKE off, removing the adoption line in handleResponderEAP goes red: Ze
refuses the IKE_AUTH Child SA because the peer has no authenticated IKE
endpoint yet.
Two more scenarios reuse the tunnel-control topology to drive show mtu across a
real NAT. mtu-tunnel-sizing-strongswan binds the tunnel to an xfrm interface,
clamps the route the box forwards over to 1400 (ip route replace ... mtu 1400,
which Linux honors when it forwards), and requires the measured path, the derived
ceiling over the negotiated transform behind UDP encapsulation, the recommended
interface MTU and its command; an iputils ping -M do at the old size MUST lose
every packet and one at the recommended size MUST lose none. mtu-nat-installed-endpoint
requires the peer measurement and the tunnel row to target the box's face, the
address the Child SA carries ESP to, and never the peer's real address, where no
ESP arrives. The ze lab image installs iputils for -M do, so every IPsec
scenario's ping is the iputils one.
ike-padded-probe-strongswan reuses the sizing topology to drive the padded IKE
path probe (docs/architecture/diagnostics/path-mtu.md, "The IKE prober") where
ICMP cannot see the clamp. The checker keeps the 1400 clamp for UDP and ESP, routes
ICMP past it through a policy-routing table (exemptICMPFromClamp, so an echo
crosses at 1500), and drops every Fragmentation Needed the box would send Ze
(dropTooBigToward). The ICMP search then measures 1500, the padded INFORMATIONAL
asks 1500 on UDP/4500, its DF copy vanishes, its DF-clear retransmission crosses
the box fragmented and strongSwan answers it, and the descent settles on 1400 with
prober: ike. Each show mtu runs DETACHED (zeShowMTUDetached): a probe the
peer never answers holds the request window for 30 s before the SA is deemed
failed, longer than one docker exec is given, so the shell writes the document to
a file and the checker polls for the marker. A second run races swanctl --rekey,
and a third cuts strongSwan's reassembly memory (dropFragmentsAtPeer,
net.ipv4.ipfrag_high_thresh) so no fragmented datagram is ever reassembled: an
iptables -f rule matches nothing on either container, because conntrack
defragments before any chain runs. The SA then fails exactly once and the row
names sa-failed and the size. Before each lossless ping the checker waits until
Ze lists the Child SA with its selectors (waitZeSelectors): after the SA is
re-established Ze lists the IKE SA before it installs the Child SA, so a ping sent
on the IKE state alone can meet no XFRM state.
The IPsec name server
A scenario that carries dns.conf starts a name server container, dns, at
172.28.0.8, keyed on that file as the NAT box is keyed on nat.conf. It exists so
Ze's remote-address can be a DNS name: RFC 7296 Section 2.23 NAT detection must
then hash over the address the name resolved to, which no scenario with a literal
reaches.
dns.conf is one record per line, <name> <ipv4>. The lab refuses an empty file, a
malformed line, a name given twice, an address outside the lab network and the
server's own address, before any container starts (readDNSConfig). The image is
Alpine with dnsmasq (Dockerfile.dns), built only when a planned scenario carries a
dns.conf.
Two choices are load-bearing:
- The container starts with no server running. A checker calls
startDNSwhen its scenario wants the names to answer, which runs dnsmasq with one--host-recordper line, no upstream and no hosts file, then waits untilnslookupover the lab network returns every record. So a scenario can boot Ze while DNS is down and bring it up later, the home CPE's "boot before DNS" case. - Ze asks this server and nothing else. The lab mounts a
resolv.confnaming 172.28.0.8 alone (options timeout:1 attempts:1) over Ze's/etc/resolv.conf, which replaces Docker's own file and its embedded resolver. While dnsmasq is down a query is refused at once, so the DNS-down phase measures Ze's backoff rather than a resolver timeout.
Ze's SPD catch-all DISCARD (vpn ipsec unmatched discard, the default) drops Ze's own
DNS query like any other clear traffic, so each scenario's ze.conf carries a
policy lab-dns bypass to 172.28.0.8 at order 1000, ahead of the tunnel's 2000. An
operator with a DNS-named peer writes the same entry for their resolver
(docs/guide/ipsec.md, "The catch-all entry").
| Scenario | Proves |
|---|---|
fqdn-remote-dynamic-local |
Ze names no interface and no local-address (the route-chosen source) and dials vpn.example.net. With DNS down, Ze logs ike: connect cycle cannot start naming the name and "does not resolve", logs it again (the retry), and charon parses no IKE_SA_INIT. Once startDNS runs, the SA establishes; charon's parsed IKE_SA_INIT request and its generated response both carry N(NATD_S_IP) and N(NATD_D_IP); charon logs no NAT condition and Ze reports nat-detected false; charon received the request from 172.28.0.2 port 500, and the IKE SA sits on Ze's port 4500 (MOBIKE, RFC 4555); both ends hold the Child SA between 172.28.0.2 and 172.28.0.3, and traffic crosses it |
fqdn-remote-real-nat |
real-nat-tunnel-control with remote-address vpn.example.net, answered with strongSwan's public 172.28.0.7. charon logs "local host is behind NAT" and "remote host is behind NAT", Ze reports the NAT verdict, the IKE SA sits on Ze's public 172.28.0.6 port 4500, Ze's inbound state carries encap type espinudp, and ESP crosses the box in all four directions |
charon answers NAT detection only when the request carried both payloads (strongSwan
6.1.0 src/libcharon/sa/ikev2/tasks/ike_natd.c build_r: "if (this->src_seen &&
this->dst_seen)"), and it compares each received hash with the addresses it observed
(process_payloads), setting COND_NAT_HERE on a destination mismatch and
COND_NAT_THERE on a source mismatch, which ike_sa.c set_condition logs as the two
"behind NAT" lines. Before spec-home-cpe-4, Ze built the payloads and checked the reply
only when remote-address parsed as an IP, so a named remote sent none: the first
scenario goes red on the parsed request and the second on charon's NAT lines and Ze's
verdict.
IPv6 inside an IPv4 Child SA on an xfrm interface
vti-ipv6-inner-ipv4-outer is the home CPE's management path. Ze initiates over the
plain IPv4 lab pair with two traffic-selector entries, 0.0.0.0/0 and then ::/0
on both sides, bound to vti0 (if-id 7), with fd00:db8:1::1/128 on vti0 and a
static route for fd00:db8:1::/112 through it. strongSwan's child config names
::/0 alone, so it narrows the answer to ::/0 === ::/0: strongSwan 6.1.0's
traffic_selector_list.c select_ intersects each configured selector with each
received one, and traffic_selector.c get_subset answers NULL when the two types
differ. Ze's unconfigured wildcard over an IPv4 outer pair would be 0.0.0.0/0
alone, which this child config cannot accept, so the established ::/0 child is
itself proof that the IPv6 entry reached the wire. Ze's esp-group sets pfs enable,
so strongSwan's esp_proposals name MODP_2048: the IKE_AUTH child carries no key
exchange either way, but the rekey below does, because strongSwan 6.1.0's
child_create.c build_i takes the child config's first key exchange method for a
CREATE_CHILD_SA, and Ze's respondChildRekey refuses an offer with no group while
PFS is on. With aes256gcm16 alone the rekey drew N(NO_PROP). The checker asserts:
| Step | State read | Fails when |
|---|---|---|
| Selectors | swanctl --list-sas, and the child of show vpn ipsec sa | json, each waited for, because Ze lists its Child SA only once the XFRM install is done (waitChildSelectors, waitZeSelectors) |
Either end holds anything but ::/0 on both halves, strongSwan holds an IPv4 selector, or Ze's child does not report if-id 7 |
| Policies | Ze's ip xfrm policy, the records on if_id 0x7 other than vti0's own final entry (action block at priority 4294967295, both families, isInterfaceSPDCatchAll) |
They are not exactly one ::/0 policy for each of out, in and fwd; any other IPv4 policy on the if_id, a BLOCK at another priority included, is refused |
| States | Ze's ip xfrm state, the records on if_id 0x7 |
Fewer than two, or one without flag af-unspec (child 20's XFRM_STATE_AF_UNSPEC) |
| Datapath | ip -6 route get fd00:db8:1::2 on Ze, a ping from, and the four directed ESP counters. fd00:db8:1::1 |
The route does not leave through vti0, the ping loses anything, or any simplex SA did not count bytes |
| Rekey | The four rows above again after swanctl --rekey --child ze-child and a new SPI |
Any of them fails |
The ping cannot pass outside the tunnel: xfrmi_xmit2
(net/xfrm/xfrm_interface_core.c) refuses a packet whose lookup returns no xfrm
state, so vti0 has no clear exit. strongSwan's lab interface has IPv6 disabled on
the IPv4-only network, and its policy-based tunnel decapsulates onto that interface,
so the checker enables IPv6 there (writesPeerSysctl) and puts fd00:db8:1::2 on
its loopback with a route through eth0. When the route or the ping fails, the
error carries both peers' kernel state (vtiIPv6Evidence): the ping repeated once
with its stderr, every XFRM policy rather than the if_id 0x7 records alone,
ip -s xfrm state, /proc/net/xfrm_stat, IPv6 routes, rules and addresses, and
disable_ipv6. The full policy dump matters because the first output lookup
(xfrm_lookup_route, called from ip6_dst_lookup_flow) runs at if_id 0, before
the packet reaches vti0, so a policy bound to no interface decides first. The red walk reverts child 20's
state.AFUnspec = true in xfrmStateFromParams, rebuilds the Ze image, and must
fail at the state row and the ping.
vti-unmatched-discard-interface-spd proves RFC 4301 Section 4.4.1's SPD selection
function and each XFRM interface SPD's own final entry under unmatched discard.
Ze binds one Child SA carrying 10.255.1.0/24 to 10.255.2.0/24 and fd00:db8:1::/64
to fd00:db8:2::/64 to vti0 (if_id 7); strongSwan's child names the same pairs, so
select_ (traffic_selector_list.c) answers them unchanged. The checker requires
IPv4 and IPv6 to cross the tunnel both ways, each Ze-initiated round trip counted
by all four simplex SAs: Linux looks Ze's packet up at if_id 0 first, where the
DISCARD catch-all would drop it, so only the selection entries hand it to vti0's
SPD. It requires vti0's SPD to hold a BLOCK at priority 4294967295 in, out and
fwd for both families. Last it pings 10.255.9.1 and fd00:db8:9::1, which ze.conf
routes into vti0 outside every selector: every echo must be lost, vti0's
tx_errors must rise (xfrmi_xmit counts the failed lookup), and
XfrmOutPolBlock must rise, so the drop happened inside vti0's SPD rather than
at if_id 0. The red walk rebuilds the Ze image without the exemptions
(xfrmInterfaceExemptions answering nothing) and must fail at the first ping.
vti-ipv6-tunnel-source-no-cleartext-leak proves the tunnel-source guard from the
WAN side. IPv6 is on for both lab interfaces because the scenario's lab network
carries the IPv6 /64 fd00:db8:a::/64 beside its IPv4 subnet (networkIPv6 on its
scenarioCheckers entry, leakWANLink), the route the BGP lab takes for OSPFv3.
Docker disables IPv6 on eth0 of a container on an IPv4-only network, and a
start-time --sysctl net.ipv6.conf.all.disable_ipv6=0 did not undo it, so Ze
keeps a read-only /proc/sys and takes no --sysctl; Docker numbers the two
interfaces fd00:db8:a::2 and fd00:db8:a::3. strongSwan's checker still writes its
sysctls (writesPeerSysctl) for the tunnel side, and the checker reads
disable_ipv6 back for Ze's eth0 before counting. The checker gives strongSwan the WAN-only
destination 2001:db8:99::1 on lo, and routes it on Ze out of eth0 through a
static neighbour with vti0's address fd00:db8:1::1 as the preferred source. Its
observer is an ip6tables INPUT counter in strongSwan's namespace per source.
Before counting, it requires the leak path to exist: the if_id 0 source BYPASS for
fd00:db8:1::1, the guard's ip -6 rule from it, and the kernel choosing eth0 and
fd00:db8:1::1 for the destination. A positive control, ICMPv6 from eth0's own
address passed by the wan-control bypass, must reach the counter. Then echoes from
the tunnel address, socket-chosen (ping -I, steered by the rule) and
kernel-chosen (no source, dropped by the postrouting term), must leave the
counter at zero. The red walk rebuilds the Ze image with the source exemption
installed whether or not its guard holds (the !guarded[source] test in
installXFRMInterfaceEntries answering false) and with no guard installed, and must
fail at the zero count. Removing the guard alone is not that break: the source
exemption waits for its guard, so the leak path itself would be missing and the run
would stop at the leak-path check instead.
IPsec virtual-IP leases
Two scenarios prove Ze as the IRAS of RFC 7296 Section 2.19: a strongSwan client
sends a CFG_REQUEST, and Ze answers with a lease from the remote-access pool its
peer's virtual-ip-pool names. Ze's inner address is 10.20.0.1 in both, and the
checker puts it on Ze's loopback.
| Scenario | Path | Proves |
|---|---|---|
virtual-ip-assignment |
EAP-MSCHAPv2, no NAT, vips = 0.0.0.0 |
charon reports an address inside 10.10.0.0/24, installed it, and narrowed its child to it. Ze's show vpn ipsec sa reports the same virtual-ip, and its Child SA's ts-remote is that address as a /32. Ze's kernel holds the /32 policy and no policy for the pool. charon wrote the pool's DNS server to. A ping from each end, from its inner address, moves ESP on all four simplex SAs. /etc/resolv.conf |
virtual-ip-uniqueness |
PSK, two clients behind the NAT box, both vips = 10.1.2.3 |
RFC 3948 Section 5.1: the two leases differ, one of them is 10.1.2.3, and both sit in 10.1.2.0/24. Each Ze peer reports its own client's lease as virtual-ip and as ts-remote. Ze's kernel holds both /32 policies and no policy for the pool. A ping to each lease moves that client's four SAs and leaves the other client's outbound SA at Ze still |
The uniqueness scenario needs two client sources in one strongSwan container. The
checker adds 172.28.0.8/32 to strongSwan's eth0 before connection ze-b
initiates. The scenario's nat.conf maps that address to 172.28.0.9 beside the two
usual rows, so it has three translations, which is the most readNATConfig accepts
(natTranslationsMax). Ze sees the clients at 172.28.0.7 and 172.28.0.9, one
respond peer each, both bound to one pool.
Both scenarios went red on 2026-10-26 with the lease substitution removed
(substituteVirtualIP in internal/component/ike/engine/cp.go returning its input,
applied through a GOFLAGS=-overlay= build so the shared tree stayed untouched),
and green again without it. In virtual-ip-assignment strongSwan still
established, held 10.10.0.1 and narrowed its own side to it, so every client-side
read passed. The failure was Ze's ts-remote, which named 10.10.0.0/24 where the
lease /32 was wanted. In virtual-ip-uniqueness the second client never got a
Child SA, because Ze could not install a second policy for 10.1.2.0/24 while
swan-a held one. That refusal is the Section 5.1 hazard itself.
IPsec inner-address uniqueness without a pool
unbound-inner-address-uniqueness is the same two-laptop figure of RFC 3948
Section 5.1 with no pool to make the addresses unique. Ze's two respond peers bind
no virtual-ip-pool and carry no traffic-selector, so each client chooses its
own inner selector. Both strongSwan connections propose local_ts = 10.1.2.3/32
with no vips; ze-a asks for remote_ts = 10.20.0.1/32 and ze-b for
remote_ts = 10.20.0.0/24. The two pairs overlap without being equal, so the
dataplane's per-selector owner check, which keys on the exact selector, cannot
refuse the second, and only the cross-peer guard can. The NAT box and the second
client address are those of virtual-ip-uniqueness. The checker puts 10.1.2.3 on
strongSwan's loopback and 10.20.0.1 on Ze's, each with a route to the other.
| Step | Proves |
|---|---|
ze-a initiates |
strongSwan's Child SA is INSTALLED, and Ze's kernel holds exactly one outbound ESP policy to 10.1.2.3/32, tunneled to 172.28.0.7 |
ze-b initiates |
Ze's kernel still holds exactly the one outbound policy to 10.1.2.3/32, the first client's |
| strongSwan's answer | strongSwan logs TS_UNACCEPTABLE and keeps IKE SA ze-b ESTABLISHED with no Child SA (RFC 7296 Sections 2.9 and 2.21.2) |
| Ze's view | show vpn ipsec sa reports swan-a's Child SA with ts-remote 10.1.2.3/32 and swan-b established with no child-sa |
| Traffic | a ping from 10.1.2.3 to 10.20.0.1 moves ESP on all four simplex SAs of the first client |
The policy count reads ip xfrm policy one outbound policy at a time
(innerOutPolicyTunnels), because the selector-pair set the other checkers use
collapses two policies to one inner address into one entry.
The red run, on 2026-10-26: with the cross-peer guard reverted through a Go
overlay and the Ze image rebuilt, the scenario FAILS at the ze-b step with
"ze's kernel holds 2 outbound ESP policies to 10.1.2.3/32 (tunnels [172.28.0.9
172.28.0.7]), want exactly one". Ze accepts the second client's overlapping pair
and installs it beside the first, so a packet to 10.1.2.3 could leave through
either tunnel: the RFC 3948 Section 5.1 confusion itself. The kernel is judged
before the notify, so that is the failure the run reports.
The strongSwan lab drop-in
Every scenario that starts a strongSwan peer mounts
test/interop-ipsec/strongswan-lab.conf read-only at
/etc/strongswan.d/98-lab.conf. It carries the charon settings the whole lab
needs, and today that is two: charon.plugins.bypass-lan.load = no, and a
stderr filelog with flush_line = yes at charon's default level.
charon loads bypass-lan by default, and that plugin installs a PASS shunt for
every locally attached subnet. This lab puts both containers on 172.28.0.0/24, so
the shunt covers the peer. Measured on 2026-08-30 in psk-site-to-site, the shunt
sits at priority 175423 against the Child SA policy's 399999, the lower number
wins, and every packet strongSwan sends to Ze leaves in the clear. A ping still
succeeds under that shunt, which is exactly why a lossless ping is necessary and
not sufficient evidence that a tunnel carried anything.
A scenario's own strongswan.conf still mounts at
/etc/strongswan.d/99-interop.conf and composes with the lab file. A scenario
MUST NOT set bypass-lan itself: TestNoScenarioCarriesItsOwnBypassLanOverride
(test/interop-ipsec/parity_test.go) refuses a second copy, because two files
setting one value is a disagreement with nothing to arbitrate it.
The logger exists because charon, with no logger configured, logs through a
default stdout logger opened without line flushing (strongSwan 6.1.0
load_loggers, src/libcharon/daemon.c), and stdio fully buffers stdout on the
pipe docker logs reads. A checker waiting for a charon line then waits on the
buffer rather than on the exchange. Each scenario that read charon's log used to
carry a private copy of this logger; mtu-negotiated-transform, which had none,
timed out in evidence-nightly run 41298258262 waiting for a
selected proposal: ESP: line the Child SA had already used. A scenario MUST NOT
set flush_line itself (TestEveryStrongSwanPeerFlushesEachLogLine); it MAY add
options to the same stderr logger, as child-rekey-refused-keeps-the-sa adds
its timestamps. The native MOBIKE lab composes the same two files, the scenario's
only when it exists (writeSwanConfig).
AES CCM on the strongSwan peer
ike-aes-ccm16 gives the IKE SA's Encrypted payload the transform RFC 5282
Section 7.2 numbers 16, AES CCM with a 16-octet ICV, and takes the Child SA to
AES GCM because Ze refuses AES CCM for ESP at config parse
(ipsec.EncryptionImplementedESP). The checker reads charon's own
selected proposal: IKE:AES_CCM_16_256/ line, then Ze's encryption field, then
ESP in both directions, so the agreement on the Transform ID and the verification
of the encrypted ICV are separate observations.
The lab image carries that transform although /usr/lib/ipsec/plugins holds no
libstrongswan-ccm.so. test/interop-ipsec/Dockerfile.strongswan builds
strongSwan 6.1.0 from source with no --enable-ccm, so the standalone ccm
plugin is absent, and two other plugins register the algorithm instead: on an
x86-64 host swanctl --list-algs in the image answers AES_CCM_16[aesni],
AES_CCM_12[aesni] and AES_CCM_8[aesni], and with aesni not loaded it answers
AES_CCM_16[openssl], AES_CCM_12[openssl] and AES_CCM_8[openssl] (measured
2026-10-22). The aesni plugin is built only on x86, because its sources need
-maes, -mpclmul and -mssse3, which gcc refuses on arm64; there openssl is
the only provider. The absent FILE is not an absent
capability, and a source build of the ccm plugin would add a second provider of
one algorithm and nothing else.
Integrity-only ESP on the strongSwan peer
esp-integrity-only-null negotiates the Child SA as ENCR_NULL with
HMAC-SHA2-256-128, the integrity-only ESP that RFC 4303 Section 1 requires to be
negotiable, over an ordinary AES GCM IKE SA (RFC 7296 Section 5 keeps ENCR_NULL
out of the IKE SA). Ze initiates and charon's esp_proposals = null-sha256
accepts nothing else. The checker reads charon's
selected proposal: ESP:NULL/HMAC_SHA2_256_128/ line, then Ze's Child SA
esp-encryption field (null), then both kernels' SAs and ESP in both
directions. The ESP datapath is the Linux kernel's ecb(cipher_null) on both
ends, which needs CONFIG_CRYPTO_NULL on the Docker host kernel.
The lab-wide strongswan-lab.conf sends charon's log to stderr with
flush_line = yes, at the unchanged default level. StrongSwan's default
stdout logger can retain the selected ESP proposal in its buffer after the
Child SA is installed. The flushed logger makes that exact proposal observable
without changing the negotiation, kernel-state or received-traffic assertions.
RFC 4739 multiple authentication on the strongSwan peer
multiple-auth-ze-initiator and multiple-auth-ze-responder run a second
authentication round in each direction inside one IKE_AUTH phase. strongSwan
takes its rounds from local-2 and remote-2, and charon announces the
extension by default (ike_auth.c multiple_auth_enabled). Ze takes them from
authentication second-round local and remote.
| Scenario | Ze role | Round 1 | Round 2 |
|---|---|---|---|
multiple-auth-ze-initiator |
initiator | X.509 both ways (client.pem, server.pem) |
pre-shared key both ways, ze-user.example.com and swan-user.example.com |
multiple-auth-ze-responder |
responder | X.509 both ways (server.pem, client.pem) |
pre-shared key both ways, swan-user.example.com and ze-user.example.com |
A tunnel coming up does not show that a second round ran, so each checker also
reads both sides' logs. charon logs authentication of '<id>' with pre-shared key successful (psk_authenticator.c) for each of Ze's rounds it verifies,
and Ze logs ike: second authentication round verified with the peer and
id attributes on one line. Every round-2 identity is one that round 1 does
not use, so the identity says which round a line belongs to.
The round-2 identities are FQDNs. Ze sends any identity that is not an address
literal as ID_FQDN (encodeIKEID), while charon reads user@domain as
ID_RFC822_ADDR, and the two types never match. charon matches a pre-shared key
against an unordered list of owner identities (mem_cred.c), so the
scenarios use one round-2 secret for both directions.
Two charon policies shape the responder scenario. charon's initiator runs the
strict check of every configured remote round as soon as its current local
round is a pre-shared key (ike_auth.c require_strict), so a strongSwan
initiator whose round 1 is a PSK refuses a responder's first IKE_AUTH response
with "insufficient authentication rounds" before the responder's second round
can come; strongSwan's round 1 is therefore a certificate. And charon counts a
configured round as done when a completed round complies with it
(do_another_auth), so a local-2 identical to local is never sent, which
is why strongSwan's second round is a pre-shared key rather than server.pem
again. Ze's x509 second round is proven Ze to Ze
(TestRFC4739ZeToZeTwoRoundsEachWay).
Each scenario was walked red: with multipleAuthRequestNotifies,
settleAnotherAuth and moreAuthRounds reverted through an overlay, charon
refuses the Ze initiator's single round ("insufficient authentication rounds"),
and the Ze responder establishes after one round while charon still owes its
second, so the strongSwan SA never comes up. Both scenarios set
send_cert = always, and both flush charon's log to stderr the way
esp-integrity-only-null does.
Legacy signature methods against a strongSwan without RFC 7427
The five x509-legacy-* scenarios set
charon.signature_authentication = no, so strongSwan 6.1.0 neither sends
SIGNATURE_HASH_ALGORITHMS nor reads Ze's (ike_init.c). Ze then receives no
notify, as initiator or as responder, RFC 7427 Section 3 leaves it no method 14, and it must
sign with its key's legacy method (RFC 8247 Section 3.1). charon signs its own
AUTH the same way (pubkey_authenticator.c build).
| Scenario | Keys on both sides | Method each side sends | PKI |
|---|---|---|---|
x509-legacy-rsa-signature |
RSA 2048 | 1, RSASSA-PKCS1-v1_5 over SHA-1 | the scenario's pki/ |
x509-legacy-ecdsa-p256-signature |
ECDSA P-256 | 9, ECDSA-256 | the shared lab PKI |
x509-legacy-ecdsa-p384-signature |
ECDSA P-384 | 10, ECDSA-384 | the scenario's pki/ |
x509-legacy-ecdsa-p521-signature |
ECDSA P-521 | 11, ECDSA-521 | the scenario's pki/ |
x509-legacy-rsa-signature-ze-responder |
RSA 2048, strongSwan initiates | 1, RSASSA-PKCS1-v1_5 over SHA-1 | the scenario's pki/ |
Ze initiates in the first four and responds in the last. Each scenario's own
pki/ holds leaf certificates issued by the shared lab CA.
The checker requires the IKE and Child SAs, both kernels' ESP states and
traffic both ways, then charon's two log lines: authentication of 'ze-test-client' with <method> successful for Ze's AUTH, and
authentication of '172.28.0.3' (myself) with <method> successful for its own,
which Ze verified before installing the Child SA. charon names a method 14 AUTH
by signature scheme rather than by method, so a Ze that sent method 14 fails the
first line.
Each scenario was walked red on 2026-10-31: with computeX509Auth reverted to
refusing when no notify arrived, the rebuilt Ze logs ike: build IKE_AUTH failed and the strongSwan SA never comes up. The RSA scenario was also walked
red with verifyLegacyRSAAuth reverted to SHA-256 only: Ze then refuses
charon's SHA-1 method 1 AUTH.
Ed25519 signatures against strongSwan (RFC 8420)
Two scenarios run with RFC 7427 left on, so both sides send
SIGNATURE_HASH_ALGORITHMS. strongSwan 6.1.0 announces Identity (5) because its
openssl and curve25519 plugins each register an Identity hasher (ike_init.c
send_supported_hash_algorithms), and Ze announces it in every IKE_SA_INIT.
Each scenario's pki/ holds one Ed25519 leaf issued by the shared lab P-256 CA,
beside copies of the shared CA and of the other side's P-256 leaf.
| Scenario | Ze's key | strongSwan's key | Who initiates | charon line the checker requires |
|---|---|---|---|---|
x509-ed25519-ze-initiator |
Ed25519 (client.pem) |
P-256 (server.pem) |
Ze | authentication of 'ze-test-client' with ED25519 successful |
x509-ed25519-ze-responder |
P-256 (client.pem) |
Ed25519 (server.pem) |
strongSwan | authentication of '172.28.0.3' (myself) with ED25519 successful |
Both checkers first require the IKE and Child SAs, both kernels' ESP states and
traffic both ways. charon names a method 14 AUTH by its signature scheme
(pubkey_authenticator.c: process logs the scheme it verified, build the
scheme it signed with). In the initiator scenario the line is charon's proof
that it verified Ze's Ed25519 signature. In the responder scenario it proves
charon sent an Ed25519 AUTH, which it can only do when Ze announced Identity,
and Ze installs the Child SA only after verifying that AUTH.
Not yet walked red. The break for the initiator scenario makes
chooseEd25519Signature refuse as if the peer had not announced Identity: Ze
then fails to build IKE_AUTH and no SA comes up. The break for the responder
scenario makes verifyEd25519 refuse every signature: Ze answers
AUTHENTICATION_FAILED and no Child SA is installed.
Ed448 signatures against strongSwan (RFC 8420)
The Ed448 twins of the two Ed25519 scenarios. strongSwan 6.1.0's openssl plugin
provides Ed448 private and public keys, PRIVKEY_SIGN and PUBKEY_VERIFY for
SIGN_ED448, and the Identity hasher (openssl_plugin.c, under OpenSSL 1.1.1 or
later), and the lab image builds with --enable-openssl. The Ed448 leaves were
minted with openssl genpkey -algorithm ed448 and issued by the shared lab P-256
CA. crypto/x509 parses such a leaf without reading its key, and Ze reads the key
from the raw SubjectPublicKeyInfo.
| Scenario | Ze's key | strongSwan's key | Who initiates | What the checker requires |
|---|---|---|---|---|
x509-ed448-ze-initiator |
Ed448 (client.pem) |
P-256 (server.pem) |
Ze | authentication of 'ze-test-client' with ED448 successful |
x509-ed448-ze-responder |
P-256 (client.pem) |
Ed448 (server.pem), then Ed448 (server-untrusted.pem) |
strongSwan | authentication of '172.28.0.3' (myself) with ED448 successful for ze; for untrusted, charon's received AUTHENTICATION_FAILED notify error, Ze's ike: initiator AUTH verification failed with peer=swan and certificate signed by unknown authority, and no new ESP state on Ze |
The responder scenario's second connection, untrusted, carries the same
identity with an Ed448 leaf from untrusted-ca.example.net
(pki/untrusted-ca.pem), a CA Ze does not hold. The lab mounts
server-untrusted.pem and server-untrusted-key.pem into strongSwan when a
scenario's pki/ carries them. The checker initiates untrusted only after the
trusted tunnel carries traffic.
Not yet walked red. The break for the initiator scenario makes
chooseEd448Signature refuse as if the peer had not announced Identity. The
break for the responder scenario makes verifyEd448 refuse every signature,
which takes the trusted tunnel down. The break for the untrusted half makes
getRemoteCert skip its cert.Verify chain check: untrusted then
establishes, and Ze logs no refusal.
Peer certificate revocation and purpose (RFC 5280, RFC 4945)
Four scenarios, two pairs. Ze initiates with the lab P-256 leaf
(client.pem) and checks the certificate strongSwan presents with
send_cert = always. The two scenarios of a pair differ by one file in pki/,
so that file is the only possible cause when one is refused and the other
establishes.
| Scenario | The file that differs | Expected |
|---|---|---|
x509-crl-revoked-peer-ze-initiator |
crl of Ze's pki ca test-ca, lists the serial number of strongSwan's server.pem. pki/ca-crl.pem |
refused, was revoked by |
x509-crl-unrevoked-peer-ze-initiator |
the same list, revoking nothing | established |
x509-eku-server-auth-only-refused-ze-initiator |
carries extendedKeyUsage serverAuth alone. pki/server.pem |
refused, extended key usage |
x509-eku-ipsec-ike-ze-initiator |
carries extendedKeyUsage id-kp-ipsecIKE. pki/server.pem |
established |
The two server.pem files of the purpose pair have the key, subject and
subjectAltName of the lab server.pem, no keyUsage, and the lab CA as issuer.
openssl issued them: strongSwan's pki --issue cannot write id-kp-ipsecIKE,
because x509_generate_eku_extension (x509_cert.c) knows only serverAuth,
clientAuth, iKEIntermediate, ocspSigning and msSmartcardLogon. charon does not
judge the purpose of its own certificate: pubkey_authenticator.c checks
keyUsage only in process, for the peer's AUTH (is_compliant_cert).
A refusal leaves strongSwan's SA in place. Ze sends no notification for a
responder AUTH it refuses on the first round, and charon's responder holds an
ESTABLISHED IKE SA from the moment it answered. So the refusing checker asserts
nothing about strongSwan's SA list or kernel. It requires Ze's
ike: remote AUTH verification failed line for peer=swan carrying the cause,
charon's two pubkey_authenticator.c lines that show it verified Ze's AUTH and
signed its own with the certificate under test, and no ESP state on Ze. The
accepting checker requires Ze's ESP state and traffic both ways, which only an
accepting Ze produces: the strongSwan SA alone would also be there after a
refusal.
The red run, on 2026-11-10, after fbdd9a82e7: with an early return nil at
the top of checkIKEPurpose and of checkPeerChainRevocation
(internal/component/ike/engine/remote_cert_checks.go) and the Ze image
rebuilt, both refusing scenarios FAIL with "ze did not refuse strongSwan's
certificate: wait for ze log "ike: remote AUTH verification failed" timed out
before the peer became ready": Ze established with the revoked certificate and
with the serverAuth-only one. With the file restored byte-exact both pass
again, and both accepting twins pass throughout.
ESP transform kernel proof
esp-aes-cbc-sha256, esp-chacha20poly1305 and esp-3des-sha1 offer one
ESP combination to strongSwan. checkESPKernel requires the negotiated
encryption and both kernels' installed transforms in both directions, then
lossless traffic and increasing receiver counters. The IKE SA stays on
AES-GCM/SHA256; the legacy 3DES/SHA1 choice applies only to ESP.
test deployment vpn-kernel selects these scenarios alongside the
existing GCM and NULL scenarios, then runs the WireGuard kernel proof.
test qemu vpn-kernel supplies the disposable Ze-system guest.
The external scenarios exercise Ze-initiator establishment and bidirectional
traffic, not both roles' rekey exchanges.
Cleanup is part of the verdict: a scenario or image cleanup failure makes the
VPN proof fail even when traffic passed. The report keeps the original Docker
cleanup error beside that failure, rather than reporting traffic as a clean pass.
L2TP and PPPoE lifecycle fault scenarios
l2tp-unknown-mandatory-avp-xl2tpd relays xl2tpd's exchange without
originating messages. An unknown optional AVP on SCCCN must permit PPP traffic.
The relay then appends the same AVP with M=1 to a real HELLO. The checker
requires a matching StopCCN with Result Code 2 and Error Code 8, route
withdrawal and removal of both peers' PPP and kernel tunnel state.
The native log retains the original and forwarded control datagrams.
The native scenario passed, failed when a private build omitted HELLO AVP
validation (no matching StopCCN 2/8), then passed after a normal rebuild.
ppp-code-reject-xl2tpd puts the same relay between xl2tpd and Ze, and it is
the one scenario where the relay originates frames. No lab peer sends a PPP
Code it does not know, so the relay injects them on the live L2TP session,
and only after the checker creates /run/code-reject once PPP traffic has
flowed. The next data message from the LAC is the trigger: the relay
forwards it, then sends Ze four PPP packets behind a copy of its L2TP header,
so they carry the live tunnel and session IDs. Two are LCP packets with Code
99 and two are IPCP packets with Code 12, all four with Identifier 0x42 and
204 octets long, against the MRU of 128 that pppd requests (mru 128). The
relay refuses a data header carrying Ns and Nr, because a frame it injected
would break the sequence. From the marker on, it logs every data datagram in
both directions, unfiltered, and the checker refuses one it cannot read, so a
frame the log holds is never passed over. The checker decodes the probes and
Ze's answers from that log. It requires two LCP Code-Rejects and two IPCP
Code-Rejects, the two of each protocol with different Identifiers (RFC 1661
Section 5.6: "The Identifier field MUST be changed for each Code-Reject
sent."). The IPCP probes reach Ze on its PPP unit descriptor, not the
channel: once NCP is up Ze connects the PPP channel to its unit, and Linux's
ppp_input then queues an IPCP frame on the unit, so the IPCP answers prove
Ze reads that descriptor.
Each Code-Reject must have an Information field of at most 128 octets,
and its Rejected-Packet must be the start of the probe, with the probe's Code, Identifier and Length unchanged (RFC 1661 Section 5.6: "The
Rejected-Packet MUST be truncated to comply with the peer's established
MRU."). The session must survive: a ping still crosses it, both PPP
interfaces are the ones from before the probes, and neither peer sends an LCP
or IPCP Configure or Terminate packet afterwards. Last, the checker creates
/run/ipcp-terminate, the relay sends one IPCP Terminate-Request with
Identifier 0x5a on the next LAC datagram, and Ze must answer with an IPCP
Terminate-Ack carrying 0x5a (RFC 1661 Section 5.5: "Upon reception of a
Terminate-Request, a Terminate-Ack MUST be transmitted."). That ends IPCP,
so it runs after every other assertion.
pppoe-chap-rechallenge-ze-client observes two accel-ppp reauthentication
exchanges after working IPCP traffic, checks each CHAP Response's Identifier
and digest, and requires the session and connectivity to survive.
pppoe-peer-terminate-ze-ac lets pppd originate LCP termination while
filtering its outgoing PADT. Ze must send the correlated Terminate-Ack and
its own delayed PADT, remove the session and release its kernel PPP state.
Both scenarios passed, failed under isolated producer breaks, then passed after
normal rebuilds. Removing network-phase CHAP handling left repeated Challenges
unanswered; removing the Stopped lower-layer notification left no Ze PADT despite
the peer's correlated Terminate-Request and Terminate-Ack. Shared product sources
were not changed by these private build overlays.
The FreeRADIUS admin-login suite
internal/le/interoplab/radius/ runs ze's operator login against a real
FreeRADIUS server at a pinned tag, pulled through ImageBuild{Pull: true}. It
exists because every other RADIUS proof ze holds runs against a mock ze wrote:
test/plugin/aaa-radius-admin.ci drives internal/test/mock/radius/radius.go,
and the L2TP lab's peer is internal/le/interoplab/l2tp/radiusmock/, which is
ze's own Go program in a container. A mock built beside ze's encoder agrees with
ze by construction, and ze now computes a CHAP digest a server must reproduce
from its own stored password. Only a server ze did not write can disagree.
It is its own suite rather than four more L2TP scenarios. The L2TP lab probes
for the l2tp_ppp or pppol2tp kernel module and refuses to run without it,
which is correct for a suite that carries PPP sessions. Admin login is ze's SSH
listener, a UDP socket and a RADIUS server, so this lab declares no preflight
beyond its own ze cross-compile, mounts no module tree, asks for no capability
and runs nothing privileged. TestSuiteNeedsNoKernelModule holds that. The
Docker host kernel check still applies: it is a rule about the host, so a
kernel without PPPoL2TP refuses this suite as it refuses every other.
Every checker reads BOTH sides. Ze's log saying source=radius is not enough on
its own, because a login the local bcrypt backend satisfied produces a line of
the same shape and no server traffic at all. The lab therefore mounts a
linelog module at /etc/raddb/mods-enabled/ze_request_log that writes one
line per answered request to /var/log/freeradius/ze-request.log, carrying the
verdict, the User-Name, the PRESENCE of a User-Password, of a CHAP-Password and
of an EAP-Message, and the NAS-Identifier. Presence and not value: a fixture
must not put a password or a digest in a log file. parseServerRecord refuses a
line missing any of the six fields, so a truncated or reformatted line is never
read as a partial verdict.
An EAP login is several requests and FreeRADIUS runs no post-auth section for
an Access-Challenge, so the reply that asked the question records nothing. A
second module, ze_state_echo_log, is called from authorize on a request
carrying both an EAP-Message and a State, and writes verdict=state-echo. That
is the server's own evidence that ze returned the State unmodified, which
RFC 2865 Section 5.24 requires, and that the login was a conversation rather
than one request.
| Scenario | Ze's side | The server's side |
|---|---|---|
radius-admin-pap-freeradius |
An operator logs in over ze's real SSH listener, ze's log says source=radius, the Filter-Id profile denies show bgp, and the local account's own password is refused |
verdict=accept with a User-Password present, a CHAP-Password absent and ze's NAS-Identifier, then verdict=reject for the wrong password |
radius-admin-chap-freeradius |
The same, with auth-method chap against a Cleartext-Password entry |
verdict=accept with a CHAP-Password present and NO User-Password beside it, which is what RFC 2865 Section 4.1 demands of an Access-Request |
radius-admin-chap-hashed-freeradius |
The CHAP login is REFUSED, and ze authenticates the user through no backend at all | A radclient probe first proves the same entry accepts the same password over PAP, then verdict=reject for the CHAP request |
radius-admin-eap-freeradius |
The same, with auth-method eap-mschapv2. Ze answers the EAP conversation itself from the operator's password, Naks the server's MD5-Challenge toward MSCHAPv2, and returns the State on every later round |
verdict=state-echo for at least one round carrying the State the server issued, then verdict=accept, both with an EAP-Message present and NEITHER password attribute beside it |
radius-admin-eap-unsigned-freeradius |
On a server row with require-message-authenticator false, so the BlastRADIUS presence check stays out of the way and the RFC 3579 check is the one that runs: the EAP login through the relay twice. With the relay passing the answers, the login completes and ze_radius_unsigned_eap_discards_total reads 0. With the relay removing the Message-Authenticator, the login fails, ze logs radius: discarding the answer with the RFC 3579 Section 3.1 reason, and the counter is above 0 |
verdict=state-echo and verdict=accept for the passed login. The relay's own log records each answer it carried: an Access-Accept with the signature present, then an Access-Challenge with an EAP-Message and the signature stripped |
radius-admin-pap-unsigned-freeradius |
BlastRADIUS (CVE-2024-3596) on a default server row: the PAP login through the relay twice. With the relay passing the signed Access-Accept, the login completes. With the relay removing its Message-Authenticator, the login fails, ze logs radius: discarding an answer without Message-Authenticator, ze_radius_unsigned_response_discards_total is above 0, and the RADIUS login count does not move |
verdict=accept for both requests: the server accepted, and only ze refused. The relay records an Access-Accept with no EAP-Message and the signature present, then the same with the signature stripped |
radius-admin-pap-unsigned-legacy-freeradius |
The same stripped Access-Accept on a server row with require-message-authenticator false: the login completes with source=radius, and ze logs no BlastRADIUS discard |
verdict=accept, and the relay records the Access-Accept it carried with the signature stripped, so the login cannot have ridden a signed answer |
Both lab clients.conf files set require_message_authenticator = yes, so
FreeRADIUS discards any Access-Request without a valid Message-Authenticator.
Ze signs every Access-Request (draft-ietf-radext-deprecating-radius-10 Section
4.2), so every scenario in this suite and in the subscriber suite also proves
that FreeRADIUS verifies the signature ze computed. The red phase was run on
2026-10-28: with the insertMessageAuthenticatorFirst call in encodeRequest
skipped and the lab binary rebuilt, radius-admin-pap-freeradius failed at its
first assertion because FreeRADIUS dropped the unsigned Access-Request before
its request log recorded anything; with the call restored it passed.
The unsigned scenarios need an answer FreeRADIUS will not send. RFC 3579 Section 3.1:
Access-Challenge, Access-Accept, or Access-Reject packets including EAP-Message attribute(s) without a Message-Authenticator attribute SHOULD be silently discarded by the NAS.
FreeRADIUS 3.2.10 writes a Message-Authenticator as the first attribute of
every Access-Accept, Access-Reject and Access-Challenge it sends over UDP:
rad_encode in src/lib/radius.c decides it from its code2ma table, which no
configuration item reaches, so neither an unlang update reply nor a proxy
server removes it. The lab therefore mounts test/interop-radius/unsigned-relay.pl
into the server's container, and the checker starts it there on port 1912, the
port the scenario's ze.conf names. The relay forwards each Access-Request
unchanged to FreeRADIUS on 1812 and each answer back to ze. In strip mode it
removes the Message-Authenticator and nothing else; in both modes it recomputes
the Response Authenticator (RFC 2865 Section 3). The mode is a file the checker
writes, and a missing or unknown mode forwards nothing. The image carries Perl
and its Digest::MD5, and no other program that speaks UDP and MD5, which is
why the relay is Perl.
The pass login is the control for the three ways the refusal could be read
wrongly. It proves the relay carries the conversation, that its recomputed
Response Authenticator is right (a wrong one would fail it, since pass mode
recomputes too), and that the counter starts at 0. The strip login then
differs from it by the missing attribute alone. The counter is read from ze's
Prometheus endpoint on loopback, and a series the exposition does not carry is
an error, never a zero.
The scenario passed on 2026-10-23 and discriminates: with
verifyResponseMessageAuthenticator answering responseIntegrityAccepted for
an unsigned EAP answer, as it did before the discard, and ze rebuilt, it fails
with "assertion 6: ze completed an EAP login for radiusop through answers whose
Message-Authenticator the relay removed".
radius-admin-pap-unsigned-freeradius passed on 2026-10-28 and discriminates:
with the presence check in dispatchResponse disabled (if w.requireMessageAuthenticator { made if false && ...) and ze rebuilt, it
fails with "assertion 6: ze completed a PAP login for radiusop on an
Access-Accept whose Message-Authenticator the relay removed", and passes again
once the check is restored.
The third scenario is the one that proves docs/guide/radius.md is telling the
truth. RFC 2865 Section 2.2:
For example, CHAP requires that the user's password be available in cleartext to the server so that it can encrypt the CHAP challenge and compare that to the CHAP response. If the password is not available in cleartext to the RADIUS server then the server MUST send an Access-Reject to the client.
A rejection on its own would also follow from a typo in the user file, which is
the fourth vacuity trap wearing another face: the asserted result would be
produced by a different cause than the one under test. The radclient PAP probe
removes it, because the storage form is then the only thing left to explain the
CHAP rejection.
Two credentials share one username on purpose. radiusop exists in the server's
user file AND in ze's local account list with a DIFFERENT password, so the
scenario can send the local password while RADIUS rejects it: a chain that fell
through to local bcrypt would accept that login. localop exists only locally
and the server answers nothing for it, which is the positive control that proves
the SSH listener and the local backend are both live before any refusal is read
as evidence.
The FreeRADIUS subscriber suite
./le test integration interop-radius-subscriber puts FreeRADIUS 3.2.10, the
release the admin-login suite pins, in front of ze's subscriber RADIUS surface:
the accountant and the CoA and Disconnect listener of
internal/component/l2tp/plugins/authradius. Both act only on a kernel L2TP
session, which no fixture can stand in for, so the lab carries its own LAC
(xl2tpd and pppd), and ze and the LAC hold NET_ADMIN and /dev/ppp. That is
why it is a suite of its own: the admin-login suite promises no capability at
all, and the L2TP lab's RADIUS peer is a mock ze wrote. FreeRADIUS writes
every accounting record it accepts to its ze_detail file, and its radclient
is the Dynamic Authorization Client.
| Scenario | What it proves |
|---|---|
radius-coa-duplicate-freeradius |
RFC 5176 Section 2.3 duplicate detection. With ze paused, radclient sends one Disconnect-Request and retransmits it on its own socket: ze answers the copies from the cache, radclient gets a Disconnect-ACK, and ze logs one teardown. On a second session, radclient sends one Disconnect-Request and gets a Disconnect-ACK, then a bash process in the FreeRADIUS container sends the very octets ze received, read from the capture, out of a socket of its own, so from a new port: ze processes it afresh and answers Disconnect-NAK, where a key blind to the port would answer the cached ACK. A tcpdump capture in ze's container confirms the retransmitted copies share port, Identifier and Authenticator, and the two clients' requests are the same octets from two ports |
radius-acct-restart-freeradius |
RFC 2866 Section 5.5 across a real restart. The Accounting-Start before ze restarts and the one after carry ids of the form tunnel-session-tag-counter, the two boot tags differ, and each Stop carries its Start's id, all read from FreeRADIUS's detail file |
radius-coa-disconnect-freeradius |
Requires both kernel shaping rates to change after CoA-ACK; a mixed valid/unsupported request must NAK without changing either rate. A valid Disconnect must ACK and produce the matching Accounting-Stop |
l2tp-radius-framed-protocol-reject |
Changes only FreeRADIUS's Framed-Protocol for one user: PPP must establish traffic, SLIP must produce authorization refusal and CHAP failure without Accounting-Start, then PPP must work again |
radius-subscriber-dns |
RFC 2548 Sections 2.7.6 and 2.7.7 through the pool (spec-home-cpe-23b AC-9). The scenario's own users file has FreeRADIUS answer alice with MS-Primary-DNS-Server 198.51.100.53 and MS-Secondary-DNS-Server 198.51.100.54 (dictionary.microsoft, Microsoft attributes 28 and 29, moved into the reply by rlm_files' file_common), and its own pppd options add usepeerdns. The pool writes no dns-source and has its own dns-primary 203.0.113.53, so its built-in list (radius first) must offer the RADIUS pair: pppd's create_resolv writes exactly those two nameserver lines to, and the pool's address must not appear. Not yet run. /etc/ppp/resolv.conf |
The server listens as a stock deployment does, type = auth on 1812 and
type = acct on 1813 (site-default), and ze sends accounting to a server
entry's accounting-port, 1813 by default, as RFC 2866 Section 3 asks. The
pinned radclient has no -i, which is why the two-client half replays
radclient's own octets rather than running a second radclient. The LAC
dials once (redial = no in xl2tpd.conf): each new session comes from the
checker restarting the LAC, so the Accounting-Start a checker reads is the
session it asked for. Ze's container sets ze.log.l2tp.auth.radius=debug,
because the teardown count reads the plugin's Info line and the default level
is WARN.
The duplicate and restart scenarios passed on 2026-10-20, and each discriminates. With replayKey
writing port 0 into the key, so blind to the source port, the two-client half fails: "ze answered Code 41,
want a fresh Disconnect-NAK (42), not the first client's cached answer". With
genSessionID writing the old tunnel-session-counter form, the restart
scenario fails: "Acct-Session-Id "1702-33224-1" before the restart carries no
boot tag".
The two additional scenarios passed on native Ze KVM guests on 2026-10-27.
Their forced-red and restored-green proofs remain owed. The framing checker
uses radclient's Access-Accept as its reload barrier and retries HUP no sooner
than six seconds after the last signal: FreeRADIUS ignores HUP within five
seconds of the preceding reload.
After copying its initial users file, the framing peer starts through the
image's entrypoint. The pinned Debian image resolves radiusd to freeradius
there; a shell executing radiusd directly bypasses that translation.
The LAC sends pppd's diagnostic messages to descriptor 2: xl2tpd closes stdout
before starting pppd in kernel mode, while its foreground stderr reaches Docker
logs. This keeps the checker's CHAP-refusal observation connected to the peer.
Independent TFTP duplicate-ACK recovery
The Linux, Docker-tagged TestTFTPCurlDuplicateACKInterop uses digest-pinned
curl 8.23.0 through labtest.Run against Ze's production TFTP handlers.
A bounded UDP relay drops DATA 2 and replays curl's previous ACK 1. It requires
no immediate DATA response, timeout recovery and an exact 1,500-byte download.
Linux receive timestamps distinguish timeout recovery from an immediate
duplicate that sat unread in a socket queue; missing timestamps or an ACK replay
outside the fault window refuse the run. The relay neither constructs client
requests nor generates acknowledgments.
Results on 2026-11-14 (arm64, Ze's kernel 7.3.0-ze in the QEMU docker-lab
guest, curl 8.23.0 aarch64 pinned by digest): the real build passes, the relay
reporting "replayed native ACK 1 once; dropped first DATA 2; kernel arrival
interval 5.007927169s; forwarded final ACK 3; curl matched all 1500 bytes
(8 relay datagrams)". With windowSender.awaitACK returning on an ACK that is
not for the outstanding block, where it now keeps waiting (the behavior before
519d9f4f2a), FAIL: the relay reports "DATA in answer to duplicate ACK after
36.333µs, before five-second timeout", and curl exits 28 at its 35-second limit
with 512 bytes received. The real build run again right after passes (interval
5.004289378s).
On a Mac that run used the Docker daemon of Ze's own docker-host image. The
test binary is cross-built for linux/arm64 with go test -c under the feature
tags plus docker. ./le test qemu docker-lab with lab words boots the
image under HVF, forwards the guest's Docker socket to the host, and runs the
host le with those words and DOCKER_HOST naming the forwarded socket. The
words run the binary in a docker:cli container on the guest's host network
with the guest's Docker socket mounted, because the relay, Ze's server and the
curl container must share the guest's loopback, and the test needs Linux
receive timestamps. The guest mounts the checkout at its host path, so the
container bind-mounts the binary from the path it was built at. The docker-lab
command mode cannot run the test, because that mode starts no Docker daemon.
No ./le action selects one docker-tagged package test for this route yet:
./le test integration docker-peers runs every docker-tagged package, against
the Docker daemon the host reaches.
Ze's TFTP server against tftp-hpa and curl options
Two Linux, Docker-tagged tests in tftp_hpa_docker_test.go drive Ze's
production TFTP handlers (ServeConn) with independent clients, through a
relay that copies every datagram both ways and records the client's RRQ
options, the server's OACK and the longest run of DATA blocks between two
ACKs. The relay constructs nothing.
TestTFTPHPAWindowsizeInteropbuilds the tftp-hpa 7.2 client from its release tag (test/interop-tftp/Dockerfile.tftp-hpa, which refuses any other commit) and downloads a 300-block file twice: with the client's default request (windowsize 64) and with-W 1000. Each must be granted windowsize 64, be sent in windows (64 DATA blocks between ACKs), and match the file's SHA-256. WithgrantWindowsizeforced to one block the OACK carries no windowsize and the test goes red.TestTFTPCurlTimeoutInteropruns curl 8.23.0 with its default options, whose RRQ carriestimeout 5for--max-time 25. The OACK must echo that timeout and curl must deliver the exact file. With the timeout left out of the OACK the test goes red.
Both run on a Mac the way the dnsmasq test below does: the test binary is
cross-built with -tags docker and run in a docker:cli container on the
ze Colima VM's host network.
Ze's TFTP client against dnsmasq
The Linux, Docker-tagged TestTFTPClientDnsmasqInterop builds dnsmasq 2.91
from test/interop-tftp/Dockerfile.dnsmasq through labtest.Start and runs it
on the host network, because dnsmasq answers each transfer from a fresh port
(its TID) that a published port does not carry. Ze's client fetches a
116-block file, a file of exactly three 1468-octet blocks and an empty file,
and must negotiate blksize 1468 and tsize equal to the file size, fall back to
a window of one block (dnsmasq acknowledges no windowsize), and deliver the
exact bytes; the last fetch runs through the scheme table with a SHA-256. With
tftp.ParseOACK broken to grant nothing, the client refuses dnsmasq's
1468-octet block 1 with ERROR 4 and the test goes red. On a Mac, where Colima's
port forwarder does not reach the container's transfer port, the test binary is
cross-built for linux/arm64 with go test -c -tags docker and run in a
docker:cli container on the VM's host network with the Docker socket mounted.
The freeRouter RSVP-TE suite
internal/le/interoplab/rsvpte/ puts up to four nodes on one Docker segment,
172.29.81.0/24, in four roles: ingress (host 2), transit (3), egress
(4) and relay (5). A scenario directory decides which implementation fills
each role by the files it carries: <role>.conf makes the role a Ze node
answering on the container address .<host>, and
<role>-hw.txt with <role>-sw.txt makes it a freeRouter node answering on
its own address .<10+host>. A role with neither file is absent. Nodes start
downstream first, so the first PATH meets nodes that already listen. The
addresses are fixed because the configurations in each scenario directory name
them. Two shapes are used: freeRouter, Ze, freeRouter, where freeRouter
originates and Ze relays; and Ze, freeRouter, Ze (and Ze, freeRouter, Ze, Ze),
where Ze originates PATH, ResvErr, ResvTear, PathErr and strict hops and
freeRouter is the independent implementation that parses, relays and
re-encodes them. A Ze head-end tunnel in this suite requests fast-reroute,
because freeRouter's packRsvp.parseDatPatReq refuses a PATH without
SESSION_ATTRIBUTE and Ze emits that object only for a protected tunnel and a
bypass. A third shape puts Ze at the point of local repair: Ze ingress, Ze
PLR, freeRouter relay on the bypass, freeRouter egress as the merge point. Each freeRouter owns its own IPv4 stack
and MAC: its container command (freeRtrScript) makes the container's eth0
promiscuous and joins it to the jar through the upstream rawInt.bin, so the
suite needs Docker alone, never host root, a TAP device or a network namespace,
and no container runs privileged: a freeRouter holds NET_ADMIN and NET_RAW.
A Ze node holds NET_ADMIN to program MPLS labels and VLANs. A VLAN is a link
of the shared core ("Separate links between daemons"): labLinks holds each
scenario's interoplab.Link declarations, and the core's start wrapper builds
a member's devices before its container command runs. The one user is
plr-backup-path-to-egress-merge-point, whose protected link is VLAN 100 on
10.0.14.0/24 with the Ze PLR (transit, host 3, so 10.0.14.3) as its one
member under the device name prot0 (Devices), because transit.conf and
the checker name it, and the freeRouter egress as its External end: the
egress's rtr-sw.txt holds interface ethernet1.100 at 10.0.14.14, reached
over rawInt.bin, so its container gets no device. The node's container
command (zeNodeScript) adds the routes the role's optional
<role>-net.txt declares, one route <prefix> via <gateway> per line, and a
line of any other shape, a vlan line included, refuses the run. The command
writes its MPLS sysctls at run time, input on eth0 and on each link device
the wrapper built (linkDevices, from interoplab.LinkMemberships), so it
runs under ze-lab-net-sysctl with /proc/sys writable. No scenario carries
a shell script: the commands are built in Go from typed declarations. The preflight
reads /proc/sys/net/mpls/platform_labels with no grant, loads no module, and
refuses a host kernel without MPLS routing with the host command that loads it.
Every assertion reads what a peer received. Every container, Ze or freeRouter,
runs tcpdump -vvv on its eth0 into /run/fr/rsvp.txt, RSVP and every
MPLS-labelled frame, because a PATH carried through a bypass LSP is labelled.
The checker parses that text: a message Ze
logs as sent counts for nothing until the peer's capture holds it.
| Scenario | What the peers observe |
|---|---|
transit-loose-ero-expansion |
freeRouter ingress, Ze transit, freeRouter egress. The ingress names Ze and the egress loopback as loose hops and prepends its own next hop, Ze, as a strict subobject (ipFwdTab.fillRsvpFrst), so it signals [Ze strict, Ze loose, egress loopback loose]. Ze's native route to the loopback runs through .14, which no subobject names, so the PATH the egress captures carries .14 ahead of the still-loose loopback (RFC 3209 Section 4.3.4.1 steps 5 and 6). The ingress captures Ze's RESV with a label and an LSP_TUNNEL_IPv4 FILTER_SPEC naming the sender (RFC 3209 Section 4.6.3.1, class 10 C-Type 7, not a class 11 SENDER_TEMPLATE), and Ze's MPLS table holds a swap via .14 |
plr-backup-path-to-egress-merge-point |
Ze ingress, Ze PLR, freeRouter relay .15, freeRouter egress .14. The ingress asks facility protection for a tunnel whose protected hop is the PLR's prot0, VLAN 100 to the egress at 10.0.14.14; the PLR's configured bypass runs untagged through the relay and merges at that egress. Before the fault the egress holds no backup PATH. The checker takes prot0 down, and the egress captures, through the bypass, a PATH whose sender and RSVP_HOP are the PLR and whose ERO is 10.0.14.14 alone; the PLR captures at least two labelled RESVs from the egress naming its own sender and answers none with a ResvErr (RFC 4090 Sections 6.1 and 6.4). The egress answers from its eth0 address .14, not from 10.0.14.14, and Ze accepts an alternate merge-point source only when a live native IGP database attributes it to the same node, so the PLR and the egress run OSPF on eth0 (freeRouter router-id 10.0.14.14) and the checker waits for a Full adjacency, read through show ospf neighbor on the PLR, and then for the egress's router-LSA to carry its point-to-point link (48 octets or more in show ospf database router), before it takes prot0 down: the egress re-originates that LSA after the adjacency forms, and until it arrives nothing ties .14 to 10.0.14.14. After the cut the ingress keeps refreshing its PATH, whose ERO still names 10.0.14.14 strict across the failed link; the PLR relays it through the bypass, and the checker fails on a Routing Problem PathErr for the protected LSP-ID 1 within three refreshes (RFC 4090 Section 6.4.3). The Notify Tunnel locally repaired for LSP-ID 1 and a Routing Problem for a new LSP-ID the ingress signals over the same ERO are correct and do not count. Because a PLR that dropped the refresh in silence would send no PathErr either, the checker then waits until the protected PSB's lifetime has passed since the cut (10.5 seconds, RFC 2205 Section 3.7 with K = 3 and the ingress's 2-second refresh) and requires a further backup PATH at the egress and a further labelled RESV from the merge point at the PLR. freeRouter is the merge point only as an egress: it keys the backup PATH by sender and LSP-ID as a new LSP, and has no transit merge |
transit-strict-hop-forwarded |
Ze ingress, freeRouter relay .15, Ze transit, freeRouter egress .14. The Ze ingress names every hop strict. The PATH the egress captures from Ze carries the ERO shortened to strict .14 alone, the Ze ingress captures a labelled RESV relayed by freeRouter, and the Ze transit holds a swap via .14 |
transit-strict-hop-outside-refused |
Ze ingress, freeRouter relay, Ze transit. The last strict hop's native route at the transit runs through a node outside both abstract nodes, so the transit sends PathErr Routing Problem / Bad strict node (24/2) and never forwards the PATH (RFC 3209 Section 4.3.3.1). The Ze ingress captures that PathErr as freeRouter relays it, naming the transit as the error node |
ingress-resv-error-relayed |
Ze ingress, freeRouter relay, Ze egress. The ingress interface reserves less than the tunnel asks, so the ingress refuses the RESV freeRouter relays and sends a ResvErr naming itself, Error Code 1 Admission Control failure, value 2. The Ze egress captures that ResvErr from freeRouter with the error node, code and value intact (RFC 2205 Sections 2.5 and 3.1.8) |
transit-resv-tear-relayed |
Ze ingress, freeRouter relay, Ze transit, Ze egress. Once the LSP is up the egress is frozen with SIGSTOP, so the transit's reservation times out while its path state is refreshed; the transit sends a ResvTear upstream, and the Ze ingress captures it as freeRouter relays it (RFC 2205 Section 3.1.6) |
transit-resv-increase-refused-in-place |
freeRouter ingress, Ze transit, patched freeRouter egress. The ingress tunnel signals 10 Mbit/s (bandwidth 10000) and the transit's eth0 reserves 100 Mbit/s. The egress's second RESV, answering the ingress's first PATH refresh (freeRouter refreshes every 120 s), raises its reservation in place to 1 Gbit/s, same session, sender and LSP-ID. The transit captures that RESV, and the egress captures Ze's ResvErr naming the transit, Error Code 1 value 2, ERROR_SPEC flags 0x01 (InPlace). The transit's swap stays installed and the ingress never receives the raised rate (RFC 2205 Section 3.1.8) |
transit-ff-resv-unknown-sender |
freeRouter ingress, Ze transit, patched freeRouter egress. Every RESV the egress sends is fixed-filter, with a second flow descriptor naming LSP-ID+1 of the same sender, which never signalled a PATH. The egress captures one ResvErr from the transit naming only that LSP-ID, Error Code 4 No sender information, and no ResvErr naming the real sender. The ingress receives a labelled RESV naming only the real sender, and the transit holds its swap (RFC 2205 Section 3.1.8) |
egress-implicit-null-penultimate-pop |
freeRouter ingress, Ze transit, patched freeRouter egress. Every RESV the egress sends distributes Implicit NULL, label 3, so Ze is the penultimate hop. The transit captures that RESV, the ingress receives Ze's RESV carrying a label Ze allocated, not 3, and Ze's MPLS table pops that label toward .14 with no outgoing label: no as to (RFC 3032 Section 2.1) |
transit-resv-state-expires |
freeRouter ingress, Ze transit, freeRouter egress. Once the LSP is up and Ze holds its swap via .14, the checker freezes the egress's Java process with SIGSTOP, so no RESV refreshes Ze's reservation while the ingress keeps refreshing the path. The ingress captures a ResvTear from Ze, none of which existed before the freeze, and Ze's MPLS table drops the swap via .14 (RFC 2205 Sections 2.3 and 3.1.6). The freeRouter egress advertises its fixed 120-second refresh period in the RESV's TIME_VALUES, so Ze, with refresh-multiplier 3, times the reservation out about six minutes after the freeze, and the checker waits that long |
The checker compares the ERROR_SPEC code and value as the numbers tcpdump
prints in parentheses, because tcpdump names no code for every value: it prints
Admission Control failure as unknown (1).
The pinned freeRouter originates only PATH, PathTear and RESV. It relays
PathErr, ResvErr and ResvTear but never originates them. It encodes its
configured ERO hops as loose (clntMplsTeP2p.workDoer,
ipFwdTab.fillRsvpPack) and prepends one strict subobject for its own next
hop unless the first hop is already strict (ipFwdTab.fillRsvpFrst), and it
routes each hop without reading the strict bit (rtrRsvpIface.getHop), so it
does not enforce strict hops. It signals one fixed bandwidth for the life of
an LSP, and sends only shared-explicit RESVs naming its own sender. In every scenario
where Ze originates a message, freeRouter is the independent implementation
that parses it, keeps the state it needs to relay it, and re-encodes it toward
the next Ze node: the evidence is that freeRouter accepts and relays what Ze
originates, not that freeRouter enforces strict hops or originates ResvErr,
ResvTear or PathErr itself.
The image is built with one patch, test/interop-rsvpte/freertr/ze-interop-resv.patch,
because the pinned jar can neither raise a reservation in place, name a
sender it has no path for, nor distribute Implicit NULL as an egress
(rtrRsvpIface always sends the label it allocated). The patch adds three knobs to the RESV a freeRouter
egress originates, each off unless its environment variable is set, so every
scenario that sets neither runs the upstream behavior. A scenario sets them in
<role>-env.txt, one NAME=value per line, which the lab passes to that
freeRouter container and refuses when a line is not NAME=value.
FREERTR_ZE_RESV_RATE_AT=n:rate sends only the n-th RESV with its FLOWSPEC
rate (bytes per second) raised; FREERTR_ZE_RESV_FF_EXTRA_SENDER=1 makes every
RESV fixed-filter and appends a second flow descriptor (FLOWSPEC, FILTER_SPEC
with LSP-ID+1, LABEL); FREERTR_ZE_RESV_IMPLICIT_NULL=1 puts label 3 in the
LABEL object of every RESV in place of the allocated label.
Two scenarios depend on objects RFC 2205 makes optional and freeRouter's parser
requires. packRsvp.parseDatPatErr refuses a PathErr without an ADSPEC, which
transit-strict-hop-outside-refused relies on, and packRsvp.parseDatResTer
refuses a ResvTear without a FLOWSPEC, which transit-resv-tear-relayed relies
on. The PathErr sender descriptor is "<SENDER_TEMPLATE> <SENDER_TSPEC>
[ buildPathErr, buildReservationControl), so both
scenarios pass against the unpatched image. Before Ze carried them, freeRouter's
capture held Ze's message and freeRouter never relayed it.
The topology suite
The topology suite (internal/le/interoplab/topology) runs the scenarios no
single protocol suite owns: several Ze nodes, each with its own name and its
own configuration file, beside peers whose images other suites declare, on
named links (see "Separate links between daemons"). A scenario directory
under test/interop-topology/scenarios/ carries only the nodes'
configuration files. Its Go register file declares the nodes in start order
(name, registered image, host number, configuration file), the links and the
checker, and Discover joins the directory with that declaration as it does
for every suite, so a directory with no declaration is an error.
| Scenario | Nodes and links | What the checker proves |
|---|---|---|
ze-static-link-chain |
Ze nodes ze-a, ze-b, ze-c on links a-b (VLAN 181) and b-c (VLAN 182), B forwarding; outside on eth0 alone, running only tcpdump on ip net 10.181.1.0/24 or arp net 10.181.1.0/24. A ping from outside to a link address proves nothing: Linux answers ARP for any local address on any interface (arp_ignore 0), so ze-b answers 10.181.1.3 from its own eth0. The checker flushes ze-a's neighbours on the link so its ARP for ze-b, a tagged broadcast, reaches outside's port. Each node's file carries interface { backend netlink; }: with no interface block the interface component loads no backend, and show interface brief answers iface: no backend loaded. 10.181.1.0/2410.181.2.0/24 |
each node answers show interface brief by name; A pings C through B; outside's capture (tcpdump -e) holds a frame of link a-b with its 802.1Q tag, and none without it |
two-ze-pppoe-link |
a Ze PPPoE client and a Ze PPPoE server on a link with no prefix; an observer on eth0 alone |
the client holds an address from the server's pool on a ppp device (assigned only once IPCP is Opened), the server's kernel PPPoE table holds the session on the link device, and the observer received no untagged PPPoE discovery frame. Its capture filter also matches the link's own tagged frames, because veth hands the tag to the packet socket out of band, so the checker reads the capture with tcpdump -e and counts only lines without an 802.1Q header |
pppoe-mss-pmtu |
Ze CPE ze-cpe dialing accel-ppp accel on link access (VLAN 184, no prefix) with MTU 1492; lan-host behind Ze on link lan (VLAN 185); server beyond accel-ppp on link core (VLAN 186), capturing SYNs. 10.185.1.0/2410.186.1.0/24 |
Ze's pppoe0 holds an IPCP address and reports MTU 1492, its table lists tcp option maxseg size set rt mtu, and the server captures the SYN lan-host offered with MSS 1460 as 1452 and the one offered with 1200 as 1200 |
access-up |
the access chain (below): Ze CPE ze-cpe, running the home-CPE lab's converted configuration with one lab addition (subnet br0, with no dns-server and no dns-source, reserving 10.77.0.50 for lan-res-c), on link access (VLAN 191, no prefix) and on link lan (VLAN 197, no prefix) with lan-client and lan-res-c (dhcpcd), to Ze LAC ze-lac, which relays the empty Service-Name over link core (VLAN 192) to Ze LNS ze-lns, which checks CHAP with freeradius on the same link. 10.77.0.0/24192.0.2.0/24 |
the LAC's access capture holds a PADO with a zero-length Service-Name and a PADS with a non-zero session; the LAC holds one tunnel to the LNS with one session, the kernel PPPoE session on its access device, and no addressed PPP device; FreeRADIUS logged an Accept for cpe@lab.example.net with a CHAP-Password, no User-Password, NAS-Identifier ze-lns and NAS-IP-Address and source 192.0.2.3; the LNS session names the user and holds 203.0.113.1, its detail reports mru 1492 and auth-method chap-md5, its core capture holds its LCP Configure-Request with MRU 1492; the CPE's pppoe0 holds 203.0.113.1 with MTU 1492, its show interface pppoe-client reports pppoe0 up with dns-servers 192.0.2.53 then 192.0.2.54, the two servers the LNS offered in IPCP, while its resolv.conf holds only the converted configuration's static system name-server 192.0.2.53, because a static server wins over every learned one; lan-res-c then binds 10.77.0.50 and its dhcpcd --dumplease holds domain_name_servers exactly 192.0.2.53 192.0.2.54, the IPCP pair in order, which only the subnet default reaching its received class hands out (the host fallback alone gives 192.0.2.53); the CPE's eth0 holds no address, its route to 100.64.0.1 leaves on pppoe0, it pings that address with no loss, and the LNS's PPP device toward the subscriber received at least the pings sent |
access-pool |
the access chain, with FreeRADIUS answering cpe@lab.example.net with no Framed-IP-Address |
the CPE's pppoe0 holds an address in, the LNS session holds the same one, and show l2tp pool reports one address allocated, to that session. 100.64.0.0/24 |
access-accounting |
the access chain, with the LNS's acct-interval 60 |
FreeRADIUS's ze_detail holds the session's Start, an Interim-Update and, once the CPE is stopped (its PPPoE client sends PADT), the Stop, all under one Acct-Session-Id and carrying 203.0.113.1, the Stop counting at least the 20 560 octets a 20-packet ping of 1000-octet payloads carried each way |
access-wholesale |
the access chain with the scenario's own LAC, which relays the realm lab.example.net (relay-realm) and so hands the LNS the proxy LCP AVPs of RFC 2661 Section 4.4.5, and an LNS set proxy-lcp renegotiate |
the CPE's pppoe0 holds 203.0.113.1; FreeRADIUS logged the LNS's CHAP Accept for cpe@lab.example.net; the LNS session detail reports mru 1492, auth-method chap-md5 and proxy-lcp renegotiated; the LNS's core capture holds its own LCP Configure-Request with MRU 1492, which an LNS that accepted the proxy never sends; the subscriber pings across |
l2tp-lns-lcp-renegotiate-pppd |
access-wholesale with pppd 2.5.5 (rp-pppoe plugin, the PPPoE suite's ze-pppoe-client image) as subscriber pppd@lab.example.net |
pppd's ppp0 holds 203.0.113.1; the LNS session reports proxy-lcp renegotiated and chap-md5; the LNS's core capture holds its LCP Configure-Request; pppd's debug log holds two received LCP Configure-Requests and two sent Configure-Acks (the LAC's negotiation and the LNS's renegotiation, RFC 1661 Section 4.3), a CHAP success and the IPCP address; pppd pings 192.0.2.3 over the default route defaultroute puts on ppp0 (its eth0 is flushed, so nothing else reaches the core) |
conversion |
the home-CPE lab (below) with lan-client on the CPE's bridge |
conversion-record.md names every line of vyos.conf with its own text and a Ze form or a dropped reason; every scenario whose ze-cpe node runs mounts the rendered conversion (only pppoe-mss-pmtu is exempt); ze config validate passes on the file the CPE mounted; the CPE's pppoe0 holds 203.0.113.1 and the kernel holds br0, wg0 and vti0 |
lan-services |
the home-CPE lab with lan-client, lan-res-a and lan-res-b (dhcpcd), lan-host (static 192.168.50.10) and web |
lan-client binds inside 192.168.50.100-199; lan-res-a binds 192.168.50.30 and lan-res-b 10.99.0.40 (the later subnet); web captures lan-host's fetch from 203.0.113.1; SYNs offered with MSS 1460 and 1300 arrive as 1400 and 1300; web gets no answer from while the CPE's default-drop counter advances, and lan-host completes TCP to; an active-mode FTP download completes. 203.0.113.1:22192.168.50.1:22 |
road-warrior |
the home-CPE lab with lan-res-a, web and wg-client |
the CPE's wg show wg0 dump lists port 51820 and the laptop's key with a preshared key and wg-client holds a handshake, pings lan-res-a and fetches from web, which captures the SYN from 203.0.113.1. 10.88.0.2/32 |
mgmt-tunnel |
the home-CPE lab with lan-client, strongswan (vpn.example.net) and a dns gated shut |
✕ |
mgmt-tunnel-ed25519 |
mgmt-tunnel with the lab issuing the CPE an Ed25519 key (PKCS #8) under the same P-256 CA |
every mgmt-tunnel clause, and strongSwan's log holds authentication of 'cpe.lab.example.net' with ED25519 successful, the line pubkey_authenticator.c process() writes only after the signature verified, so Ze signed its IKE AUTH with Ed25519 under RFC 8420 |
access-wholesale-accel-lns |
the access chain with access-wholesale's realm-relaying LAC, accel-ppp 1.14.0 as the LNS on link core at 192.0.2.3 in place of ze-lns, checking CHAP from its own chap-secrets, and no FreeRADIUS |
the CPE's pppoe0 holds 203.0.113.1, the address accel-ppp's chap-secrets names; accel-ppp's verbose log holds an LCP Configure-Request and a Configure-Ack it received (the CPE, Opened with the LAC, renegotiated when accel-ppp, which skips the proxy LCP AVPs in l2tp_recv_ICCN, sent its own request), a CHAP Response naming cpe@lab.example.net and its CHAP Success; the CPE pings 100.64.0.1 across and accel-ppp's PPP device received at least the pings sent |
l2tp-lac-realm-relay-accel-ppp |
spec-home-cpe-14's realm path: pppd 2.5.5 subscribers pppd (pppd@lab.example.net) and pppd-stray (pppd@other.example.net) on link access (VLAN 191, no prefix) to Ze LAC ze-lac, which relays the realm lab.example.net (relay-realm) over link core (VLAN 192) to accel-ppp 1.14.0 at 192.0.2.3, whose chap-secrets holds pppd only; no FreeRADIUS. 192.0.2.0/24 |
pppd's ppp0 holds 203.0.113.1, the address accel-ppp's chap-secrets names; the LAC holds one L2TP session, reporting relay-realm lab.example.net, relay-remote lns, proxy-lcp-sent and proxy-auth-sent true; accel-ppp's verbose log holds an LCP Configure-Request and a Configure-Ack it received, a CHAP Response naming pppd@lab.example.net and its CHAP Success (it skips the proxy AVPs in l2tp_recv_ICCN and renegotiates); pppd pings 100.64.0.1 across; pppd-stray's log holds the LAC's own CHAP Failure no users configured (the local auth handler, with no user configured) and no CHAP success, and accel-ppp's log never names it |
l2tp-lac-realm-relay-xl2tpd |
l2tp-lac-realm-relay-accel-ppp with xl2tpd 1.3.20 and pppd 2.5.5 as the LNS (xl2tpd-lns, the RADIUS subscriber suite's ze-radius-subscriber-lac image), answering the LAC's tunnel challenge from l2tp-secrets and checking CHAP from its pppd's chap-secrets, which holds pppd only |
pppd's ppp0 holds 203.0.113.1, handed by xl2tpd's ip range; the LAC session reports the same four keys; xl2tpd logs Call established with 192.0.2.2 (control_finish, on the ICCN, whose proxy AVPs it ignores); the LNS pppd's debug log holds a received LCP Configure-Request and Configure-Ack, a CHAP Response naming pppd@lab.example.net, the CHAP Success it sent and remote IP address 203.0.113.1; pppd pings 100.64.0.1; pppd-stray is refused by the LAC as above and the LNS pppd's log never names it |
pppoe-dhcpv6-pd-ze-client |
a Ze CPE with its own file (pppoe0, ipv6 with accept-ra 2, autoconf, forwarding and dhcpv6 pd length 56; ethernet lan0 unit 0 with forwarding, delegated-prefix pppoe0 subnet-id 1 and router-advertisement at intervals 3 to 4 seconds and router-lifetime 1800) dialing accel-ppp 1.14.0 on link access (VLAN 198, no prefix), accel-ppp running IPv6CP, ipv6_nd and ipv6_dhcp with a /60 delegate pool and on its loopback; slaac-host (the Knot image's dhcpcd 10.5.2, kernel Router Discovery and SLAAC off) on link delegated-lan (VLAN 199, no prefix), which the CPE reaches as lan0; the CPE records every RA it sends on lan0 from its prestart. 2001:db8:ffff::1 |
spec-home-cpe-8. Phases 1 to 3: pppoe0 holds one fe80::/64 from a non-zero identifier and no pppN holds one; pppoe0's accept_ra is 2, forwarding and autoconf 1; ::/0 goes via (accel-ppp's ipv6-intf-id) on pppoe0; pppoe0 holds a SLAAC /64 from; accel-ppp's log holds a Solicit carrying IA-PD with the hint ::/56, then either a Request for a /60 from and the Reply delegating it, or, when the Solicit carried Rapid Commit, accel-ppp's Rapid Commit Reply delegating a /60 from that pool and a later Renew or Rebind from the CPE naming that exact prefix (RFC 7084 WPD-3). The two-message exchange sends no Request, so the CPE's Renew is its word that it kept the prefix: RFC 9915 Section 18.2.4 forbids a client to include a prefix it did not obtain, and Ze's client renews with the ::/56 hint when it holds none. accel-ppp's pref-lifetime 60 puts T1 at 30 seconds, inside the checker's wait. Phases 4 and 5 (checkDelegatedLAN): lan0 holds the delegated /60's subnet 1 with, its lifetimes finite and no longer than accel-ppp's 120 and 60 seconds; the CPE's own forwarding units hold net.ipv6.conf.all.forwarding at 1 (Linux forwards no IPv6 packet without it, and the checker writes no sysctl, so a Ze that leaves it off fails the ping), and the checker starts dhcpcd, which holds a SLAAC address in that /64, ::/0 via lan0's link-local and a route to the /60 from the RIO, and pings across the CPE. Every captured RA carrying the /64 is judged against the accel-ppp Reply before it: the PIO's valid and preferred lifetimes and the RIO's lifetime are no longer than what remained of the lease, a second and a half of clock rounding allowed, the RA carries the RIO and Router Lifetime 1800, and at least one such RA left ten seconds or more after its Reply, so lifetimes that never count down fail (red walk: the RFC 9915 Section 6.3 bound removed). The checker then sets accel-ppp's PADO-delay to -1, so the CPE cannot dial again, and runs accel-cmd terminate username alice: the first RA with Router Lifetime 0 leaves within five seconds of the termination, no later RA carries more (red walk: the W input forced true), and dhcpcd drops ::/0 via lan0 and logs no longer a default router (lifetime = 0) (RFC 7084 G-5). The client writes the three sysctls on each new session device at run time, through the sysctl component, so the CPE runs as a run-time net sysctl writer (netSysctlWriter: /proc/sys writable under the net-only AppArmor profile) and the checker polls each value. fe80::12001:db8:198::/482001:db8:2000::/40::12001:db8:ffff::1 |
access-reconnect |
the home-CPE lab with lan-host, web, strongswan and dns, FreeRADIUS answering from a writable copy of its users file |
after FreeRADIUS is changed to answer 203.0.113.2, restarted, and the LNS clears the session, the CPE holds 203.0.113.2, strongSwan's SA sits on it, ESP advances both ways, and web captures lan-host's fetch from it |
The home-CPE lab (spec-home-cpe-11) runs the CPE every home-CPE scenario
shares, as a run-time net sysctl writer the way a deployed CPE writes its
devices' sysctls (netSysctlWriter): test/interop-topology/cpe/ze-cpe.conf, the model's conversion of
the invented vyos.conf with conversion-record.md beside it. The file holds
placeholders, never values: <<SECRET:name>> (filled from lab-values) and
<<LAB:name>> (issued by the run: a fresh P-256 CA with path length 0, the
vpn.example.net certificate with a P-256 key, the cpe.lab.example.net
certificate with the key algorithm the scenario declares, P-256 in SEC 1 or
Ed25519 in PKCS #8 for mgmt-tunnel-ed25519, and the laptop's X25519 public
key derived from its lab private key). The scenario's prepare
step renders it into the session scratch before any container starts, and
the CPE mounts the rendered file, so no scenario carries a CPE file of its
own: the access chain's scenarios below run it too. A scenario that needs a case the VyOS source cannot state inserts lab-only text into the rendered file at an anchor the file holds once (cpeAddition, home_lab_render.go), so ze-cpe.conf and its record stay the conversion: access-up adds a subnet left on the dns-source default, because a VyOS subnet with no option name-server converts to source configured. The lab adds a lan link (VLAN 194, no prefix: the CPE numbers it,
br0 on eth2) and an internet link (VLAN 196, 198.51.100.0/24) the LNS
routes, beside the access chain's access (VLAN 193) and core (VLAN 195);
dns (Knot, 192.0.2.53) answers vpn.example.net. The prepare step also
writes a resolv.conf naming the converted configuration's static
system name-server, which the CPE mounts at /etc/resolv.conf: on the
appliance that path links to /tmp/resolv.conf, the file Ze writes, so Go's
resolver (the one IKE uses for the remote-address name) asks the configured
server, while Docker's own file names only 127.0.0.11, and Ze's write by
rename cannot be followed by a bind mount. access-up proves Ze's own file
names the same single server. The converted configuration sets
vpn ipsec unmatched bypass, because VyOS installs no SPD catch-all and Ze's
defaults to discard, which would drop the CPE's own DNS and ping in the
clear. It names no application-override, so the default deny attaches
Ze's setsockopt guard, and the CPE holds BPF beside NET_ADMIN
(capabilityBPF, internal/le/interoplab/topology/topology.go): without it
the guard's map cannot be created and ze exits at startup. It also holds the
host's /dev/net/tun beside /dev/ppp (convertedCPE,
register_home_lab.go): a Child SA that receives UDP-encapsulated ESP makes
ze create the ze-espform TUN device, whose sysctls it writes under the same
run-time net sysctl grant. A lan-client dhcpcd node starts with
net.ipv4.conf.default.promote_secondaries=1, because dhcpcd's if_init
writes that sysctl for its interface when it reads anything else, and Docker's
/proc/sys is read-only. Every node beyond the
LNS flushes its eth0, so the links are its only path. The web and FTP
server and lan-host run ze-topology-webftp (Debian, busybox httpd,
vsftpd active-only, curl, bash), registered from
test/interop-topology/home-lab/Dockerfile.webftp; wg-client runs
ze-topology-wireguard, registered from the deployment proof's
Dockerfile.wireguard.
The config corpus renders the converted CPE through the same issuer and placeholder substitution as the topology lab. Both readers use the lab values and fresh certificate material. The corpus validates that rendered config, rather than accepting or excluding its unresolved placeholders.
The access chain is the first end-to-end proof that a subscriber's PPP
session crosses Ze's LAC (relay "") and terminates on Ze's LNS: the LAC
terminates nothing, so LCP, CHAP and IPCP run between the CPE and the LNS
across the LAC's kernel channel bridge. Every value is invented: core
192.0.2.0/24, subscriber 203.0.113.1/32, pool 100.64.0.0/24, realm
lab.example.net. The three scenarios share one declaration
(accessChain) and the files under test/interop-topology/access-chain/:
the LAC configuration, which no scenario changes, and FreeRADIUS's
clients.conf, a site-default that adds a ze_auth_log line per answered
login (mods-ze-auth-log: the presence of a User-Password and a
CHAP-Password, the NAS-Identifier, the NAS-IP-Address and the request's
source), and the subscriber lab's ze_detail module mounted unchanged. Each
scenario directory carries what differs: the LNS configuration and
FreeRADIUS's users. The CPE is the home-CPE lab's (convertedCPE, with
the same prepare step), and since its bridge holds eth2 it also joins link
lan (VLAN 197) with lan-client, a link needing two members;
l2tp-lns-lcp-renegotiate-pppd dials with pppd instead and has no lan, and the two l2tp-lac-realm-relay-* scenarios dial with two pppd subscribers to a third-party LNS and run no FreeRADIUS. FreeRADIUS joins the core link, so it runs
ze-topology-freeradius, the pinned release built with iproute2 for the
link's start wrapper. Each clause reads a different producer (vacuity trap
5), and the CPE's prestart flushes its eth0, so the session is its only path
to the LNS (trap 4). The LAC and the LNS start their captures in their
prestart, before their daemon, and the LNS's and FreeRADIUS's readiness gate
the next node's start. The interim waits the full 60 seconds: RFC 2869
Section 5.16 says of Acct-Interim-Interval "The value MUST NOT be smaller
than 60.", Ze clamps a received value to that floor and the acct-interval
leaf's range starts there. The scenarios need a Docker host running Ze's
kernel, with PPPoE, L2TP and PPPoL2TP (spec-docker-hosts-run-the-ze-kernel);
the core's kernel check refuses another.
./le test integration interop-topology runs every scenario;
./le test integration interop-topology scenario <name>, or
TOPOLOGY_INTEROP_SCENARIO=<name>, runs one. A checker that asks a Ze
node's CLI does so through interoplab.ZeCLICommand, the same argv the IPsec
lab sends, and that node's configuration serves the SSH account on
127.0.0.1 port 2222. A node whose configuration is not the lab's own names
its account with interoplab.ZeCLICommandAs: the home CPE's converted
configuration serves user admin on port 22, its password read from
test/interop-topology/cpe/lab-values, and access-up asks it
show interface pppoe-client that way. The CPE's input ruleset admits SSH
only from its LAN, so that login dials the CPE's LAN gateway 192.168.50.1
rather than loopback, and the builder sets ze.ssh.insecure for a
non-loopback host: the SSH client trusts an unknown host key only on
loopback, the lab seeds none, and the session still never leaves the
container. The access chain's checkers read the
LAC and the LNS through their REST surface instead (POST /api/v1/execute on 127.0.0.1 port 17012,
the api-server block its configurations carry), which answers the
show l2tp commands as JSON.
Images are registered by name in the core. Each suite registers the images it
builds from init() in its register_images.go (interoplab.RegisterImage),
under a name no other suite uses (the image's tag), and derives its own list
from the registry (SuiteImages, or ResolveImages where it also runs a
core registration such as interoplab.FreeRADIUSImageName). A topology node
names any registered image, and the suite resolves the names its plans use
once each, so an image another suite declares is built from that one
declaration. A name nobody registered is refused before any build, naming
the registered images; two registrations of one name panic at init, naming
both owners. FRR is not registered: its reference comes from each run's
environment.
Typed checker operations
checkers.go is the complete scenario catalogue. Each operation identifies the
peer, exact command, required or forbidden evidence, proof for negative
assertions, and a bound. check_engine.go executes those operations through the
shared CheckerLab interface. FRR, BIRD, GoBGP, Ze, speaker logs, and kernel
state are all queried through explicit typed branches.
An absent value never proves a negative assertion by itself. The operation must also name positive evidence that the query mechanism ran. Failed and empty queries remain errors rather than becoming plausible empty protocol state.
That rule decides which command a negative assertion sends. opBIRDRouteAbsent
reads the whole table with birdc show route, and never show route for <prefix>: BIRD answers a lookup for a network it does not hold with "Network
not found", and birdc exits 1. A lookup therefore fails in the exact state the
assertion exists to observe, and the failure is not absence. Ask a question the
peer answers in both states, then read the absence out of the answer.
A checker that reads a BGP attribute from BIRD reads BIRD 3's spelling. birdc show route ... all prints each attribute on its own line as a tab, the
attribute's name, a colon, a space and the value (ea_show, nest/rt-attr.c at
v3.4.0), and the names are proto/bgp/attrs.c's bgp_origin, bgp_path,
bgp_next_hop, bgp_med, bgp_local_pref, bgp_community,
bgp_large_community, bgp_originator_id and bgp_cluster_list. BIRD 2's
BGP.as_path family and its Type: BGP univ line are gone; BIRD 3 prints
preference:, from: and source: lines instead, and closes each route with
Internal route handling values:. An AS path is space-separated ASNs and a
community list is (65001,100) (65001,200), as in BIRD 2. Captured from the lab
image on 2026-10-23, for an iBGP route reflected once:
10.77.0.0/24 unicast [b 17:15:20.935 from 10.99.0.3] * (100) [AS65020i]
via 10.99.0.2 on eth0
preference: 100
from: 10.99.0.3
source: BGP
bgp_origin: IGP
bgp_path: 65010 65020
bgp_next_hop: 10.99.0.2
bgp_med: 77
bgp_local_pref: 250
bgp_community: (65001,100) (65001,200)
bgp_originator_id: 10.99.0.2
bgp_cluster_list: 10.99.0.3
bgp_large_community: (65001, 0, 1) (65001, 2, 3)
Internal route handling values: 0L 2G 0S id 1
Scenarios with non-linear behavior register a bespoke checker in
specialCheckers (check_special.go). Each one owes a named subtest in
TestBespokeCheckerBranches that drives its predicate in BOTH polarities: the
true case, and a false case written against one stated wrong reading, such as
two tokens matched across two log lines, a peer-originated event passing as a
received one, or an absence with no proof that the query ran.
Raw frame injection from inside a peer container
A checker that must put a hand-built Ethernet frame on a peer's wire, as that
peer, sends it with interoplab.SendFrameInContainer. The helper writes the
frame as a one-record pcap inside the named container and runs tcpreplay
there through docker exec, so the image MUST carry tcpreplay. Ze's BGP lab
image does, for the isis-purge-reorig-frr own-LSP purge, and so does the
PPPoE client image, for pppoe-padr-replay, pppoe-pap-ze-ac and the IPv6CP
replays.
The send runs in the container, never from the checker process. On a rootful
Docker daemon a container's network namespace belongs to root: an unprivileged
checker cannot open /proc/<pid>/ns/net, because the kernel's ptrace access
check refuses another user's process, and could not setns into it without
CAP_SYS_ADMIN. docker exec already runs inside the namespace and needs
neither.
The octets FRR received from Ze
When FRR prints nothing for the attribute a check is about, read the bytes
instead. A scenario's frr.conf names dump bgp updates <file>, and FRR then
writes every UPDATE it receives to that file as MRT, octet for octet as Ze sent
it. The check runs od -An -v -tx1 <file> | tr -d ' \n' in the FRR container
and asserts on the hex. bgp-prefix-sid-malformed-frr does this for the
Originator SRGB TLV, which FRR parses and never shows. Its absence assertion
names two proofs: the NLRI of the UPDATE it judges, so that UPDATE is in the
capture, and a TLV Ze relays unchanged in another UPDATE, so the capture holds
attribute bytes at all.
Querying Ze
Ze.cli(command) is the only way to ask the Ze daemon anything. It runs
ze cli -c <command> --user ... --format json inside the container.
Two properties of that line are load-bearing and neither is obvious.
ze cli -c rather than the verb form ze show bgp rib status: --user and
--format are flags of ze cli, and the verb form has no slot for either.
The daemon starts an SSH listener only when its config asks for one
(infraSetup, cmd/ze/hub/infra_setup.go), and ze cli reaches the daemon over
SSH. No scenario ze.conf asks. The harness appends ZE_CLI_CONFIG -- the
listener plus the account it authenticates against -- to the RENDERED copy of
every ze.conf (renderScenario), so no scenario carries the boilerplate and
none can forget it. The native IPsec plan appends the same blocks in
renderZeConfig (internal/le/interoplab/ipsec/ipsec.go).
Rendered configurations stay under tmp/interop-rendered in the checkout.
Docker must share the checkout with its Linux VM. An unshared system-temp path
gives the container a directory instead of the configuration file.
IPsec and L2TP create private directories there even without an AI session ID.
A Ze helper never converts a failed query into a plausible number.
Ze.rib_count raises when the command fails or answers without a routes-in
field, because 0 is a legitimate RIB size and a failed query is not
(ai/rules/evidence.md). It returned 0 on failure until 2026-08-07 and three
separate faults hid behind that one number for three days
(spec-fixit-test-harness-fail-open-guards, guard 3). Write new Ze
helpers the same way.
A checker asserts on the STRUCTURED answer, never on the rendered text. The
IPsec suite asks with show vpn ipsec sa | json and decodes the list of SA
records (zeIKESAs, internal/le/interoplab/ipsec/helpers.go). The text
rendering is a table, and its column order follows the field names, so a
line-anchored <field> <value> regex over that table matches by accident. On
2026-09-06 one new field, behind-nat, took first place from child-sa in that
order. Every nested Child SA key moved off the line start, and
child-rekey-narrowing, peer-reload-narrowing and
initiator-rekey-answer-narrows went red with no daemon behavior changed.
The VPN withdrawal checker receives its phase from private named constants,
not from FRR output. An unknown internal phase is a BUG assertion; malformed
or unsupported external table data remains an ordinary rejected observation.
A number two readers print in two bases is normalized to one TYPE, and the
decode is typed. The dataplane-readback scenario joins the SPI set
show vpn ipsec dataplane sa | json answers against the set ip xfrm state
prints in the same container. iproute2 writes an SPI as 0xc1a2b3c4 and Ze
answers a JSON number, so a string comparison is false for every SPI. The
checker decodes the Ze answer into a struct whose spi field is a uint32, and
parses the printed form to the same type. Decoding into map[string]any would
route the number through float64, which holds a uint32 today and says nothing
about the uint64 byte counter beside it.
An agreement between two readers asserts NON-EMPTY before it asserts EQUAL.
Two empty sets are equal, so a read-only comparison passes over a kernel that
holds nothing, which is what the dump answers with its whole body deleted.
requireSameSPISet refuses an empty side first and names which side was empty.
requireSPISetChanged refuses any old SPI that remains after rekey. The checker
polls through the bounded old-SA cleanup window until Ze and iproute2 agree on a
nonempty replacement set, and checks strongSwan's kernel against that set too.
RFC 7296 Section 2.8 permits overlap during rekey; overlap after cleanup fails.
The same scenario reads all four Child SA byte and packet counters as uint64
values and compares them with the directed lifetime-current records from
ip -s xfrm state. It samples before and after bidirectional traffic and requires
each counter to advance. A bounded one-way probe makes the directions differ,
so swapping inbound and outbound counters cannot pass on a symmetric ping.
Dataplane observation also crosses the live operator surfaces. With the
Prometheus exporter enabled, the checker scrapes the installed-SA count and
clean drift gauge, deletes one kernel SA, and requires CLI drift, degraded
show health, and the corresponding gauge transition. It restores the kernel
state and waits for recovery. Removing the configured peer then removes both
the peer and interface-label series.
An unreadable observation is induced inside the Ze scenario container by
temporarily setting only the daemon's soft RLIMIT_NOFILE to zero. The netlink
package's default handle opens a socket for each XFRM dump. A Python helper
finds the daemon as the one ze process whose parent pid in /proc/<pid>/stat
is tini, because /proc/<pid>/task/<tid>/children exists only with
CONFIG_PROC_CHILDREN, which Ze's runtime kernel does not set. The helper
opens and warms an HTTP connection before changing the limit, scrapes that same
connection until both dataplane metric families have no series, and restores
the original limit in finally. Reconnection is refused during the fault.
During the fault, live health must identify an unknown or unreadable dataplane;
the preceding known-drift diagnosis cannot satisfy that assertion. Restoring
the limit must restore both the drift diagnosis and the published drift gauges.
The checker requires published series before the fault and their return after
restoration, so an exporter that never publishes cannot satisfy the test.
This helper needs Python in the lab image; it changes no production backend.
All session waiters use explicit bounds (default 90 seconds, override via
SESSION_TIMEOUT). The harness passes that value into the Ze container so a
compiled process helper can size its barriers against the same budget.
A compiled process helper that fails
Scenario process personalities use the Go plugin SDK. A helper failure writes
the ZE-OBSERVER-FAIL sentinel to stderr and requests daemon shutdown. The
checker failure path reads the last 2,000 Ze log lines and appends that measured
cause when the sentinel is present.
An unreadable log is not a plugin verdict. checkerFailure retains the original
scenario assertion when the log read fails, so a Docker diagnostic cannot
replace the protocol failure that triggered it.
Ordered-checker failures include the selected IPv4 and IPv6 networks. A JSON wait retains its last answer, and a pmacct failure captures the collector log plus the last 80 msglog rows before container teardown. These diagnostics retain the original assertion failure; they do not substitute a guessed route state.
The external SDK carrier srv6-service-export-control announces two VPN
services with different route targets and SRv6 SIDs. A destination's export
policy withholds RT 10 while permitting RT 20; the unfiltered control peer must
receive both SIDs. Both peers remain connected while the observer checks the
socket-write counters, and the filtered peer continues rejecting RT 10 after
its initial End-of-RIB. These are BGP export observations, not SRv6 dataplane
reachability evidence.
The BMP Route Monitoring collector also waits past initial End-of-RIB messages. It completes only after a received-direction UPDATE carries the test prefix in its announced IPv4 NLRI. A matching byte sequence in attributes, a withdrawal, or an Adj-RIB-Out report cannot establish reception of that route.
Conditional MED observed by a foreign collector
bgp-addpath-best-path-pmacct retains its two-identifier baseline and adds a
separate prefix, 10.99.77.0/24, with three sources. FRR sends A from AS65004
with MED0 and Router ID198.51.100.3. GoBGP sends B from AS65005 with MED50 and
Router ID198.51.100.2. The existing injector sends C from AS65004 with MED100
and Router ID198.51.100.1. All three are external, have one AS in their path,
and carry ORIGIN IGP. The checker reads the exact received attributes from
pmacct's latest Adj-RIB-In route, including the absence of AIGP and received
LOCAL_PREF; Ze explicitly configures their common preference base. Router ID is
carried by Peer Up, not Route Monitoring. The checker joins the route to its
preceding Peer Up using the BMP router, connection port, peer address, RIB view
and increasing collector sequence, and requires the expected ASN in both rows.
A Peer Down, a new Peer Up or a BMP stream restart invalidates the old route.
An earlier correct identifier cannot satisfy a route from a different epoch.
A removes C at MED, then B defeats A at Router ID. The checker requires B, withdraws non-best A through FRR, requires C, restores A, and requires B again. Before each change it checkpoints the collector's append-only log. The selection query keeps pre-checkpoint Peer Up context and every epoch boundary, rejecting snapshots beyond its 256-line bound rather than silently truncating them. Only the newest route in the current Loc-RIB epoch may satisfy the assertion, and its absolute file position must follow the checkpoint. A matching Peer Down, stream reset or replacement Peer Up invalidates an earlier winner. AS_PATH, NEXT_HOP and ORIGIN identify the source because Loc-RIB reports omit MED. All four original baseline assertions run before and after the cycle.
DelayOpen timing against native FRR
bgp-delayopen-frr keeps Ze and both FRR peers administratively held until a
bounded packet capture is active inside Ze's container. Native administrative
commands then release the peers; no relay or payload rewrite sits between them.
The fixture pairs a shorter Ze delay with a longer FRR delay to observe timer
expiry, and a longer Ze delay with a shorter FRR delay to observe early-OPEN
release. Both speakers must report Established.
The checker measures OPEN timing from each completed TCP handshake and checks
every observed connection, including losers. Each peer must have one surviving
connection with OPEN followed by KEEPALIVE in both directions. Missing bytes,
incomplete handshakes, truncation, capture drops or an unfinished capture
cannot prove that no premature OPEN was sent. Retained wire.pcap and
proof.json carry the wire history, configurations and native answers; retain
the suite's image-identity JSON alongside them. Opposite-initiator overlap is
reported only as a collision opportunity, not proof of an internal arbitration
order. The real-Peer tests retain that detailed ordering responsibility.
Timing uses each byte's first captured arrival. ACK and FIN boundaries must already have complete payload coverage at their capture time; a later retransmission or FIN cannot repair earlier evidence. Every packet needs a nonzero capture timestamp, and timestamps must be monotone across both directions of each complete address-and-port tuple. Independent connections may have records with out-of-order timestamps because packets can be timestamped and queued on different CPUs, as described in the libpcap timestamp manual. The checker retains the original packet order and timestamps. Missing times or a reversal within one connection refuse the timing proof, not the peer's protocol behavior.
Scenario Inventory
The suite has grown to over 100 scenario directories in test/interop/scenarios/. The table
below lists the core BGP scenarios (01-37); beyond these, the suite also covers route
reflection, policy import/export, RPKI origin validation, BMP monitoring
(bmp-locrib-pmacct puts Ze's RFC 9069 Loc-RIB feed in front of pmacct and requires
pmacct to print the configured Peer AS, the configured Peer BGP ID, the global
VRF/Table Name TLV and reason code 6 on the Peer Down; bmp-locrib-receiver-frr
turns the direction around, so FRR's bmpd drives Ze's BMP receiver and
show bmp peers must report the third party's Loc-RIB peer and its address
family; bgp-addpath-best-path-pmacct has the raw injector announce one prefix
under two ADD-PATH identifiers, MED 10 then MED 50, and requires pmacct's reading
of the Loc-RIB stream to carry the MED 10 best and never the MED 50 path, told
apart by their AS_PATHs since Ze's Loc-RIB Route Monitoring carries no MED, because
Ze advertises no best path to a BGP peer and the Loc-RIB feed is where a foreign
implementation can observe its election), an RFC 8277 labelled withdrawal
(bgp-labeled-withdraw-compatibility-frr has the raw injector announce 10.10.0.0/24 under
label 100 and 10.11.0.0/24 under label 101 in ipv4/mpls-label, then withdraw 10.10.0.0/24 alone
with the Compatibility value 0x800000 in the label field. FRR, fed through bgp-rs, must hold both
routes, then lose 10.10.0.0/24 with exactly one logged withdrawal while the injector session stays
Established. The checker then stops the injector, FRR must lose 10.11.0.0/24 through bgp-rs's
peer-down withdrawals, and its log must still hold one withdrawal of 10.10.0.0/24. bgp-rs relays
the withdrawal's bytes unchanged, so FRR losing the route alone cannot tell whether Ze read it:
Ze's reading decides what bgp-rs keeps in its route inventory, and a misread withdrawal leaves
10.10.0.0/24 there to be withdrawn a second time at peer down), PATHS-LIMIT,
max-prefix cease, a reload of the global router-id (bgp-reload-global-router-id
starts Ze with 10.255.0.1, reloads it to 10.255.0.2, and requires BIRD, a static
peer, and FRR, which Ze adds after the reload with create bgp peer, each to
report 10.255.0.2 as Ze's BGP Identifier), GTSM (bgp-gtsm-frr for the session, and gtsm-related-icmp-ttl
for RFC 5082 Section 3's related ICMP messages: FRR's kernel reads the hop limit of an
ICMPv6 error ze's kernel generates about the session, and its TCPMinTTLDrop counter is
what goes red when ze's host-route metric is absent), AS112, the RFC 7454 Section 9 transit leak
(bgp-path-asn-leak-frr gives FRR two prefixes that differ only in their AS_PATH, and requires
ze to drop the one reached through a listed transit ASN, keep the other, and keep the session),
an import prefix-list over multiprotocol UPDATEs (bgp-policy-modify-mp-frr: the raw injector
sends each case as one UPDATE, an IPv6 MP_REACH_NLRI with one prefix kept and one refused, IPv4
unicast inside MP_REACH_NLRI, an all-refused MP_REACH_NLRI, and thirty kept /64s with one refused
so the rebuilt attribute is 291 octets, then an MP_UNREACH_NLRI for the kept /48; FRR, behind
the OPEN-only relay at .11, must hold exactly the kept prefixes and lose the withdrawn one, and
the relay's capture of Ze's frames must show the extended-length MP_REACH_NLRI and no
MP_REACH_NLRI without NLRI),
next-hop self under local ip auto (bgp-nexthop-self-local-auto-frr configures no local
address toward FRR, and requires FRR to hold both ze's own route and a route relayed from a raw
injector with NEXT_HOP 172.30.0.2, ze's connected endpoint, never the injector's 172.30.0.9),
ADD-PATH re-advertisement (bgp-addpath-readvertise-collision-frr
proves a receiver keeps two paths whose sources both chose one Path Identifier, and
bgp-addpath-rail-agreement-speaker proves the live forward and the peer-up replay emit the same
bytes for one path), the RFC 6793 mixed-width relay
(as-path-mixed-width-relay-frr gives ze a route from a two-octet injector whose AS_PATH carries
AS_TRANS and whose AS4_PATH carries the real four-octet AS number, and requires FRR to report that
AS number and never 23456; as-path-prepend-two-octet-peer turns the direction around, so ze's own
non-mappable AS is prepended toward an FRR that refused the four-octet AS capability), the
RFC 9774 AS_SET deprecation (bgp-rfc9774-as-set-frr has the raw injector announce a route whose
AS_PATH holds an AS_SET and then a clean control, and Ze originate a static route with a configured
AS_SET and then its clean twin; FRR, running no bgp reject-as-sets so it would keep a set, must
hold both clean prefixes and neither set prefix, and Ze's own show bgp rib must hold the control
and not the injected set route, because the send guard alone already keeps a relayed set off FRR
and only Ze's own RIB can show the receive half), the
RFC 6368 ATTR_SET under RFC 9774 (bgp-attr-set-included-as-set-frr-bird has the raw injector
announce a route whose ATTR_SET carries an included AS_PATH holding an AS_SET, then one whose
included AS_PATH holds AS 0, then a clean control; RFC 9774 binds only the UPDATE's own paths, and
FRR's bgp_attr_unknown and BIRD's bgp_decode_unknown keep code 128 opaque, so FRR and BIRD
must both hold the set route, FRR's MRT capture of what Ze sent must hold the included AS_SET
segment, and neither peer nor Ze's RIB may hold the AS 0 route, which Ze treats as withdrawn), the
RFC 9774 Section 3 operator override (bgp-as-set-accept-frr has FRR and frr-sink each build a
summary-only as-set aggregate and send it to Ze, FRR to a peer with session as-set accept and
frr-sink to a twin peer without it; each router's own table must show the set, Ze's RIB must hold
both control routes and the accepted aggregate with its set member, and must not hold the twin's
aggregate, which the receive verdict line shows Ze treated as withdrawn; with the accept path forced
off the run fails at the RIB wait, the accepted aggregate never arriving), the
RFC 8669 Section 6 repeated Prefix-SID TLV (bgp-prefix-sid-duplicate-tlv-frr gives ze an SRv6
L3VPN route whose Prefix-SID carries the SRv6 L3 Service TLV twice, and requires FRR, whose own
parser refuses a repeated type-5 TLV, to hold the route with the first SID only), the
Software Version capability (frr-software-version holds two sessions to one FRR: peer legacy
sends the length-prefixed form over IPv4 and must reach Established with FRR showing ze's version,
while peer draft sends the draft's bare form over IPv6 and must be refused, with ze recording
FRR's NOTIFICATION OPEN Message Error/Unspecific, because FRR 10.3.1 reads the first octet as a
length, and with ze's log carrying the one WARN notification received line that names it), the
RFC 9072 Section 2 boundary (bgp-open-extended-params-boundary-frr pads ze's capabilities to 254
octets with a 200-octet hostname, so the Optional Parameters field is 256 octets and must take the
extended form, and requires FRR to reach Established and report a prefix of that hostname), and full
IS-IS (auth, convergence, dual-stack, LAN DIS,
P2P, redistribution) and OSPFv2/OSPFv3 (auth, BFD, TE, LFA/TI-LFA, graceful restart,
segment routing, opaque LSAs, stub/NSSA, virtual links, and more) interop families.
ospf-stub-nssa-frr observes Ze's first NSSA exchange, not a replacement
adjacency. Its startup wrapper waits for tcpdump's activated eth0 packet
socket before executing Ze, so even a startup-only Type 5 flood is in scope.
The original configuration still omits default-originate. Native FRR must
reach Full and hold the exact selected, installed OSPF default through Ze.
The complete Ethernet capture must independently contain both peers' NSSA
Hellos and initial database descriptions, Ze's matching border-router LSA,
and its non-MaxAge Type 7 default. Every outbound update is checked for Type 5;
FRR's empty external database alone cannot prove no flood, because FRR
discards incoming Type 5 LSAs in an NSSA.
The observation window is intentionally bounded at 120 seconds. The lab image
explicitly installs GNU coreutils 9.11-r0; its fixed /usr/bin/timeout owns
and joins tcpdump after SIGINT. Only the normal timeout completion, complete
packet counts, zero kernel drops, valid checksums and matching capture
timestamps, interface, MACs, addresses, router IDs and area support absence.
Truncation, forced termination and missing positive traffic refuse proof.
Retained packet/configuration/binary hashes and native FRR answers attribute
this current session; they do not replace missing historical originals or
establish kernel, VM or conformance results.
ospf-virtual-link-frr and ospfv3-vlink-frr put a transit FRR between Ze and
the far virtual-link endpoint. VLAN 100 connects Ze eth1 to transit eth1;
VLAN 200 connects transit eth2 to the far endpoint's eth2. Docker eth0
remains the management link and carries no OSPF adjacency. The routed data
links use 10.200.0.0/30 and 10.200.0.4/30, with IPv6
2001:db8:100::/64 and 2001:db8:200::/64.
The OSPFv2 endpoint is FRR. OSPFv3 uses the existing BIRD peer image because
FRR 10.3.1 does not implement the OSPFv3 virtual-link configuration command.
Ze advertises 192.0.2.1/32 and 2001:db8:10::1/128 from the dedicated
passive interface backbone0, so the IPv4 host prefix is the interface's primary
address rather than an unadvertised secondary beside 127.0.0.1 on lo.
The checker requires the far endpoint's address in Ze's Full backbone neighbor record and an OSPF route at the independent endpoint for Ze's backbone host prefix. Changing the transit egress cost from 10 to 30 must change the virtual cost from 20 to 40 without changing the database-exchange identity or persistent adjacency-reset counters. Taking that data interface down must withdraw the route and virtual reachability; bringing it up must restore both through a new database exchange.
Ze's existing RIB and kernel FIB plugins install the transit route. The checker
reads the actual kernel route to the far data subnet, requires the intermediate
router as next hop on eth1, and requires withdrawal and restoration across the
same down/up cycle. IPv6 uses the transit router's observed link-local address.
Neither endpoint receives a preinstalled transit or backbone route; FRR uses
its normal zebra forwarding and BIRD exports OSPF routes through its kernel protocol.
ospf-self-lsa-flush-frr proves RFC 2328 Section 13.4 for an LSA Ze no
longer wishes to originate. Two reloads leave FRR holding Ze's default
AS-external-LSA at sequence 0x80000003 or later. FRR's eth0 then goes down,
Ze is stopped and started (opStop, then opStart), so it originates the
default again at 0x80000001, and ze-reload-withdraw.conf withdraws it. With
no neighbour to acknowledge the flush, Ze keeps its own copy at MaxAge,
0x80000002, and the checker reads that copy out of show ospf database external before FRR comes back. In the database exchange that follows, FRR
hands Ze its newer instance. Ze must flush it at MaxAge with FRR's own
sequence number, so the LSA must leave FRR's database and the default must
leave show ip route ospf json, whose answer is {} once FRR's OSPF table is
empty. The text form prints nothing then, and the lab never reads an empty
answer as an absence. A Ze that re-originates the withdrawn body at age 0
leaves both in place, and the absences time out. opStop returns only once
Docker reports the container stopped, so the start that follows cannot race a
container that is still shutting down. Results on 2026-11-14 (arm64, QEMU
docker-lab guest, FRR 10.7.1): with the if local.Age.IsMaxAge() branch removed
from (*LSDB).handleSelfReceived, FAIL at assertion 21, the withdrawn LSA never
leaving FRR's database; the real build passes (2m24s).
ospf-gr-frr and ospf-gr-helper-broadcast-frr prepare FRR 10.7.1 with
graceful-restart prepare ip ospf, save ospfd's PID, and send that process
SIGTERM. Preparation alone does not restart the daemon. Watchfrr owns the
restart; the checker neither starts a competing ospfd nor stops zebra, the
container, or its network. The existing post-restart Full wait also requires
a different ospfd PID. The broadcast helper still requires no prior helper
exit, then reason="flushed" within 100 seconds, shorter than the 120-second
grace period, and Full on both sides.
The helper fixture enables Prometheus so those assertions read the daemon's
actual counters; Netdata collectors are disabled because they are not involved.
Without the registry, show metrics values refuses instead of proving that
no earlier helper exit occurred.
FRR's ospf_gr_prepare and ospf_gr_restart_exit
originate and flush the Grace-LSA;
ospf_finish_final
preserves LSAs and routes during prepared shutdown, and
daemon_down, wakeup_down, and try_restart
drive the supervised restart.
ospf-sr-frr and ospfv3-sr-frr read the kernel label table with the installed
iproute2 command ip -f mpls route show 16100, not the absent mpls -ls
utility. The destination filter selects label 16100; the existing content
assertion still requires that label, so an empty successful query fails.
Their 60-second label deadlines and adjacency assertions are unchanged.
In ospf-sr-frr, FRR's Prefix-SID prefix 10.255.255.3/32 sits on FRR's
loopback: FRR's sr_prefix_sid looks the prefix up on an interface and, when
none holds it, defers the Prefix-SID without an error, so no label 16100 would
reach Ze.
ospf-sr-anycast-nexthop-frr proves RFC 8665 Section 5's rule that a router
"MUST take into account ... the E-, NP-, and M-Flags advertised by the next-hop
router if that router advertised the SID for the prefix", for a next hop that is
not the binding's Originator. Ze and two FRRs share one broadcast LAN with SR on.
Each peer enables segment-routing on inside router ospf, before configuring
its SRGB and Prefix-SID. FRR 10.7.1 registers the enable command only in that
context; placing it at top level leaves SR disabled even when adjacency forms.
Both FRRs bind index 100 to the anycast 10.255.255.100/32 on lo with PHP
requested: frr (router-id 192.0.2.3) at stub cost 1, so it is Ze's only next
hop, and frr-sink (192.0.2.1) at cost 50. Ze reads Extended Prefix LSAs in
LSDB key order, and both FRRs use the same Opaque ID, so 192.0.2.1 becomes the
Originator and the next hop is the second advertiser. The checker first requires
that order from Ze's show ospf database opaque-area (a lab where the next hop
became the Originator passes either way, so it fails instead). It then reads
ip -f mpls route show in Ze's container until in-label 16100 is popped toward
the next hop, holds that pop for 15 seconds (three times OSPF's 5-second SPF
hold ceiling), failing on the first read that is not the pop, and re-reads the
LSDB order at the end. A single read cannot discriminate: the first pop can be
the route Ze installed before the second advertiser's LSA reached its SR table,
when the next hop was the binding's only advertiser, and a Ze applying only the
Originator's flags pops then too. Such a Ze later swaps 16100 to 17100, the next
hop's SRGB base plus the index. Results on 2026-11-14 (arm64, QEMU docker-lab
guest, FRR 10.7.1): with the body of (*srInstaller).forwarding restored to its
form before 19de202493, FAIL at assertion 3, "the pop held for 5.046s, then
sample 5 found in-label 16100 is swapped, not popped: 16100 as to 17100"; the
real build passes (1m49s).
ospf-ext-prefix-link-frr is the Extended Prefix/Link scenario: it proves both
directions of the RFC 7684 exchange. FRR enables capability opaque and
segment-routing on inside router ospf and holds its Prefix-SID prefix on a
loopback, so it originates Extended Prefix and Extended Link LSAs. Without the
loopback FRR defers the Prefix-SID and originates no Extended Prefix LSA. Ze
originates both types too, so a substring match on Ze's show ospf database opaque-area passes on Ze's own LSAs alone. The checker decodes that answer and
binds each decoded row to its advertising-router: it requires an Extended
Prefix row and an Extended Link row from FRR's router-id, and one of each from
Ze's. On FRR's side it reads show ip ospf database opaque-area json and
requires, under areaLocalOpaqueLsa.areas, an LSA whose Link State ID has
opaque type 7 and one with opaque type 8, each with advertisingRouter equal
to Ze's router id. A match on Ze's address alone is not enough, because FRR's
own Extended Link LSA names Ze as its Link ID. The Ze read waits rather than
reading once at Full, because FRR schedules each opaque type min_ls_interval
after the previous once an opaque-capable neighbor appears
(ospf_opaque_lsa_originate_schedule). Results on 2026-11-14 (arm64, QEMU
docker-lab guest, FRR 10.7.1): with FRR's Prefix-SID on no loopback, FAIL at
assertion 2, "no decoded Extended Prefix LSA advertised by 172.30.0.3"; with
the fixture corrected, pass (1m13s).
bgp-graceful-restart-frr retains its original session, route, GR and EOR
checks and adds two strict input speakers with FRR as the independent receiver.
One source omits IPv4 from GR while advertising a 40-second IPv4 LLST; its
IPv6 family has a 20-second GR period followed by the same LLST. The other
source advertises zero LLST. After fencing initial receipt and bilateral FRR
LLGR negotiation, the checker sends each native source USR1 through
Lab.Signal. Only this oracle handles that signal: it closes its BGP socket
without NOTIFICATION, stops KEEPALIVEs, and keeps PID 1, the container and its
next-hop interfaces alive until its existing bounded lifetime ends or lab cleanup.
Unexpected peer closure, NOTIFICATION, or lifetime expiry before control remains
a failure. The main loss timestamp precedes the signal; the DOWN poll and
three-second observation margin are unchanged. The checker requires immediate
removal for zero LLST, immediate IPv4 LLGR_STALE, and IPv6's conventional-to-LLGR
transition.
IPv4 must expire at its original DOWN+40 deadline while IPv6 survives until
DOWN+60. FRR's established/dropped connection counters must remain unchanged;
its recomputed wall-clock epoch is not a session identity. Complete observation
cycles, including final expiry probes, must stay within the sampling bound.
An explicit FRR-origin received-route check preserves the original reverse
direction proof despite the added native sources. This does not cover
NOTIFICATION, reconnect F-bit or received-stale-policy behavior.
The receipt query is show bgp rib received | json; its CLI output is a
top-level array of route rows, not an object containing a routes field.
An empty array is a measured RIB with no matching route; null or an object
is not a valid CLI observation.
The received-route fence retains the last completed CLI result (stdout, stderr, exit status and parse error), the last successfully measured RIB, and probe counts when it fails. A final cancelled Docker exec cannot overwrite this earlier evidence. Capturing it adds no queries or time to the original 30-second receipt fence and changes none of the 40/60-second expiry windows. FRR route-observation errors retain the queried prefix and raw JSON response, so an invalid route can be distinguished from an unexpected response shape.
bgp-nexthop-self-local-auto-frr retains both original self-next-hop checks
and adds AIGP on the general forwarding path. The received metric is 100;
source-link cost 7 and destination-link cost 43 must produce 107 and Ze's
next hop at FRR. GoBGP is the unchanged-next-hop control and must retain 100.
The fixture's ASes are in one administrative domain. FRR 10.3.1 requires both
neighbor ... aigp and neighbor ... oad to retain received AIGP on eBGP;
aigp alone leaves the route installed but discards its AIGP attribute.
oad keeps the session eBGP, including its AS-path and next-hop assertions.
See the pinned FRR receive gate.
A third-party next hop initially has no distance: the rewritten route must be absent while the unchanged control holds it. An SDK route installation with metric 11 restores 111, then metric 7 produces 107; replacing that distance with zero withdraws the whole rewritten route, and restoring 7 recovers it again. Source UPDATE, EOR and session counters fence the recovery against a new advertisement. FRR's established/dropped counters and local/foreign socket ports are also captured before the initial withheld inventory and checked after every metric transition and the final baseline checks. Recovery through a replacement recipient session cannot satisfy that fence.
A separate iBGP wire recipient uses the existing speaker-args container.
Its passive Ze peer binds the explicit lab address; local ip auto supplies
no address for a passive listener. The original FRR local-auto checks remain
unchanged. The wire recipient records one connection's original received frames,
never reconnects, and
must observe exact AIGP TLVs 107 and 111, Ze's next-hop bytes, and a complete
withdrawal of 10.10.3.0/24. Its history also checks the subsequent
107/withdrawal/107 transitions. Both the FRR readiness prefix and a checker
release route precede injection; the release follows all recipients' initial
EORs. This carries the former aigp-source-cost-recovery draft's wire contract
on a legal source topology: 127/8 NEXT_HOPs are rejected before AIGP, even when
unassigned to a host interface. Run it with
INTEROP_SCENARIO=bgp-nexthop-self-local-auto-frr ./le test integration interop.
bgp-labeled-withdraw-compatibility-frr also exercises VPNv4 and VPNv6,
with each prefix announced under three distinct RDs. Compatibility values
800000 and 123456 withdraw two RDs while the third survives. FRR's
RD-keyed table and exact withdrawal log identify each route. After the
injector goes down, only the surviving RD may receive another withdrawal;
the previously removed routes must not be withdrawn twice. Receiver session
generation and a subsequent received KEEPALIVE fence the final observation.
bgp-vpn-withheld-withdraw-frr covers the sending half of RFC 8277 Section
2.4 on the forward rail. The injector announces two VPNv4 routes at one RD and
two labelled-unicast routes, each under one label, then re-announces one of
each under three labels with NO_EXPORT, which ze withholds from FRR (an
external peer) and converts into withdrawals. FRR must lose exactly those two
routes, keep both siblings, decode one withdrawal of each, and keep its
session generation, having decoded only the one-label VPN announcement. A
VPN withdrawal that copied the three-label stack would leave FRR a 72-bit
IPv4 prefix and be refused. The labelled-unicast leg does not discriminate:
FRR walks a copied stack to its bottom-of-stack bit.
bgp-flowspec-sctp-gobgp supplies both a covering unicast route and a FlowSpec
from GoBGP, checks the installed kernel rule, then withdraws and restores only
the unicast route. Its checker requires the existing FlowSpec filter to
disappear and return without another FlowSpec announcement. Finally it
withdraws the FlowSpec itself. Empty-ruleset checks require nft's JSON response,
not a table that should have been removed.
bgp-flowspec-gobgp retains its originated-rule and OR-of-AND checks and adds
a received rule with a nonzero next hop. A byte-preserving relay captures
the UPDATE delivered to GoBGP: its MP_REACH next-hop length must be zero,
and GoBGP must decode the rule and traffic-rate action. Withdrawing and
restoring only the covering route drives stored replay without a new source
FlowSpec UPDATE. The checker requires the same rule and action after recovery.
This independent-peer scenario covers IPv4 FlowSpec; the four-family wire
matrix remains a separate internal proof.
The Linux integration-tagged
TestSelectedFlowSpecKernelPacketSemantics uses an isolated network namespace
and actual UDP packets to distinguish ordered terminal and continuing actions,
deferred DSCP marking, withdrawal, and excess-rate drops. It requires
CAP_SYS_ADMIN, CAP_NET_ADMIN, and nftables kernel support. A capability skip
is not packet-forwarding evidence.
| # | Scenario | Daemons | What It Tests |
|---|---|---|---|
| 01 | bgp-ebgp-ipv4-frr | Ze, FRR | Basic eBGP session establishment |
| 02 | bgp-ebgp-ipv4-bird | Ze, BIRD | Basic eBGP session with BIRD |
| 03 | bgp-ibgp-frr | Ze, FRR | iBGP session (same AS) |
| 04 | bgp-4byte-asn-frr | Ze, FRR | 4-byte ASN negotiation (RFC 6793) |
| 05 | bgp-routes-from-frr | Ze, FRR | Ze receives routes originated by FRR |
| 06 | bgp-routes-from-bird | Ze, BIRD | Ze receives routes originated by BIRD |
| 07 | bgp-routes-to-frr | Ze, FRR | FRR receives routes originated by Ze |
| 08 | bgp-triangle | Ze, FRR, BIRD | Three-way topology, multi-peer stability |
| 09 | bgp-route-withdrawal-frr | Ze, FRR | Route withdrawal propagation |
| 10 | bgp-ipv6-ebgp-frr | Ze, FRR | IPv6 eBGP session and route exchange |
| 11 | bgp-addpath-frr | Ze, FRR | ADD-PATH capability (RFC 7911) |
| 12 | bgp-route-refresh-frr | Ze, FRR | Route Refresh (RFC 2918) |
| 13 | bgp-graceful-restart-frr | Ze, FRR | Graceful Restart negotiation (RFC 4724) |
| 14 | bgp-route-server-frr | Ze, FRR, BIRD | Route server: forwards without inserting own ASN |
| 15 | bgp-community-frr | Ze, FRR | Standard community propagation |
| 16 | bgp-extended-community-frr | Ze, FRR | Extended community propagation |
| 17 | bgp-md5-auth-frr | Ze, FRR | TCP MD5 authentication (RFC 2385) |
| 18 | bgp-ebgp-gobgp | Ze, GoBGP | eBGP session with GoBGP |
| 19 | bgp-routes-gobgp | Ze, GoBGP | Route exchange with GoBGP |
| 20 | bgp-role-frr | Ze, FRR | RFC 9234 Role capability negotiation |
| 21 | bgp-role-gobgp | Ze, GoBGP | RFC 9234 Role capability negotiation |
| 22 | bgp-evpn-frr | Ze, FRR | EVPN Type-2 route exchange; FRR accepts an IP Prefix route (Type 5) whose Overlay Index is the ESI. FRR 10.3.1 prints no Type 5 ESI in any view, so bgp-evpn-gobgp asserts its octets |
| 23 | bgp-vpn-frr | Ze, FRR, BIRD | VPN-IPv4 route exchange with FRR; RFC 4659 Section 3.2.1.1 VPN-IPv6 next hops, read at BIRD (the 48-octet field's link-local, which FRR 10.3.1 does not display) and at FRR (the off-link route's sole global) |
| 24 | bgp-flowspec-frr | Ze, FRR | FlowSpec rule exchange |
| 25 | bgp-ipv6-ebgp-bird | Ze, BIRD | IPv6 eBGP route exchange |
| 26 | bgp-ipv6-ebgp-gobgp | Ze, GoBGP | IPv6 eBGP route exchange |
| 27 | bgp-multihop-ebgp-frr | Ze, FRR | Multi-hop eBGP with outgoing-ttl |
| 28 | bgp-evpn-gobgp | Ze, GoBGP | EVPN Type-2 route exchange, Ethernet A-D ESI Labels, and the IP Prefix route (Type 5) sent with overlay-index esi: GoBGP reads the non-zero ESI and a zero GW IP from the NLRI (RFC 9136 Section 3.1) |
| 29 | bgp-vpn-gobgp | Ze, GoBGP | VPN (L3VPN) route exchange |
| 30 | bgp-flowspec-gobgp | Ze, GoBGP | FlowSpec rule exchange |
| 31 | bgp-multihop-ebgp-bird | Ze, BIRD | Multi-hop eBGP with outgoing-ttl |
| 32 | bgp-multihop-ebgp-gobgp | Ze, GoBGP | Multi-hop eBGP with outgoing-ttl |
| 33 | bfd-frr | Ze, FRR | BFD opt-in and BFD-triggered BGP teardown |
| 34 | bgp-ecmp-frr | Ze, FRR, GoBGP | FRR ECMP selection for the same prefix from Ze and GoBGP |
| 35 | bgp-srv6-frr | Ze, FRR | SRv6 VPNv6 route exchange and Prefix-SID handling |
| 36 | bgp-remove-private-as-frr | Ze, FRR, GoBGP | remove-private-as export policy to FRR |
| 37 | bgp-remove-private-as-as4path-frr | Ze, FRR, BIRD | remove-private-as handling for AS4_PATH private ASNs |
| 38 | bgp-vpn-addpath-frr | Ze, FRR | ipv4/mpls-vpn over ADD-PATH with no path-information: FRR holds the exact RD, prefix and label, so ze wrote Path Identifier 0 (RFC 7911 Section 3) |
| 39 | ldp-keepalive-expiry-frr | Ze, FRR ldpd | ze's LDP KeepAlive Timer expires after an ldplab nft table in ze's container rewrites the TCP window ze advertises on the session to zero, so FRR can send ze no PDU while its kernel still acknowledges ze's segments (a drop of FRR's segments would also drop the ACKs they carry and stall ze's own stream); FRR logs msg[in]: notification: ... status KeepAlive Timer Expired (fatal error) (RFC 5036 Section 3.5.1.2.3) and leaves OPERATIONAL. Red with the send reverted: FRR logs only connection closed by remote end. Ships its own daemons (ldpd=yes) and gives ze transport address 172.30.0.250 so ze is the active LSR |
| 53 | ldp-init-unknown-tlv-frr | Ze, FRR ldpd | RFC 5036 Section 3.3 on ze's receive side: "if U is clear (=0), a notification MUST be returned to the message originator and the entire message MUST be ignored". FRR cannot send an unknown TLV, so an ldplab nft rule in ze's container rewrites the type of the first Capability TLV in FRR's real Initialization (0x8506, the Dynamic Capability Announcement ldpd's send_init writes at LDP offset 36) to 0x3f00, U and F clear, and counts each match. FRR, the originator, logs msg[in]: notification: ... status Unknown TLV, and stays short of OPERATIONAL for 10 s because ze never answers the ignored Initialization with a KeepAlive. With the rule removed the same peers reach OPERATIONAL, the positive control. FRR proves the Notification reached the originator and decodes as Unknown TLV; the rule's counter proves ze was shown the TLV. Red with skipTLV returning nil for every TLV: FRR logs no received status Unknown TLV Notification. Shares the daemons and the 172.30.0.250 transport address of ldp-keepalive-expiry-frr |
| 66 | ldp-backoff-clear-frr | Ze, FRR ldpd | A session that reached OPERATIONAL leaves no setup backoff when it ends (RFC 5036 Section 2.5.3, "After such a configuration action, there is no further need to throttle subsequent session establishment attempts"). Once FRR has reported the session OPERATIONAL for 3 s, FRR runs clear mpls ldp neighbor <ze's transport address>: ldpd's nbr_clear_ctl calls session_shutdown with S_SHUTDOWN, so FRR sends a Shutdown Notification and closes the TCP connection while the adjacency stays. FRR holds the passive role and ze dials on each Hello FRR sends (processDiscoveryPacket, startSessionForAdj), every 1 s by discovery hello interval 1. FRR's show mpls ldp neighbor json must report ze OPERATIONAL again with an upTime no older than the clear (nbr_to_ctl restarts it on each entry to OPERATIONAL) within 8 s of the clear: the released backoff lands near 2 s, a recorded one forbids the dial for 15 s. ze's own log, read at info (environment { log { ldp info } }), must carry a new fatal Notification and a new session end, the proof that the clear reached ze and that info lines are read, and no new ldp: session setup failed, next attempt delayed. Red with the operational arm of startSessionForAdj's session-end callback recording a setup failure (one line changed in the working tree for the run, because the ze image builds from the tree; image sha256:a218f6cd97d5 and sha256:c58b3f2734ff, 2026-11-13): ze logged "ldp: session setup failed, next attempt delayed" 1 time(s) after FRR cleared a session that had reached OPERATIONAL: its end recorded a setup backoff, which delays the redial by 15 s or more (the probes ended 15.454s after the clear). Shares the daemons of ldp-keepalive-expiry-frr. The lab adds ze's 172.30.0.250 transport address in ze's prestart (scenarioHostAddresses), because ze dials from it on FRR's first Hello: added by the checker, the address could arrive after that dial, and each dial that fails to bind costs a backoff step of 15 s or more. sha256:3c4dee38ac2e |
| 40 | bgp-attr-flags-low-nibble-gobgp | Ze, GoBGP, raw injector | RFC 4271 Section 4.3: attribute 241 sent to Ze at flags 0xCF reaches GoBGP at 0xE0 (low nibble cleared, Partial stamped by Section 5), never 0xEF. Results on 2026-11-14 (arm64, QEMU docker-lab guest): with attrs[pos] &^= attrFlagsUnusedMask deleted from normalizeTransmittedUpdate, FAIL after 2m41s at assertion 3, the route never reaching GoBGP; the real build passes (1m5s) |
| 41 | bgp-large-community-no-repeat-gobgp | Ze, GoBGP, raw injector, native speaker | RFC 8092 Section 3: an export policy adding values the route already carries leaves GoBGP with each Large Community once, and the native speaker's no-repeated-large-community oracle finds no value sent twice on the wire. GoBGP alone cannot go red: it removes repeats on receipt, as the same section requires of a receiver (v4.10.0 decodes every value, then ValidateAttribute in drops each repeat). Its ze.conf lists bgp-filter-community:egress in the export chain, which Ze requires since a965ee38fb. Results on 2026-11-14 (arm64, QEMU docker-lab guest): with the AttrLargeCommunity branch of normalizeTransmittedUpdate disabled, FAIL after 3m8s at assertion 6, the oracle reporting "LARGE_COMMUNITY value 0000fde80000000100000001 transmitted twice in one UPDATE" and the same for 65000:2:2, while GoBGP's own view (assertion 4) still passed; the real build passes (1m42s). pkg/packet/bgp/validate.go |
| 42 | bgp-mup-malformed-t1st-withdraw-gobgp | Ze, GoBGP, raw injector | draft-ietf-bess-mup-safi Section 3.1.3.1: a malformed T1ST withdraws the earlier generation at GoBGP, a well-formed sibling stays, the session stays up. Results on 2026-11-14 (arm64, QEMU docker-lab guest): with recognizerFor returning nlritype.Keep where it returned Withdraw, FAIL after 3m4s at assertion 4, the wait for the valid route sharing the UPDATE timing out; the real build passes (2m2s) |
| 43 | bgp-role-rs-client-learned-not-upward-frr | Ze, FRR 10.7.1, raw injector | RFC 9234 Section 3.1: a route learned on a session where Ze is the Route Server never reaches FRR, Ze's Provider; Ze's own control route does. Results on 2026-11-14 (arm64, QEMU docker-lab guest): with || srcRole == roleRS removed from OTCEgressFilter, FAIL after 1m22s at assertion 4, FRR holding 10.10.0.0/24, which must be absent; the real build passes (48s) |
| 44 | bgp-rfc7999-blackhole-readvertise-frr | Ze, BIRD, FRR 10.7.1 | RFC 7999 Section 3.1: a BLACKHOLE-tagged route from BIRD is never re-advertised to FRR, an eBGP neighbor with no agreement; the untagged control is. Results on 2026-11-14 (arm64, QEMU docker-lab guest): with blackholeReadvertiseRefused returning false, FAIL after 2m2s at assertion 5, FRR holding 10.200.0.1/32, which must be absent; the real build passes (1m34s) |
| 45 | bgp-sr-policy-gobgp | Ze, GoBGP | RFC 9830: GoBGP's RIB, read through gobgp-paths (Dockerfile.gobgp, a gRPC ListPath reader, because no gobgp CLI release names an SR Policy family), holds Ze's configured IPv4 and IPv6 SR Policies with their distinguisher, color and endpoint, Binding SID label 1048575 and Type A segment labels 16 and 1048575, and a third IPv4 policy (color 300) whose SRv6 BSID 2001:db8::b5 rides the type-13 Binding SID sub-TLV at Length 18 (RFC 9830 Section 2.4.2), which GoBGP prints through its type-13 decoder with no B-Flag. Every policy must also carry the configured Route Target as GoBGP types it, (RFC 9830 Section 1, RFC 4360 Section 4). Red with parseConfigRoute writing sub-type 3 (Route Origin) into the community it appends: GoBGP still holds every policy and String still prints. Removing the append is not a red for this assertion, because the config parser then refuses the configured extended community and Ze never starts. Goal validation for origination: in-range labels never reach the 20-bit range check, so no revert turns the label checks red. {"type":1,"subtype":2,"value":"172.30.0.5:0"}[172.30.0.5:0] |
| 46 | bgp-sr-policy-malformed-receive-gobgp | Ze, GoBGP, raw injector | RFC 9830 Section 5: after policies A, B and control C reach GoBGP, A beside a 13-octet NLRI and then B without its Tunnel Encapsulation attribute are each treat-as-withdraw, so GoBGP loses A, then B, keeps C, and its session with Ze stays up: Ze's peer detail for GoBGP shows one connection established and none dropped (GoBGP resets a session that carries an SR Policy NLRI whose length is neither 96 nor 192 bits) |
| 47 | bfd-link-local-frr | Ze, FRR bfdd | RFC 5881 Section 3: a BFD-plugin-pinned single-hop session to FRR's IPv6 link-local on eth0 reaches Up, each side holding the other's discriminator. Acceptance, not discrimination (see "BFD over link-local") |
| 48 | bfd-link-local-two-links-frr | Ze, FRR bfdd | RFC 5881 Section 3: two sessions to ONE link-local address, one each on eth1 and eth2, reach Up on their own link with their own discriminators crossed, while Ze's Your-Discriminator-zero sends are dropped. Discriminates the receive-side zone reconciliation and the send-side link pinning (see "BFD over link-local") |
| 49 | bgp-exabgp-bridge-sr-policy-gobgp | Ze (exabgp bridge), GoBGP | RFC 9830: an ExaBGP API script writes announce ipv4 sr-policy ... preference 300 with two Type A segment lists and a Route Target; the bridge hands it to update text, whose sr-policy section reaches the registered config-route parser, and GoBGP's RIB (through gobgp-paths) holds the NLRI (color 300, endpoint 10.0.0.3), the Preference sub-TLV, labels 16001 and 16002, and the Route Target typed as sub-type 2. Red with the sr-policy registration's NLRISectionCarriesAttributes set false, which stops the hand-off. The Route Target on this path comes from update text's own attribute block, not from parseConfigRoute: red with the extended-community case of update_text.go writing sub-type 3, and green with the same change made in parseConfigRoute |
| 50 | bgp-exabgp-bridge-rtc-gobgp | Ze (exabgp bridge), GoBGP | RFC 4684 Section 4: an ExaBGP API script writes announce ipv4 rtc origin-as 65001 route-target 65001:100 next-hop. ; the bridge hands it to update text. mp ipv4/rtc add, and GoBGP's RIB (through gobgp-paths 1 132) holds membership. The 0-bit default membership is not sent: GoBGP v3.31.0 panicked on it (nil Route Target in BgpServer.propagateUpdate); v4.10.0 handles a nil Route Target (processRTCMembership), and the scenario still sends the specific membership only. Red with the bridge's rtc row of bridgeSAFI removed, which refuses the line. 65001:65001:100pkg/server/server.go |
| 51 | bgp-exabgp-bridge-vpnv4-frr | Ze (exabgp bridge), FRR | RFC 4364: an ExaBGP API script writes ExaBGP 6.0's announce ipv4 mpls-vpn 10.0.0.0/24 next-hop ... rd 65001:100 label 100; the bridge hands it to update text ... mp ipv4/mpls-vpn add, and FRR holds the route under RD 65001:100 with remote label 100, its session staying up. Red with the bridge's mpls-vpn row of bridgeSAFI removed, which refuses the line |
| 52 | bgp-flowspec-redirect-check-frr | Ze (route server, bgp-rib), FRR, raw injector | draft-simpson-idr-flowspec-redirect-02 Section 3: the injector (AS 65004) sends a cover for 192.0.2.0/24 originated in AS 65009, then a FlowSpec rule for that prefix with the 0x0800 redirect-to-IP community and next hop 192.0.2.77. The longest match for the next hop is not originated in the peer's AS, so FRR must hold the rule (show bgp ipv4 flowspec detail json) with no FS:redirect in its ecomlist. No lab daemon originates a 0x0800 rule, hence the raw frames; both reach FRR through Ze's peer-up replay. The plain show bgp ipv4 flowspec json is FRR's generic route printer and carries no extended communities, so a check on it passes either way. Red with the RIB's refusal disabled (flowSpecRedirectRefused answering false, by Go overlay): FRR holds "ecomlist":"FS:redirect IP 0x0" |
| 54 | bgp-route-server-replacement-frr | Ze (route server, bgp-rib), FRR, GoBGP, BIRD | RFC 4271 Section 9.2 and RFC 7947 at a client that negotiated no ADD-PATH: BIRD holds one route for 10.98.0.0/24, and Ze decides whose. The checker first asserts from BIRD's show protocols all ze_peer that no ADD-PATH capability was exchanged. GoBGP announces the prefix with one prepended AS, and BIRD holds AS_PATH 65004 65004. FRR announces it, better and later, and BIRD moves to 65002. GoBGP re-announces it, still worse, with community 65004:2; Ze's show bgp rib received community 65004:2 must list the prefix, the proof the revision reached the route server, and BIRD must keep 65002 through a 20 s window read every second (assertion 12). FRR withdraws it with no network; BIRD must reach 65004 65004 with no reading in between that lacks the prefix (assertion 14), then keep it through the window. The two sources are attached receive-only: in a run where Ze also relayed to them, FRR re-advertised GoBGP's route back as 65002 65004 65004 (a route server adds no AS, so nothing stops the loop), and FRR's no network became an implicit replacement that never exercised the withdrawal half. Red on 2026-10-26, each from a git archive export of HEAD with the Ze image rebuilt: oweAnnouncement answering false fails assertion 12 at reading 1, BIRD holding 65004 65004; owesWithdrawal answering false fails assertion 14 at reading 1, BIRD answering the prefix lookup with no route |
| 55 | bgp-route-server-control-community-frr | Ze (route server), FRR, BIRD | RFC 7947 Section 2.2.2.1 and the route server's large control communities: FRR (AS 65002) announces 10.99.0.0/24 with, asking Ze (AS 65001) to prepend once toward BIRD (AS 65003), and 10.98.0.0/24 with, naming a client that is not BIRD; both carry FRR's own. BIRD must decode 65001 65002 for the first and 65002 for the second, and its bgp_large_community must hold (65002, 1, 1) and no value under 65001. Red on 2026-10-27, working tree with the Ze image rebuilt by the harness: RSPrepend returning its input fails assertion 5, BIRD decoding 65002 for 10.99.0.0/24; StripLargeControlCommunities returning nil fails assertion 6, BIRD holding (65001, 101, 65003) (65002, 1, 1). 65001:101:6500365001:101:6509965002:1:1 |
| 56 | bgp-paths-limit-receive-bird | Ze, BIRD 3.4.0 | draft-abraitis-idr-addpath-paths-limit Section 3, the receive ceiling against a sender that ignores it: BIRD has no capability 76 and, with add paths tx, exports every route its table holds, so four static protocols give 10.70.0.0/24 four paths. Ze advertises limit { receive 2; } and must hold exactly two in show bgp rib received, count two in ze_bgp_paths_limit_refused_total, install 10.71.0.0/24 and keep the session. The wait ends once every offered path is held or refused, so the held count is read after the last offer. Red on 2026-10-27, processValidatedMessage skipping receiveCeiling (a GOFLAGS=-overlay build, image rebuilt by the harness): assertion 3, "ze holds 4 paths of 10.70.0.0/24, want the advertised ceiling 2 (refused 0)", which is also the reading that BIRD offered four |
| 57 | bgp-paths-limit-best-path-frr | Ze (route server, bgp-rib), FRR 10.7.1, BIRD 3.4.0 | draft-abraitis-idr-addpath-paths-limit Section 1, best paths under a peer's limit: FRR advertises PATHS-LIMIT 2 for ipv4/unicast, and BIRD (add paths tx, allow bgp_med yes) gives Ze four paths of 10.72.0.0/24 one at a time, worst first, by enabling static protocols MED 300, 200, 100, then 50. After each step FRR's show bgp ipv4 unicast 10.72.0.0/24 json must hold exactly the two best MEDs from Ze; more than two fails at once. Disabling MED 50 must bring MED 200 back under the path identifier FRR held it with before its demotion. Without allow bgp_med, BIRD strips a MED on an EBGP export that its own export filter did not set, and every path arrives with none. Red on 2026-10-27, owePathsLimitElections returning before it owes an election (a GOFLAGS=-overlay build, image rebuilt by the harness): assertion 6, FRR holding MED 300 and MED 200 after MED 100 arrived |
| 58 | bgp-session-protection-frr | Ze (GTSM ttl { max 1; }, control-plane-protection bgp), FRR 10.7.1 |
RFC 5082 Section 2.2 and Section 3, and RFC 7454 Section 4, read at the sender: FRR runs ttl-security hops 1, and the checker opens python3 connects to Ze's port 179 inside FRR's container, because FRR's image has no other tool that sets a socket's TTL. A SYN from FRR's address at TTL 254 must time out (only the ze_gtsm listen-port term can drop it: FRR is a CoPP neighbor); the same address at TTL 255 must complete the handshake, the control that proves the probe can see a SYN-ACK; a SYN at TTL 255 from 172.30.0.40, an address the checker adds to FRR's interface and Ze does not know, must time out, never reset (only CoPP's unknown-source-drop can drop it: GTSM leaves an unknown source alone), and that term's counter in nft list table inet ze_copp must rise. FRR's session must keep its established and dropped counts and both ports across the probes. Red on 2026-10-27, each producer broken by a GOFLAGS=-overlay build with the image rebuilt by the harness: peerTerms without the listen-port term, assertion 4, "a SYN from 172.30.0.3 at TTL 254 got a completed handshake from ze"; translatePolicy without unknown-source-drop, assertion 6, "a SYN from the unconfigured 172.30.0.40 got a completed handshake from ze" |
| 59 | rtr-stayrtr-preference | Ze (two RTR caches, preference 10 and 20), StayRTR v0.6.4 twice, FRR 10.7.1, BIRD 3.4.0 | RFC 8210 Section 10, cache preference, failover and fail-back observed at BIRD's export: the preferred cache's set is in use first, the standby's after the checker stops the preferred container, the preferred cache's again after it starts it ("RTR cache preference, failover and fail-back" above). Red on 2026-10-27 in phase 1 with the preference sort reversed, and in phase 3 with fail-back disabled. Green in 10 of 10 runs once the StayRTR image carried its Error Report patch and a readiness probe that waits for RTR |
| 60 | bgp-panic-cease | Ze (the ze_le daemon), FRR 10.7.1, BIRD 3.4.0 | RFC 4271 Section 6.7 and Section 4.5, a panic on one session's read path contained to that session: the scenario's ze-le-daemon file starts the ze_le build, the checker creates /run/ze-inject/bgp-read-panic-<FRR address> in ze's container and makes FRR redistribute 10.49.1.0/24, and the injected panic on that UPDATE must reach FRR as a received NOTIFICATION Cease / Unspecific (lastResetDueTo naming a received notification, lastNotificationReason a Cease with no named subcode). FRR's session must come back (two establishments, one drop), while ze counts one establishment and no drop toward BIRD and BIRD stays Established. Green on 2026-10-27, twice. Red on 2026-10-27, ceaseAfterPanic without its tryNotificationWithin send (a GOFLAGS=-overlay build, image rebuilt by the harness): assertion 5, FRR's lastResetDueTo "Peer closed the session" after ze logged the contained panic. The shipped ze-linux holds neither injectReadPanic nor the trigger path; only ze-le-linux does |
| 61 | isis-import-policy-frr | Ze, FRR 10.7.1 twice (FRR-A, FRR-B on separate links) | isis filter import decides what Ze installs and nothing else: the hidden prefix is out of Ze's RIB and FIB, listed by show rib candidates as hidden by the isis import chain, and still flooded on, so FRR-B learns it through Ze ("Separate links between daemons") |
| 62 | ospf-import-policy-bird | Ze, BIRD 3.4.0 | ospf filter import sets distance 200 on BIRD's tagged external, which then loses to a static path at 150, while the untagged one keeps 110 and wins, judged per path in show rib candidates |
| Paired import | isis-import-policy-frr-forked | Ze with external system RIB, FRR 10.7.1 twice | Same source-hidden/control and onward-flooding assertions as the internal scenario, with exact subprocess placement and consumed peer identities. Live paired validation remains required. |
| Paired import | ospf-import-policy-bird-forked | Ze with external system RIB, BIRD 3.4.0 | Same tagged-distance/control assertions as the internal scenario, plus the central-only reject on 10.88.3.0/24 in both placements. Live paired validation remains required. |
| 63 | bgp-rfc8212-default-reject-frr | Ze (route server, bgp-rib), FRR 10.7.1 twice (FRR at host 3 with no Ze policy, the FRR sink at host 15 with an explicit Ze policy) | RFC 8212 Section 3, the EBGP default reject in both directions, against an FRR that keeps its own bgp ebgp-requires-policy and holds explicit route-maps toward Ze: with no policy on Ze's FRR session, Ze's show bgp rib best must hold the sink's 10.81.15.0/24 and not FRR's 10.81.1.0/24 (after FRR's advertised-routes names it and Ze counts its UPDATE and End-of-RIB), FRR must hold neither Ze's config-originated 10.81.0.0/24 nor the sink's route, and the sink must receive 10.81.0.0/24 from the same config update path. A reload adding a bgp-level final accept in both directions must then deliver every route across the FRR session, with FRR and the sink each counting one establishment and no drop. Results on 2026-10-31, each image built by the harness: unmodified, assertions 1 to 15 pass, so both default rejects hold and the reload's export recovery reaches FRR with no reset, and assertion 16 fails: FRR's 10.81.1.0/24 never reaches Ze's RIB after the reload, because the import policy rising did not yet re-learn the routes the absent policy discarded. Ze's FRR peer now negotiates Route Refresh (route-refresh enable), since a rising import on a peer without it is refused (spec D-1). On 2026-10-31, after the import-rise refresh (e9385df360 through 16744a9e7e), the unmodified build passes all assertions in 25s: the reload sends ROUTE-REFRESH to FRR and FRR's 10.81.1.0/24 reaches Ze's RIB with no session reset. Policy.AcceptsAtEnd answering true for an unconfigured policy (a GOFLAGS=-overlay build): red at assertion 7, Ze's RIB holds 10.81.1.0/24. An export-only break, the PolicyFilterChain final verdict skipped for export and egressFinalAccepts, forward_rs.go and egress_inject_filter.go admitting (overlay): red at assertion 8, FRR holds 10.81.0.0/24. The same break without egressFinalAccepts left assertion 8 green, so the session write gate is the guard Ze's config-originated route meets. sha256:16bb31efdbb8sha256:7a6490c76e9bsha256:41f076b8fae9 |
| 64 | isis-export-policy-frr | Ze, FRR 10.7.1 | isis filter export decides what enters Ze's LSP and at what metric: FRR installs the tagged /32 at its link metric plus the 77 the modify set, the untagged /32 at the link metric plus Ze's default 100, and no route for the /24 the prefix-list rejects ("IGP export-policy scenarios"). Results on 2026-10-31 at 697dc6a618, each image built by the harness: unmodified, pass; exportVerdict answering accepted at DefaultRedistMetric, assertion 2 red with "FRR's IS-IS route for 10.66.0.1/32 has metric 110, want 87"; restored, pass. sha256:310feaf0c24f |
| 65 | ospf-export-policy-frr | Ze, FRR 10.7.1 | ospf filter export decides which AS-External-LSAs Ze originates and what they carry: FRR holds the tagged route's LSA at metric 50, E1, tag 200, the untagged one at metric 20, E2, tag 0, and none for the rejected route ("IGP export-policy scenarios"). Results on 2026-10-31 at 697dc6a618, each image built by the harness: unmodified, pass; applyExportSets returning its input, assertion 2 red with "FRR's AS-External-LSA for 10.67.1.0/24 has metric 20, want 50"; restored, pass. The first run found FRR 10.7.1 writing "E2 (Larger than any link state path)" for type 2, which the checker now reads by its first word (697dc6a618). sha256:7459d9f4decasha256:49ebad4b55ccsha256:8cfc88af0ea1 |
| 67 | bgp-rapid-reconnect-initial-sync-frr | Ze, FRR 10.7.1 | A session's initial sync ends with its session: FRR runs clear bgp <ze> three times, each on the first probe (every 100 ms) that shows the next session Established by its connectionsEstablished count, so each clear lands inside ze's initial sync. On the fourth session FRR's show bgp neighbor <ze> json must count four establishments and three drops, report gracefulRestartInfo.endOfRibRecv for ipv4Unicast and ipv6Unicast and no other family, and, in that same answer, an acceptedPrefixCounter equal to the prefixes ze.conf originates in each family (three IPv4, two IPv6). show bgp <afi> unicast neighbors <ze> routes json must then hold each of those prefixes with exactly one path from ze and nothing else, and a last read of the neighbor must still pass. ze advertises Graceful Restart because FRR prints End-of-RIB received only for a peer whose Graceful Restart capability it received (bgp_show_peer_gr_extra_info), and FRR clears those flags when a session stops (bgp_stop), so the flags read belong to the fourth session. Claimed for the positive outcome only (spec R-9). Results, 2026-11-14, QEMU docker-lab on arm64: pass with ze image. Revert run with the join of the initial sync removed from the runOnce session-end defer through a build overlay, ze image: also pass, so the scenario does not discriminate the fix; the unit test TestPredecessorSyncWritesNothingOnSuccessor does. sha256:5f80bd3c10b4sha256:484c9683e046 |
| 68 | peer-stops-reading-bird | Ze, BIRD 3.4.0 | RFC 4271 Section 8.2.2, Event 10, when the peer has stopped reading and ze still has routes to write to it: the hold timer must drop the TCP connection behind the blocked writer. A frozen BIRD and an idle ze cannot show that, because with nothing to write no writer holds ze's write path and the close succeeds on any build (bgp-holdtime-deadpeer-frr is that case). So the checker first requires the session up with two routes held on each side, freezes BIRD's container, and reloads ze with ze-reload.conf, which adds 131072 routes to the peer (two /8 split into /24, one UPDATE of 451 octets each, about 56 MiB). The reload delivers them on the running session and is not waited for: it is the writer that blocks. Ze must log session closed for BIRD (Peer.sessionDown, logged once Session.Run has returned) between 5 s and 20.5 s after the freeze. The hold time is 9 s and BIRD's last KEEPALIVE is at most 3 s old at the freeze, so the timer expires 6 s to 9 s after it and the close may take two more; a close that waits for the blocked write instead comes no sooner than the 30 s write budget (ze.fwd.write.deadline, which the lab leaves at its default), and 20.5 s is halfway between the two. Two readings then prove the writer was blocked, because without one the timing proves nothing: /proc/net/tcp in ze's container showed at least 16 KiB unsent toward BIRD, and ze logged no delivered Hold Timer Expired NOTIFICATION, which it can write only on a free write path. The checker thaws BIRD on every exit path. After the thaw BIRD must log ze_peer: State changed to up once more, ze must report the peer Established, BIRD must hold exactly 131074 routes from ze (show route protocol ze_peer count), and a last count of both logs must show exactly one session ended and one returned. The scenario's ze.conf logs bgp.reactor.peer at info for the two lines the checker counts |
| 69 | bgp-manual-eor-requested-frr | Ze, FRR 10.7.1 | On a behavior { manual-eor true; } peer, Ze sends no End-of-RIB nobody asked for, and sends the one a process asks for from inside the initial sync after the route that process announced first. Session 1: the process announces inside its peer-UP handler and asks for nothing; FRR's log must hold the route decode and no bgp_update_receive_eor line through a 10 s hold. After clear bgp <ze>, session 2: the process announces and asks for the IPv4 unicast End-of-RIB inside its peer-UP handler; the log written after the clear must hold the route decode, then exactly one marker through a 10 s hold. FRR's connectionsEstablished must read 1, then 2, and the process's records must show each session's requests answered within 4 s. Recorded 2026-11-14 in the Docker lab, each run with the Ze image rebuilt from the tree under test: green on the fix (ze image); red with the requested form suppressed during the sync again in announceEOR (assertion 7: FRR never logged the End-of-RIB after the second route; image); red with the sync's marker forced on the manual-eor peer in sendInitialRoutes (assertion 3: FRR decoded 1 End-of-RIB on the first session, where nothing asked for one; image sha256:fe2dd16c5cfc. ). 10.84.1.0/2410.84.2.0/24sha256:4da4a47314f2...sha256:883e8bd4bf3b... |
Running
./le test integration interop
INTEROP_SCENARIO=bgp-ebgp-ipv4-frr ./le test integration interop
VERBOSE=1 ./le test integration interop
NO_BUILD=1 ./le test integration interop
FRR_IMAGE=quay.io/frrouting/frr:10.3 ./le test integration interop
BUILD_TIMEOUT=7200 ./le test integration interop
Interop tests require Docker and are not part of the offline precommit gate. They are a separate protocol-validation action.
The first run cross-compiles the lab binaries on the host, then builds the Docker
images. No Dockerfile.ze carries a Go compiler: each one is an Alpine (alpine:3.24.3 in the BGP, IPsec, L2TP, PPPoE, RADIUS and RSVP-TE labs)
base, one apk add, and a COPY of a binary the suite's preflight has already
written into the build context (internal/le/interoplab/zebuild.go,
StageBinaries). Each lab declares the binaries it needs beside the images it
needs, and the bgp lab declares three. Its scenarios also run the harness
from inside the container: a linux le built by internal/le/linuxle
(linuxle.Base), staged at test/interop/le-linux and copied to
/usr/local/bin/le. The third is the daemon built with ze_le added to
DaemonBase, staged at test/interop/ze-le-linux and copied to
/usr/local/lib/ze-le/ze, outside PATH: only a scenario holding a
ze-le-daemon file runs it (see "Optional sidecars"). Harness peers start with entrypoint le and a command that
begins test (le test interop-bgp speaker ...). The image holds no program
under a retired harness name, so a scenario ze.conf that still runs one fails
to start it.
Measured on 2026-09-06 on a 32-core workstation: 6.1s and 4.8s for the two
cross-compiles against a warm cache/go-cache, at a peak resident set of 1.03
GiB and 0.98 GiB, then 54.7s for the docker build, which is now context
transfer rather than compilation.
Measured again on 2026-10-09 on the same workstation, from a git archive
export of 1cfc44afd7 with an empty cache/go-cache, while other sessions ran
labs: INTEROP_SCENARIO=as-path-prepend-two-octet-peer ./le test integration interop
reported 1 passed, 0 failed in 9m27s of wall time, which includes building
the launcher itself from cold. The staged binaries were written about 18s and
9s apart. The largest process of the whole run peaked at 1524736 KiB
(1.45 GiB) under /usr/bin/time -v. A separate docker build of
test/interop/Dockerfile.ze over those staged binaries took 199.4s at a load
average of 65, with the client peaking at 48.5 MiB, so the time is the
contended daemon rather than any compile. Inside that image ze --version and
le test both ran, and /etc/alpine-release read 3.21.7.
The shape before 2026-09-06 is what those numbers are read against. Dockerfile.ze
copied the whole tree and compiled ze twice with no cache mount. One colima VM of
2 CPUs and 2 GB built it in 2m48s on 2026-09-04, and the same VM took 40m39s for
it earlier that day, when the host disk was full and the guest was thrashing. On
2026-09-06 the kernel killed that build three times on an idle 31 GiB
workstation, once with 23 GiB free and nothing else running, so the compiler in
the container was the thing that did not fit rather than the machine being busy.
A consequence: docker build -f test/interop/Dockerfile.ze . on a clean checkout
now fails at the COPY until ./le test integration interop has run its preflight.
Each converted Dockerfile's header names the action that writes its binary.
Docker.Build never hands docker the context as a directory. Ze walks it,
applies its ignore file, writes it as a tar stream, and runs
docker build -f <Dockerfile inside the context> - with that stream on stdin.
The reason is the macOS host. There the QEMU guest sees the checkout over 9p,
and QEMU's darwin 9p backend answers llistxattr and lgetxattr on a symlink
with ELOOP. BuildKit's directory walker accepts only ENOTSUP from those calls,
so a guest build whose context held a symlink (the checkout's cache, the
migrated tmp/ children) failed with too many levels of symbolic links,
even with cache/ in .dockerignore. Ze's walk reads each entry with lstat
and readlink and sends a symlink as a link entry, never following it, and it
reads no extended attribute. The ignore file is the one Docker would read:
<Dockerfile>.dockerignore beside the Dockerfile wins over the context's
.dockerignore. An excluded directory is not entered unless a ! pattern
could match inside it, and the Dockerfile and the ignore file are always
sent. Owners are sent as root, as the docker CLI sends them. A socket is
skipped, also as the docker CLI does, while a device or a named pipe fails
the build with its path. A Dockerfile outside the context is refused, because
the stream carries the context alone. A context named through a symlink to
its directory (a migrated tmp/ child) is resolved before the walk, as the
docker CLI resolves it, so its contents are sent rather than an empty
archive. A walk that fails part way fails the build with its cause. Docker
may still build an image from the part it read, so that failure also
releases the build's own tag, and the image is never answered as built. A
named context (--build-context) is still passed as a path.
Each build is bounded at 90 minutes, and BUILD_TIMEOUT sets that bound in whole
seconds for a machine slower or faster than that one. The bound stops a wedged
Docker daemon and is not a budget for the build, so a build that finishes returns
at once and a generous bound costs nothing. A value that does not parse, or that
is not positive, keeps the 90 minutes.
An image that needs more than the machine bound declares its own
ImageBuild.Timeout. The build runs under the larger of that field and the
machine bound, so the field only ever lengthens the bound and a value below it
changes nothing; BUILD_TIMEOUT stays the one knob that lowers it. No suite
declares one. The PPPoE suite did until 2026-09-04: 10 minutes for its ze image,
15 for accel-ppp and 10 for the client, each written when the shipped default was
10 minutes and each a cap once the default became 90, because the field then
replaced the machine bound instead of raising it.
Subsequent runs with NO_BUILD=1 skip rebuilds. Once the images exist, the full
suite takes roughly 5-10 minutes depending on session establishment times.
Source-built peers
Owner decision (ai/META-DECISIONS.md): every third-party lab peer runs its latest stable upstream
release. A peer that publishes no stable release is built from source, and only
from source less than one month old. Two do today: tac_plus-ng
(test/interop-tacacs/Dockerfile.tacplusng, used by
internal/component/tacacs/tacplusng_docker_test.go) and freeRouter
(test/interop-rsvpte/Dockerfile.freertr for the RSVP-TE suite, and
test/interop/Dockerfile.freertr for the perf DUT).
The Dockerfile is the declaration, and no list of peers exists beside it. A source-built peer's Dockerfile carries Docker build arguments that its own build also reads:
| Build argument | Meaning |
|---|---|
ZE_SOURCE_REPOSITORY |
The upstream GitHub repository, owner/name |
ZE_SOURCE_COMMIT |
The pinned upstream commit, the full hex id the codeload URL fetches |
ZE_SOURCE_DATE |
That commit's committer date, RFC 3339 |
ZE_SOURCE_CHECKOUT |
Instead of a pin: the named build context that holds a local checkout the build follows |
Docker.Build is where every lab image is built (a suite, a labtest
Spec.Dockerfile, the perf DUTs), and it calls RequireFreshSource before
docker build runs:
- A pinned commit more than one calendar month old refuses the build. The
refusal names the Dockerfile, the commit, its date and age, and the command
that moves it:
./le lab pin update dockerfile <Dockerfile>. Bump asks the GitHub API for the newest commit of the default branch and rewritesZE_SOURCE_COMMITandZE_SOURCE_DATE. The next image build is the proof that the new source still builds and that a Ze patch, such asfreertr/ze-interop-resv.patch, still applies. - A followed checkout (the perf DUT's
freertrcontext,FREERTR_DIR, default../rtr) whose HEAD commit is more than one month old is updated first:git fetch, thengit merge --ff-only @{upstream}. A checkout holding uncommitted changes is never touched: the build is refused, naming the changes andgit -C <checkout> pull --ff-onlyas the fix. A fetch or a fast-forward that fails refuses the build the same way. - A Dockerfile that declares
ZE_SOURCE_REPOSITORYwith neither a complete pin nor a checkout is refused, so a half-declared pin never builds unjudged.
Lab pins against upstream
./le lab pin check lists every lab peer pin the tree holds and compares each
with upstream's latest stable release. It is a report over the network, never a
commit gate. The pins are derived from the files that carry them, under test/
and internal/ (vendor and testdata trees skipped):
| Pin | Read from | Upstream asked |
|---|---|---|
| Image tag | FROM in a Dockerfile (not a stage name, not scratch), an image string literal in Go (an official image with no repository path, such as, only with a dotted version tag on a line whose code names an image). Comments are never read: a comment that quotes an image, or an example of what is not one, pins nothing. Also a scenario's *-image file. alpine:3.24.3 |
The registry's OCI tag list, with the anonymous token it offers (Docker Hub, ghcr.io, quay.io) |
| Go module | go install <path>@<version> in a Dockerfile, or go mod download <path>@<version> where the build patches the release source before compiling it (StayRTR) |
proxy.golang.org, including the next major version paths, since @latest of a /v3 path never names v4 |
| Git tag | git clone --branch <tag> https://github.com/..., a codeload or archive tag URL |
The GitHub repository's latest release |
| Source commit | ZE_SOURCE_COMMIT |
The newest commit of the default branch |
An image tag is compared only with tags of its own shape (the same prefix,
number count and suffix), so golang:1.25-alpine is judged against
1.27-alpine, never 1.27.1-trixie, and a pre-release never wins. A moving tag
such as latest is reported floating with the newest plain release beside it.
An upstream that does not answer leaves its row unresolved with the cause.
A package test's own server container (labtest)
A package _test.go that needs one real server, such as stayrtr for the RPKI
client, starts it with labtest.Start(t, spec) from
internal/le/interoplab/labtest, never with its own exec.Command("docker").
Start pulls the image unless it is digest-pinned (name:tag@sha256:...)
and the daemon already holds it, since the digest makes the local copy exactly
the named image and a host holding it then runs offline; a tag can move, so a
tag reference is always pulled. It then starts one labeled container through
Docker.RunDetached (so a sweep removes it when the test binary is killed),
answers the host address the server listens on, and returns only once
Spec.Ready accepts that address. Spec.Port names the container port with
its transport, interoplab.TCPPort(3323) or interoplab.UDPPort(123), and a
port with no transport is refused before any docker command runs. The
container either publishes that port on the host loopback at a port the daemon
picks (Docker.PublishedPort reads which, under the same transport), or runs
on the host network (Spec.HostNetwork) where it listens on 127.0.0.1:Port
and reaches a listener the test opened. Spec.Entrypoint replaces the image's
entry point, for an image whose entry point writes its own configuration and
ignores the command. t.Cleanup writes the log tail of a failed test, then
removes the container.
| Machine state | What the test does |
|---|---|
| ✕ | skips, naming the install route; fails under ZE_REQUIRE_DOCKER=1 |
docker info fails |
fails, naming the probe error |
| The image cannot be pulled | fails; under the live tag it skips an ad hoc run and fails under ZE_REQUIRE_DOCKER=1 (owner decision Q-6, pull_live.go) |
A Spec.Dockerfile image does not build |
fails, naming the image, the Dockerfile and the docker build command that shows the failing step |
The container does not start, or Ready never accepts |
fails, then the log tail follows |
Two package tests start their server this way. The RPKI live tests start
stayrtr over TCP (internal/component/bgp/plugins/rpki/rpki_live_test.go, tag
live). The NTP client tests start chrony (dockurr/chrony:4.9) over UDP
(internal/plugins/ntp/ntp_chrony_docker_test.go, tag docker) and run
syncWorker.doSync with the real query: a server on its local reference at
stratum 7 is accepted and reported at stratum 7 with a near-zero offset, and a
server with no synchronised source (leap indicator 3, stratum 0 on the wire)
is rejected with no clock step. The container reads the host's own kernel
clock, so the offset is bounded, never a configured value.
A server with no published image is built from its Dockerfile:
Spec.Dockerfile names it from the module root, and Start then builds
Spec.Image through Docker.Build, with the Dockerfile's directory as the
context, instead of pulling it, and releases the tag the run owns after the
container is removed; a build that fails fails the test. A server no probe from
the host can observe, such as a UDP listener behind a published port that
accepts datagrams before the server binds, names Spec.ReadyInContainer
instead of Spec.Ready: a command run in the container whose exit 0 is ready.
A test that reads what the server wrote calls labtest.StartContainer, which
answers the Container with its address and Container.Exec. The flow-export
collector tests run this way (internal/plugins/flowexport/collectors_docker_test.go,
tag docker): nfcapd/nfdump and sflowtool built from
test/interop-flowexport/Dockerfile.collectors, ready once the collector's
socket is in the container's UDP table, their decoded records read with
docker exec (docs/architecture/flowexport/flow-export-1-counter-export.md,
"Independent collector tests").
A package test whose real peer is a client, against a server the test runs
in-process, runs it with labtest.Run(t, client). Run pulls the image as
Start does, runs
one labeled container to its exit through Docker.RunOneShot, and answers the
client's stdout, stderr and exit code, a non-zero exit included, so a test can
assert a refused request. Client.HostNetwork puts the client in the host's
network namespace (OneShotContainer.Network), where it reaches the test's
listener on 127.0.0.1, and Client.Mounts bind-mounts files the test wrote,
such as a CA certificate (OneShotContainer.Mounts). The machine states in the
table below are handled as Start handles them; a run the docker client did
not finish (interoplab.OneShotDidNotExit), or an exit code docker run keeps
for its own failure (125 to 127), fails the test. The gNMI server tests run
gnmic this way (internal/component/gnmi/gnmic_docker_test.go, tag docker):
Capabilities, Get, Subscribe ONCE and Set over TLS with a bearer token,
decoded from gnmic's protojson output, and a request with no token or a wrong
one refused Unauthenticated.
labtest runs no Docker host kernel check: Ze runs in the test process on the
host, and that check guards a containerised Ze. A peer that needs a Ze-kernel
feature belongs in an interoplab.Suite lab. Only a _test.go file, or tooling
under internal/le, imports labtest (./le arch tier check,
docs/architecture/module-tiers.md).
Debugging Failures
The shared orchestrator does not read container logs. Each suite's checker adds them to the failure it returns:
| Suite | What a failed checker carries |
|---|---|
BGP (interop) |
the last 80 log lines of each peer the scenario started (checkerFailure), pmacct's message log, and for speaker and speaker2 a section headed extended relay files: the relay's -result.json verdict and the last 8 rows of its -ze.jsonl and -frr.jsonl captures under the extendedCaptureBase prefix, each row cut to 512 characters and the section to its last 8192 octets. A file that does not exist is printed missing, and a prefix with none of the three files says so in one line. The section is read even when the speaker logged nothing (extendedRelayDump). internal/le/interoplab/bgp/check_engine.gointernal/le/interoplab/bgp/extended_relay.go |
IPsec (interop-ipsec) |
the last 60 log lines of Ze, strongSwan, the NAT box and FRR, for each one the scenario started, read after the checker's own deadline has passed, all of them within one 20 s bound (checkerFailure). internal/le/interoplab/ipsec/failure.go |
RSVP-TE (interop-rsvpte) |
each peer's capture tail and the last 60 log lines (diagnosticError). internal/le/interoplab/rsvpte/checkers.go |
A package test's own server (labtest.Start) |
the last Spec.LogLines lines (1 to 1000) of its container, written to the test log by t.Cleanup when the test failed, before the container is removed (host.release). internal/le/interoplab/labtest/labtest.go |
A docker exec that runs past its 30 s bound names the container, the command it
ran there (cut at its first line and at 160 bytes), and the bound
(execOperation, internal/le/interoplab/docker.go).
For more detail:
VERBOSE=1enables debug output (polling status, container commands, raw CLI output)SESSION_TIMEOUT=120increases the session establishment timeout (default 90s)- Single-scenario runs isolate the problem:
INTEROP_SCENARIO=bgp-graceful-restart-frr ./le test integration interop
Writing a New Scenario
- Create a descriptively named directory under
test/interop/scenarios/. - Add
ze.confand whichever peer configs the scenario needs. - Add the scenario's ordered assertions to
scenarioOperationsandscenarioExtras, or register a bespoke checker inspecialCheckerswhen the control flow is non-linear. - For a bespoke checker, put each decision in a pure predicate in
check_rfc_predicate.goand add its both-polarity subtest toTestBespokeCheckerBranches. - Run
INTEROP_SCENARIO=<name> ./le test integration interop.
TestCheckerPopulationMatchesProducer compares every scenario directory with
the package-local registry. TestEveryCheckerFailsClosedWithoutPeerEvidence
rejects a checker that can pass without reading a peer. Each negative assertion
must carry positive proof that its query mechanism ran.
A scenario that reads the RIB attaches the RIB plugin
plugin { internal rib { use bgp-rib; } } loads the plugin. It does NOT feed it.
A peer delivers an event to a process only where both halves agree, and the
peer's half is its attach block (Server.PeerScopedProcs,
internal/component/plugin/server/delivery_graph.go). A peer with no
attach process rib block grants nothing, so the plugin sees no peer at all and
every RIB question answers empty:
attach process rib {
receive [ update state refresh ];
}
The tell is "peers": 0 from show bgp rib status while show bgp peer list
reports both sessions Established. Ze also logs it at startup: "the plugin
declared events and no peer attaches it".
Explicit configuration is authoritative at every start
ze start <file> reads that file on every start, including a container restart.
Persisted configuration from a previous run does not replace the explicit input.
The labs mount individual configuration files, so replacing a source file by
rename can leave an existing container's bind mount on the old inode. Recreating
the container refreshes the mount; the harness creates each container once.
For Ze's configuration syntax, see docs/architecture/config/syntax.md.
Copy an existing scenario's ze.conf as a starting point.
ExaBGP Wire Compatibility (test/exabgp-compat/)
A separate test suite validates that Ze's wire encoding matches the reviewed ExaBGP API 6.0.0 contract fixtures. The compiled Go server negotiates each BGP session and compares every received frame byte-for-byte.
What It Tests
The harness migrates each ExaBGP-derived configuration, runs Ze, and compares
its wire bytes with the known-good fixture. The 42 .ci cases in
test/exabgp-compat/encoding/ use option=file:, option=serial, 1:cmd:,
1:raw:, 1:signal:, and 1:json: records rather than the standard .ci
format. option=serial marks process-driven fixtures that must not overlap
other ExaBGP harness instances; the runner executes those after the parallel
batch.
The mock replies with the client's OPEN capabilities, changing only its AS
number and router ID; it does not infer capabilities from expected UPDATEs.
Migration expands an omitted or empty ExaBGP family block to every family
registered in the current binary. An explicit family block stays restricted
to its entries. conf-vpn explicitly declares ipv4 mpls-vpn, which migrates
to session > family ipv4/mpls-vpn, so both OPENs advertise AFI 1, SAFI 128.
The negotiated-family guard remains active: RFC 4760 Section 8 requires both
speakers to advertise the family for bidirectional exchange.
<prefix>:signal:<NAME> marks the point in a connection's script where the
runner reloads Ze. It divides the script: every raw frame written before it
must match before the reload happens, and the frames after it are matched only
once it has. Each signal step consumes the NEXT option=file: config the case
names, so a case owes one config more than it has signals, and the runner
refuses a case where those two counts disagree. test/exabgp-compat/api/api-reload.ci
is the case that drives this: its withdrawal of 2.0.0.0/24 is produced BY the
reload.
The fixtures name ExaBGP's reload signal, SIGUSR1. Ze reloads on SIGHUP, so
the runner translates the name (deliverExaBGPReloads), writes the next
migrated config over the path Ze reads, and signals Ze's own pid rather than the
process group, which also holds the bridge's scripts.
Coverage includes:
| Category | Examples |
|---|---|
| Address families | IPv4/IPv6 unicast, VPN, FlowSpec, FlowSpec VPN, EVPN, VPLS, MPLS labeled, MUP, MVPN |
| Path attributes | ORIGIN, AS_PATH, NEXT_HOP, MED, LOCAL_PREF, communities (standard, extended, large), AGGREGATOR, ORIGINATOR_ID, PREFIX_SID, SRv6 |
| Capabilities | 4-byte ASN, ADD-PATH, link-local next-hop, software version, hostname |
| Edge cases | Generic/unknown attributes, self-referencing routes, group limits, IPv4+IPv6 mixed configs, deferred announcement (watchdog) |
Running
./le test functional exabgp
ExaBGP compatibility is part of the offline precommit gate.
Test Hierarchy
| Workflow | Includes Interop? | Includes ExaBGP? | Requires Docker? |
|---|---|---|---|
| Offline precommit gate | ✕ | ✓ | ✕ |
| Standard functional sweep | ✕ | ✓ | ✕ |
./le test integration interop |
✓ | ✕ | ✓ |
./le test functional exabgp |
✕ | ✓ | ✕ |
Interop tests are intentionally separate from the pre-commit gate because they require Docker and take longer to run. ExaBGP wire compatibility tests run as part of the standard verification suite.
Current Scope
Interop scenarios cover core BGP: session establishment, route exchange, withdrawal, capabilities (4-byte ASN, ADD-PATH, GR, route refresh, PATHS-LIMIT), communities, MD5 auth, route server behavior, route reflection, policy import/export, RPKI origin validation, BMP monitoring, BFD failover, ECMP, SRv6 VPNv6, remove-private-as export policy, GTSM, AS112, and non-unicast address families (EVPN, VPN, FlowSpec). The suite also includes full IS-IS and OSPFv2/OSPFv3 interop families (adjacency, flooding, SPF, dual-stack, authentication, TE, LFA/TI-LFA, graceful restart, and segment routing). ExaBGP compat covers wire encoding for all supported address families.
Known Vendor Limitations
| Vendor | Limitation | Affected Scenario | Workaround |
|---|---|---|---|
BIRD (observed at 2.15; unverified at 3.4.0 until bgp-ipv6-ebgp-bird runs) |
Enforces next-hop reachability for IPv6 routes. On IPv4-only Docker networks, IPv6 next-hops are unreachable and BIRD rejects routes as invalid (RFC 7606 treat-as-withdraw). | bgp-ipv6-ebgp-bird | Add multihop; to BIRD config to disable the directly-connected next-hop check. |
Related Documents
.citest format -- Ze's standard functional test file format- Functional test system -- complete guide to the functional test system
- BGP implementation comparison -- feature matrix comparing Ze with FRR, BIRD, GoBGP, ExaBGP, and others
- ExaBGP comparison report -- detailed implementation differences between Ze and ExaBGP