Guide

Benchmarking

Ze includes le perf, a tool for measuring BGP route propagation latency through a device under test (DUT). It works with any BGP implementation, including Ze.

le perf send runs one benchmark against a DUT, le perf compare compares result files, and le perf track follows the history. Every word after the verb reaches the program unchanged, and the program's exit code is le's.

Architecture

                  +-----------+
  le perf send    |           |         le perf send
  (sender)  ----> |    DUT    | ---->  (receiver)
  AS 65001        |  AS 65000 |         AS 65002
                  +-----------+

The sender establishes a BGP session with the DUT, injects routes, and the receiver measures when those routes arrive. Both sessions are managed by le perf send in a single process. Timing starts when the sender writes the first UPDATE and stops when the receiver has collected all expected prefixes.

Quick Start

Run a benchmark against Ze on localhost:

# Start ze with a config that accepts peers from 127.0.0.1 and 127.0.0.2
ze start test-config.conf &

# Run the benchmark
./le perf send --dut-addr 127.0.0.1 --dut-asn 65000 --dut-name ze --routes 1000

With JSON output saved to a file:

./le perf send --dut-addr 127.0.0.1 --dut-asn 65000 --dut-name ze \
  --routes 1000 --output result-ze.json

Running Benchmarks

For a complete flag reference, see le perf send.

Encoding Modes

Three encoding modes measure different code paths through the DUT:

Mode Flag What It Tests
IPv4 inline NLRI --family ipv4/unicast (default) Standard IPv4 unicast path with NLRI at the end of the UPDATE
IPv4 force-MP --family ipv4/unicast --force-mp IPv4 encoded via MP_REACH_NLRI attribute (exercises multiprotocol path)
IPv6 MP --family ipv6/unicast IPv6 unicast via MP_REACH_NLRI (standard for non-IPv4 families)

Multi-Iteration

By default, le perf send executes 5 iterations with 1 warmup run. The warmup run is discarded, and outliers beyond 2 standard deviations from the median convergence time are removed (minimum 3 iterations kept). Final results report median and standard deviation across the kept iterations.

Flag Default Purpose
--repeat 5 Total iterations (including warmup)
--warmup-runs 1 Warmup iterations discarded from results
--iter-delay 3s Pause between iterations for clean separation

More iterations improve statistical confidence. For reliable results, use at least --repeat 10:

./le perf send --dut-addr 172.31.0.2 --dut-asn 65000 --repeat 10 --warmup-runs 2

Timing

Flag Default Purpose
--warmup 2s Delay after session establishment before injecting routes
--connect-timeout 10s TCP connection timeout
--duration 60s Maximum wait time for convergence per iteration
--iter-delay 3s Delay between iterations
--reconnect-wait 0 How long to keep dialing a DUT that refuses the next iteration's session

The --iter-delay flag controls whether iterations run back-to-back or with a pause. Longer delays give the DUT time to settle between measurements.

Every iteration ends its sessions with a Cease and opens new ones. A DUT may hold a peer in Idle after a reset, as RFC 4271 Section 8.1.1 allows: OpenBGPD holds it 30 s after the second reset and doubles the hold after each later one, closing every connection the peer opens meanwhile. With --reconnect-wait, le perf send dials such a DUT again every second until it accepts the session or the wait runs out; timing starts only once the session is up, so the wait is not measured. With the default 0, the run fails at the first connection the DUT closes before its OPEN, and the error names the flag. ./le perf run passes --reconnect-wait 10m.

Cross-Implementation Comparison

Automated Docker Runner

The included test runner benchmarks all eight supported implementations in Docker:

DUT Image Config Forwarding mechanism
Ze ze-interop (built) test/perf/configs/ze.conf bgp-rs plugin, import and export final accept (RFC 8212 refuses an EBGP direction with no policy)
FRR quay.io/frrouting/frr:10.7.1, pinned by digest (interoplab.FRRImage) test/perf/configs/frr.conf route-map PERMIT
BIRD bird-interop (built) test/perf/configs/bird.conf import/export all
GoBGP gobgp-interop (built) test/perf/configs/gobgp.toml default accept policy
rustbgpd rustbgpd-interop (built from source) test/perf/configs/rustbgpd.toml route_server_client
rustybgp rustybgp-interop (built) test/perf/configs/rustybgp.toml default accept policy
FreeRtr freertr-interop (built) test/perf/configs/freertr-sw.txt eBGP + redistribute connected
OpenBGPD openbgpd-interop (built) test/perf/configs/openbgpd.conf allow from/to any

The Ze DUT runs on the Docker daemon's kernel, so a run that tests it first builds test/interop/ze-linux, the ze the image carries, for the daemon's architecture, then checks that kernel with it and refuses one that lacks any feature Ze enrolls, naming each. The check and its routes are in docs/architecture/testing/interop.md, "The Docker host kernel check".

# Build every DUT image, measure every DUT, and record the run.
./le perf run

# Measure one DUT, or several.
./le perf run dut ze
./le perf run dut "ze bird rustbgpd"

# Run one step: build the images only, or measure with the images that exist.
./le perf run step build
DUT_ROUTES=10000 DUT_REPEAT=5 ./le perf run step test dut ze

# Append the results of the last measurement to the committed NDJSON history.
./le perf record history

# The release evidence gate: measure ze, append the result, fail on a regression.
./le perf record evidence

A run with no step keyword builds the images and measures. step build records nothing, because it measured nothing. Any other step value exits 2 and names the value. A run that measured and passed records the commit, so the perf nudge (./le perf suggest) stops asking for one. A run that failed, for any cause below, records nothing.

An image that fails to build or pull fails the run, and the run prints an error naming the DUT and the cause. In a step build run the image is the whole answer, so the run names every DUT that has no image, prints the rebuild command, exits 1, and never reports the images as built. In a run that also measures, the test step still measures every DUT whose image was made, but it never measures a DUT whose build failed, even from an image an earlier run left. The summary line reads FAIL, counts that DUT as failed, gives its cause, and the run exits 1.

A DUT is skipped only when the run built no image (NO_BUILD=1, or step test) and none exists. The summary names the skipped DUTs and the ./le perf run step build dut "..." command that builds them. Under NO_BUILD=1 that command is env -u NO_BUILD ./le perf run step build dut "...", because the plain form would build nothing. Skips beside a measured DUT still pass, but a run that measured no DUT prints FAIL no DUT was measured and exits 1, because a run of skips has no result.

A DUT whose measurement lost a route is failed, because its timings describe the routes that arrived, or nothing, and a DUT that forwarded nothing reads as the fastest. After each le perf send pass the runner reads the result file and judges it with the same check that sets le perf send's own exit code: every route sent must reach the receiver, and a result that sent none measured nothing. The ze DUT's propagation pass (ze-propagation.json) is judged the same way. For a pass that lost routes the run prints the DUT's last log lines under its name, while the DUT still runs. The summary line reads FAIL, counts the DUT as failed, and names the pass and the count, as in ze: the measurement lost routes: 0 of 100000 routes reached the receiver. It then prints the ./le perf run dut "..." command to run once the DUT is fixed, and the run exits 1. A result file that does not read or parse fails the pass the same way. A pass that lost routes is left out of the comparison table, the HTML report and the doc snapshot, so no row of 0ms values reads as a fast result, and a DUT left with no rendered pass is named on the run's no result for line. A complete pass of the same DUT is still rendered, and when both passes of ze are complete the comparison shows both, one row each (see "Passes" below).

NO_BUILD=1 with step build is refused: the run would build nothing, so it names NO_BUILD as the reason, prints the env -u NO_BUILD ./le perf run step build command that unsets it, and exits 1 without claiming any image was built.

The runner builds no host benchmark program. It cross-builds le for linux with CGO_ENABLED=0 into tmp/perf-run/linux-<arch>/le, and prints how long that build took. It mounts the file at /usr/local/bin/le in the sender container, which runs le perf send. The le that started the run renders the report as le perf compare. The terminal-demo recorder container runs a linux le built by the same recipe (docs/contributing/gh-pages.md).

Every container, network and image tag the run creates carries the lab's owner labels. Ctrl-C (SIGINT) or SIGTERM stops the run, and the run still removes the DUT, the sender and the network before it exits 1. A removal that fails is printed on stderr with the resource's name, and the run exits 1; a run that died before its removals (SIGKILL, a crash) leaves resources that ./le test integration clean, or the next lab start, removes. The DUT images themselves are kept, so the next step test finds them.

Results are written to test/perf/results/ as JSON files. An HTML comparison report is generated automatically.

Passes

Every result records the pass it measured in its pass field:

pass le perf send What it measures
live-forwarding without -sender-first The receiver connects first, so the DUT forwards each UPDATE as it arrives
rib-propagation with -sender-first The sender connects first, so the DUT holds every route before the receiver connects and is sent the table

The perf runner measures every DUT in the live-forwarding pass (<dut>.json, named the measurement in its summary), and ze also in the rib-propagation pass (ze-propagation.json, the propagation pass). The pass, its file name and its summary phrase are declared once, in perf.Pass.

le perf compare keeps the newest result for each DUT, family, force-mp and pass, so both passes of one DUT render, each in its own row. The markdown table, the HTML table and the doc snapshot carry a Pass column, and each HTML chart bar is labelled with the DUT and the pass, as in ze (rib-propagation). A result written before the field existed has no pass: its Pass cell reads not recorded, it is never shown as either pass, and it does not displace a result that records one. A pass value that is not one of the two above is refused: le perf compare prints the file name, the value and the accepted values, and exits 1. The doc snapshot's DUT setup section lists a DUT once, however many passes it has.

Manual Reports

After running benchmarks, generate reports from the result files:

# Markdown report (default)
./le perf compare result-ze.json result-gobgp.json result-rustbgpd.json

# HTML report
./le perf compare --html result-ze.json result-gobgp.json > comparison.html

Comparison and trend reports validate every decoded row before selecting the latest sample, applying --last, or writing output. Any nonzero routes-lost refuses the whole report, naming the input and measurement; an older lossy row cannot be hidden by a newer clean row. Missing or null convergence-ms is not a measurement and is refused; an explicitly measured 0 is valid. Raw JSON and NDJSON remain diagnostic evidence and retain loss rather than refusing it.

All comparison formats (Markdown, HTML tables and charts, and --doc) and both trend formats use this rule. Compare has no JSON comparison format. Invalid input writes no report bytes. A stdout write error fails the command, but bytes already written cannot be rolled back; shell redirection can truncate its target before the command starts. The automated runner publishes its HTML report and doc snapshot by replacing each destination only after successful generation; failure retains that file's old bytes, not a transaction across all artifacts.

For the full flag reference, see le perf compare.

Filter throughput

./le perf filter is the throughput harness of the route filter paths. It covers three sets of Go benchmarks, and needs no Docker and no DUT:

Path Benchmarks Package
In-process: the typed DirectBridge filter slot BenchmarkFilterFastPath pkg/plugin/sdk
External: one filter-update batch of 1, 16 and 64 routes BenchmarkFilterForkedBatch pkg/plugin/sdk
External slow path: a filter that modifies the route, from the received UPDATE to the rebuilt one, in batches of 1, 16 and 64 routes BenchmarkFilterSlowPathModify internal/component/bgp/reactor (tag ze_bgp)

BenchmarkFilterSlowPathModify drives each route through the real import chain. The engine projects the declared fields, and PluginConn.CallRPC with MuxConn framing carries the batch over net.Pipe to a real SDK plugin, whose OnFilter evaluator sets local-preference 200. The engine then decodes the verdicts, applies the set and rebuilds the UPDATE body. The plugin server's process lookup and the batcher's queue are left out, so every exchange carries exactly the batch size. Next to the per-exchange figures it reports ns/route, B/route and allocs/route.

# Check the harness builds: compile each package, list the benchmarks,
# measure nothing, and print the go command of each set.
./le perf filter

# Measure: every set, ten runs each, with allocations. macOS only.
./le perf filter measure

measure refuses to run anywhere but macOS, because performance is measured on the owner's Mac alone, whose hardware is known. The bare form is the check an agent runs: it fails when a package no longer builds its benchmarks or a benchmark the harness names was renamed.

Tracking Performance Over Time

NDJSON History

Each le perf send --json invocation produces a single JSON object. Append results to an NDJSON (newline-delimited JSON) file to build a history:

./le perf send --dut-addr 127.0.0.1 --dut-asn 65000 --json >> history.ndjson

Trend Reports

Generate a trend report from a history file:

./le perf track history.ndjson
./le perf track --html history.ndjson > trend.html

Regression Detection

Use --check in CI to detect performance regressions. The tool compares the most recent entry against the previous one using stddev-aware thresholds:

./le perf track --check history.ndjson

Exit code 0 means no regression. Exit code 1 means a regression was detected.

Regressions are flagged when a metric exceeds its threshold percentage AND the delta exceeds the combined standard deviation of the two measurements. This prevents false positives from noisy measurements.

Default thresholds:

Metric Default Threshold Direction
Convergence time 20% Higher is worse
Throughput (avg) 20% Lower is worse
P99 latency 30% Higher is worse
Routes lost Any loss Always flagged

Custom thresholds:

./le perf track --check --threshold-convergence 15 --threshold-throughput 10 --threshold-p99 25 history.ndjson

Limit the comparison window to the last N entries with --last:

./le perf track --check --last 5 history.ndjson

--last limits the displayed or regression-comparison window, not input validation. A lossy or missing-convergence row anywhere in the history refuses the command, including --check, even when it lies outside that window.

For the full flag reference, see le perf track.

Understanding Results

JSON Output Fields

The JSON result object contains these key fields:

Field Unit Description
convergence-ms ms Time from first UPDATE sent to last route received (median); explicit zero is measured, missing/null is absent
convergence-stddev-ms ms Standard deviation of convergence across iterations
first-route-ms ms Time to first route arrival
throughput-avg routes/s Average route propagation rate
throughput-avg-stddev routes/s Standard deviation of throughput
throughput-peak routes/s Peak routes/s in any 1-second window
latency-p50-ms ms 50th percentile per-route latency
latency-p90-ms ms 90th percentile per-route latency
latency-p99-ms ms 99th percentile per-route latency
latency-p99-stddev-ms ms Standard deviation of P99 across iterations
latency-max-ms ms Maximum per-route latency
routes-sent count Routes injected by sender
routes-received count Routes received by receiver
routes-lost count Difference between sent and received
pass name live-forwarding or rib-propagation (see "Passes"); absent in a result written before the field existed
repeat count Total iterations run
repeat-kept count Iterations kept after outlier removal

Raw result readers and writers preserve convergence presence: absent/null input remains absent on serialization, while an explicitly measured zero remains 0, including when Go results are serialized as values, pointers, slice entries or map values. Direct Go measurement producers call Result.RecordConvergence, which records zero explicitly; existing nonzero raw values carry their own presence. An uninitialized Result has no convergence measurement.

Interpreting Results

Convergence time is the primary metric. It measures how long the DUT takes to forward all injected routes from sender to receiver. Lower is better.

Standard deviation indicates measurement stability. A high stddev relative to the median suggests noisy measurements. Increase --repeat and --iter-delay for more stable results.

Throughput shows sustained forwarding rate. The average is computed over the full convergence window. The peak shows the maximum 1-second burst rate.

Latency percentiles show per-route propagation time distribution. P50 is the typical latency, P99 captures tail latency, and max shows the worst case.

Routes lost should always be zero. Any loss indicates the DUT dropped routes: le perf send then exits 1 (in --json mode it exits 0 once the result file is written, and the file carries the loss), ./le perf run fails the DUT, and the regression checker always flags it.

Before you report a number

le perf send already discards the warmup run, drops outliers and reports a median with its standard deviation. That makes one run repeatable. It does not say what the number measured, and a result that looks like a win can be a measurement of something else. Answer these before you report a number, act on it, or pick between two builds or two implementations.

  1. Write the claim first, in the words you would ship: "ze converges 100k IPv4 routes 25% faster with change X". The questions below test that sentence.
  2. Why not twice as good? Name what bounds the result: one core, a lock, the socket, the sender, or the receiver. Read it from per-process CPU (top, pidstat) or a CPU profile taken during a run you do not report. Start ze with --pprof 127.0.0.1:6060 to profile it. If the sender or the receiver saturates first, you measured le perf, not the DUT.
  3. Was every side tuned? Compare release builds with the same flags, the same family and encoding mode, and the same route count. When one side runs on its defaults, you compared configurations, not implementations.
  4. Did the work happen? Check that routes-lost is zero and that routes-received matches routes-sent. A run that dropped or rejected routes is often the fastest run. ./le perf run fails such a DUT for you; a result you produced with le perf send --json you check yourself.
  5. Is the gap bigger than the noise? To compare two builds, alternate them: A, B, A, B, at least five times each, appending every --json result. Then warm caches and machine drift favour neither side. A gap smaller than the spread between runs of the same build is no measurable difference.
  6. Does it matter end to end? A change to a step that takes 10% of convergence can make convergence at most about 11% faster. A result past that limit measured a cache, a no-op or a bug.

Report the verdict first (faster, slower, no measurable difference, or inconclusive). Then give the number with its unit, the run count, the spread and the bound you found. Without a run count, a spread and a bound, call the result inconclusive and say which answer is missing.