Benchmarking
Ze includes le perf, a tool for measuring BGP route propagation latency
through a device under test (DUT). It works with any BGP implementation,
including Ze.
le perf send runs one benchmark against a DUT, le perf compare compares
result files, and le perf track follows the history. Every word after the verb reaches the program
unchanged, and the program's exit code is le's.
Architecture
+-----------+
le perf send | | le perf send
(sender) ----> | DUT | ----> (receiver)
AS 65001 | AS 65000 | AS 65002
+-----------+
The sender establishes a BGP session with the DUT, injects routes, and the
receiver measures when those routes arrive. Both sessions are managed by
le perf send in a single process. Timing starts when the sender writes the first
UPDATE and stops when the receiver has collected all expected prefixes.
Quick Start
Run a benchmark against Ze on localhost:
# Start ze with a config that accepts peers from 127.0.0.1 and 127.0.0.2
ze start test-config.conf &
# Run the benchmark
./le perf send --dut-addr 127.0.0.1 --dut-asn 65000 --dut-name ze --routes 1000
With JSON output saved to a file:
./le perf send --dut-addr 127.0.0.1 --dut-asn 65000 --dut-name ze \
--routes 1000 --output result-ze.json
Running Benchmarks
For a complete flag reference, see le perf send.
Encoding Modes
Three encoding modes measure different code paths through the DUT:
| Mode | Flag | What It Tests |
|---|---|---|
| IPv4 inline NLRI | --family ipv4/unicast (default) |
Standard IPv4 unicast path with NLRI at the end of the UPDATE |
| IPv4 force-MP | --family ipv4/unicast --force-mp |
IPv4 encoded via MP_REACH_NLRI attribute (exercises multiprotocol path) |
| IPv6 MP | --family ipv6/unicast |
IPv6 unicast via MP_REACH_NLRI (standard for non-IPv4 families) |
Multi-Iteration
By default, le perf send executes 5 iterations with 1 warmup run. The warmup
run is discarded, and outliers beyond 2 standard deviations from the median
convergence time are removed (minimum 3 iterations kept). Final results report
median and standard deviation across the kept iterations.
| Flag | Default | Purpose |
|---|---|---|
--repeat |
5 |
Total iterations (including warmup) |
--warmup-runs |
1 |
Warmup iterations discarded from results |
--iter-delay |
3s |
Pause between iterations for clean separation |
More iterations improve statistical confidence. For reliable results, use at
least --repeat 10:
./le perf send --dut-addr 172.31.0.2 --dut-asn 65000 --repeat 10 --warmup-runs 2
Timing
| Flag | Default | Purpose |
|---|---|---|
--warmup |
2s |
Delay after session establishment before injecting routes |
--connect-timeout |
10s |
TCP connection timeout |
--duration |
60s |
Maximum wait time for convergence per iteration |
--iter-delay |
3s |
Delay between iterations |
--reconnect-wait |
0 |
How long to keep dialing a DUT that refuses the next iteration's session |
The --iter-delay flag controls whether iterations run back-to-back or with
a pause. Longer delays give the DUT time to settle between measurements.
Every iteration ends its sessions with a Cease and opens new ones. A DUT may
hold a peer in Idle after a reset, as RFC 4271 Section 8.1.1 allows: OpenBGPD
holds it 30 s after the second reset and doubles the hold after each later
one, closing every connection the peer opens meanwhile. With
--reconnect-wait, le perf send dials such a DUT again every second until
it accepts the session or the wait runs out; timing starts only once the
session is up, so the wait is not measured. With the default 0, the run
fails at the first connection the DUT closes before its OPEN, and the error
names the flag. ./le perf run passes --reconnect-wait 10m.
Cross-Implementation Comparison
Automated Docker Runner
The included test runner benchmarks all eight supported implementations in Docker:
| DUT | Image | Config | Forwarding mechanism |
|---|---|---|---|
| Ze | ze-interop (built) | test/perf/configs/ze.conf |
bgp-rs plugin, import and export final accept (RFC 8212 refuses an EBGP direction with no policy) |
| FRR | quay.io/frrouting/frr:10.7.1, pinned by digest (interoplab.FRRImage) |
test/perf/configs/frr.conf |
route-map PERMIT |
| BIRD | bird-interop (built) | test/perf/configs/bird.conf |
import/export all |
| GoBGP | gobgp-interop (built) | test/perf/configs/gobgp.toml |
default accept policy |
| rustbgpd | rustbgpd-interop (built from source) | test/perf/configs/rustbgpd.toml |
route_server_client |
| rustybgp | rustybgp-interop (built) | test/perf/configs/rustybgp.toml |
default accept policy |
| FreeRtr | freertr-interop (built) | test/perf/configs/freertr-sw.txt |
eBGP + redistribute connected |
| OpenBGPD | openbgpd-interop (built) | test/perf/configs/openbgpd.conf |
allow from/to any |
The Ze DUT runs on the Docker daemon's kernel, so a run that tests it first
builds test/interop/ze-linux, the ze the image carries, for the daemon's
architecture, then checks that kernel with it and refuses one that lacks any
feature Ze enrolls, naming each. The check and its
routes are in docs/architecture/testing/interop.md, "The Docker host kernel
check".
# Build every DUT image, measure every DUT, and record the run.
./le perf run
# Measure one DUT, or several.
./le perf run dut ze
./le perf run dut "ze bird rustbgpd"
# Run one step: build the images only, or measure with the images that exist.
./le perf run step build
DUT_ROUTES=10000 DUT_REPEAT=5 ./le perf run step test dut ze
# Append the results of the last measurement to the committed NDJSON history.
./le perf record history
# The release evidence gate: measure ze, append the result, fail on a regression.
./le perf record evidence
A run with no step keyword builds the images and measures. step build
records nothing, because it measured nothing. Any other step value exits 2
and names the value. A run that measured and passed records the commit, so the
perf nudge (./le perf suggest) stops asking for one. A run that failed, for
any cause below, records nothing.
An image that fails to build or pull fails the run, and the run prints an error
naming the DUT and the cause. In a step build run the image is the whole
answer, so the run names every DUT that has no image, prints the rebuild
command, exits 1, and never reports the images as built. In a run that also
measures, the test step still measures every DUT whose image was made, but it
never measures a DUT whose build failed, even from an image an earlier run
left. The summary line reads FAIL, counts that DUT as failed, gives its
cause, and the run exits 1.
A DUT is skipped only when the run built no image (NO_BUILD=1, or
step test) and none exists. The summary names the skipped DUTs and the
./le perf run step build dut "..." command that builds them. Under
NO_BUILD=1 that command is env -u NO_BUILD ./le perf run step build dut "...", because the plain form would build nothing. Skips beside a measured DUT
still pass, but a run that measured no DUT prints FAIL no DUT was measured
and exits 1, because a run of skips has no result.
A DUT whose measurement lost a route is failed, because its timings describe
the routes that arrived, or nothing, and a DUT that forwarded nothing reads
as the fastest. After each le perf send pass the runner reads the result
file and judges it with the same check that sets le perf send's own exit
code: every route sent must reach the receiver, and a result that sent none
measured nothing. The ze DUT's propagation pass (ze-propagation.json) is
judged the same way. For a pass that lost routes the run prints the DUT's last
log lines under its name, while the DUT still runs. The summary line reads
FAIL, counts the DUT as failed, and names the pass and the count, as in
ze: the measurement lost routes: 0 of 100000 routes reached the receiver.
It then prints the ./le perf run dut "..." command to run once the DUT is
fixed, and the run exits 1. A result file that does not read or parse fails
the pass the same way. A pass that lost routes is left out of the comparison
table, the HTML report and the doc snapshot, so no row of 0ms values reads as
a fast result, and a DUT left with no rendered pass is named on the run's
no result for line. A complete pass of the same DUT is still rendered, and
when both passes of ze are complete the comparison shows both, one row each
(see "Passes" below).
NO_BUILD=1 with step build is refused: the run would build nothing, so it
names NO_BUILD as the reason, prints the env -u NO_BUILD ./le perf run step build command that unsets it, and exits 1 without claiming any image was built.
The runner builds no host benchmark program. It cross-builds le for linux
with CGO_ENABLED=0 into tmp/perf-run/linux-<arch>/le, and prints how long
that build took. It mounts the file at /usr/local/bin/le in the sender
container, which runs le perf send. The le that started the run renders
the report as le perf compare. The terminal-demo recorder container runs a
linux le built by the same recipe (docs/contributing/gh-pages.md).
Every container, network and image tag the run creates carries the lab's
owner labels. Ctrl-C (SIGINT) or SIGTERM stops the run, and the run still
removes the DUT, the sender and the network before it exits 1. A removal that
fails is printed on stderr with the resource's name, and the run exits 1; a
run that died before its removals (SIGKILL, a crash) leaves resources that
./le test integration clean, or the next lab start, removes. The DUT
images themselves are kept, so the next step test finds them.
Results are written to test/perf/results/ as JSON files. An HTML comparison report is generated automatically.
Passes
Every result records the pass it measured in its pass field:
pass |
le perf send |
What it measures |
|---|---|---|
live-forwarding |
without -sender-first |
The receiver connects first, so the DUT forwards each UPDATE as it arrives |
rib-propagation |
with -sender-first |
The sender connects first, so the DUT holds every route before the receiver connects and is sent the table |
The perf runner measures every DUT in the live-forwarding pass (<dut>.json,
named the measurement in its summary), and ze also in the rib-propagation
pass (ze-propagation.json, the propagation pass). The pass, its file name
and its summary phrase are declared once, in perf.Pass.
le perf compare keeps the newest result for each DUT, family, force-mp and
pass, so both passes of one DUT render, each in its own row. The markdown
table, the HTML table and the doc snapshot carry a Pass column, and each HTML
chart bar is labelled with the DUT and the pass, as in ze (rib-propagation).
A result written before the field existed has no pass: its Pass cell reads
not recorded, it is never shown as either pass, and it does not displace a
result that records one. A pass value that is not one of the two above is
refused: le perf compare prints the file name, the value and the accepted
values, and exits 1. The doc snapshot's DUT setup section lists a DUT once,
however many passes it has.
Manual Reports
After running benchmarks, generate reports from the result files:
# Markdown report (default)
./le perf compare result-ze.json result-gobgp.json result-rustbgpd.json
# HTML report
./le perf compare --html result-ze.json result-gobgp.json > comparison.html
Comparison and trend reports validate every decoded row before selecting the
latest sample, applying --last, or writing output. Any nonzero routes-lost
refuses the whole report, naming the input and measurement; an older lossy row
cannot be hidden by a newer clean row. Missing or null convergence-ms is not a
measurement and is refused; an explicitly measured 0 is valid. Raw JSON and
NDJSON remain diagnostic evidence and retain loss rather than refusing it.
All comparison formats (Markdown, HTML tables and charts, and --doc) and both
trend formats use this rule. Compare has no JSON comparison format. Invalid
input writes no report bytes. A stdout write error fails the command, but bytes
already written cannot be rolled back; shell redirection can truncate its target
before the command starts. The automated runner publishes its HTML report and
doc snapshot by replacing each destination only after successful generation;
failure retains that file's old bytes, not a transaction across all artifacts.
For the full flag reference, see le perf compare.
Filter throughput
./le perf filter is the throughput harness of the route filter paths. It
covers three sets of Go benchmarks, and needs no Docker and no DUT:
| Path | Benchmarks | Package |
|---|---|---|
| In-process: the typed DirectBridge filter slot | BenchmarkFilterFastPath |
pkg/plugin/sdk |
External: one filter-update batch of 1, 16 and 64 routes |
BenchmarkFilterForkedBatch |
pkg/plugin/sdk |
| External slow path: a filter that modifies the route, from the received UPDATE to the rebuilt one, in batches of 1, 16 and 64 routes | BenchmarkFilterSlowPathModify |
internal/component/bgp/reactor (tag ze_bgp) |
BenchmarkFilterSlowPathModify drives each route through the real import
chain. The engine projects the declared fields, and PluginConn.CallRPC with
MuxConn framing carries the batch over net.Pipe to a real SDK plugin,
whose OnFilter evaluator sets local-preference 200. The engine then decodes
the verdicts, applies the set and rebuilds the UPDATE body. The plugin
server's process lookup and the batcher's queue are left out, so every
exchange carries exactly the batch size. Next to the per-exchange figures it
reports ns/route, B/route and allocs/route.
# Check the harness builds: compile each package, list the benchmarks,
# measure nothing, and print the go command of each set.
./le perf filter
# Measure: every set, ten runs each, with allocations. macOS only.
./le perf filter measure
measure refuses to run anywhere but macOS, because performance is measured
on the owner's Mac alone, whose hardware is known. The bare form is the check
an agent runs: it fails when a package no longer builds its benchmarks or a
benchmark the harness names was renamed.
Tracking Performance Over Time
NDJSON History
Each le perf send --json invocation produces a single JSON object. Append
results to an NDJSON (newline-delimited JSON) file to build a history:
./le perf send --dut-addr 127.0.0.1 --dut-asn 65000 --json >> history.ndjson
Trend Reports
Generate a trend report from a history file:
./le perf track history.ndjson
./le perf track --html history.ndjson > trend.html
Regression Detection
Use --check in CI to detect performance regressions. The tool compares the
most recent entry against the previous one using stddev-aware thresholds:
./le perf track --check history.ndjson
Exit code 0 means no regression. Exit code 1 means a regression was detected.
Regressions are flagged when a metric exceeds its threshold percentage AND the delta exceeds the combined standard deviation of the two measurements. This prevents false positives from noisy measurements.
Default thresholds:
| Metric | Default Threshold | Direction |
|---|---|---|
| Convergence time | 20% | Higher is worse |
| Throughput (avg) | 20% | Lower is worse |
| P99 latency | 30% | Higher is worse |
| Routes lost | Any loss | Always flagged |
Custom thresholds:
./le perf track --check --threshold-convergence 15 --threshold-throughput 10 --threshold-p99 25 history.ndjson
Limit the comparison window to the last N entries with --last:
./le perf track --check --last 5 history.ndjson
--last limits the displayed or regression-comparison window, not input
validation. A lossy or missing-convergence row anywhere in the history refuses
the command, including --check, even when it lies outside that window.
For the full flag reference, see le perf track.
Understanding Results
JSON Output Fields
The JSON result object contains these key fields:
| Field | Unit | Description |
|---|---|---|
convergence-ms |
ms | Time from first UPDATE sent to last route received (median); explicit zero is measured, missing/null is absent |
convergence-stddev-ms |
ms | Standard deviation of convergence across iterations |
first-route-ms |
ms | Time to first route arrival |
throughput-avg |
routes/s | Average route propagation rate |
throughput-avg-stddev |
routes/s | Standard deviation of throughput |
throughput-peak |
routes/s | Peak routes/s in any 1-second window |
latency-p50-ms |
ms | 50th percentile per-route latency |
latency-p90-ms |
ms | 90th percentile per-route latency |
latency-p99-ms |
ms | 99th percentile per-route latency |
latency-p99-stddev-ms |
ms | Standard deviation of P99 across iterations |
latency-max-ms |
ms | Maximum per-route latency |
routes-sent |
count | Routes injected by sender |
routes-received |
count | Routes received by receiver |
routes-lost |
count | Difference between sent and received |
pass |
name | live-forwarding or rib-propagation (see "Passes"); absent in a result written before the field existed |
repeat |
count | Total iterations run |
repeat-kept |
count | Iterations kept after outlier removal |
Raw result readers and writers preserve convergence presence: absent/null input
remains absent on serialization, while an explicitly measured zero remains 0,
including when Go results are serialized as values, pointers, slice entries or
map values. Direct Go measurement producers call Result.RecordConvergence,
which records zero explicitly; existing nonzero raw values carry their own
presence. An uninitialized Result has no convergence measurement.
Interpreting Results
Convergence time is the primary metric. It measures how long the DUT takes to forward all injected routes from sender to receiver. Lower is better.
Standard deviation indicates measurement stability. A high stddev relative
to the median suggests noisy measurements. Increase --repeat and
--iter-delay for more stable results.
Throughput shows sustained forwarding rate. The average is computed over the full convergence window. The peak shows the maximum 1-second burst rate.
Latency percentiles show per-route propagation time distribution. P50 is the typical latency, P99 captures tail latency, and max shows the worst case.
Routes lost should always be zero. Any loss indicates the DUT dropped
routes: le perf send then exits 1 (in --json mode it exits 0 once the
result file is written, and the file carries the loss), ./le perf run
fails the DUT, and the regression checker always flags it.
Before you report a number
le perf send already discards the warmup run, drops outliers and reports a
median with its standard deviation. That makes one run repeatable. It does
not say what the number measured, and a result that looks like a win can be a
measurement of something else. Answer these before you report a number, act on
it, or pick between two builds or two implementations.
- Write the claim first, in the words you would ship: "ze converges 100k IPv4 routes 25% faster with change X". The questions below test that sentence.
- Why not twice as good? Name what bounds the result: one core, a lock,
the socket, the sender, or the receiver. Read it from per-process CPU
(
top,pidstat) or a CPU profile taken during a run you do not report. Start ze with--pprof 127.0.0.1:6060to profile it. If the sender or the receiver saturates first, you measuredle perf, not the DUT. - Was every side tuned? Compare release builds with the same flags, the same family and encoding mode, and the same route count. When one side runs on its defaults, you compared configurations, not implementations.
- Did the work happen? Check that
routes-lostis zero and thatroutes-receivedmatchesroutes-sent. A run that dropped or rejected routes is often the fastest run../le perf runfails such a DUT for you; a result you produced withle perf send --jsonyou check yourself. - Is the gap bigger than the noise? To compare two builds, alternate them:
A, B, A, B, at least five times each, appending every
--jsonresult. Then warm caches and machine drift favour neither side. A gap smaller than the spread between runs of the same build is no measurable difference. - Does it matter end to end? A change to a step that takes 10% of convergence can make convergence at most about 11% faster. A result past that limit measured a cache, a no-op or a bug.
Report the verdict first (faster, slower, no measurable difference, or inconclusive). Then give the number with its unit, the run count, the spread and the bound you found. Without a run count, a spread and a bound, call the result inconclusive and say which answer is missing.