Benchmarking

le perf send runs one benchmark against a DUT, le perf report compares result files, and le perf track follows the history. Every word after the verb reaches the program unchanged, and the program's exit code is le's.

Architecture

                  +-----------+
  le perf send    |           |         le perf send
  (sender)  ----> |    DUT    | ---->  (receiver)
  AS 65001        |  AS 65000 |         AS 65002
                  +-----------+

The sender establishes a BGP session with the DUT, injects routes, and the receiver measures when those routes arrive. Both sessions are managed by le perf send in a single process. Timing starts when the sender writes the first UPDATE and stops when the receiver has collected all expected prefixes.

Quick Start

Run a benchmark against Ze on localhost:

# Start ze with a config that accepts peers from 127.0.0.1 and 127.0.0.2
ze start test-config.conf &

# Run the benchmark
./le perf send --dut-addr 127.0.0.1 --dut-asn 65000 --dut-name ze --routes 1000

With JSON output saved to a file:

./le perf send --dut-addr 127.0.0.1 --dut-asn 65000 --dut-name ze \
  --routes 1000 --output result-ze.json

Running Benchmarks

For a complete flag reference, see le perf send.

Encoding Modes

Three encoding modes measure different code paths through the DUT:

Multi-Iteration

By default, le perf send executes 5 iterations with 1 warmup run. The warmup run is discarded, and outliers beyond 2 standard deviations from the median convergence time are removed (minimum 3 iterations kept). Final results report median and standard deviation across the kept iterations.

More iterations improve statistical confidence. For reliable results, use at least --repeat 10:

./le perf send --dut-addr 172.31.0.2 --dut-asn 65000 --repeat 10 --warmup-runs 2

Timing

The --iter-delay flag controls whether iterations run back-to-back or with a pause. Longer delays give the DUT time to settle between measurements.

Cross-Implementation Comparison

Automated Docker Runner

The included test runner benchmarks all eight supported implementations in Docker:

# Build every DUT image, measure every DUT, and record the run.
./le perf run

# Measure one DUT, or several.
./le perf run dut ze
./le perf run dut "ze bird rustbgpd"

# Run one step: build the images only, or measure with the images that exist.
./le perf run step build
DUT_ROUTES=10000 DUT_REPEAT=5 ./le perf run step test dut ze

# Append the results of the last measurement to the committed NDJSON history.
./le perf history-record

# The release evidence gate: measure ze, append the result, fail on a regression.
./le perf evidence-record

A run with no step keyword builds the images and measures. step build records nothing, because it measured nothing. Any other step value exits 2 and names the value. A run that measured records the commit, so the perf nudge (./le perf suggest) stops asking for one.

The runner builds no host benchmark program. It cross-builds le for linux with CGO_ENABLED=0 into tmp/perf-run/linux-<arch>/le, and prints how long that build took. It mounts the file at /usr/local/bin/le in the sender container, which runs le perf send. The le that started the run renders the report as le perf report. The terminal-demo recorder container runs a linux le built by the same recipe (docs/contributing/gh-pages.md).

Results are written to test/perf/results/ as JSON files. An HTML comparison report is generated automatically.

Manual Reports

After running benchmarks, generate reports from the result files:

# Markdown report (default)
./le perf report result-ze.json result-gobgp.json result-rustbgpd.json

# HTML report
./le perf report --html result-ze.json result-gobgp.json > comparison.html

For the full flag reference, see le perf report.

Tracking Performance Over Time

NDJSON History

Each le perf send --json invocation produces a single JSON object. Append results to an NDJSON (newline-delimited JSON) file to build a history:

./le perf send --dut-addr 127.0.0.1 --dut-asn 65000 --json >> history.ndjson

Trend Reports

Generate a trend report from a history file:

./le perf track history.ndjson
./le perf track --html history.ndjson > trend.html

Regression Detection

Use --check in CI to detect performance regressions. The tool compares the most recent entry against the previous one using stddev-aware thresholds:

./le perf track --check history.ndjson

Exit code 0 means no regression. Exit code 1 means a regression was detected.

Regressions are flagged when a metric exceeds its threshold percentage AND the delta exceeds the combined standard deviation of the two measurements. This prevents false positives from noisy measurements.

Default thresholds:

Custom thresholds:

./le perf track --check --threshold-convergence 15 --threshold-throughput 10 --threshold-p99 25 history.ndjson

Limit the comparison window to the last N entries with --last:

./le perf track --check --last 5 history.ndjson

For the full flag reference, see le perf track.

Understanding Results

JSON Output Fields

The JSON result object contains these key fields:

Interpreting Results

Convergence time is the primary metric. It measures how long the DUT takes to forward all injected routes from sender to receiver. Lower is better.

Standard deviation indicates measurement stability. A high stddev relative to the median suggests noisy measurements. Increase --repeat and --iter-delay for more stable results.

Throughput shows sustained forwarding rate. The average is computed over the full convergence window. The peak shows the maximum 1-second burst rate.

Latency percentiles show per-route propagation time distribution. P50 is the typical latency, P99 captures tail latency, and max shows the worst case.

Routes lost should always be zero. Any loss indicates the DUT dropped routes, which the regression checker always flags.