Chaos Testing

Quick Start

# Run the native chaos unit suites.
./le chaos selftest unit
./le chaos selftest cli-unit

# Run the chaos orchestrator. le builds itself; no separate binary is needed.
./le chaos run --seed 42 --peers 4 --duration 30s

./le chaos run <options> runs the chaos orchestrator. A help word after ./le chaos run reaches the orchestrator, which prints its own option list.

Flags

Chaos Tool

The chaos tool, ./le chaos run, is a chaos simulator that runs multiple BGP peers against a ze route server, validates route propagation, and injects faults.

The web dashboard shows real-time peer status, per-family route propagation, convergence progress, and fault triggers. Color coding indicates propagation state: green = complete, orange = partial, red = zero.

# Fork mode (default): le chaos run starts ze as a child process
./le chaos run --seed 42 --peers 8 --duration 60s

# Specify ze binary path
./le chaos run --binary ./bin/ze --seed 42 --peers 8 --duration 60s

Fork mode runs the ze daemon that --binary names, else the ze on PATH. It refuses when neither names a program, and it refuses a path that is the running le itself, because le is not a ze daemon.

# Pipeline mode: config on stdout, diagnostics on stderr
./le chaos run --pipe --seed 42 --peers 8 --duration 60s | ./bin/ze -

# Write config to file
./le chaos run --config-out chaos.conf --seed 42 --peers 8
ze start chaos.conf

# In-process mode: mock network + virtual clock (fully deterministic)
./le chaos run --in-process --seed 42 --duration 30s

# In-process with chaos and route dynamics
./le chaos run --in-process --seed 42 --duration 60s --chaos-rate 0.1 --route-rate 0.05

# Multi-family
./le chaos run --families ipv4/unicast,ipv6/unicast --chaos-rate 0.2 --pipe | ./bin/ze -

Multi-Daemon Testing (FRR, BIRD)

./le chaos run can generate configs for and fork FRR (bgpd) or BIRD, so the same chaos scenario runs against different BGP implementations.

# Generate FRR config to inspect
./le chaos run --config-only --application frr --seed 42 --peers 4

# Generate BIRD config to inspect
./le chaos run --config-only --application bird --seed 42 --peers 4

# Fork FRR bgpd (auto-discovers bgpd in PATH)
./le chaos run --application frr --seed 42 --peers 4 --duration 60s

# Fork BIRD with explicit binary path
./le chaos run --application bird --binary /usr/sbin/bird --seed 42 --peers 4 --duration 60s

# Write config to file, start daemon manually
./le chaos run --config-only --application frr --config-out chaos-frr.conf
bgpd -f chaos-frr.conf -p 1850 -l 127.0.0.1 -P 0 -n -Z -S

FRR and BIRD use a single BGP port with peers identified by source IP address (127.0.0.x), unlike Ze's per-peer port model. On Linux, the 127.0.0.0/8 range is routed to loopback by default. On macOS, loopback aliases are needed:

# macOS only: create loopback aliases for each peer
for i in $(seq 2 $((peers+1))); do
  sudo ifconfig lo0 alias 127.0.0.$i
done

./le setup install adds 127.0.0.2 through 127.0.0.5 (and the IPv6 address the functional suite binds), so the loop above is needed only for a run with more than four peers. ./le setup check reports which addresses are missing. Neither route survives a reboot; re-run ./le setup install after one.

Running via Docker

When FRR or BIRD is not installed locally, use Docker. The --network host flag shares the host network so le chaos run simulators can connect directly (Linux only; macOS Docker Desktop does not support host networking).

# Start FRR in Docker, le chaos run on the host
./le chaos run --config-only --application frr --seed 42 --peers 4 \
  --config-out chaos-frr.conf
docker run --rm --network host \
  -v ./chaos-frr.conf:/etc/frr/bgpd.conf:ro \
  quay.io/frrouting/frr:10.3.1 \
  /usr/lib/frr/bgpd -f /etc/frr/bgpd.conf -p 1850 -l 127.0.0.1 -P 0 -n -Z -S

# In another terminal: run le chaos run against the FRR instance
./le chaos run --application frr --seed 42 --peers 4 --duration 60s \
  --config-out /dev/null

# Same pattern for BIRD
./le chaos run --config-only --application bird --seed 42 --peers 4 \
  --config-out chaos-bird.conf
docker run --rm --network host \
  -v ./chaos-bird.conf:/etc/bird.conf:ro \
  --entrypoint bird bird-interop:latest \
  -f -c /etc/bird.conf -s /tmp/bird.ctl

Config Validation (no live session)

Validate generated configs without running a full session:

# FRR: -C flag checks config and exits
./le chaos run --config-only --application frr > frr.conf
docker run --rm -v ./frr.conf:/etc/frr/bgpd.conf:ro \
  quay.io/frrouting/frr:10.3.1 \
  /usr/lib/frr/bgpd -C -f /etc/frr/bgpd.conf -n -Z -S -p 0 -P 0

# BIRD: -p flag parses config and exits
./le chaos run --config-only --application bird > bird.conf
docker run --rm -v ./bird.conf:/etc/bird.conf:ro \
  --entrypoint sh bird-interop:latest \
  -c 'bird -p -c /etc/bird.conf'

Event Logging and Replay

# Record events
./le chaos run --event-log run.ndjson --seed 42 | ./bin/ze -

# Replay a recorded failure
./le chaos run --replay run.ndjson

# Shrink to minimal reproduction
./le chaos run --shrink run.ndjson

Property Validation

./le chaos run --properties all --convergence-deadline 5s | ./bin/ze -
./le chaos run --properties list    # Show available properties

Properties validated:

• Convergence: All routes reach all peers despite faults
• State consistency: RIB matches peer state after recovery
• No data loss: Updates eventually delivered
• Graceful degradation: Sessions restart on critical faults

Fault Types

Deterministic Replay

With a fixed seed, chaos testing is fully reproducible:

# These two runs produce identical fault sequences
ze --chaos-seed 42 --chaos-rate 0.1 config.conf
ze --chaos-seed 42 --chaos-rate 0.1 config.conf

Seed 0 disables chaos entirely (zero overhead). Seed -1 uses the current time (non-reproducible).

Native Commands

When to Use

• Before deployment: Validate your config handles peer failures
• CI pipeline: Catch race conditions and edge cases
• Debugging: Reproduce intermittent failures with a fixed seed
• Benchmarking: Measure convergence time under fault conditions