Operations
Running Ze in production: SSH access, signals, health checks, environment variables, and troubleshooting.
SSH Server
Ze exposes its CLI over SSH. ze cli, ze cli -c, and ze signal commands connect to this server. Most ze show subcommands do too, except local ones (version, bgp decode/encode, env, schema, yang, completion) which run in-process.
If a session reaches an internal Ze defect, only that session ends: its user
reads on the session that an internal defect ended it and which
ze show crashes name <record> command prints the crash record, and the session
exits with status 1. A defect met while a login is checked refuses that login.
The listener and every other session keep running, and each such panic is
logged, crash recorded, and counted in
ze_task_panics_total{component="ssh"} (monitoring).
Setup
ze init # interactive: prompts for username, password, host, port
Defaults: 127.0.0.1:2222, ED25519 host key auto-generated.
Credentials are stored under database/ in the resolved configuration folder,
with bcrypt-hashed passwords. Initialization seeds every key before publishing
the directory, so an interrupted initialization cannot expose partial credentials.
| Init flag | Effect |
|---|---|
--managed |
Enable managed fleet mode |
--force |
Replace an existing store, preserving a .replaced-<stamp> backup |
--yes |
Confirm --force without a terminal prompt |
--web-cert <address> |
Generate a web TLS certificate for the listen address |
--web-cert-name <name> |
Add a DNS name to the generated certificate |
--seed |
Build database.zefs as an appliance seed artifact, without build-host interface discovery |
Reinitializing
Running ze init a second time fails with error: database already exists. This is intentional -- it prevents accidental credential loss.
To reinitialize:
ze signal stop # daemon must be stopped first
ze init --force # prompts for confirmation interactively
--force stages the new store before moving the old tree to
database.replaced-<stamp>. An existing blob is backed up as
database.zefs.replaced-<stamp>. The backup retains the previous credentials
and configs. Confirmation requires a terminal unless --yes is supplied.
Replacement acquires the same ownership lock as the daemon and refuses while
any process owns the store, including a web-only daemon. Changing the SSH target
does not bypass this protection.
The initial config is written under the name the store's instance name gives it. When that name cannot be read, ze init fails with name initial config: ... and closes the interface backend it opened, rather than writing the config under ze.conf.
Store folder and ownership
An explicit config path selects its parent folder. Without one, clients and
daemons use ze.config.dir, then the binary's default configuration folder.
- uses the same environment/default resolution. To reach credentials for a
daemon started on a non-default folder, set ZE_CONFIG_DIR to that folder.
ze start <file> creates database/ when neither store shape exists and reads
the explicit file on every start. Its daemon commits update that file and stored
history. Bare ze start reads the stored active configuration. A database.zefs
beside the live location is refused without conversion. The diagnostic names
ze init --from <source>, which imports a blob from a local path or from an
http or https URL, optionally pinned with --sha256 <hex>.
Appliance first boot already imports its seed explicitly.
Stored history keeps each committed config once, named by its SHA-256, and every
read of a version checks that hash. A store that ze data repair salvaged can
keep an active pointer whose version was dropped. Bare ze start then exits 1
and names meta/config/<name>/active and the stamp; it never serves the mirror,
the rollback or an empty config. ze start <file> rebuilds the active version
from the file, leaves rollback on the version that still reads, and logs the
stamp that would not resolve. ze data check lists what repair left behind.
Without a file, ze start names its config after the instance name (meta/instance/name), or ze.conf when that key is absent, blank or not a valid file stem. When the key exists but cannot be read (a CRC mismatch, an I/O error), ze start exits 1 with an error that names ze doctor and ze init, rather than starting ze.conf on a device whose own name it could not read.
The tree root and every directory beneath it must be 0700, each frame file 0600,
and all must belong to the caller. Symlinks and non-regular leaves are refused.
Root has no ownership exception. Permission errors name the path and the repair.
Only one process may own the store for writing. Read-only credential lookup and
inspection remain available while that owner runs; offline writers such as
ze connect add, remove, and default refuse until it stops.
The owner holds database.lock until the store closes. The lock stays outside
the replaceable tree. Removing this file would break writer exclusion.
Initialization uses a private random staging directory and a no-replace rename.
Offline data access
ze data --path <folder>/database addresses a tree, while an explicit .zefs
file addresses a blob artifact. Run mutating commands only after stopping the
owning daemon and as the store owner, except ze data rm, ze data write and
ze data import: while the daemon runs, rm sends each removal to that daemon
as delete data key <key>, write sends the file as
update data key <key> content <base64>, import sends each file as
update data config name <name> content <base64>, and each prints the same
lines. These
commands expose decoded values, never the tree's framing bytes.
| Command | Effect |
|---|---|
ze data list |
List keys recursively |
ze data cat <key> |
Read a key |
ze data write <key> <file> |
Write a file's bytes into a raw key (- reads stdin); while ze runs, through the daemon, which writes runtime-state keys only, up to 195,840 bytes (see update data key) |
ze data import <file>... |
Import each file as the config named by its file name; a config that exists gets a new active version, the previous one kept as rollback; while ze runs, through the daemon, up to 195,840 bytes per file (see update data config) |
ze data rm <key>... |
Remove keys; while ze runs, through the daemon, which removes runtime-state keys only (see delete data key) |
ze data registered |
Show registered key patterns |
ze data check |
Check integrity; exits 0 clean, 1 corrupt, 2 unreadable; runs while ze runs |
ze data repair --output <path> |
Copy recoverable keys into a new destination; runs while ze runs, and installing the copy needs ze stopped |
ze data backup <file> [spare <n>] |
Copy every key to one exact-fit 0600 blob artifact under the store lock; offline only |
ze data restore <file> config [name <source-name>] |
Commit the artifact's config as a new active version; offline only |
ze data restore <file> full |
Replace the whole tree with the artifact's keys; the previous tree is kept as database.replaced-<stamp> and the artifact is unchanged; offline only |
request data backup path <absolute-file> [spare <n>] [force] |
Live backup of the daemon's store, written on the daemon's host |
request data restore path <absolute-file> config [name <source-name>] |
Live config restore: staged as the candidate, active only when the reload accepts it; refused while a commit confirmed window is open |
request data restore path <absolute-file> config [name <source-name>] client <name> |
On a hub: restore the config it serves to managed client <name>; the hub pushes config-changed and does not reload |
delete data key <key> |
Live removal of one runtime-state key (under meta/ outside) through the daemon's own store; a configuration key, a key whose granted plugin runs, and an absent key are refused. meta/config/ |
update data key <key> content <base64> |
Live write of one runtime-state key, created or replaced with the decoded bytes; content is at most 261,120 base64 characters (195,840 bytes, one SSH packet with the command); refused as delete data key refuses, and recorded with <redacted> for the content |
update data config name <name> content <base64> |
Live import of one config as a new version: for the daemon's own config, staged and active only when the reload accepts it, refused while a commit confirmed window is open or a change is staged; for another name, committed with no reload; content has the bound of update data key, and a larger config is restored with request data restore path <file> config |
request data repair path <absolute-output> |
Live salvage: the daemon writes a new store holding every entry that reads, on its own host, with its writes held for the walk; answers the recovered and skipped counts |
show data check |
The store's integrity verdict through the daemon: the entry counts and one row per corrupt entry and per history finding |
ze data encode [--crc|--header] [--cap N] <string|-> |
Encode a netcapstring for inspection |
Connection
ze cli # interactive CLI
ze cli -c "show bgp peer list" # single command
ze show bgp peer list # read-only shorthand
ze cli -c "request peer transit teardown 2" # one-shot command
An operator can use a standalone or downloaded Ze binary without creating a
local database. Supply the remote endpoint and username explicitly, with the
password in ZE_SSH_PASSWORD or the interactive password prompt:
ZE_SSH_PASSWORD='your-password' ./ze cli --remote 127.0.0.1:2222 --user operator -c "show version"
An available local store is an optional credential source. A nonempty store that is corrupt or unsafe still causes an error; explicit credentials do not bypass its validation or the remote host-key checks.
Override Host/Port
An SSH tunnel is an alternative when Ze listens only on the device's loopback
interface. The example below assumes
the daemon's SSH listener is 127.0.0.1:2222 on a Linux host reachable through
the host's ordinary SSH service:
# Keep this tunnel running in a separate terminal.
ssh -N -L 12222:127.0.0.1:2222 operator@router.example
# In the operator terminal:
export ZE_SSH_HOST=127.0.0.1
export ZE_SSH_PORT=12222
export ZE_SSH_USERNAME=operator
# Prompts for the Ze account's password unless ZE_SSH_PASSWORD is set.
ze signal reload --host 127.0.0.1 --port 12222
For direct remote access, Ze verifies the server against the current user's
OpenSSH ~/.ssh/known_hosts. Use ZE_SSH_KNOWN_HOSTS=/absolute/path/to/known_hosts
to select another trust file. Provision that file from a trusted source and
compare the server fingerprint with your trusted device inventory or operator
before trusting it. Ze refuses unknown, changed, or revoked keys and never
enrolls a key automatically. OpenSSH hashed hosts and [host]:port entries
work through the same parser. Loopback keeps its existing local trust policy;
ze.ssh.insecure remains an explicit verification override.
ZE_SSH_KNOWN_HOSTS="$HOME/.ssh/known_hosts" ZE_SSH_PASSWORD='your-password' \
./ze cli --remote router.example:2222 --user operator -c "show version"
Signals
Via SSH (preferred)
| Command | Effect |
|---|---|
ze signal reload |
Transactionally reload configuration (add/remove/update peers) |
ze signal stop |
Graceful shutdown: stop drain, then Cease to every peer (no GR marker) |
ze signal restart |
Graceful restart: writes the GR marker (RFC 4724), drops BGP sessions without a Cease, drain kind restart |
ze signal status |
Dump process status |
ze signal quit |
Goroutine dump to stderr, then the same stop as ze signal stop |
Via Unix Signals
| Signal | Effect |
|---|---|
SIGHUP |
Transactionally reload configuration |
SIGTERM / SIGINT |
Graceful shutdown: stop drain, then NOTIFICATION Cease to all peers. A second one skips or cuts the drain, a third forces exit 1 (see Stopping) |
SIGUSR1 |
Dump status to stderr |
Reload writes the edited config as a candidate version. The active pointer moves only after verification, apply, and subsystem reload succeed. A failed reload clears the candidate and keeps the previous active config.
A SIGHUP that arrives while the process is still in Go package initialization,
before main registers any handler, kills ze. That window is the first few
hundred milliseconds after exec, and nothing in ze runs earlier. The remedy is
the supervisor's: start ze with SIGHUP ignored. Go keeps an inherited ignored
SIGHUP until ze registers it, so the signal is lost in that window instead of
fatal, and every later SIGHUP reloads. ze install systemd writes this into the
unit (sh -c 'trap "" HUP; exec ze start'), because systemd has no directive
that ignores a signal. A supervisor you write yourself does the same, or never
sends SIGHUP before ze is ready. Details: docs/architecture/behavior/signals.md.
Stopping
Every stop takes the same path: SIGTERM, Ctrl+C, ze signal stop,
request shutdown, request halt and a plugin failure under the fatal
policy. Ze first lets a config commit already running finish, then runs the
stop drain: each plugin that declared a drain budget is asked, in parallel, to
make Ze unattractive to its neighbours before the sessions close, and Ze waits
for the answers. The wait ends at the latest 60 s after the stop began. Only
then does teardown start, and BGP sends Cease (Administrative Shutdown) to every
peer. With no drain-registered plugin the stop is as fast as it was.
ze signal restart, ze signal reboot and request reboot drain with kind
restart: BGP sessions are dropped without a Cease so peers keep the routes for
graceful restart. Any IGP wait during that drain runs while BGP peers hold the
routes as stale, so the BGP restart time must exceed the drain plus teardown and
start.
Each phase prints what the next Ctrl+C or SIGTERM does:
| Phase | Printed line | Next signal |
|---|---|---|
| Config fence | Stopping: finishing config changes (Ctrl+C again to skip the drain)... |
skips the drain ("drain skipped by signal") |
| Drain | Draining routing protocols for up to N s (Ctrl+C again to stop now)... |
aborts the drain ("drain aborted by signal"), teardown goes on |
| Teardown | Shutting down (Ctrl+C again to force)... |
forces exit 1 |
After request shutdown the first signal is the one that aborts the drain. Once
the drain has begun the stop is final: a plugin that exits is not restarted.
Exit Codes (signal command)
| Code | Meaning |
|---|---|
| 0 | Success |
| 1 | Daemon not running |
| 2 | ✕ |
| 4 | Signal delivery failed |
Health Checks
Liveness
ze status # exit 0 = running, exit 1 = not running
This dials the SSH port without completing a handshake. Suitable for systemd watchdog or load balancer TCP check.
Scripting
if ze status --host 127.0.0.1 --port 2222; then
echo "ze is running"
else
echo "ze is down"
fi
Peer Health
ze cli -c "show bgp peer list" # brief peer list with state
ze cli -c "show bgp" # summary table with uptime and prefix counts
Kernel Capability Refusal
Ze refuses to start when the configuration uses a subsystem this kernel cannot carry. The refusal prints one line on stderr and exits 1.
error: kernel capability: ipsec: the kernel holds no CONFIG_XFRM_USER, which vpn ipsec requires: protocol not supported
Every failing subsystem is named, not the first one, so one repair cycle covers
them all. ze doctor reports the same verdict before you start, and
ze config validate fails on it.
| Subsystem | Kernel feature | What to do |
|---|---|---|
ipsec |
CONFIG_XFRM_USER, and CONFIG_INET_ESP to carry the packets |
Run a kernel with both. Ze's appliance kernel builds them in |
mpls |
CONFIG_MPLS_ROUTING and CONFIG_MPLS_IPTUNNEL, or the mpls_router module |
Load the module or run a kernel with both built in |
There is no override. A NOS that half-works on a kernel missing a required feature is the hazard this removes, and an override is what an operator reaches for under pressure.
Three answers are possible and only one refuses:
ze doctor reports |
Severity | ze |
|---|---|---|
| nothing | - | starts |
doctor-<subsystem>-unavailable |
error | refuses, exit 1 |
doctor-<subsystem>-unknown |
warning | starts |
The third answer means ze could not reach the probe's evidence, most often
because the process lacked a privilege. No kernel rebuild fixes that, so ze warns
and starts. Read the reason in the message: ze show diagnostic doctor-ipsec-xfrm-unknown
carries the rest.
A configuration that does not use the subsystem is never probed and never
reported. An empty vpn { ipsec { } } block installs no Security Association, so
it is not IPsec in use.
A reload is refused the same way, and a refused reload leaves the running
configuration serving. The operator sees the refusal through ze config commit.
ze config validate answers about the host running the command. A config written
for another machine and validated on a workstation is judged against the
workstation.
Environment Variables
Ze environment variables use dot notation (ze.log.bgp) and are case-insensitive. All forms are equivalent: ze.log.bgp, ZE_LOG_BGP, ze_log_bgp.
List all registered variables:
ze env list # all vars with types and defaults
ze env list -v # include current values
ze env get ze.log # details for one var
Key Variables
| Variable | Default | Purpose |
|---|---|---|
ze.log |
warn |
Base log level |
ze.log.<subsystem> |
-- | Per-subsystem log level |
ze.log.backend |
stderr |
Log output: stderr, stdout, syslog, kmsg, or a comma-separated list of them |
ze.log.destination |
-- | Syslog address (when backend=syslog) |
ze.log.relay |
warn |
Plugin stderr relay threshold |
ze.ssh.host |
-- | Override SSH host for CLI commands |
ze.ssh.port |
-- | Override SSH port for CLI commands |
ze.ssh.known.hosts |
~/.ssh/known_hosts |
OpenSSH trust file for remote SSH |
ze.config.dir |
-- | Override config directory |
CLI Flags
| Flag | Purpose |
|---|---|
-d, --debug |
Enable debug logging (sets ze.log=debug) |
ze config edit -f <file> |
Edit a loose file without opening its store |
--plugin <name> |
Load additional plugin for YANG/native configs (repeatable). Hub/orchestrator configs reject this; use plugin { internal ... } or plugin { external ... } in the config instead. |
--pprof <addr:port> |
Start pprof HTTP server for profiling |
--chaos-seed <N> |
Chaos testing seed (0=off, -1=time-based) |
--chaos-rate <0-1> |
Chaos fault probability |
-V, --version |
Show version |
Install and remove a standalone Linux daemon
From the directory containing the selected Ze executable, run:
sudo ./ze install
The default prefix is /usr/local. The installer prompts for the initial
username, password, SSH address, SSH port and instance name. Keep the SSH
address on loopback. The default listener is 127.0.0.1:2222. This bootstrap
does not configure the host's interfaces.
On a running systemd host, installation creates and starts the service. Connect with the account and store path the installer prints:
sudo -u ze env ZE_CONFIG_DIR=/etc/ze XDG_RUNTIME_DIR=/run/ze ze cli
sudo -u ze env ZE_CONFIG_DIR=/etc/ze XDG_RUNTIME_DIR=/run/ze ze doctor
An existing private store, credentials and compatible administrator unit are preserved during repeat installation. Conflicting services, package ownership markers and symlink paths refuse installation before executable replacement.
Choose another prefix or foreground operation with:
./ze install --dry-run --prefix /opt/ze
sudo ./ze install --prefix /opt/ze
./ze install --prefix "$HOME/ze" --no-service
"$HOME/ze/bin/ze" start
Without a running systemd manager, the installer also prints the foreground
start command. A foreground daemon must be stopped before replacement or
removal. --no-service does not bypass the store ownership lock.
A service installation needs a root-owned executable path. Existing ancestors
and the executable must not permit group or other write access. Use a
protected prefix such as /opt/ze, or select --no-service for a user-owned
prefix. Installation refuses conflicting runtime units, masks, indirect
configuration sources and unsupported service execution overrides before
changing files or service state.
To remove the default installation, run:
sudo ze uninstall
Removal asks for confirmation, stops and removes the selected service, and
removes the executable. It retains /etc/ze and the service account.
A stop failure retains the executable and service. Use the installation's
prefix when removing a custom installation. --yes confirms removal in scripts.
A foreground process that still owns the selected store also refuses removal. The installer takes the store lock before disabling or deleting the unit. That refusal preserves the previous service policy and resumes only a service that was active before the attempt.
Explicit purge also removes the selected config directory, including its backups. Copy required backups elsewhere first. The service account remains.
sudo ze uninstall --prefix /opt/ze --purge
Upgrade and recover a standalone installation
Keep the previous executable and a live store backup before upgrading:
sudo install -m 0755 /usr/local/bin/ze /var/tmp/ze-before-upgrade
sudo -u ze env ZE_CONFIG_DIR=/etc/ze XDG_RUNTIME_DIR=/run/ze \
ze cli -c "request data backup path /etc/ze/pre-upgrade.zefs"
Choose a new backup filename when that file already exists. The backup command runs on the daemon and preserves the complete store in a private blob.
Run the new executable's native installer, then verify the running version and diagnostic results:
sudo ./ze install
sudo -u ze env ZE_CONFIG_DIR=/etc/ze XDG_RUNTIME_DIR=/run/ze \
ze cli -c "show version"
sudo -u ze env ZE_CONFIG_DIR=/etc/ze XDG_RUNTIME_DIR=/run/ze ze doctor
The installer stops an active managed service before replacement and starts it afterward. A previously inactive service remains inactive. It retains the store and compatible administrator service settings. A failed replacement restores the prior executable and resumes a service that was previously active. The existing binary is replaced atomically. Preserve the backup executable until the new version and required routes are verified.
To restore the previous executable with the current configuration, run its installer:
sudo /var/tmp/ze-before-upgrade install
If configuration restoration is also required, stop the service first. Restore the saved store as its owner, then install the previous executable:
sudo systemctl stop ze.service
sudo -u ze env ZE_CONFIG_DIR=/etc/ze \
/var/tmp/ze-before-upgrade data restore /etc/ze/pre-upgrade.zefs full
sudo /var/tmp/ze-before-upgrade install
Full restoration retains the replaced tree as a recovery directory. Verify the restored version, login and routes after startup. Package-managed installations use their package manager for upgrade and removal.
systemd
Use ze install systemd on standard Linux hosts:
sudo ze init
sudo ze install systemd --start
This writes /etc/systemd/system/ze.service, creates the ze user/group if
missing, and transfers the complete store tree and its ownership lock to that
account before enabling or starting the service. A running store owner refuses
the transfer. Root performs the system changes; after transfer, storage
maintenance runs as ze.
Inspect the generated unit without writing anything:
ze install systemd --dry-run --config /etc/ze
Run doctor after installation to verify the service account and binary path:
sudo -u ze env ZE_CONFIG_DIR=/etc/ze ze doctor
When /etc/systemd/system/ze.service exists, doctor checks that the unit's
ExecStart binary exists and is executable, and that the configured User and
Group exist on the host.
The service runs with XDG_RUNTIME_DIR=/run/ze, so the daemon socket is
/run/ze/ze.socket. Local commands using stored credentials run as its owner:
sudo -u ze env ZE_CONFIG_DIR=/etc/ze XDG_RUNTIME_DIR=/run/ze ze cli
Other users can supply --user, ze.ssh.password, and an explicit remote
address instead of reading the service account's credentials. Offline
maintenance first stops ze.service as root, then runs the storage command
with sudo -u ze; restarting the service is a root system operation.
Remove only the service unit:
sudo ze uninstall systemd
Generated unit shape:
[Unit]
Description=Ze Network OS
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=ze
Group=ze
ExecStart=/bin/sh -c 'trap "" HUP; exec /usr/local/bin/ze start'
ExecReload=/bin/sh -c 'kill -HUP $MAINPID'
Restart=on-failure
RestartSec=5
TimeoutStopSec=90
KillMode=mixed
LimitNOFILE=65536
LimitCORE=infinity
LimitMEMLOCK=infinity
WorkingDirectory=/etc/ze
Environment=ZE_CONFIG_DIR=/etc/ze
Environment=XDG_RUNTIME_DIR=/run/ze
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_RAW CAP_NET_BIND_SERVICE CAP_BPF
CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_RAW CAP_NET_BIND_SERVICE CAP_BPF
# Host tuning may need CAP_SYS_NICE. Future VRF/netns support may need CAP_SYS_ADMIN.
# Add those capabilities only when the configured feature requires them.
NoNewPrivileges=true
ProtectSystem=true
ProtectHome=true
RuntimeDirectory=ze
[Install]
WantedBy=multi-user.target
CAP_BPF is granted for the eBPF programs of the IPsec ESP form fast path,
the IPsec application override guard and traffic-usage, which the kernel
refuses to load without it.
Troubleshooting
Peer Won't Connect
Symptom: peer stays in Connect/Active state
-
Check connectivity: Can you reach the peer IP on port 179?
nc -zv 10.0.0.2 179 -
Check config: Validate before debugging network issues
ze config validate config.conf -
Enable debug logging:
ze cli -c "request log level bgp.fsm debug" ze cli -c "bgp monitor event state" -
Common causes:
- Firewall blocking TCP 179
- Peer not configured for your address
- AS number mismatch (check
local { as }andremote { as }) - Hold time rejected (must be 0 or >= 3)
Peer Drops After OPEN
Symptom: session establishes briefly then NOTIFICATION received
-
Look for the
notification receivedline in ze's log first. Every NOTIFICATION a peer sends leaves one WARN line, visible at the default log level with noze.logoverride, including a peer that refuses Ze's OPEN before the session ever comes up. It names the peer, the FSM state the NOTIFICATION arrived in (OPENSENTorOPENCONFIRMwhen the OPEN was refused), the code and subcode as names and numbers, and the peer's shutdown message when it sent one:level=WARN msg="notification received" subsystem=bgp.reactor.session peer=192.0.2.1 state=OPENCONFIRM code="OPEN Message Error" code-number=2 subcode=Unspecific subcode-number=0 -
Read the code and subcode against this table (the same values appear in monitor output):
Code Subcode Meaning 2 2 Bad Peer AS -- AS number mismatch 2 3 Bad BGP Identifier -- duplicate router-id 2 6 Unacceptable Hold Time 2 11 Role Mismatch (RFC 9234) 6 2 Administrative Shutdown 6 5 Connection Rejected -
Role mismatch: If using
role { import customer; strict true; }, both peers must advertise compatible roles. -
Capability issues: Enable debug to see negotiated vs offered capabilities:
ze cli -c "request log level bgp.reactor debug"
Daemon Won't Start
- Config parse error: Run
ze config validate config.conffirst - Port in use: Check if another ze instance or BGP daemon holds port 179
- SSH port conflict: Default SSH is 2222. Check with
netstat -tlnp | grep 2222 - Missing credentials: Run
ze initbefore starting
Plugin Not Working
-
Check plugin is loaded:
ze show plugin list # list compiled-in plugins -
Check plugin is bound to peer: Config must have
attach process <name> { receive [...] }on the peer -
Check plugin logs: Set
ZE_LOG_PLUGIN_RELAY=debugto see plugin stderr output -
Plugin not reaching Ready state: Enable
ZE_LOG_PLUGIN=debugand look for startup stage failures
No Routes in RIB
-
Check RIB plugin is loaded and bound:
ze cli -c "show bgp rib status" -
Check peer is sending updates:
ze cli -c "bgp monitor peer transit-a event update direction received" -
Check RPKI validation: If bgp-rpki is loaded, routes may be pending validation. Check:
ze cli -c "show bgp rpki status"
Crash Capture
Ze automatically captures panic stack traces and forwards them to syslog. On
restart after a crash, show crashes displays saved crash reports with ring
buffer context (the last 64 log entries before the panic).
Configuration (environment variables):
| Variable | Default | Description |
|---|---|---|
ze.crash.dir |
autodetect | Override crash file directory |
ze.crash.keep |
5 |
Number of crash files to retain (1-100) |
Crash dir autodetection probes in order: /perm/ze/crash/ (gokrazy),
<config-dir>/crash/, /var/lib/ze/crash/, /tmp/ze-crash/. The resolved
path is printed at startup.
Syslog forwarding uses ze.log.destination (no separate config). Crash
messages use LOG_CRIT facility for filtering.
CLI (online, daemon running):
show crashes # list crash files with timestamps
show crashes latest # display most recent crash report
CLI (works with or without a daemon; falls back to in-process read when the daemon is down):
ze show crashes # list crash files (JSON)
ze show crashes latest # display most recent crash report
Collecting Debug Information
For bug reports, collect:
ze --version # ze version
ze config validate config.conf # config validity
ze status # daemon state
ze cli -c "show bgp peer list" # peer states
ze cli -c "show bgp" # session summary
ze env list -v # effective configuration
ze signal quit # goroutine dump (kills daemon!)