Guide

Operations

Running Ze in production: SSH access, signals, health checks, environment variables, and troubleshooting.

SSH Server

Ze exposes its CLI over SSH. ze cli, ze cli -c, and ze signal commands connect to this server. Most ze show subcommands do too, except local ones (version, bgp decode/encode, env, schema, yang, completion) which run in-process.

If a session reaches an internal Ze defect, only that session ends: its user reads on the session that an internal defect ended it and which ze show crashes name <record> command prints the crash record, and the session exits with status 1. A defect met while a login is checked refuses that login. The listener and every other session keep running, and each such panic is logged, crash recorded, and counted in ze_task_panics_total{component="ssh"} (monitoring).

Setup

ze init                        # interactive: prompts for username, password, host, port

Defaults: 127.0.0.1:2222, ED25519 host key auto-generated.

Credentials are stored under database/ in the resolved configuration folder, with bcrypt-hashed passwords. Initialization seeds every key before publishing the directory, so an interrupted initialization cannot expose partial credentials.

Init flag Effect
--managed Enable managed fleet mode
--force Replace an existing store, preserving a .replaced-<stamp> backup
--yes Confirm --force without a terminal prompt
--web-cert <address> Generate a web TLS certificate for the listen address
--web-cert-name <name> Add a DNS name to the generated certificate
--seed Build database.zefs as an appliance seed artifact, without build-host interface discovery

Reinitializing

Running ze init a second time fails with error: database already exists. This is intentional -- it prevents accidental credential loss.

To reinitialize:

ze signal stop                 # daemon must be stopped first
ze init --force                # prompts for confirmation interactively

--force stages the new store before moving the old tree to database.replaced-<stamp>. An existing blob is backed up as database.zefs.replaced-<stamp>. The backup retains the previous credentials and configs. Confirmation requires a terminal unless --yes is supplied. Replacement acquires the same ownership lock as the daemon and refuses while any process owns the store, including a web-only daemon. Changing the SSH target does not bypass this protection.

The initial config is written under the name the store's instance name gives it. When that name cannot be read, ze init fails with name initial config: ... and closes the interface backend it opened, rather than writing the config under ze.conf.

Store folder and ownership

An explicit config path selects its parent folder. Without one, clients and daemons use ze.config.dir, then the binary's default configuration folder. - uses the same environment/default resolution. To reach credentials for a daemon started on a non-default folder, set ZE_CONFIG_DIR to that folder.

ze start <file> creates database/ when neither store shape exists and reads the explicit file on every start. Its daemon commits update that file and stored history. Bare ze start reads the stored active configuration. A database.zefs beside the live location is refused without conversion. The diagnostic names ze init --from <source>, which imports a blob from a local path or from an http or https URL, optionally pinned with --sha256 <hex>. Appliance first boot already imports its seed explicitly.

Stored history keeps each committed config once, named by its SHA-256, and every read of a version checks that hash. A store that ze data repair salvaged can keep an active pointer whose version was dropped. Bare ze start then exits 1 and names meta/config/<name>/active and the stamp; it never serves the mirror, the rollback or an empty config. ze start <file> rebuilds the active version from the file, leaves rollback on the version that still reads, and logs the stamp that would not resolve. ze data check lists what repair left behind.

Without a file, ze start names its config after the instance name (meta/instance/name), or ze.conf when that key is absent, blank or not a valid file stem. When the key exists but cannot be read (a CRC mismatch, an I/O error), ze start exits 1 with an error that names ze doctor and ze init, rather than starting ze.conf on a device whose own name it could not read.

The tree root and every directory beneath it must be 0700, each frame file 0600, and all must belong to the caller. Symlinks and non-regular leaves are refused. Root has no ownership exception. Permission errors name the path and the repair. Only one process may own the store for writing. Read-only credential lookup and inspection remain available while that owner runs; offline writers such as ze connect add, remove, and default refuse until it stops.

The owner holds database.lock until the store closes. The lock stays outside the replaceable tree. Removing this file would break writer exclusion. Initialization uses a private random staging directory and a no-replace rename.

Offline data access

ze data --path <folder>/database addresses a tree, while an explicit .zefs file addresses a blob artifact. Run mutating commands only after stopping the owning daemon and as the store owner, except ze data rm, ze data write and ze data import: while the daemon runs, rm sends each removal to that daemon as delete data key <key>, write sends the file as update data key <key> content <base64>, import sends each file as update data config name <name> content <base64>, and each prints the same lines. These commands expose decoded values, never the tree's framing bytes.

Command Effect
ze data list List keys recursively
ze data cat <key> Read a key
ze data write <key> <file> Write a file's bytes into a raw key (- reads stdin); while ze runs, through the daemon, which writes runtime-state keys only, up to 195,840 bytes (see update data key)
ze data import <file>... Import each file as the config named by its file name; a config that exists gets a new active version, the previous one kept as rollback; while ze runs, through the daemon, up to 195,840 bytes per file (see update data config)
ze data rm <key>... Remove keys; while ze runs, through the daemon, which removes runtime-state keys only (see delete data key)
ze data registered Show registered key patterns
ze data check Check integrity; exits 0 clean, 1 corrupt, 2 unreadable; runs while ze runs
ze data repair --output <path> Copy recoverable keys into a new destination; runs while ze runs, and installing the copy needs ze stopped
ze data backup <file> [spare <n>] Copy every key to one exact-fit 0600 blob artifact under the store lock; offline only
ze data restore <file> config [name <source-name>] Commit the artifact's config as a new active version; offline only
ze data restore <file> full Replace the whole tree with the artifact's keys; the previous tree is kept as database.replaced-<stamp> and the artifact is unchanged; offline only
request data backup path <absolute-file> [spare <n>] [force] Live backup of the daemon's store, written on the daemon's host
request data restore path <absolute-file> config [name <source-name>] Live config restore: staged as the candidate, active only when the reload accepts it; refused while a commit confirmed window is open
request data restore path <absolute-file> config [name <source-name>] client <name> On a hub: restore the config it serves to managed client <name>; the hub pushes config-changed and does not reload
delete data key <key> Live removal of one runtime-state key (under meta/ outside) through the daemon's own store; a configuration key, a key whose granted plugin runs, and an absent key are refused. meta/config/
update data key <key> content <base64> Live write of one runtime-state key, created or replaced with the decoded bytes; content is at most 261,120 base64 characters (195,840 bytes, one SSH packet with the command); refused as delete data key refuses, and recorded with <redacted> for the content
update data config name <name> content <base64> Live import of one config as a new version: for the daemon's own config, staged and active only when the reload accepts it, refused while a commit confirmed window is open or a change is staged; for another name, committed with no reload; content has the bound of update data key, and a larger config is restored with request data restore path <file> config
request data repair path <absolute-output> Live salvage: the daemon writes a new store holding every entry that reads, on its own host, with its writes held for the walk; answers the recovered and skipped counts
show data check The store's integrity verdict through the daemon: the entry counts and one row per corrupt entry and per history finding
ze data encode [--crc|--header] [--cap N] <string|-> Encode a netcapstring for inspection

Connection

ze cli                         # interactive CLI
ze cli -c "show bgp peer list"  # single command
ze show bgp peer list                # read-only shorthand
ze cli -c "request peer transit teardown 2" # one-shot command

An operator can use a standalone or downloaded Ze binary without creating a local database. Supply the remote endpoint and username explicitly, with the password in ZE_SSH_PASSWORD or the interactive password prompt:

ZE_SSH_PASSWORD='your-password' ./ze cli --remote 127.0.0.1:2222 --user operator -c "show version"

An available local store is an optional credential source. A nonempty store that is corrupt or unsafe still causes an error; explicit credentials do not bypass its validation or the remote host-key checks.

Override Host/Port

An SSH tunnel is an alternative when Ze listens only on the device's loopback interface. The example below assumes the daemon's SSH listener is 127.0.0.1:2222 on a Linux host reachable through the host's ordinary SSH service:

# Keep this tunnel running in a separate terminal.
ssh -N -L 12222:127.0.0.1:2222 operator@router.example

# In the operator terminal:
export ZE_SSH_HOST=127.0.0.1
export ZE_SSH_PORT=12222
export ZE_SSH_USERNAME=operator

# Prompts for the Ze account's password unless ZE_SSH_PASSWORD is set.
ze signal reload --host 127.0.0.1 --port 12222

For direct remote access, Ze verifies the server against the current user's OpenSSH ~/.ssh/known_hosts. Use ZE_SSH_KNOWN_HOSTS=/absolute/path/to/known_hosts to select another trust file. Provision that file from a trusted source and compare the server fingerprint with your trusted device inventory or operator before trusting it. Ze refuses unknown, changed, or revoked keys and never enrolls a key automatically. OpenSSH hashed hosts and [host]:port entries work through the same parser. Loopback keeps its existing local trust policy; ze.ssh.insecure remains an explicit verification override.

ZE_SSH_KNOWN_HOSTS="$HOME/.ssh/known_hosts" ZE_SSH_PASSWORD='your-password' \
    ./ze cli --remote router.example:2222 --user operator -c "show version"

Signals

Via SSH (preferred)

Command Effect
ze signal reload Transactionally reload configuration (add/remove/update peers)
ze signal stop Graceful shutdown: stop drain, then Cease to every peer (no GR marker)
ze signal restart Graceful restart: writes the GR marker (RFC 4724), drops BGP sessions without a Cease, drain kind restart
ze signal status Dump process status
ze signal quit Goroutine dump to stderr, then the same stop as ze signal stop

Via Unix Signals

Signal Effect
SIGHUP Transactionally reload configuration
SIGTERM / SIGINT Graceful shutdown: stop drain, then NOTIFICATION Cease to all peers. A second one skips or cuts the drain, a third forces exit 1 (see Stopping)
SIGUSR1 Dump status to stderr

Reload writes the edited config as a candidate version. The active pointer moves only after verification, apply, and subsystem reload succeed. A failed reload clears the candidate and keeps the previous active config.

A SIGHUP that arrives while the process is still in Go package initialization, before main registers any handler, kills ze. That window is the first few hundred milliseconds after exec, and nothing in ze runs earlier. The remedy is the supervisor's: start ze with SIGHUP ignored. Go keeps an inherited ignored SIGHUP until ze registers it, so the signal is lost in that window instead of fatal, and every later SIGHUP reloads. ze install systemd writes this into the unit (sh -c 'trap "" HUP; exec ze start'), because systemd has no directive that ignores a signal. A supervisor you write yourself does the same, or never sends SIGHUP before ze is ready. Details: docs/architecture/behavior/signals.md.

Stopping

Every stop takes the same path: SIGTERM, Ctrl+C, ze signal stop, request shutdown, request halt and a plugin failure under the fatal policy. Ze first lets a config commit already running finish, then runs the stop drain: each plugin that declared a drain budget is asked, in parallel, to make Ze unattractive to its neighbours before the sessions close, and Ze waits for the answers. The wait ends at the latest 60 s after the stop began. Only then does teardown start, and BGP sends Cease (Administrative Shutdown) to every peer. With no drain-registered plugin the stop is as fast as it was.

ze signal restart, ze signal reboot and request reboot drain with kind restart: BGP sessions are dropped without a Cease so peers keep the routes for graceful restart. Any IGP wait during that drain runs while BGP peers hold the routes as stale, so the BGP restart time must exceed the drain plus teardown and start.

Each phase prints what the next Ctrl+C or SIGTERM does:

Phase Printed line Next signal
Config fence Stopping: finishing config changes (Ctrl+C again to skip the drain)... skips the drain ("drain skipped by signal")
Drain Draining routing protocols for up to N s (Ctrl+C again to stop now)... aborts the drain ("drain aborted by signal"), teardown goes on
Teardown Shutting down (Ctrl+C again to force)... forces exit 1

After request shutdown the first signal is the one that aborts the drain. Once the drain has begun the stop is final: a plugin that exits is not restarted.

Exit Codes (signal command)

Code Meaning
0 Success
1 Daemon not running
2 ✕
4 Signal delivery failed

Health Checks

Liveness

ze status                      # exit 0 = running, exit 1 = not running

This dials the SSH port without completing a handshake. Suitable for systemd watchdog or load balancer TCP check.

Scripting

if ze status --host 127.0.0.1 --port 2222; then
    echo "ze is running"
else
    echo "ze is down"
fi

Peer Health

ze cli -c "show bgp peer list"  # brief peer list with state
ze cli -c "show bgp"            # summary table with uptime and prefix counts

Kernel Capability Refusal

Ze refuses to start when the configuration uses a subsystem this kernel cannot carry. The refusal prints one line on stderr and exits 1.

error: kernel capability: ipsec: the kernel holds no CONFIG_XFRM_USER, which vpn ipsec requires: protocol not supported

Every failing subsystem is named, not the first one, so one repair cycle covers them all. ze doctor reports the same verdict before you start, and ze config validate fails on it.

Subsystem Kernel feature What to do
ipsec CONFIG_XFRM_USER, and CONFIG_INET_ESP to carry the packets Run a kernel with both. Ze's appliance kernel builds them in
mpls CONFIG_MPLS_ROUTING and CONFIG_MPLS_IPTUNNEL, or the mpls_router module Load the module or run a kernel with both built in

There is no override. A NOS that half-works on a kernel missing a required feature is the hazard this removes, and an override is what an operator reaches for under pressure.

Three answers are possible and only one refuses:

ze doctor reports Severity ze
nothing - starts
doctor-<subsystem>-unavailable error refuses, exit 1
doctor-<subsystem>-unknown warning starts

The third answer means ze could not reach the probe's evidence, most often because the process lacked a privilege. No kernel rebuild fixes that, so ze warns and starts. Read the reason in the message: ze show diagnostic doctor-ipsec-xfrm-unknown carries the rest.

A configuration that does not use the subsystem is never probed and never reported. An empty vpn { ipsec { } } block installs no Security Association, so it is not IPsec in use.

A reload is refused the same way, and a refused reload leaves the running configuration serving. The operator sees the refusal through ze config commit.

ze config validate answers about the host running the command. A config written for another machine and validated on a workstation is judged against the workstation.

Environment Variables

Ze environment variables use dot notation (ze.log.bgp) and are case-insensitive. All forms are equivalent: ze.log.bgp, ZE_LOG_BGP, ze_log_bgp.

List all registered variables:

ze env list                    # all vars with types and defaults
ze env list -v                 # include current values
ze env get ze.log              # details for one var

Key Variables

Variable Default Purpose
ze.log warn Base log level
ze.log.<subsystem> -- Per-subsystem log level
ze.log.backend stderr Log output: stderr, stdout, syslog, kmsg, or a comma-separated list of them
ze.log.destination -- Syslog address (when backend=syslog)
ze.log.relay warn Plugin stderr relay threshold
ze.ssh.host -- Override SSH host for CLI commands
ze.ssh.port -- Override SSH port for CLI commands
ze.ssh.known.hosts ~/.ssh/known_hosts OpenSSH trust file for remote SSH
ze.config.dir -- Override config directory

CLI Flags

Flag Purpose
-d, --debug Enable debug logging (sets ze.log=debug)
ze config edit -f <file> Edit a loose file without opening its store
--plugin <name> Load additional plugin for YANG/native configs (repeatable). Hub/orchestrator configs reject this; use plugin { internal ... } or plugin { external ... } in the config instead.
--pprof <addr:port> Start pprof HTTP server for profiling
--chaos-seed <N> Chaos testing seed (0=off, -1=time-based)
--chaos-rate <0-1> Chaos fault probability
-V, --version Show version

Install and remove a standalone Linux daemon

From the directory containing the selected Ze executable, run:

sudo ./ze install

The default prefix is /usr/local. The installer prompts for the initial username, password, SSH address, SSH port and instance name. Keep the SSH address on loopback. The default listener is 127.0.0.1:2222. This bootstrap does not configure the host's interfaces.

On a running systemd host, installation creates and starts the service. Connect with the account and store path the installer prints:

sudo -u ze env ZE_CONFIG_DIR=/etc/ze XDG_RUNTIME_DIR=/run/ze ze cli
sudo -u ze env ZE_CONFIG_DIR=/etc/ze XDG_RUNTIME_DIR=/run/ze ze doctor

An existing private store, credentials and compatible administrator unit are preserved during repeat installation. Conflicting services, package ownership markers and symlink paths refuse installation before executable replacement.

Choose another prefix or foreground operation with:

./ze install --dry-run --prefix /opt/ze
sudo ./ze install --prefix /opt/ze
./ze install --prefix "$HOME/ze" --no-service
"$HOME/ze/bin/ze" start

Without a running systemd manager, the installer also prints the foreground start command. A foreground daemon must be stopped before replacement or removal. --no-service does not bypass the store ownership lock.

A service installation needs a root-owned executable path. Existing ancestors and the executable must not permit group or other write access. Use a protected prefix such as /opt/ze, or select --no-service for a user-owned prefix. Installation refuses conflicting runtime units, masks, indirect configuration sources and unsupported service execution overrides before changing files or service state.

To remove the default installation, run:

sudo ze uninstall

Removal asks for confirmation, stops and removes the selected service, and removes the executable. It retains /etc/ze and the service account. A stop failure retains the executable and service. Use the installation's prefix when removing a custom installation. --yes confirms removal in scripts.

A foreground process that still owns the selected store also refuses removal. The installer takes the store lock before disabling or deleting the unit. That refusal preserves the previous service policy and resumes only a service that was active before the attempt.

Explicit purge also removes the selected config directory, including its backups. Copy required backups elsewhere first. The service account remains.

sudo ze uninstall --prefix /opt/ze --purge

Upgrade and recover a standalone installation

Keep the previous executable and a live store backup before upgrading:

sudo install -m 0755 /usr/local/bin/ze /var/tmp/ze-before-upgrade
sudo -u ze env ZE_CONFIG_DIR=/etc/ze XDG_RUNTIME_DIR=/run/ze \
  ze cli -c "request data backup path /etc/ze/pre-upgrade.zefs"

Choose a new backup filename when that file already exists. The backup command runs on the daemon and preserves the complete store in a private blob.

Run the new executable's native installer, then verify the running version and diagnostic results:

sudo ./ze install
sudo -u ze env ZE_CONFIG_DIR=/etc/ze XDG_RUNTIME_DIR=/run/ze \
  ze cli -c "show version"
sudo -u ze env ZE_CONFIG_DIR=/etc/ze XDG_RUNTIME_DIR=/run/ze ze doctor

The installer stops an active managed service before replacement and starts it afterward. A previously inactive service remains inactive. It retains the store and compatible administrator service settings. A failed replacement restores the prior executable and resumes a service that was previously active. The existing binary is replaced atomically. Preserve the backup executable until the new version and required routes are verified.

To restore the previous executable with the current configuration, run its installer:

sudo /var/tmp/ze-before-upgrade install

If configuration restoration is also required, stop the service first. Restore the saved store as its owner, then install the previous executable:

sudo systemctl stop ze.service
sudo -u ze env ZE_CONFIG_DIR=/etc/ze \
  /var/tmp/ze-before-upgrade data restore /etc/ze/pre-upgrade.zefs full
sudo /var/tmp/ze-before-upgrade install

Full restoration retains the replaced tree as a recovery directory. Verify the restored version, login and routes after startup. Package-managed installations use their package manager for upgrade and removal.

systemd

Use ze install systemd on standard Linux hosts:

sudo ze init
sudo ze install systemd --start

This writes /etc/systemd/system/ze.service, creates the ze user/group if missing, and transfers the complete store tree and its ownership lock to that account before enabling or starting the service. A running store owner refuses the transfer. Root performs the system changes; after transfer, storage maintenance runs as ze.

Inspect the generated unit without writing anything:

ze install systemd --dry-run --config /etc/ze

Run doctor after installation to verify the service account and binary path:

sudo -u ze env ZE_CONFIG_DIR=/etc/ze ze doctor

When /etc/systemd/system/ze.service exists, doctor checks that the unit's ExecStart binary exists and is executable, and that the configured User and Group exist on the host.

The service runs with XDG_RUNTIME_DIR=/run/ze, so the daemon socket is /run/ze/ze.socket. Local commands using stored credentials run as its owner:

sudo -u ze env ZE_CONFIG_DIR=/etc/ze XDG_RUNTIME_DIR=/run/ze ze cli

Other users can supply --user, ze.ssh.password, and an explicit remote address instead of reading the service account's credentials. Offline maintenance first stops ze.service as root, then runs the storage command with sudo -u ze; restarting the service is a root system operation.

Remove only the service unit:

sudo ze uninstall systemd

Generated unit shape:

[Unit]
Description=Ze Network OS
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=ze
Group=ze
ExecStart=/bin/sh -c 'trap "" HUP; exec /usr/local/bin/ze start'
ExecReload=/bin/sh -c 'kill -HUP $MAINPID'
Restart=on-failure
RestartSec=5
TimeoutStopSec=90
KillMode=mixed
LimitNOFILE=65536
LimitCORE=infinity
LimitMEMLOCK=infinity
WorkingDirectory=/etc/ze
Environment=ZE_CONFIG_DIR=/etc/ze
Environment=XDG_RUNTIME_DIR=/run/ze
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_RAW CAP_NET_BIND_SERVICE CAP_BPF
CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_RAW CAP_NET_BIND_SERVICE CAP_BPF
# Host tuning may need CAP_SYS_NICE. Future VRF/netns support may need CAP_SYS_ADMIN.
# Add those capabilities only when the configured feature requires them.
NoNewPrivileges=true
ProtectSystem=true
ProtectHome=true
RuntimeDirectory=ze

[Install]
WantedBy=multi-user.target

CAP_BPF is granted for the eBPF programs of the IPsec ESP form fast path, the IPsec application override guard and traffic-usage, which the kernel refuses to load without it.

Troubleshooting

Peer Won't Connect

Symptom: peer stays in Connect/Active state
  1. Check connectivity: Can you reach the peer IP on port 179?

    nc -zv 10.0.0.2 179
    
  2. Check config: Validate before debugging network issues

    ze config validate config.conf
    
  3. Enable debug logging:

    ze cli -c "request log level bgp.fsm debug"
    ze cli -c "bgp monitor event state"
    
  4. Common causes:

    • Firewall blocking TCP 179
    • Peer not configured for your address
    • AS number mismatch (check local { as } and remote { as })
    • Hold time rejected (must be 0 or >= 3)

Peer Drops After OPEN

Symptom: session establishes briefly then NOTIFICATION received
  1. Look for the notification received line in ze's log first. Every NOTIFICATION a peer sends leaves one WARN line, visible at the default log level with no ze.log override, including a peer that refuses Ze's OPEN before the session ever comes up. It names the peer, the FSM state the NOTIFICATION arrived in (OPENSENT or OPENCONFIRM when the OPEN was refused), the code and subcode as names and numbers, and the peer's shutdown message when it sent one:

    level=WARN msg="notification received" subsystem=bgp.reactor.session peer=192.0.2.1 state=OPENCONFIRM code="OPEN Message Error" code-number=2 subcode=Unspecific subcode-number=0
    
  2. Read the code and subcode against this table (the same values appear in monitor output):

    Code Subcode Meaning
    2 2 Bad Peer AS -- AS number mismatch
    2 3 Bad BGP Identifier -- duplicate router-id
    2 6 Unacceptable Hold Time
    2 11 Role Mismatch (RFC 9234)
    6 2 Administrative Shutdown
    6 5 Connection Rejected
  3. Role mismatch: If using role { import customer; strict true; }, both peers must advertise compatible roles.

  4. Capability issues: Enable debug to see negotiated vs offered capabilities:

    ze cli -c "request log level bgp.reactor debug"
    

Daemon Won't Start

  1. Config parse error: Run ze config validate config.conf first
  2. Port in use: Check if another ze instance or BGP daemon holds port 179
  3. SSH port conflict: Default SSH is 2222. Check with netstat -tlnp | grep 2222
  4. Missing credentials: Run ze init before starting

Plugin Not Working

  1. Check plugin is loaded:

    ze show plugin list               # list compiled-in plugins
    
  2. Check plugin is bound to peer: Config must have attach process <name> { receive [...] } on the peer

  3. Check plugin logs: Set ZE_LOG_PLUGIN_RELAY=debug to see plugin stderr output

  4. Plugin not reaching Ready state: Enable ZE_LOG_PLUGIN=debug and look for startup stage failures

No Routes in RIB

  1. Check RIB plugin is loaded and bound:

    ze cli -c "show bgp rib status"
    
  2. Check peer is sending updates:

    ze cli -c "bgp monitor peer transit-a event update direction received"
    
  3. Check RPKI validation: If bgp-rpki is loaded, routes may be pending validation. Check:

    ze cli -c "show bgp rpki status"
    

Crash Capture

Ze automatically captures panic stack traces and forwards them to syslog. On restart after a crash, show crashes displays saved crash reports with ring buffer context (the last 64 log entries before the panic).

Configuration (environment variables):

Variable Default Description
ze.crash.dir autodetect Override crash file directory
ze.crash.keep 5 Number of crash files to retain (1-100)

Crash dir autodetection probes in order: /perm/ze/crash/ (gokrazy), <config-dir>/crash/, /var/lib/ze/crash/, /tmp/ze-crash/. The resolved path is printed at startup.

Syslog forwarding uses ze.log.destination (no separate config). Crash messages use LOG_CRIT facility for filtering.

CLI (online, daemon running):

show crashes           # list crash files with timestamps
show crashes latest    # display most recent crash report

CLI (works with or without a daemon; falls back to in-process read when the daemon is down):

ze show crashes          # list crash files (JSON)
ze show crashes latest   # display most recent crash report

Collecting Debug Information

For bug reports, collect:

ze --version                   # ze version
ze config validate config.conf        # config validity
ze status                      # daemon state
ze cli -c "show bgp peer list"  # peer states
ze cli -c "show bgp"            # session summary
ze env list -v                 # effective configuration
ze signal quit                 # goroutine dump (kills daemon!)