Health Checks and System Readiness
Ze provides two complementary systems for monitoring operational health: offline doctor checks (pre-start readiness) and runtime health monitoring (continuous anomaly detection during operation).
Offline: ze doctor
Run ze doctor before starting the daemon to verify the environment is ready.
Add --json for machine-readable output with stable diagnostic codes.
ze doctor
ze doctor --json
ze doctor --json config /path/to/config.conf
ze doctor --json kernel-capabilities
With no argument, ze doctor checks the active config in the store. config <file> names a config file to check instead, and config - reads it from
stdin. The file is always the value of the config keyword: a bare path such as
ze doctor router.conf is refused with an error that names
ze doctor config <file>.
ze doctor kernel-capabilities asks a different question: it probes every
kernel feature Ze enrolls, whatever the configuration uses, prints one row per
feature (subsystem, CONFIG_ symbol, state, reason) and exits 0 only when every
row is present. It is how a host is checked before it runs Ze's Docker labs.
Each check produces a diagnostic with a code (e.g., doctor-config-reference),
severity (error or warning), and message. When a command clears the
problem, the finding also names it as its fix, drawn on a line of its own under
the message (JSON carries it under fix):
ERROR [doctor-fixture-down] fixture is down
fix: ze start
Use ze show diagnostic <code> for remediation guidance.
Checks
| Code | What it checks |
|---|---|
doctor-store-integrity |
database/ store corruption |
doctor-config-missing |
Config file resolution |
doctor-config-parse |
The config file parses, and then passes the same load validation the daemon applies at startup. A syntax error or a load refusal is reported under this code, and no post-config check runs after it |
doctor-config-reference |
Dangling policy/filter references, reported before load validation can refuse the configuration |
doctor-tls-* |
TLS cert existence, expiry, validity |
doctor-plugin-missing |
Plugin binary on PATH |
doctor-plugin-shell-missing |
The shell an external plugin is started with. Ze runs the run string as /bin/sh -c <run>, so a host with no shell, such as a gokrazy appliance image, starts none of them |
doctor-plugin-setup-failed |
A plugin's setup step recorded a failure: an error when the failure stops ze starting, a warning when the plugin runs degraded. The fix is the remedy the failure names, else ze show plugin list |
doctor-plugin-fatal-stop |
The last stop of ze was a plugin whose failure policy is fatal, or a boot ze refused: the plugin, what happened and when, read from the record the stop left |
doctor-plugin-stop-cleanup-late |
A warning: the last stop went on before a plugin finished its cleanup, so state that plugin installs may remain. The fix searches the log up to that stop |
doctor-plugin-autoload-unresolved |
A plugin the config loads needs a plugin this build does not carry, so ze would refuse to start; the row carries the refusal ze would give |
doctor-plugin-no-dataplane-writer |
The config loads a plugin whose routes need a data-plane writer (static, bgp-epe) and no plugin programs the data plane the config selects, or this build was compiled without it, so ze would refuse to start or reload; an explicit fib { ... } block is never reported |
doctor-plugin-removal-unrecovered |
A plugin a refused reload removed could not be restored and stays removed; ze request reload retries it |
doctor-plugin-capabilities-not-restored |
A refused reload could not restore the plugins' capability declarations, so a BGP OPEN may carry what it declared; kept until a reload commits |
doctor-audit-record-failed |
The audit log's last write failed (file, error, time, count); the records stay in memory until a write lands |
doctor-plugin-stopped |
Inside the daemon only (show doctor): each configured plugin that is not running, with why (its failure policy after an exit, the respawn limit, a failed restart, a failed startup handshake), the exit status and since when. An error, fixed by ze request plugin restart <name> |
doctor-plugin-command-not-registered |
Inside the daemon only: a command or deprecated alias a running plugin declared was refused because another plugin, or a builtin, owns the name. A warning naming both, fixed by removing or renaming one of them |
doctor-plugin-subscription-dropped |
Inside the daemon only: a plugin's event subscriptions were dropped (unknown namespace, no default namespace, ready parameters that did not parse), so it receives none of those events. An error |
doctor-plugin-delivery-disagreement |
Inside the daemon only: a peer's attach process block and the events a plugin declared do not meet, so the event is never delivered. A warning per finding, fixed by the attach process change it names |
doctor-plugin-claim-unbacked |
Inside the daemon only: an exclusive role is claimed and no claimant runs, so nobody performs it. An error naming the role and the claimants |
doctor-plugin-poststartup-undelivered |
Inside the daemon only: a running plugin's post-startup callback failed, so the work it starts after startup never ran. An error, fixed by ze request plugin restart <name> |
doctor-plugin-panic |
Inside the daemon only: a plugin failed to apply an event, or an engine event handler panicked and was recovered. An error naming the plugin, or the handler's namespace and event type, and the first error |
doctor-plugin-delivery-failed |
Inside the daemon only: in the last five minutes, event batches failed with no caller waiting, or took over 500 ms. A warning naming the counts, the slowest batch and the last error |
doctor-plugin-enricher-unavailable |
Inside the daemon only: a plugin's show enricher failed to register or its last call failed, so the command answers without its field. A warning naming plugin, command and key, and one counting the failing enrichers past the 16 listed |
doctor-plugin-config-unapplied |
Inside the daemon only: a running plugin was not told its last transaction committed, so it keeps that transaction's undo set. A warning naming the transaction and the daemon's last one, fixed by ze request plugin restart <name>: a reload of an unchanged config tells no plugin anything |
doctor-hub-managed-collision |
A plugin/hub/server block declares managed clients on the address the plugin acceptor binds, so its managed listener cannot bind and its clients are refused |
doctor-service-executable |
Installed systemd unit ExecStart points to an executable ze binary, and every ExecReload program exists and is executable |
doctor-service-user / doctor-service-group |
User/group referenced by installed ze.service exists |
doctor-service-stop |
The stop settings systemd applies to the installed ze.service (systemctl show -p TimeoutStopUSec -p KillMode, drop-ins included) give ze its whole stop: a timeout of at least 90 s (60 s drain ceiling plus 30 s teardown) and KillMode=mixed. A host with no systemctl is told it does not run systemd and that the unit file is stale; any other failed query carries systemctl's own error |
doctor-ssh-hostkey-missing |
SSH host key |
doctor-listen-unavailable |
Port binding |
doctor-listener-set-unknown |
The listen addresses to check cannot be derived: the YANG schema built into ze failed to load, so no web, SSH, gNMI or API listener was checked. An error, because it means an incomplete ze binary |
doctor-iface-missing / doctor-iface-down |
Interface existence and state |
doctor-iface-selector-unmatched / doctor-iface-selector-ambiguous |
A mac { match } selector names no device, or several |
doctor-iface-mac-override-by-name |
An ethernet overrides its MAC but binds by name, so the address lands on whichever device holds that name |
doctor-disk-space |
Partition free space (<5%) |
doctor-dns-resolver |
Name server reachability |
doctor-*-unreachable |
External-service reachability probes that warn (never fail startup) when a configured peer is down: RADIUS admin (doctor-radius-admin-unreachable), RPKI cache (doctor-rpki-unreachable), BMP collector (doctor-bmp-unreachable), NTP (doctor-ntp-server-unreachable), management hub (doctor-hub-unreachable) |
doctor-clock-skew |
System clock vs NTP (>5 min) |
doctor-gtsm-kernel-state |
A BGP peer or dynamic group range with connection ttl (RFC 5082 GTSM) whose kernel state is short: inside the daemon (show doctor), the host route carrying hop limit 255, or a ze_gtsm term the daemon published (the listen-port drop or a related-ICMP drop), is missing from the kernel. The report names each missing term and the fix, ze signal restart, because a reload leaves an unchanged peer set alone. In ze doctor, before a start, the kernel resolves no route to the peer, so that route cannot be installed at apply |
doctor-vpp-unreachable |
VPP API socket (Linux) |
doctor-vpp-version |
VPP version compatibility (Linux) |
doctor-module-missing |
Kernel modules (Linux) |
doctor-platform-detect |
Platform detection failure |
doctor-platform-unknown |
Unrecognized runtime platform |
doctor-platform-perm |
Gokrazy /perm not writable |
doctor-platform-container-ro |
Container with read-only root |
Exit code 0 means ready; exit code 1 means at least one error-severity issue.
Runtime: show health
The health registry aggregates component status from the running daemon.
show health
show health | json
The /health HTTP endpoint returns the same data as JSON, with HTTP 503
when any component is down.
Registered Components
| Component | What it monitors |
|---|---|
bgp |
Session stuck, flap, EOR timeout warnings |
fib |
Sync failure, orphan routes, programming lag |
firewall |
Stale ze_* tables, chain drift vs config, a ze_* table the kernel lost, and an audit that could not read the kernel |
iface |
RX/TX error counter increases |
plugins |
Plugin crashes and disabled-by-respawn-limit |
vpp |
Present only while interface { backend vpp; } is the loaded backend. Down, with the socket path and ze doctor as the next step, when the vpp component is not running, the configured vpp { api-socket } does not exist, or it refuses connections |
ipsec |
IPsec SA state |
pki |
Certificate expiry |
l2tp |
L2TP subsystem running, and every listener programming sessions into the kernel data plane. Degraded, with the reason and ze doctor to check the kernel modules, when the l2tp generic netlink family could not be resolved, ze.l2tp.disable-kernel-dataplane is set, or the platform is not Linux |
flow-export |
Exporter state and collector send errors |
report-bus |
Report bus registration |
Each check completes within 1 second. Components report healthy, degraded,
or down. The overall status is the worst of all components. A component that
has nothing to measure on this host, such as vpp when another interface
backend is loaded, has no row at all rather than a healthy row nobody
measured.
Runtime: show warnings / show errors
The report bus surfaces individual anomalies:
- Warnings are state-based (raised when a condition exists, cleared when it
resolves). Example:
session-stuckclears when the peer reaches Established. - Errors are event-based (recorded when something happens). Example:
fib-sync-failurefires once per failed route install.
show warnings
show warnings | json
show errors
show errors | json
show warnings source bgp
Report Bus Codes
| Code | Source | Type | Trigger |
|---|---|---|---|
session-stuck |
bgp | warning | Peer non-Established >5 min |
session-flap |
bgp | warning | >3 transitions in 5 min |
eor-timeout |
bgp | warning | EOR incomplete after GR restart-time |
route-count-anomaly |
bgp | error | >50% prefix drop (min 100 prefixes) |
prefix-threshold |
bgp | warning | Prefix count at warning level |
prefix-stale |
bgp | warning | Prefix data >180 days old |
fib-sync-failure |
fib | error | Backend add/replace/delete failed |
fib-orphan |
fib | warning | Orphan routes swept at startup |
fib-programming-lag |
fib | warning | Routes pending >30s |
doctor-fib-backup-not-programmed |
fib | warning | Installed routes hold a fast-reroute backup the kernel does not carry; the detail counts them. Cleared when none is left |
doctor-fib-flush-incomplete |
fib | error | flush-on-stop left Ze's routes in the kernel when fib-kernel stopped: a delete failed, the kernel still listed a flushed route, or the listing failed. The detail counts them and names the first; the fix is ip route flush proto 250 table all; ip -6 route flush proto 250 table all. Not in the fib health row |
doctor-fib-sweep-incomplete |
fib | error | The startup sweep found routes a previous run of Ze left in the kernel that no protocol announced again, and the kernel refused to delete one or more, or the kernel's routes could not be listed at startup, so nothing was swept. Each route left keeps forwarding towards a next hop nothing maintains. The detail counts the prefixes whose delete failed and names the first with the kernel's error, or carries the listing's error (operation list); the fix is ip route show proto 250 table all; ip -6 route show proto 250 table all; ip route del <prefix> proto 250 table <table>. Not in the fib health row |
doctor-iface-address-unattributed |
iface | error | The netlink monitor received an address for an interface it had not seen yet, and the kernel lookup of that interface failed, so Ze did not announce the address: its connected route is missing until the address changes. The subject is the address and the interface index, the detail carries the kernel's error; the fix is ip -o addr show to <address>; ip addr del <address> dev <device>; ip addr add <address> dev <device>. Raised once per lost address and never cleared, so not in the iface health row |
firewall-stale-table |
firewall | warning | ze_* table in kernel not in config |
firewall-drift |
firewall | warning | Chain count mismatch vs config, or a ze_* table the kernel no longer has |
doctor-plugin-crashed |
plugin | error | Plugin process exited unexpectedly; carries the exit status and the failure policy ze applies, and clears every warning the plugin raised itself |
doctor-plugin-stopped |
plugin | warning | A configured plugin is down on any path (failure policy, respawn limit, failed restart, failed startup handshake); the fix is ze request plugin restart <name>, and the plugins health row reads down. Cleared when the plugin runs again or leaves the configuration |
doctor-plugin-stderr-relay-stopped |
plugin | error | The relay copying a plugin's stderr into ze's log stopped while the plugin runs (a read error, a line over the scanner's limit); the fix is ze request plugin restart <name> |
doctor-plugin-restart-overlap |
plugin | error | A plugin was restarted before ze finished stopping its previous process (2s after the stop for a forked plugin, 1s for one in the daemon). The message says what was left: an engine that had not returned or a process that had not exited, which may hold the same resources as the replacement, or only the event delivery or stderr relay of one that had ended. ze takes no further step to end it, and a second restart does not help, because it replaces the new process. The fix shows what is left: ze show system goroutines full for a plugin in the daemon, ze show log recent component plugin.relay for a forked plugin |
doctor-config-reload-refused |
plugin-server | error | A configuration reload was refused: its transaction did not commit (a plugin refused the change or did not answer), or the plugins the new configuration needs could not be resolved or did not start. ze keeps the configuration it was running; an event raised while ze restored that configuration says the restore did not complete. The fix is the refusal's own command when it names one (a plugin dependency this build does not carry), else ze show errors |
iface-errors |
iface | warning | RX/TX error counters increasing |
The bus holds at most ze.report.warnings.max active warnings (default 1024)
and ze.report.errors.max error events (default 256), both clamped to 10000.
Past the warning cap the oldest warning is dropped, and past the error cap the
oldest event is overwritten. ze doctor then reports doctor-report-bus-full
with the cap, how many entries were lost and the export that raises the cap. A
health row whose code may have been dropped reads degraded "report bus full"
rather than healthy, until the owner clears or raises that warning again.
Adding Doctor Checks
Plugins and components extend ze doctor by registering their own checks.
There are two registration paths depending on whether the owner is a plugin.
Internal plugins (preferred)
Declare checks in registry.Registration.DoctorChecks. The doctor runner
bridges these at execution time. Each check receives a DoctorCheckContext
(config tree + platform) and returns diagnostics.
reg := registry.Registration{
Name: "my-plugin",
// ...
DoctorChecks: []registry.DoctorCheckDef{{
Name: "my-plugin-reachable",
Phase: rpc.DoctorPhasePostConfig,
Order: 700,
Dependencies: []string{"config-parse"},
Platforms: []string{"any"},
Codes: []string{"doctor-my-plugin-unreachable"},
Check: checkMyService,
}},
}
Reference example: internal/component/l2tp/plugins/authradius/register.go.
Non-plugin components
Components that are not plugins (appliance, web, SSH) use
diagnostic.RegisterDoctorCheck() from their owning package's init().
func init() {
diagnostic.RegisterDoctorCheck(diagnostic.DoctorCheck{
Name: "my-component-ready",
Phase: "post-config",
Order: 500,
Component: "my-component",
Dependencies: []string{"config-parse"},
Platforms: []string{"any"},
Codes: []string{"doctor-my-component-missing"},
Check: checkMyComponent,
})
}
External plugins (RPC)
External plugins declare checks in DoctorChecks during Stage 1
registration, then handle the doctor-check callback at runtime.
Declaration (in sdk.Registration):
p.Run(ctx, &sdk.Registration{
DoctorChecks: []rpc.DoctorCheckDecl{{
Name: "my-cache-reachable",
Phase: rpc.DoctorPhasePostConfig,
Order: 700,
Dependencies: []string{"config-parse"},
Platforms: []string{"any"},
Codes: []string{"doctor-my-cache-unreachable"},
}},
})
Callback handler:
p.OnDoctorCheck(func(name string) ([]rpc.DoctorCheckDiagnostic, error) {
if name == "my-cache-reachable" {
if err := pingCache(); err != nil {
return []rpc.DoctorCheckDiagnostic{{
Code: "doctor-my-cache-unreachable",
Severity: "error",
Message: "cache server not reachable: " + err.Error(),
Fix: "show my-cache status",
}}, nil
}
}
return nil, nil
})
The engine invokes plugin checks only at runtime (show doctor), not
during offline ze doctor. Set Fix to the command that clears the finding,
alone, with no fix: label. A check that reads live state the daemon holds
(not the config file) declares Phase: rpc.DoctorPhaseRuntime; the same phase
is open to an in-tree check registered with diagnostic.RegisterDoctorCheck,
and only show doctor runs it. A plugin's runtime check is declared here and
answered by OnDoctorCheck, never as a registry.DoctorCheckDef: the plugin
registry refuses that form in the runtime phase, because its function would
run in the daemon's process rather than the plugin's. Each check has one
declaration: Stage 1 refuses a name the plugin's registry row already declares.
See
docs/architecture/api/process-protocol.md for the full protocol.
Requirements
Every new doctor check must:
- Use a diagnostic code with the
doctor-prefix for a code the check introduces. A check that bridges another surface's validator declares the codes that validator emits: the semantic-validation check ininternal/component/configdeclaresconfig-mcp-invalidand its siblings, because adoctor-alias for them would be a second code for one fact. The registry accepts any well-formed lower-kebab code (diagnostic.RegisterDoctorCheck). - Register the code in
internal/core/diagnostic/codes.gosoze show diagnostic <code>works. The entry names the code through an exported constant, and the check that emits it references that constant. A code spelled a second time in the emitting package is a second declaration of one fact, and./le arch enumeration checkrefuses it. - Include a unit test proving the check fires when the relevant config is present and emits the registered code.