Guide

VM Appliance

Build a bootable VM image with Ze baked in using gokrazy. The default target is x86_64, and image.arch in appliance.json selects another architecture for ./ze-host appliance build. At runtime, the appliance is minimal: Linux kernel, gokrazy init, and Ze as the only application, with no package manager, no general shell (except authenticated emergency serial console), no unused distro daemons, and automatic process supervision.

Suitable for N100-class mini PCs, Proxmox VMs, or QEMU testing.

Prepare the build-host command

Work from the Ze source checkout with Go and Git installed. Build the native host driver and select the patched Go environment before the image commands:

./le build host-driver
GOROOT="$(./le go stdpatch goroot)"
export GOROOT GOTOOLCHAIN=local PATH="$GOROOT/bin:$PATH"

The examples below use ./ze-host, which runs on the build machine and carries the appliance and PXE provisioning commands. The ordinary ./ze launcher builds the Linux daemon; it does not carry appliance. Build kernels and cold-cache images on a host matching the image architecture. Binaries can be cross-built, but selecting another image.arch does not make a foreign kernel build possible.

What's in the image

Component Purpose
Linux kernel Boot and hardware drivers
gokrazy init Starts Ze, supervises it, seeds entropy, sends watchdog heartbeat
Ze BGP daemon with DHCP client and all internal plugins
ze-serial-shell Authenticated emergency shell on serial console (login required)

Ze owns network configuration in the appliance. The gokrazy default DHCP and NTP packages are excluded from the image; the shipped Ze seed template enables interface DHCP auto-discovery with set interface dhcp-auto true and leaves Ze NTP disabled (set environment ntp enabled false) until the operator enables it in Ze config.

The root filesystem is read-only (SquashFS). Persistent data lives on a separate ext4 partition mounted at /perm.

Prerequisites

Install once on the build machine.

macOS:

brew install e2fsprogs    # ext4 filesystem tools
brew install qemu         # VM runtime (testing only)

Linux:

sudo apt-get install -y e2fsprogs qemu-system-x86   # Debian, Ubuntu
sudo dnf install -y e2fsprogs qemu-system-x86       # Fedora

The build needs mkfs.ext4 and debugfs from e2fsprogs. If e2fsck is available, the builder also checks ext4 structure; otherwise it warns and skips that check. The native builder resolves each tool independently from the Homebrew keg, standard sbin directories, and PATH.

For appliance ISO creation, install grub-mkstandalone (or grub2-mkstandalone) with the GRUB EFI modules for the target (x86_64-efi or arm64-efi), plus xorriso. ./ze-host appliance iso checks those tools and modules before it stages an ISO. On macOS, build the ISO on a Linux host: see ISO prerequisites.

The vendored gokrazy command (github.com/gokrazy/tools/gok) runs in process. ./ze-host appliance build calls it directly. A developer runs it with ./le build gokrazy <gok args>, which uses the checked-in module cache, forbids network module lookups, and builds overwrite from a prepared copy of the instance under tmp/. No separate gokrazy installation or first-party script is required.

The image's ze compiles only against Ze's patched standard library, so the build runs under the patched GOROOT. ./le build gokrazy inherits it from le. ./ze-host appliance build refuses unless its environment already selects it: GOROOT set to the tree, GOTOOLCHAIN=local, and $GOROOT/bin/go first on PATH. The refusal ends with the command line that sets all three:

GOROOT="$(./le go stdpatch goroot)" && export GOROOT GOTOOLCHAIN=local PATH="$GOROOT/bin:$PATH" && ./ze-host appliance build <name>

First-time setup

Initialize the named appliance before its first build:

./ze-host appliance init edge-01

The build resolves pinned system packages through the Go module graph.

Runtime Kernel Requirements

The runtime kernel is pinned by two manifests, gokrazy/kernel/kernel.require and gokrazy/kernel/runtime.require. Each is checked against the resolved config after the build, and enforce_required_symbols accepts =y alone. A Kconfig answer of m therefore fails the BUILD rather than shipping an appliance where a feature Ze accepts in config cannot work. Runtime images also package the matching module tree; a module from a stock distribution is not a substitute for that kernel's modules.

Each symbol has a producer in Ze rather than a test that wanted it.

Group Symbols Producer in Ze
Subscriber PPP, PPPOL2TP, PPPOE, L2TP, L2TP_V3 The L2TP LNS, and the PPPoE server and client
Firewall NF_TABLES, NF_TABLES_INET, NF_TABLES_IPV4, NF_TABLES_IPV6, NF_CONNTRACK, NF_NAT, NFT_CT, NFT_NAT, NFT_MASQ, NFT_REDIR, NFT_LIMIT, NFT_LOG, NFT_REJECT, NF_FLOW_TABLE, NF_FLOW_TABLE_INET, NFT_FLOW_OFFLOAD The nftables backend, and translatePolicy, which returns FamilyInet unconditionally. Without NF_TABLES_INET the kernel answers EOPNOTSUPP, Apply's flush fails, the firewall plugin fails startup, and the daemon exits. The three flowtable symbols carry a table's flowtable and a term's flow-offload action (applyFlowtable); without them the kernel answers ENOENT and refuses the whole table. internal/plugins/copp/translate.go
Tunnels NET_IPGRE_DEMUX, NET_IPGRE, IPV6_GRE, NET_IPIP, IPV6_TUNNEL, IPV6_SIT, VXLAN The nine tunnel kinds ze-iface-conf.yang models. Six symbols plus the GRE demux gate. Measured cost: vmlinuz 16352256 to 16450560 bytes, +0.60%
SRv6 IPV6_SEG6_LWTUNNEL The kernel FIB backend's Service SID routes: buildSEG6Encap installs each RFC 9252 service route as a seg6 encap-mode lightweight tunnel. Without it the kernel answers EOPNOTSUPP and no SRv6 service route is installed. internal/plugins/fib/kernel/nexthop_linux.go
Policy routing IP_MULTIPLE_TABLES, IPV6_MULTIPLE_TABLES, IP_ROUTE_MULTIPATH The policy-routing engine. Without IPV6_MULTIPLE_TABLES the kernel folds every table id into the main table and says nothing
Traffic control NET_SCH_HTB, TBF, FQ_CODEL, HFSC, FQ, SFQ, NETEM, PRIO, INGRESS, NET_CLS_U32, NET_CLS_FW, NET_CLS_MATCHALL, NET_ACT_MIRRED, NET_ACT_POLICE Class of service, rate limiting, and mirroring; subscriber upload rates use the police action through shaper.applyTC and ingressPolicerFilter
Interfaces and VPN DUMMY, VETH, MACVLAN, VLAN_8021Q, INET_ESP, INET6_ESP, XFRM_STATISTICS, HUGETLBFS, BPF_SYSCALL, BPF_JIT Interface creation, IPsec, VPP hugepages, and the eBPF surfaces
IPsec and WireGuard XFRM_USER, XFRM_INTERFACE, XFRM_MIGRATE, INET_AH, INET6_AH, WIREGUARD, CRYPTO_CBC, CRYPTO_DES, CRYPTO_NULL, CRYPTO_GCM, CRYPTO_CHACHA20POLY1305, CRYPTO_SHA1, CRYPTO_SHA256, CRYPTO_SHA512 Every SA install (XFRM netlink), route-based IPsec, MOBIKE, OSPFv3 authentication, WireGuard links, and one symbol for each ESP transform Ze can install, so every cipher Ze accepts in config is carried by the kernel

CONFIG_NET_UDP_TUNNEL and CONFIG_TUN are =y in the config fragment and are not pinned in a manifest.

VPN Kernel Proof

./le test qemu vpn-kernel boots Ze's docker-host image with its own kernel and base, on the host's native architecture. It uses the shared Docker lab transport: KVM on Linux or HVF on macOS, never a stock guest or foreign-architecture emulation. This is a manual kernel proof, not a check of the default appliance image's packaged config.

The action runs AES-GCM, AES-CBC/SHA256, ChaCha20-Poly1305, 3DES/SHA1 and NULL/SHA256 against strongSwan. The IPsec checks require installed SAs and received tunnel traffic. A separate pair of Linux WireGuard peers must complete a handshake and pass a ping, with received bytes at both ends.

Before services start, the disposable guest disables module loading, refuses any preloaded module, and removes its /sbin/modprobe helper. The shipped image and any existing Docker host are unchanged. The host checks the emitted kernel config against both requirement manifests; the WireGuard step checks the guest root again, not the container root.

The WireGuard step proves kernel traffic, not Ze's configuration-commit path. An amd64 run does not replace the required arm64/HVF run, and no result is implied by the command's presence.

L2TP Kernel Support

Ze's L2TP LNS path needs kernel PPPoL2TP support in the appliance runtime kernel: CONFIG_PPP, CONFIG_PPPOL2TP, CONFIG_L2TP, and CONFIG_L2TP_V3. The shared runtime proof kernel also keeps CONFIG_PPPOE built in for PPPoE evidence. The appliance image boots this kernel, so every image carries these options. To fill the cache before a build, on an amd64 host:

./ze-host appliance kernel --target runtime --arch amd64

A kernel is built only on a host of its own architecture, so the arm64 kernel is built on an arm64 host (a Mac, for example). Select the QEMU builder there when Docker is absent:

./ze-host appliance kernel --target runtime --arch arm64 --builder qemu

On a host of the other architecture both commands are refused before anything starts. To build an image for another architecture, build it on a host of that architecture, or set image.arch to the host's in the appliance's config file; the refusal names that file and ends with the build command to run. ./le build gokrazy with GOARCH naming another architecture is refused the same way, and ends with the same command run with GOARCH set to the host's. This is the kernel every appliance image boots, and no other kernel can reach an image. ./ze-host appliance build and ./le build gokrazy resolve it for the image's architecture themselves: from the cache when an entry is there, else by building it, so these commands only fill the cache ahead of a build. A cold cache also builds the kernel before the image. It needs network access to kernel.org and Docker or QEMU on a host of the image's architecture.

image.kernel-profile selects the installer kernel. Appliance images use the shared runtime profile for their architecture. The installer qemu fragment requests virtio drivers and is merged onto Linux defconfig, which can enable additional drivers. The emitted kernel configuration records the complete driver selection.

./ze-host appliance kernel --target runtime calls the Go driver in internal/appliance/kernelbuilder. The driver reads internal/appliance/kernel.version, selects Docker or QEMU, resolves the tracked config fragments and .require manifests, and writes the runtime kernel tree. The Go worker enforces every required symbol before the artifact can be used.

On a Linux runner with QEMU, xl2tpd, pppd, /dev/ppp, and PPPoL2TP kernel support, the deployment proof target builds an L2TP-enabled appliance image and drives a real LAC against it:

./le test deployment gokrazy-l2tp-ppp

The proof image is built from a temporary gokrazy instance config so the normal appliance config is left unchanged. It disables IPv6CP in that proof image because the current static L2TP pool is IPv4-only. Set ZE_GOKRAZY_SKIP_BUILD=1 to run against an existing tmp/gokrazy/ze.img that was already built with the L2TP proof template, the proof runtime environment, and an L2TP-capable kernel: skip-build bypasses the proof's check of the runtime kernel cache, and an image whose kernel lacks PPPoL2TP crash-loops at first boot instead of serving.

Build an image

Build the appliance initialized above. If starting here, first run ./ze-host appliance init edge-01 once:

./ze-host appliance build edge-01

The named appliance keeps its config and secrets between builds. The build assembles its ZeFS seed artifact, builds the disk image through vendored gokrazy, formats the persistent partition, injects database.zefs, and writes the build manifest. Seed assembly uses the explicit blob-artifact storage API, so image building continues to produce the same blob format.

./ze-host appliance show edge-01
./ze-host appliance config edge-01 --merged

Test in QEMU

./ze-host appliance run edge-01

The command boots the named image in QEMU. Quit QEMU with Ctrl-A X. The Gokrazy management UI is exposed through Ze's authenticated web UI at /gokrazy/; the proxy reads Gokrazy's password from its standard password-file locations.

Deploy to hardware

Write the image to a USB drive or internal disk on your N100 machine:

Use the exact .img path printed by the named build's image ready: line. Named builds write into the selected appliance directory, normally ~/.config/ze/appliances/edge-01/; tmp/gokrazy/ze.img belongs to other build and proof workflows. Set ZE_IMAGE to the named image you just built, verify its accompanying .sha256 file, and identify the target device before writing.

ZE_IMAGE='/absolute/path/from-image-ready/ze-YYYYMMDD-HHMMSS.img'

# Linux
sudo dd if="$ZE_IMAGE" of=/dev/sdX bs=4M status=progress

# macOS
sudo dd if="$ZE_IMAGE" of=/dev/rdiskN bs=4m

Or import into Proxmox:

qm importdisk <vmid> "$ZE_IMAGE" <storage>

The machine boots to a serial console (115200 baud). On first boot Ze explicitly imports /perm/ze/database.zefs into /perm/ze/database/, logs the import, and retires the seed as database.zefs.replaced-*. Subsequent starts use the live tree. Ze gets a DHCP address and loads its active configuration from that tree. The serial console requires authentication with the local admin credentials before granting shell access. If the store is missing or unreadable, or the credentials cannot be read, the login is refused with the error named. When admin-enabled: false is set in the appliance config, the serial console denies the built-in admin (fail-closed) and prints "local admin login disabled".

Configuration

Seed config

The initial Ze config is stored as the seed template in gokrazy/ze/ze.conf. ./ze-host appliance assemble uses it when no base or per-appliance overlay exists. The seed database holds no active config that can shadow the template, so the appliance builds its effective config on first boot.

set environment log level info
set environment web enabled true
set environment web server default ip 0.0.0.0
set environment web server default port 8080
set environment ssh enabled true
set environment ssh server default ip 0.0.0.0
set environment ssh server default port 22
set environment ntp enabled false
set interface dhcp-auto true

To change the seed config, edit gokrazy/ze/ze.conf, or use the structured workflow's config-base and per-appliance ze.conf files.

Runtime config

Once booted, use ze config edit over SSH to modify the running configuration. Changes are stored in /perm/ze/database/ and persist across reboots and image updates. The daemon owns the store for its lifetime; offline maintenance must run as the store owner after the daemon stops.

Environment variables

Ze's environment is set in gokrazy/ze/config.json under PackageConfig:

Variable Value Purpose
ze.config.dir /perm/ze Configuration folder containing the live database/ tree
ze.bgp.api.socketpath /tmp/ze.socket API socket location
ze.bgp.daemon.drop false ✕
ze.log info Log level
ze.log.backend kmsg,stderr Logs go to kmsg and gokrazy ring buffers
ze.gokrazy.enabled true Enables appliance auto-init fallback and the. /gokrazy/

Kernel crash capture

An appliance that suffers a kernel panic reboots and tells you nothing. Root is read-only SquashFS, the image holds no busybox and no shell, and the kernel that faulted is gone. Crash capture asks the kernel to write its own panic message and backtrace into a reserved memory region before it goes down, so the evidence survives the warm reboot.

Two halves, and both are needed:

Half Where What it does
The reservation image.crash-dump.reserve in the appliance config Renders reserve_mem=<N>M:4096:zecrash and ramoops.mem_name=zecrash onto the built image's kernel command line
The intent system crash-dump enabled true in the Ze config Tells the running daemon to harvest the record at boot and store it as a crash report
"image": {"arch": "amd64", "size-bytes": 2147483648, "crash-dump": {"reserve": "16mb"}}

The reservation is a boot argument, so it takes effect at the next boot of an image built with it, never at a config commit. show crashes reports the two states separately as configured and armed, and ze doctor raises doctor-crash-capture-unarmed while they disagree.

The reserve is 4mb to 256mb and is taken from RAM on every boot, so it is sized for a backtrace: 16mb is the default and holds a full one. The build refuses a value outside that range, and refuses a size that is not a whole number of megabytes, because the appliance has no shell to diagnose a silently ineffective reservation with.

The region is named rather than addressed. Size-named reservation has been in the kernel since 6.12 and internal/appliance/kernel.version pins a kernel above that floor, so no per-machine physical address is needed. The runtime kernel carries CONFIG_PSTORE and CONFIG_PSTORE_RAM, and the build fails if either stops resolving to =y: without them the kernel would accept the reservation, take the RAM, and expose nothing to read the record back from.

Harvested records land in the crash directory, which on an appliance is /perm/ze/crash. Gokrazy's A/B updates replace the root partition and leave /perm, so a record survives the update that follows a crash. Records share the ze.crash.keep retention count with Go panic reports, so a panic loop cannot fill /perm.

Updating

Push a rebuilt image to the running appliance over authenticated HTTPS:

./ze-host appliance build <name>
./ze-host appliance push --testboot <name>

Push validates the disk image, sends its root filesystem to the inactive root slot, updates the shared boot filesystem and boot code, then reboots. It keeps the device's disk identities and persistent configuration. --testboot asks gokrazy for a temporary boot of the new root, with the previous root selected for a subsequent reboot. It does not verify routing health. Use --no-reboot to schedule the restart separately.

The two root slots protect the previous root filesystem. The boot filesystem is shared, so keep installer media and a configuration/secret backup for an interrupted boot write. A partition-layout change requires reinstalling the disk image rather than an OTA push.

Before the update, record the running version and archive the active configuration using the lifecycle guide. Keep the named appliance's secrets and the previous image. After reboot, query the appliance through its SSH management endpoint, using its provisioned address, SSH port and credentials:

ssh -p <ssh-port> <username>@<device-address> "show version --extended"
ssh -p <ssh-port> <username>@<device-address> "show doctor"

Compare the returned version with the selected image, then verify the hostname, committed configuration, peers and routes. A completed upload alone does not establish that the new runtime is reachable or routing correctly.

For a --testboot update, a subsequent reboot selects the previous root slot. Check the version and retained configuration again after recovery. Ordinary push selects the new root persistently. Both modes leave the persistent store in place; a root-slot recovery does not undo configuration edits. The installation guide states the architecture and deployment checks completed so far.

Removing an appliance

Before retiring an appliance, back up its live store using the Operations guide and keep the backup outside the disk you will remove. Also export the named appliance directory to retain its build configuration and secrets. Appliance exports exclude disk images and ZeFS databases, so keep the live-store backup separately.

Use the graceful stop procedure, then power off the VM or physical device. Wait for shutdown to finish before removing its storage.

For a VM, remove its definition through the hypervisor and explicitly choose whether to retain or delete the associated disks. The QEMU command uses the selected .img file as its writable disk; retaining that file keeps the device's stored state. For dedicated hardware, boot the replacement system's installer and identify the appliance disk before overwriting it. Follow the storage device's erasure procedure before disposing of it or transferring it to another owner.

Retained disks, appliance directories, seeded images and installer ISOs contain configuration and authentication material. Protect any copies kept for recovery, and revoke credentials that should no longer grant access after retirement.

Architecture notes

Internal plugins only

Gokrazy has no shell and no PATH. Ze's external plugin mechanism (which uses /bin/sh -c to fork processes) does not work. All Ze plugins (bgp-rib, bgp-gr, bgp-adj-rib-in, etc.) are compiled into the ze binary as internal plugins and run as goroutines. This is the default and covers all standard BGP functionality.

A config that names an external plugin anyway is reported: ze doctor raises doctor-plugin-shell-missing, naming the absent shell and the plugins that depend on it, and a start that reaches the fork names the shell rather than the plugin.

Process supervision

Gokrazy's init restarts Ze if it exits with a non-zero status (except 125, which means "don't restart"). Ze handles SIGTERM for graceful shutdown. Logs (stdout/stderr) are captured in ring buffers visible through the gokrazy web UI.

Persistent storage

The /perm partition (ext4) survives image updates. The configuration folder /perm/ze, selected by ze.config.dir, contains the live database/ tree. Credentials, TLS material and config history are keys in that tree, with 0700 directories and 0600 frame files owned by the daemon user. The retired seed remains beside it for recovery.

Repo layout

gokrazy/
  .gitignore              # excludes *.img
  ze/
    config.json           # gokrazy instance config (what to build, how to start)
    builddir/
      github.com/ze-software/ze/
        go.mod            # ze dependency pins + relative replace directive
        go.sum
      ze.invalid/kernel/
        go.mod            # requires ze's runtime kernel, replaced per build
      github.com/gokrazy/gokrazy/
        go.mod, go.sum    # gokrazy init system version pin
        cmd/dhcp/         # DHCP client
        cmd/ntp/          # NTP client
        cmd/heartbeat/    # watchdog heartbeat
        cmd/randomd/      # entropy seeder
cmd/ze-serial-shell/        # serial console login gate (replaces serial-busybox)
  main.go                   # gokrazy wrapper: symlink + DontStartOnBoot
  _gokrazy/                 # renamed busybox extrafiles per arch
internal/le/build/gokrazy/  # `le build gokrazy`: the vendored gokrazy command wrapper

The gok source is vendored under vendor/github.com/gokrazy/. The small builddir/ modules pin the system-package versions.

Builds never run from this directory

The tree above is the build input; no build runs inside it. Every image build first copies the instance it names (gokrazy/ze for the appliance, gokrazy/docker-host for ./le setup docker host build) to a fresh directory under the project tmp/, including the whole builddir, rewriting each filesystem-path replace to an absolute path so it still resolves from the new depth. gok is then pointed at the copy, and the copy is deleted afterwards.

./ze-host appliance build prepares an isolated copy through resolveBuildParentDir. A build leaves the working tree unchanged, and two builds in one checkout use separate prepared instances.

The image boots ze's own runtime kernel and no other. config.json names ze.invalid/kernel as its KernelPackage, a path under a reserved domain that never resolves on its own. Each build resolves the runtime kernel for the image's architecture (the cache entry, else a native build), assembles a gokrazy kernel package from that cache entry inside the prepared copy, and replaces the kernel module's require to point at it. The package carries the kernel, its modules and a cmdline.txt naming root=/dev/sda2, which gokrazy rewrites to the image's root partition UUID before it appends the instance's extra kernel arguments. It also carries an empty config.txt, the Raspberry Pi bootloader file gokrazy reads from every kernel package and no ze target uses. Preparation with no resolved kernel is refused, so gok never fetches a kernel. No setting selects another kernel package.

Build-host command

Appliance build commands (./ze-host appliance) and PXE provisioning (./ze-host install remote) use the native driver prepared at the beginning of this guide. Rebuild it after changing the source:

./le build host-driver

Building and installing an appliance (end to end)

The running appliance's kernel host name comes from system host in its Ze config, which Ze applies at start and on every reload. identity.hostname in appliance.json names the image's CA certificate and listings, and the image boots as ze until Ze applies system host; with system host unset it stays ze (see docs/guide/configuration.md, "System Host Name").

Write an appliance config file (arch, kernel profile, credentials, networking):

{
    "credentials": { "username": "exa", "admin-enabled": true },
    "ssh":   { "host": "0.0.0.0", "port": "2222" },
    "web":   { "enabled": true, "host": "0.0.0.0", "port": "8080" },
    "tls":   { "cert-name": "router.local", "validity-years": 10 },
    "identity": { "hostname": "ze-prod" },
    "device": { "address": "10.12.104.10", "update-port": 443 },
    "image": { "arch": "amd64", "size-bytes": 2147483648, "kernel-profile": "hardware-kms" }
}

For SSH key login, add credentials.ssh-authorized-keys, an array of complete plain OpenSSH public-key lines, before initializing the named appliance. These keys log in as credentials.username; password recovery remains available. Supply unrestricted recovery public keys: assembly refuses SSH certificates and authorized_keys options such as from= or command= because Ze cannot enforce them.

Reserving hugepages for VPP

When the appliance runs VPP, reserve hugepages at boot by adding image.hugepages to the config. ./ze-host appliance build bakes default_hugepagesz/hugepagesz/hugepages into the boot cmdline (via a derived gokrazy instance config; the checked-in gokrazy/ze/config.json is never modified). Declare image.memory so the build rejects a reservation over 50% of target RAM and ./ze-host appliance run sizes QEMU's -m to match. Sizes are byte-size strings (10b, 512mb, 1gb, 1tb; case-insensitive, 1024-based):

"image": {
    "arch": "amd64",
    "size-bytes": 2147483648,
    "memory": "8gb",
    "hugepages": { "size": "1gb", "page-size": "2mb" }
}

hugepages.size is the total reservation and page-size is 2mb or 1gb; the page count is size / page-size (so size must be a whole multiple of page-size). The reservation is bounded to 512 GiB and, when memory is set, to 50% of it. 1gb pages need CPU pdpe1gb support and CONFIG_HUGETLBFS in the kernel profile (both surfaced by ./ze-host doctor).

Build the full ISO through the structured appliance actions:

./ze-host appliance init --config prod.json prod
./ze-host appliance kernel prod
./ze-host appliance initrd
./ze-host appliance build prod
./ze-host appliance iso prod

The named appliance retains its config and secrets for subsequent builds.

Manual steps

Use the same commands individually when you need to inspect an intermediate artifact:

# Create the appliance with config and secrets.
./ze-host appliance init --config prod.json prod

# Check and prepare ISO prerequisites.
./ze-host appliance iso --check prod
./ze-host appliance kernel prod
./ze-host appliance initrd

# Build the disk image and installer ISO.
./ze-host appliance build prod
./ze-host appliance iso prod

# Provision the resulting image over the network when needed.
ZE_IMAGE='/absolute/path/from-image-ready/ze-YYYYMMDD-HHMMSS.img'
sudo ./ze-host install remote \
  --interface eth0 \
  --network 10.0.0.0/24 \
  --image "$ZE_IMAGE" \
  --ssh-username admin \
  --ssh-password 'choose-a-strong-one'

The kernel and initrd commands first check the XDG cache. If ze.appliance.kernel.url or ze.appliance.initrd.url is set, the matching command then tries that configured prebuilt-artifact URL; otherwise it builds locally. Kernel local builds use the shared Docker-or-QEMU builder selection, and initrd local builds compile and pack cmd/ze-installer. Cached artifacts avoid rebuilding their matching inputs. See "ISO prerequisites" below for details.

ze appliance (structured workflow)

The ./ze-host appliance command provides structured appliance management. Each appliance has its own directory with a JSON config, secrets that are optionally encrypted at rest, and a TLS certificate.

Quick start

./ze-host appliance init lab                  # interactive wizard
./ze-host appliance build lab                 # full image
./ze-host appliance kernel lab                # installer kernel
./ze-host appliance initrd                    # installer initrd
./ze-host appliance iso lab                   # bootable installer ISO
./ze-host appliance list                      # show all appliances
./ze-host appliance show lab                  # config summary and cert expiry

Appliance directory

By default, appliances live in ~/.config/ze/appliances/. Override with --dir or ZE_APPLIANCE_DIR.

~/.config/ze/appliances/
  _shared/
    ze.conf                    # optional base config for all appliances
  lab/
    appliance.json             # config (no credentials)
    ze.conf                    # per-device config overrides
    secrets/                   # 0700 permissions
      .encrypted               # marker (present = secrets encrypted)
      tls/
        cert.pem               # device certificate, then the appliance CA root (plaintext)
        key.pem                # private key (encrypted if passphrase set)
        ca-cert.pem            # appliance CA root certificate (plaintext)
        ca-key.pem             # appliance CA root key (encrypted if passphrase set)
      password.hash            # bcrypt hash (encrypted if passphrase set)
      update.token             # gokrazy OTA token (encrypted if passphrase set)
      authorized_keys          # SSH public keys (plaintext)

Encryption

Secrets are encrypted at rest with Argon2id + XChaCha20-Poly1305 when an encryption passphrase is set during ./ze-host appliance init. The passphrase is never stored on disk. For fleet operations, ./ze-host appliance unlock starts a passphrase agent (like ssh-agent) that holds the derived key in memory.

./ze-host appliance unlock                    # start agent
./ze-host appliance unlock --duration 15m     # auto-expire after 15 minutes
./ze-host appliance unlock --stop             # stop agent

Day-2 operations

./ze-host appliance passwd lab
./ze-host appliance replace-cert lab
./ze-host appliance replace-cert lab --cert ca.pem --key ca.key
./ze-host appliance rekey lab
./ze-host appliance clone lab lab2

Without --cert and --key, replace-cert issues a new device certificate from the appliance's own certificate authority, which ./ze-host appliance init generated. The root does not change, so a device you already pushed to stays reachable: ./ze-host appliance push trusts the issuer rather than one certificate.

replace-cert validates the material before it writes anything. It refuses a certificate and a key that are not a pair. It refuses a file that holds no PEM data. It refuses a certificate that is past its not-after date, and the message gives both validity dates. A certificate whose validity starts in the future is accepted, because a staged renewal is copied into an image that boots later. --cert and --key must be given together.

A refusal leaves cert.pem and key.pem byte-identical to what the appliance already held. Both files are written through a temp file and a rename, so an interrupted run leaves neither file truncated. When the key write fails after the certificate write, the command puts the previous certificate back and reports the restore. ./ze-host appliance init validates and writes the same way.

A ze older than this validation could store a certificate and a key that do not load as a pair. On such an appliance the web listener does not start. Run ze doctor to find it: the stored pair is reported as doctor-tls-invalid, "certificate and key in storage are not a usable pair". replace-cert fixes it. On an appliance that holds no store yet, ze doctor does not report the pair at all: it prints doctor-storage-unavailable as a WARNING for the folder and one more for each feature that needs the store (web TLS pair: refused, this doctor run resolved no storage), and exits 0. A store it cannot read is an ERROR.

Config layering

Set config-base in appliance.json to share a base config across appliances:

{
  "config-base": "../_shared/ze.conf"
}

The base config is read first, then per-appliance ze.conf is appended. Later set commands override earlier ones; delete commands remove settings from the base.

Commands reference

Command Purpose
init <name> Create appliance with config + secrets (encrypted when a passphrase is set)
assemble [--keep] <name> Build ZeFS database only (auto-deletes; use --keep to retain)
build <name> Full image: assemble + gok + ext4 inject + checksum + manifest
build --all Build all appliances
kernel [--target] [--arch] [--profile] [--builder] [--version] [<name>] Download or build an installer or runtime kernel; with <name>, reads arch from appliance config and its profile for the installer target
initrd Download or build the installer initrd
iso [--image] [--output] [--kernel] [--initrd] [--target] [--builder] [<name>] Bootable installer ISO from an existing image
iso --check Check ISO prerequisites without building
passwd <name> Change SSH password
replace-cert <name> Replace TLS cert (reissue from the appliance CA, or --cert/--key for material from another CA); refuses material that is not a valid pair
rekey <name> Change encryption passphrase
clone <src> <dst> Copy config, not secrets
list List appliances with hostname and arch
show <name> Show config, cert expiry, managed status
run <name> Boot in QEMU with port forwarding
unlock Start passphrase agent
push [--image] [--testboot] [--no-reboot] <name> Push image to device via gokrazy OTA update
push --all [--parallel N] Push to all appliances with device.address
config <name> --merged Show effective config (base + overlay)
config-push --dry-run <name> Print the merged configuration without contacting the device
config-push <name> / --all [--parallel N] Validate and apply the merged configuration through authenticated SSH
init --batch <manifest> Batch init from JSON manifest
export <name> Export appliance to encrypted archive (.ze.enc)
export --all Export all appliances to single encrypted archive
import [--force] [--dir <path>] <archive> Import appliance from encrypted archive

ISO prerequisites

The ISO build requires an installer kernel, an initrd, grub-mkstandalone, its EFI modules for the target architecture, and xorriso. Use ./ze-host appliance iso --check lab to check the named appliance's target; omitting the name checks the host architecture. An installed GRUB executable alone is insufficient. Debian and Ubuntu supply the modules in grub-efi-amd64-bin or grub-efi-arm64-bin, according to the target.

./ze-host appliance iso cannot build an ISO on a stock macOS host. Homebrew has no GRUB package with arm64-efi modules. Its x86_64-elf-grub builds the x86_64-efi platform, but installs the modules under lib/x86_64-elf/grub, which the ISO build does not search: it looks for <target>/modinfo.sh under lib/grub, lib/grub2, lib64/grub, lib64/grub2 or share/grub, below the prefix of the builder, /usr, /usr/local and /opt/homebrew. Build the ISO on a Linux host, or pass --builder a grub-mkstandalone from a GRUB installation that has the target's modules in one of those places.

The kernel and initrd commands handle downloading or building these artifacts automatically:

./ze-host appliance iso --check lab
./ze-host appliance kernel lab
./ze-host appliance kernel --profile hardware lab
./ze-host appliance kernel --builder qemu --arch arm64 lab    # a build needs an arm64 host
./ze-host appliance kernel --target runtime
./ze-host appliance initrd
./ze-host appliance iso lab

A custom --builder path must exist and be executable. --check applies the same executable check as a build, even when matching EFI modules are present. If that check fails, use the printed repair to select a complete GRUB installation for the same appliance and directory.

The installer searches optical media as well as removable block media for its matching ISO. Optical devices remain excluded from installation targets. On completion it powers off; remove the installer media before booting the target disk.

./ze-host appliance kernel defaults to the installer target. --target runtime builds the gokrazy runtime kernel tree and enforces the runtime requirement floor. The installer target tries the cache, then an optional configured artifact URL, then the local Go driver. The driver selects Docker when available and otherwise QEMU; --builder docker and --builder qemu force one backend. Resolved artifacts are cached under $XDG_CACHE_HOME/ze/.

./ze-host appliance initrd uses the same cache, optional configured URL, then local build pattern for the initrd artifact. The download URL has no built-in release server default; set ze.appliance.kernel.url or ze.appliance.initrd.url to use prebuilt artifacts.

./ze-host doctor includes checks for kernel, initrd, grub, xorriso, and e2fsprogs availability, reporting warnings with actionable hints when prerequisites are missing.

ISO installer media

Create an installer ISO from an image already produced by ./ze-host appliance build. By default the command selects the latest ze-*.img in the appliance directory, verifies its .sha256 sidecar, and writes ze-*.iso next to the image. Use --image to select a specific image filename and --output to write the ISO elsewhere. The output path must not overwrite the selected .img, and the image filename must stay within [A-Za-z0-9._-] so the initrd can pass it on the kernel command line. By default, ./ze-host appliance iso resolves a matching installer kernel from the cache, or from build/kernel/Image only when its variant metadata matches the appliance architecture, profile, and version. Pass --kernel to select a specific installer kernel.

The kernel cache key includes target, architecture, profile, config, and version. Repeating the same request reuses the verified artifact; a different request gets a separate cache entry.

./ze-host appliance build lab
./ze-host appliance kernel --builder docker --profile hardware lab
./ze-host appliance iso lab
./ze-host appliance iso --image ze-20260601-120000.img lab
./ze-host appliance iso --output /path/to/lab.iso lab
./ze-host appliance iso --kernel build/kernel/Image lab

The ISO is an installer envelope around the existing raw gokrazy image. The image is gzip-compressed inside the ISO to reduce media size (a 2 GiB image with ~100 MiB of content compresses to roughly 100 MiB). The installer initrd decompresses the image during installation. The ISO does not rebuild the appliance, regenerate credentials, fetch a separate ZeFS database, or mutate /perm after writing the disk image. The installed disk receives the selected image bytes, including the /perm/ze/database.zefs and /perm/ze/build.json manifest that build already injected.

The ISO boot path accepts an optional explicit target disk. If no target is set, the installer writes only when exactly one non-removable candidate disk remains after excluding the ISO source media. The initrd also matches the booted ISO by a builder-generated ze.media-id token before it trusts a mounted installer volume, so identical image filenames on multiple attached installer media do not confuse the source selection. With multiple fixed disks, pass a whole disk path such as /dev/vda at ISO creation time:

./ze-host appliance iso --target /dev/vda lab

After the installer writes the disk in ISO mode, it powers off instead of rebooting. Remove the installer media, then power the target back on so the firmware boots from the written disk.

The ISO contains the full provisioned appliance image, including the embedded ZeFS database. Handle the ISO with the same care as the .img file.

USB write method: the ISO can be written with dd, Etcher, or Rufus in DD mode. Ventoy is also supported when the installer kernel includes loop device and FAT/exFAT filesystem support (the hardware kernel profile has this). The initrd detects the ISO file on the Ventoy data partition, loop-mounts it, and proceeds with the installation. When using the qemu kernel profile, Ventoy is not supported.

Remote operations (push, config-push)

Push a built image to a running gokrazy device via its HTTPS update endpoint:

./ze-host appliance push lab
./ze-host appliance push --image ze-20260427-143022.img lab
./ze-host appliance push --all
./ze-host appliance push --all --parallel 4

Push uses the update token (from secrets/update.token) for HTTP basic auth, and verifies the device TLS certificate against the stored cert.pem. That file holds the device certificate and the appliance CA root that issued it, so the anchor is the ISSUER: a certificate reissued with ./ze-host appliance replace-cert still verifies, and the device does not have to be pushed again first. No system CA pool is consulted.

When --image is set, the file name must resolve to a regular file inside the appliance directory. Path traversal and symlinks escaping that directory are rejected after the local update token is read, but before TLS setup or any network request starts.

Preview the effective configuration (base + overlay merged) without building:

./ze-host appliance config lab --merged

Inspect the merged configuration without contacting the device:

./ze-host appliance config-push --dry-run lab

Apply the merged replacement to one device or all addressed devices:

./ze-host appliance config-push lab
./ze-host appliance config-push --all --parallel 4

The device address, SSH port, and username come from the named appliance configuration. The command authenticates through the operator's ssh-agent and verifies the device against ~/.ssh/known_hosts. Verify the device fingerprint through a trusted channel before adding its key. A missing or changed key refuses the connection.

The authenticated account needs configuration commit permission. The daemon receives the complete replacement, validates it, and applies it through its normal configuration transaction. A successful push updates both the running configuration and its selected persistent source. The replacement must retain the device's credentials and required management listeners. Configuration text can contain secrets, so protect merged output and any retained copy.

A push succeeds only after the daemon returns its acceptance receipt and a zero SSH exit status. If the connection ends without that receipt, the outcome is uncertain. Inspect the live configuration through a fresh authenticated connection before retrying. The error prints the SSH command for that check.

Developers can run ./le test qemu appliance-config-push to check this command against a newly built native appliance. It uses an owned SSH agent and a host key verified from the stopped disk, checks live publication and retained password/key login after reboot, and checks malformed-input, permission and host-pin refusals. It requires native KVM or HVF and builds its own image; unset ZE_INSTALL_IMAGE. Use ZE_INSTALL_KEEP=1 to retain its disk and serial logs. This action proves configuration publication on a raw image; use the installer actions for ISO installation evidence.

Device-side config behavior

At boot, unmanaged devices resolve the stored configuration through its active version pointer. The legacy active entry is a fallback only when that pointer is absent. An unresolved pointer fails rather than selecting the legacy entry. If the legacy entry is missing, startup creates one from the seed template and interface discovery, or from discovery alone. The stored instance name selects the configuration name, with ze.conf as the default.

The separate boot inbox /perm/ze/config-pushed.conf validates its contents and writes the legacy active entry. Invalid contents cause deletion of that inbox file. The startup hash records this legacy entry in meta/config/active-hash inside the live store. It does not establish the current canonical version. Use authenticated show config running to inspect the running configuration.

Last-known-good hash: at build time, ./ze-host appliance build writes the SHA-256 of the assembled seed config to meta/config/last-known-good in the seed blob. First boot imports this build-time integrity baseline into the live tree.

Boot-time pushed-file monitor: When a pushed config file is consumed at boot, the monitor watches BGP peer close events for 30 seconds. A close event writes the previous config back into the legacy active entry, falling back to the seed if necessary. This store write neither promotes the active pointer nor reloads the running daemon. Completing the window records that entry's hash in meta/config/last-known-good-pushed. Live config-push has no automatic health-window rollback.

Batch init

Initialize multiple appliances from a JSON manifest:

./ze-host appliance init --batch manifest.json

Manifest format (array of entries):

[
  {"name": "edge-01", "hostname": "edge-01.lab", "password": "secret1", "device.address": "10.0.0.1"},
  {"name": "edge-02", "hostname": "edge-02.lab", "password": "generate"}
]

Use "password": "generate" for per-device random passwords (printed to stdout once, never stored in plaintext). When an encryption passphrase is set, each encrypted secret write receives a fresh random salt and nonce.

Disaster recovery (export/import)

Export creates an encrypted archive of an appliance directory for offsite backup or bastion migration. Archives include config, secrets, and build metadata, but exclude images and ZeFS databases (both are rebuildable).

Export a single appliance:

./ze-host appliance export lab
# creates lab.ze.enc in the current directory

Export all appliances:

./ze-host appliance export --all
# creates appliances-YYYYMMDD-HHMMSS.ze.enc

Import restores from an archive:

./ze-host appliance import lab.ze.enc

Import to a different bastion (migration):

./ze-host appliance import lab.ze.enc --dir /path/to/new/bastion

Archives are always encrypted using the same Argon2id + XChaCha20-Poly1305 scheme as secrets at rest. The archive passphrase can differ when provided separately from the secrets passphrase. import --force overwrites files present in the archive, but it does not delete extra files already present in existing appliance directories.