VM Appliance
Build a bootable VM image with Ze baked in using gokrazy. The default target is x86_64, and image.arch in appliance.json selects another architecture for ./ze-host appliance build. At runtime, the appliance is minimal: Linux kernel, gokrazy init, and Ze as the only application, with no package manager, no general shell (except authenticated emergency serial console), no unused distro daemons, and automatic process supervision.
Suitable for N100-class mini PCs, Proxmox VMs, or QEMU testing.
Prepare the build-host command
Work from the Ze source checkout with Go and Git installed. Build the native host driver and select the patched Go environment before the image commands:
./le build host-driver
GOROOT="$(./le go stdpatch goroot)"
export GOROOT GOTOOLCHAIN=local PATH="$GOROOT/bin:$PATH"
The examples below use ./ze-host, which runs on the build machine and carries
the appliance and PXE provisioning commands. The ordinary ./ze launcher builds
the Linux daemon; it does not carry appliance. Build kernels and cold-cache
images on a host matching the image architecture. Binaries can be cross-built,
but selecting another image.arch does not make a foreign kernel build possible.
What's in the image
| Component | Purpose |
|---|---|
| Linux kernel | Boot and hardware drivers |
| gokrazy init | Starts Ze, supervises it, seeds entropy, sends watchdog heartbeat |
| Ze | BGP daemon with DHCP client and all internal plugins |
| ze-serial-shell | Authenticated emergency shell on serial console (login required) |
Ze owns network configuration in the appliance. The gokrazy default DHCP and
NTP packages are excluded from the image; the shipped Ze seed template enables
interface DHCP auto-discovery with set interface dhcp-auto true and leaves Ze
NTP disabled (set environment ntp enabled false) until the operator enables it
in Ze config.
The root filesystem is read-only (SquashFS). Persistent data lives on a separate ext4 partition mounted at /perm.
Prerequisites
Install once on the build machine.
macOS:
brew install e2fsprogs # ext4 filesystem tools
brew install qemu # VM runtime (testing only)
Linux:
sudo apt-get install -y e2fsprogs qemu-system-x86 # Debian, Ubuntu
sudo dnf install -y e2fsprogs qemu-system-x86 # Fedora
The build needs mkfs.ext4 and debugfs from e2fsprogs. If e2fsck is
available, the builder also checks ext4 structure; otherwise it warns and skips
that check. The native
builder resolves each tool independently from the Homebrew keg, standard sbin
directories, and PATH.
For appliance ISO creation, install grub-mkstandalone (or grub2-mkstandalone)
with the GRUB EFI modules for the target (x86_64-efi or arm64-efi), plus
xorriso. ./ze-host appliance iso checks those tools and modules before it
stages an ISO. On macOS, build the ISO on a Linux host: see
ISO prerequisites.
The vendored gokrazy command (github.com/gokrazy/tools/gok) runs in process.
./ze-host appliance build calls it directly. A developer runs it with
./le build gokrazy <gok args>, which uses the checked-in module cache, forbids
network module lookups, and builds overwrite from a prepared copy of the
instance under tmp/. No separate gokrazy installation or
first-party script is required.
The image's ze compiles only against Ze's patched standard library, so the build
runs under the patched GOROOT. ./le build gokrazy inherits it from le.
./ze-host appliance build refuses unless its environment already selects it: GOROOT
set to the tree, GOTOOLCHAIN=local, and $GOROOT/bin/go first on PATH. The
refusal ends with the command line that sets all three:
GOROOT="$(./le go stdpatch goroot)" && export GOROOT GOTOOLCHAIN=local PATH="$GOROOT/bin:$PATH" && ./ze-host appliance build <name>
First-time setup
Initialize the named appliance before its first build:
./ze-host appliance init edge-01
The build resolves pinned system packages through the Go module graph.
Runtime Kernel Requirements
The runtime kernel is pinned by two manifests, gokrazy/kernel/kernel.require
and gokrazy/kernel/runtime.require. Each is
checked against the resolved config after the build, and enforce_required_symbols
accepts =y alone. A Kconfig answer of m therefore fails the BUILD rather than
shipping an appliance where a feature Ze accepts in config cannot work. Runtime
images also package the matching module tree; a module from a stock distribution
is not a substitute for that kernel's modules.
Each symbol has a producer in Ze rather than a test that wanted it.
| Group | Symbols | Producer in Ze |
|---|---|---|
| Subscriber | PPP, PPPOL2TP, PPPOE, L2TP, L2TP_V3 |
The L2TP LNS, and the PPPoE server and client |
| Firewall | NF_TABLES, NF_TABLES_INET, NF_TABLES_IPV4, NF_TABLES_IPV6, NF_CONNTRACK, NF_NAT, NFT_CT, NFT_NAT, NFT_MASQ, NFT_REDIR, NFT_LIMIT, NFT_LOG, NFT_REJECT, NF_FLOW_TABLE, NF_FLOW_TABLE_INET, NFT_FLOW_OFFLOAD |
The nftables backend, and translatePolicy, which returns FamilyInet unconditionally. Without NF_TABLES_INET the kernel answers EOPNOTSUPP, Apply's flush fails, the firewall plugin fails startup, and the daemon exits. The three flowtable symbols carry a table's flowtable and a term's flow-offload action (applyFlowtable); without them the kernel answers ENOENT and refuses the whole table. internal/plugins/copp/translate.go |
| Tunnels | NET_IPGRE_DEMUX, NET_IPGRE, IPV6_GRE, NET_IPIP, IPV6_TUNNEL, IPV6_SIT, VXLAN |
The nine tunnel kinds ze-iface-conf.yang models. Six symbols plus the GRE demux gate. Measured cost: vmlinuz 16352256 to 16450560 bytes, +0.60% |
| SRv6 | IPV6_SEG6_LWTUNNEL |
The kernel FIB backend's Service SID routes: buildSEG6Encap installs each RFC 9252 service route as a seg6 encap-mode lightweight tunnel. Without it the kernel answers EOPNOTSUPP and no SRv6 service route is installed. internal/plugins/fib/kernel/nexthop_linux.go |
| Policy routing | IP_MULTIPLE_TABLES, IPV6_MULTIPLE_TABLES, IP_ROUTE_MULTIPATH |
The policy-routing engine. Without IPV6_MULTIPLE_TABLES the kernel folds every table id into the main table and says nothing |
| Traffic control | NET_SCH_HTB, TBF, FQ_CODEL, HFSC, FQ, SFQ, NETEM, PRIO, INGRESS, NET_CLS_U32, NET_CLS_FW, NET_CLS_MATCHALL, NET_ACT_MIRRED, NET_ACT_POLICE |
Class of service, rate limiting, and mirroring; subscriber upload rates use the police action through shaper.applyTC and ingressPolicerFilter |
| Interfaces and VPN | DUMMY, VETH, MACVLAN, VLAN_8021Q, INET_ESP, INET6_ESP, XFRM_STATISTICS, HUGETLBFS, BPF_SYSCALL, BPF_JIT |
Interface creation, IPsec, VPP hugepages, and the eBPF surfaces |
| IPsec and WireGuard | XFRM_USER, XFRM_INTERFACE, XFRM_MIGRATE, INET_AH, INET6_AH, WIREGUARD, CRYPTO_CBC, CRYPTO_DES, CRYPTO_NULL, CRYPTO_GCM, CRYPTO_CHACHA20POLY1305, CRYPTO_SHA1, CRYPTO_SHA256, CRYPTO_SHA512 |
Every SA install (XFRM netlink), route-based IPsec, MOBIKE, OSPFv3 authentication, WireGuard links, and one symbol for each ESP transform Ze can install, so every cipher Ze accepts in config is carried by the kernel |
CONFIG_NET_UDP_TUNNEL and CONFIG_TUN are =y in the config fragment and
are not pinned in a manifest.
VPN Kernel Proof
./le test qemu vpn-kernel boots Ze's docker-host image with its own
kernel and base, on the host's native architecture. It uses the shared
Docker lab transport: KVM on Linux or HVF on macOS, never a stock guest or
foreign-architecture emulation. This is a manual kernel proof, not a check
of the default appliance image's packaged config.
The action runs AES-GCM, AES-CBC/SHA256, ChaCha20-Poly1305, 3DES/SHA1 and NULL/SHA256 against strongSwan. The IPsec checks require installed SAs and received tunnel traffic. A separate pair of Linux WireGuard peers must complete a handshake and pass a ping, with received bytes at both ends.
Before services start, the disposable guest disables module loading,
refuses any preloaded module, and removes its /sbin/modprobe helper.
The shipped image and any existing Docker host are unchanged. The host
checks the emitted kernel config against both requirement manifests;
the WireGuard step checks the guest root again, not the container root.
The WireGuard step proves kernel traffic, not Ze's configuration-commit path. An amd64 run does not replace the required arm64/HVF run, and no result is implied by the command's presence.
L2TP Kernel Support
Ze's L2TP LNS path needs kernel PPPoL2TP support in the appliance runtime
kernel: CONFIG_PPP, CONFIG_PPPOL2TP, CONFIG_L2TP, and
CONFIG_L2TP_V3. The shared runtime proof kernel also keeps CONFIG_PPPOE
built in for PPPoE evidence. The appliance image boots this kernel, so every
image carries these options. To fill the cache before a build, on an amd64
host:
./ze-host appliance kernel --target runtime --arch amd64
A kernel is built only on a host of its own architecture, so the arm64 kernel is built on an arm64 host (a Mac, for example). Select the QEMU builder there when Docker is absent:
./ze-host appliance kernel --target runtime --arch arm64 --builder qemu
On a host of the other architecture both commands are refused before anything
starts. To build an image for another architecture, build it on a host of that
architecture, or set image.arch to the host's in the appliance's config file;
the refusal names that file and ends with the build command to run.
./le build gokrazy with GOARCH naming another architecture is refused the
same way, and ends with the same command run with GOARCH set to the host's.
This is the kernel every appliance image boots, and no other kernel can reach
an image. ./ze-host appliance build and ./le build gokrazy resolve it for the
image's architecture themselves: from the cache when an entry is there, else by
building it, so these commands only fill the cache ahead of a build. A cold
cache also builds the kernel before the image. It needs network access to
kernel.org and Docker or QEMU on a host of the image's architecture.
image.kernel-profile selects the installer kernel. Appliance images use the
shared runtime profile for their architecture. The installer qemu fragment
requests virtio drivers and is merged onto Linux defconfig, which can enable
additional drivers. The emitted kernel configuration records the complete
driver selection.
./ze-host appliance kernel --target runtime calls the Go driver in
internal/appliance/kernelbuilder. The driver reads
internal/appliance/kernel.version, selects Docker or QEMU, resolves the
tracked config fragments and .require manifests, and writes the runtime
kernel tree. The Go worker enforces every required symbol before the artifact
can be used.
On a Linux runner with QEMU, xl2tpd, pppd, /dev/ppp, and PPPoL2TP kernel
support, the deployment proof target builds an L2TP-enabled appliance image and
drives a real LAC against it:
./le test deployment gokrazy-l2tp-ppp
The proof image is built from a temporary gokrazy instance config so the normal
appliance config is left unchanged. It disables IPv6CP in that proof image
because the current static L2TP pool is IPv4-only. Set
ZE_GOKRAZY_SKIP_BUILD=1 to run against an existing tmp/gokrazy/ze.img that
was already built with the L2TP proof template, the proof runtime environment,
and an L2TP-capable kernel: skip-build bypasses the proof's check of the
runtime kernel cache, and an image whose kernel lacks PPPoL2TP crash-loops at
first boot instead of serving.
Build an image
Build the appliance initialized above. If starting here, first run
./ze-host appliance init edge-01 once:
./ze-host appliance build edge-01
The named appliance keeps its config and secrets between builds. The build
assembles its ZeFS seed artifact, builds the disk image through vendored gokrazy,
formats the persistent partition, injects database.zefs, and writes the build
manifest. Seed assembly uses the explicit blob-artifact storage API, so image
building continues to produce the same blob format.
./ze-host appliance show edge-01
./ze-host appliance config edge-01 --merged
Test in QEMU
./ze-host appliance run edge-01
The command boots the named image in QEMU. Quit QEMU with Ctrl-A X.
The Gokrazy management UI is exposed through Ze's authenticated web UI at
/gokrazy/; the proxy reads Gokrazy's password from its standard password-file
locations.
Deploy to hardware
Write the image to a USB drive or internal disk on your N100 machine:
Use the exact .img path printed by the named build's image ready: line.
Named builds write into the selected appliance directory, normally
~/.config/ze/appliances/edge-01/; tmp/gokrazy/ze.img belongs to other build
and proof workflows. Set ZE_IMAGE to the named image you just built, verify
its accompanying .sha256 file, and identify the target device before writing.
ZE_IMAGE='/absolute/path/from-image-ready/ze-YYYYMMDD-HHMMSS.img'
# Linux
sudo dd if="$ZE_IMAGE" of=/dev/sdX bs=4M status=progress
# macOS
sudo dd if="$ZE_IMAGE" of=/dev/rdiskN bs=4m
Or import into Proxmox:
qm importdisk <vmid> "$ZE_IMAGE" <storage>
The machine boots to a serial console (115200 baud). On first boot Ze explicitly
imports /perm/ze/database.zefs into /perm/ze/database/, logs the import, and
retires the seed as database.zefs.replaced-*. Subsequent starts use the live
tree. Ze gets a DHCP address and loads its active configuration from that tree.
The serial console requires authentication with the local admin credentials
before granting shell access. If the store is missing or unreadable, or the
credentials cannot be read, the login is refused with the error named. When
admin-enabled: false is set in the appliance config, the serial console denies
the built-in admin (fail-closed) and prints "local admin login disabled".
Configuration
Seed config
The initial Ze config is stored as the seed template in
gokrazy/ze/ze.conf. ./ze-host appliance assemble uses it when no base or
per-appliance overlay exists. The seed database holds no active config that can
shadow the template, so the appliance builds its effective config on first
boot.
set environment log level info
set environment web enabled true
set environment web server default ip 0.0.0.0
set environment web server default port 8080
set environment ssh enabled true
set environment ssh server default ip 0.0.0.0
set environment ssh server default port 22
set environment ntp enabled false
set interface dhcp-auto true
To change the seed config, edit gokrazy/ze/ze.conf, or use the structured
workflow's config-base and per-appliance ze.conf files.
Runtime config
Once booted, use ze config edit over SSH to modify the running configuration.
Changes are stored in /perm/ze/database/ and persist across reboots and image
updates. The daemon owns the store for its lifetime; offline maintenance must
run as the store owner after the daemon stops.
Environment variables
Ze's environment is set in gokrazy/ze/config.json under PackageConfig:
| Variable | Value | Purpose |
|---|---|---|
ze.config.dir |
/perm/ze |
Configuration folder containing the live database/ tree |
ze.bgp.api.socketpath |
/tmp/ze.socket |
API socket location |
ze.bgp.daemon.drop |
false |
✕ |
ze.log |
info |
Log level |
ze.log.backend |
kmsg,stderr |
Logs go to kmsg and gokrazy ring buffers |
ze.gokrazy.enabled |
true |
Enables appliance auto-init fallback and the. /gokrazy/ |
Kernel crash capture
An appliance that suffers a kernel panic reboots and tells you nothing. Root is read-only SquashFS, the image holds no busybox and no shell, and the kernel that faulted is gone. Crash capture asks the kernel to write its own panic message and backtrace into a reserved memory region before it goes down, so the evidence survives the warm reboot.
Two halves, and both are needed:
| Half | Where | What it does |
|---|---|---|
| The reservation | image.crash-dump.reserve in the appliance config |
Renders reserve_mem=<N>M:4096:zecrash and ramoops.mem_name=zecrash onto the built image's kernel command line |
| The intent | system crash-dump enabled true in the Ze config |
Tells the running daemon to harvest the record at boot and store it as a crash report |
"image": {"arch": "amd64", "size-bytes": 2147483648, "crash-dump": {"reserve": "16mb"}}
The reservation is a boot argument, so it takes effect at the next boot of an
image built with it, never at a config commit. show crashes reports the two
states separately as configured and armed, and ze doctor raises
doctor-crash-capture-unarmed while they disagree.
The reserve is 4mb to 256mb and is taken from RAM on every boot, so it is sized for a backtrace: 16mb is the default and holds a full one. The build refuses a value outside that range, and refuses a size that is not a whole number of megabytes, because the appliance has no shell to diagnose a silently ineffective reservation with.
The region is named rather than addressed. Size-named reservation has been in the
kernel since 6.12 and internal/appliance/kernel.version pins a kernel above
that floor, so no per-machine physical address is needed. The runtime kernel
carries CONFIG_PSTORE and CONFIG_PSTORE_RAM, and the build fails if either
stops resolving to =y: without them the kernel would accept the reservation,
take the RAM, and expose nothing to read the record back from.
Harvested records land in the crash directory, which on an appliance is
/perm/ze/crash. Gokrazy's A/B updates replace the root partition and leave
/perm, so a record survives the update that follows a crash. Records share the
ze.crash.keep retention count with Go panic reports, so a panic loop cannot
fill /perm.
Updating
Push a rebuilt image to the running appliance over authenticated HTTPS:
./ze-host appliance build <name>
./ze-host appliance push --testboot <name>
Push validates the disk image, sends its root filesystem to the inactive root
slot, updates the shared boot filesystem and boot code, then reboots. It keeps
the device's disk identities and persistent configuration. --testboot asks
gokrazy for a temporary boot of the new root, with the previous root selected
for a subsequent reboot. It does not verify routing health. Use --no-reboot
to schedule the restart separately.
The two root slots protect the previous root filesystem. The boot filesystem is shared, so keep installer media and a configuration/secret backup for an interrupted boot write. A partition-layout change requires reinstalling the disk image rather than an OTA push.
Before the update, record the running version and archive the active configuration using the lifecycle guide. Keep the named appliance's secrets and the previous image. After reboot, query the appliance through its SSH management endpoint, using its provisioned address, SSH port and credentials:
ssh -p <ssh-port> <username>@<device-address> "show version --extended"
ssh -p <ssh-port> <username>@<device-address> "show doctor"
Compare the returned version with the selected image, then verify the hostname, committed configuration, peers and routes. A completed upload alone does not establish that the new runtime is reachable or routing correctly.
For a --testboot update, a subsequent reboot selects the previous root slot.
Check the version and retained configuration again after recovery. Ordinary
push selects the new root persistently. Both modes leave the persistent store
in place; a root-slot recovery does not undo configuration edits. The
installation guide states the architecture
and deployment checks completed so far.
Removing an appliance
Before retiring an appliance, back up its live store using the Operations guide and keep the backup outside the disk you will remove. Also export the named appliance directory to retain its build configuration and secrets. Appliance exports exclude disk images and ZeFS databases, so keep the live-store backup separately.
Use the graceful stop procedure, then power off the VM or physical device. Wait for shutdown to finish before removing its storage.
For a VM, remove its definition through the hypervisor and explicitly choose
whether to retain or delete the associated disks. The QEMU command uses the
selected .img file as its writable disk; retaining that file keeps the device's
stored state. For dedicated hardware, boot the replacement system's installer
and identify the appliance disk before overwriting it. Follow the storage
device's erasure procedure before disposing of it or transferring it to another
owner.
Retained disks, appliance directories, seeded images and installer ISOs contain configuration and authentication material. Protect any copies kept for recovery, and revoke credentials that should no longer grant access after retirement.
Architecture notes
Internal plugins only
Gokrazy has no shell and no PATH. Ze's external plugin mechanism (which uses /bin/sh -c to fork processes) does not work. All Ze plugins (bgp-rib, bgp-gr, bgp-adj-rib-in, etc.) are compiled into the ze binary as internal plugins and run as goroutines. This is the default and covers all standard BGP functionality.
A config that names an external plugin anyway is reported: ze doctor raises doctor-plugin-shell-missing, naming the absent shell and the plugins that depend on it, and a start that reaches the fork names the shell rather than the plugin.
Process supervision
Gokrazy's init restarts Ze if it exits with a non-zero status (except 125, which means "don't restart"). Ze handles SIGTERM for graceful shutdown. Logs (stdout/stderr) are captured in ring buffers visible through the gokrazy web UI.
Persistent storage
The /perm partition (ext4) survives image updates. The configuration folder
/perm/ze, selected by ze.config.dir, contains the live database/ tree.
Credentials, TLS material and config history are keys in that tree, with 0700
directories and 0600 frame files owned by the daemon user. The retired seed
remains beside it for recovery.
Repo layout
gokrazy/
.gitignore # excludes *.img
ze/
config.json # gokrazy instance config (what to build, how to start)
builddir/
github.com/ze-software/ze/
go.mod # ze dependency pins + relative replace directive
go.sum
ze.invalid/kernel/
go.mod # requires ze's runtime kernel, replaced per build
github.com/gokrazy/gokrazy/
go.mod, go.sum # gokrazy init system version pin
cmd/dhcp/ # DHCP client
cmd/ntp/ # NTP client
cmd/heartbeat/ # watchdog heartbeat
cmd/randomd/ # entropy seeder
cmd/ze-serial-shell/ # serial console login gate (replaces serial-busybox)
main.go # gokrazy wrapper: symlink + DontStartOnBoot
_gokrazy/ # renamed busybox extrafiles per arch
internal/le/build/gokrazy/ # `le build gokrazy`: the vendored gokrazy command wrapper
The gok source is vendored under vendor/github.com/gokrazy/. The small
builddir/ modules pin the system-package versions.
Builds never run from this directory
The tree above is the build input; no build runs inside it. Every image build
first copies the instance it names (gokrazy/ze for the appliance,
gokrazy/docker-host for ./le setup docker host build) to a fresh directory
under the project tmp/, including the whole builddir, rewriting each filesystem-path replace to an
absolute path so it still resolves from the new depth. gok is then pointed at the
copy, and the copy is deleted afterwards.
./ze-host appliance build prepares an isolated copy through
resolveBuildParentDir. A build leaves the working tree unchanged, and two
builds in one checkout use separate prepared instances.
The image boots ze's own runtime kernel and no other. config.json names
ze.invalid/kernel as its KernelPackage, a path under a reserved domain that
never resolves on its own. Each build resolves the runtime kernel for the image's
architecture (the cache entry, else a native build), assembles a gokrazy kernel
package from that cache entry inside the prepared copy, and replaces the kernel
module's require to point at it. The package carries the kernel, its modules and
a cmdline.txt naming root=/dev/sda2, which gokrazy rewrites to the image's
root partition UUID before it appends the instance's extra kernel arguments. It
also carries an empty config.txt, the Raspberry Pi bootloader file gokrazy
reads from every kernel package and no ze target uses. Preparation with no resolved kernel is refused,
so gok never fetches a kernel. No setting selects another kernel package.
Build-host command
Appliance build commands (./ze-host appliance) and PXE provisioning
(./ze-host install remote) use the native driver prepared at the beginning
of this guide. Rebuild it after changing the source:
./le build host-driver
Building and installing an appliance (end to end)
From a JSON config (recommended)
The running appliance's kernel host name comes from system host in its Ze
config, which Ze applies at start and on every reload. identity.hostname in
appliance.json names the image's CA certificate and listings, and the image
boots as ze until Ze applies system host; with system host unset it stays
ze (see docs/guide/configuration.md, "System Host Name").
Write an appliance config file (arch, kernel profile, credentials, networking):
{
"credentials": { "username": "exa", "admin-enabled": true },
"ssh": { "host": "0.0.0.0", "port": "2222" },
"web": { "enabled": true, "host": "0.0.0.0", "port": "8080" },
"tls": { "cert-name": "router.local", "validity-years": 10 },
"identity": { "hostname": "ze-prod" },
"device": { "address": "10.12.104.10", "update-port": 443 },
"image": { "arch": "amd64", "size-bytes": 2147483648, "kernel-profile": "hardware-kms" }
}
For SSH key login, add credentials.ssh-authorized-keys, an array of complete
plain OpenSSH public-key lines, before initializing the named appliance. These keys
log in as credentials.username; password recovery remains available. Supply
unrestricted recovery public keys: assembly refuses SSH certificates and
authorized_keys options such as from= or command= because Ze cannot enforce them.
Reserving hugepages for VPP
When the appliance runs VPP, reserve hugepages at boot by adding image.hugepages
to the config. ./ze-host appliance build bakes default_hugepagesz/hugepagesz/hugepages
into the boot cmdline (via a derived gokrazy instance config; the checked-in
gokrazy/ze/config.json is never modified). Declare image.memory so the build
rejects a reservation over 50% of target RAM and ./ze-host appliance run sizes QEMU's
-m to match. Sizes are byte-size strings (10b, 512mb, 1gb, 1tb;
case-insensitive, 1024-based):
"image": {
"arch": "amd64",
"size-bytes": 2147483648,
"memory": "8gb",
"hugepages": { "size": "1gb", "page-size": "2mb" }
}
hugepages.size is the total reservation and page-size is 2mb or 1gb; the
page count is size / page-size (so size must be a whole multiple of
page-size). The reservation is bounded to 512 GiB and, when memory is set, to
50% of it. 1gb pages need CPU pdpe1gb support and CONFIG_HUGETLBFS in the
kernel profile (both surfaced by ./ze-host doctor).
Build the full ISO through the structured appliance actions:
./ze-host appliance init --config prod.json prod
./ze-host appliance kernel prod
./ze-host appliance initrd
./ze-host appliance build prod
./ze-host appliance iso prod
The named appliance retains its config and secrets for subsequent builds.
Manual steps
Use the same commands individually when you need to inspect an intermediate artifact:
# Create the appliance with config and secrets.
./ze-host appliance init --config prod.json prod
# Check and prepare ISO prerequisites.
./ze-host appliance iso --check prod
./ze-host appliance kernel prod
./ze-host appliance initrd
# Build the disk image and installer ISO.
./ze-host appliance build prod
./ze-host appliance iso prod
# Provision the resulting image over the network when needed.
ZE_IMAGE='/absolute/path/from-image-ready/ze-YYYYMMDD-HHMMSS.img'
sudo ./ze-host install remote \
--interface eth0 \
--network 10.0.0.0/24 \
--image "$ZE_IMAGE" \
--ssh-username admin \
--ssh-password 'choose-a-strong-one'
The kernel and initrd commands first check the XDG cache. If
ze.appliance.kernel.url or ze.appliance.initrd.url is set, the matching
command then tries that configured prebuilt-artifact URL; otherwise it builds
locally. Kernel local builds use the shared Docker-or-QEMU builder selection, and
initrd local builds compile and pack cmd/ze-installer. Cached artifacts avoid
rebuilding their matching inputs. See "ISO prerequisites" below for details.
ze appliance (structured workflow)
The ./ze-host appliance command provides structured appliance management. Each
appliance has its own directory with a JSON config, secrets that are optionally
encrypted at rest, and a TLS certificate.
Quick start
./ze-host appliance init lab # interactive wizard
./ze-host appliance build lab # full image
./ze-host appliance kernel lab # installer kernel
./ze-host appliance initrd # installer initrd
./ze-host appliance iso lab # bootable installer ISO
./ze-host appliance list # show all appliances
./ze-host appliance show lab # config summary and cert expiry
Appliance directory
By default, appliances live in ~/.config/ze/appliances/. Override with --dir or ZE_APPLIANCE_DIR.
~/.config/ze/appliances/
_shared/
ze.conf # optional base config for all appliances
lab/
appliance.json # config (no credentials)
ze.conf # per-device config overrides
secrets/ # 0700 permissions
.encrypted # marker (present = secrets encrypted)
tls/
cert.pem # device certificate, then the appliance CA root (plaintext)
key.pem # private key (encrypted if passphrase set)
ca-cert.pem # appliance CA root certificate (plaintext)
ca-key.pem # appliance CA root key (encrypted if passphrase set)
password.hash # bcrypt hash (encrypted if passphrase set)
update.token # gokrazy OTA token (encrypted if passphrase set)
authorized_keys # SSH public keys (plaintext)
Encryption
Secrets are encrypted at rest with Argon2id + XChaCha20-Poly1305 when an encryption passphrase is set during ./ze-host appliance init. The passphrase is never stored on disk. For fleet operations, ./ze-host appliance unlock starts a passphrase agent (like ssh-agent) that holds the derived key in memory.
./ze-host appliance unlock # start agent
./ze-host appliance unlock --duration 15m # auto-expire after 15 minutes
./ze-host appliance unlock --stop # stop agent
Day-2 operations
./ze-host appliance passwd lab
./ze-host appliance replace-cert lab
./ze-host appliance replace-cert lab --cert ca.pem --key ca.key
./ze-host appliance rekey lab
./ze-host appliance clone lab lab2
Without --cert and --key, replace-cert issues a new device certificate
from the appliance's own certificate authority, which ./ze-host appliance init
generated. The root does not change, so a device you already pushed to stays
reachable: ./ze-host appliance push trusts the issuer rather than one certificate.
replace-cert validates the material before it writes anything. It refuses a
certificate and a key that are not a pair. It refuses a file that holds no PEM
data. It refuses a certificate that is past its not-after date, and the message
gives both validity dates. A certificate whose validity starts in the future is
accepted, because a staged renewal is copied into an image that boots later.
--cert and --key must be given together.
A refusal leaves cert.pem and key.pem byte-identical to what the appliance
already held. Both files are written through a temp file and a rename, so an
interrupted run leaves neither file truncated. When the key write fails after
the certificate write, the command puts the previous certificate back and
reports the restore. ./ze-host appliance init validates and writes the same way.
A ze older than this validation could store a certificate and a key that do
not load as a pair. On such an appliance the web listener does not start. Run
ze doctor to find it: the stored pair is reported as doctor-tls-invalid,
"certificate and key in storage are not a usable pair". replace-cert fixes it.
On an appliance that holds no store yet, ze doctor does not report the pair
at all: it prints doctor-storage-unavailable as a WARNING for the folder and
one more for each feature that needs the store (web TLS pair: refused, this doctor run resolved no storage), and exits 0. A store it cannot read is an
ERROR.
Config layering
Set config-base in appliance.json to share a base config across appliances:
{
"config-base": "../_shared/ze.conf"
}
The base config is read first, then per-appliance ze.conf is appended. Later set commands override earlier ones; delete commands remove settings from the base.
Commands reference
| Command | Purpose |
|---|---|
init <name> |
Create appliance with config + secrets (encrypted when a passphrase is set) |
assemble [--keep] <name> |
Build ZeFS database only (auto-deletes; use --keep to retain) |
build <name> |
Full image: assemble + gok + ext4 inject + checksum + manifest |
build --all |
Build all appliances |
kernel [--target] [--arch] [--profile] [--builder] [--version] [<name>] |
Download or build an installer or runtime kernel; with <name>, reads arch from appliance config and its profile for the installer target |
initrd |
Download or build the installer initrd |
iso [--image] [--output] [--kernel] [--initrd] [--target] [--builder] [<name>] |
Bootable installer ISO from an existing image |
iso --check |
Check ISO prerequisites without building |
passwd <name> |
Change SSH password |
replace-cert <name> |
Replace TLS cert (reissue from the appliance CA, or --cert/--key for material from another CA); refuses material that is not a valid pair |
rekey <name> |
Change encryption passphrase |
clone <src> <dst> |
Copy config, not secrets |
list |
List appliances with hostname and arch |
show <name> |
Show config, cert expiry, managed status |
run <name> |
Boot in QEMU with port forwarding |
unlock |
Start passphrase agent |
push [--image] [--testboot] [--no-reboot] <name> |
Push image to device via gokrazy OTA update |
push --all [--parallel N] |
Push to all appliances with device.address |
config <name> --merged |
Show effective config (base + overlay) |
config-push --dry-run <name> |
Print the merged configuration without contacting the device |
config-push <name> / --all [--parallel N] |
Validate and apply the merged configuration through authenticated SSH |
init --batch <manifest> |
Batch init from JSON manifest |
export <name> |
Export appliance to encrypted archive (.ze.enc) |
export --all |
Export all appliances to single encrypted archive |
import [--force] [--dir <path>] <archive> |
Import appliance from encrypted archive |
ISO prerequisites
The ISO build requires an installer kernel, an initrd, grub-mkstandalone, its
EFI modules for the target architecture, and xorriso. Use
./ze-host appliance iso --check lab to check the named appliance's target; omitting
the name checks the host architecture. An installed GRUB executable alone is
insufficient. Debian and Ubuntu supply the modules in grub-efi-amd64-bin
or grub-efi-arm64-bin, according to the target.
./ze-host appliance iso cannot build an ISO on a stock macOS host. Homebrew
has no GRUB package with arm64-efi modules. Its x86_64-elf-grub builds the
x86_64-efi platform, but installs the modules under lib/x86_64-elf/grub,
which the ISO build does not search: it looks for <target>/modinfo.sh under
lib/grub, lib/grub2, lib64/grub, lib64/grub2 or share/grub, below the
prefix of the builder, /usr, /usr/local and /opt/homebrew. Build the ISO
on a Linux host, or pass --builder a grub-mkstandalone from a GRUB
installation that has the target's modules in one of those places.
The kernel and initrd
commands handle downloading or building these
artifacts automatically:
./ze-host appliance iso --check lab
./ze-host appliance kernel lab
./ze-host appliance kernel --profile hardware lab
./ze-host appliance kernel --builder qemu --arch arm64 lab # a build needs an arm64 host
./ze-host appliance kernel --target runtime
./ze-host appliance initrd
./ze-host appliance iso lab
A custom --builder path must exist and be executable. --check applies the
same executable check as a build, even when matching EFI modules are present.
If that check fails, use the printed repair to select a complete GRUB
installation for the same appliance and directory.
The installer searches optical media as well as removable block media for its matching ISO. Optical devices remain excluded from installation targets. On completion it powers off; remove the installer media before booting the target disk.
./ze-host appliance kernel defaults to the installer target. --target runtime
builds the gokrazy runtime kernel tree and enforces the runtime requirement
floor. The installer target tries the cache, then an optional configured
artifact URL, then the local Go driver. The driver selects Docker when
available and otherwise QEMU; --builder docker and --builder qemu force one
backend. Resolved artifacts are cached under $XDG_CACHE_HOME/ze/.
./ze-host appliance initrd uses the same cache, optional configured URL, then local
build pattern for the initrd artifact. The download URL has no built-in release
server default; set ze.appliance.kernel.url or ze.appliance.initrd.url to use
prebuilt artifacts.
./ze-host doctor includes checks for kernel, initrd, grub, xorriso, and e2fsprogs
availability, reporting warnings with actionable hints when prerequisites are
missing.
ISO installer media
Create an installer ISO from an image already produced by ./ze-host appliance build. By default the command selects the latest ze-*.img in the appliance
directory, verifies its .sha256 sidecar, and writes ze-*.iso next to the
image. Use --image to select a specific image filename and --output to write
the ISO elsewhere. The output path must not overwrite the selected .img, and
the image filename must stay within [A-Za-z0-9._-] so the initrd can pass it
on the kernel command line. By default, ./ze-host appliance iso resolves a matching
installer kernel from the cache, or from build/kernel/Image only when its
variant metadata matches the appliance architecture, profile, and version.
Pass --kernel to select a specific installer kernel.
The kernel cache key includes target, architecture, profile, config, and version. Repeating the same request reuses the verified artifact; a different request gets a separate cache entry.
./ze-host appliance build lab
./ze-host appliance kernel --builder docker --profile hardware lab
./ze-host appliance iso lab
./ze-host appliance iso --image ze-20260601-120000.img lab
./ze-host appliance iso --output /path/to/lab.iso lab
./ze-host appliance iso --kernel build/kernel/Image lab
The ISO is an installer envelope around the existing raw gokrazy image. The image
is gzip-compressed inside the ISO to reduce media size (a 2 GiB image with ~100
MiB of content compresses to roughly 100 MiB). The installer initrd decompresses
the image during installation. The ISO does not rebuild the appliance, regenerate
credentials, fetch a separate ZeFS database, or mutate /perm after writing the
disk image. The installed disk receives the selected image bytes, including the
/perm/ze/database.zefs and /perm/ze/build.json manifest that build already
injected.
The ISO boot path accepts an optional explicit target disk. If no target is set,
the installer writes only when exactly one non-removable candidate disk remains
after excluding the ISO source media. The initrd also matches the booted ISO by
a builder-generated ze.media-id token before it trusts a mounted installer
volume, so identical image filenames on multiple attached installer media do not
confuse the source selection. With multiple fixed disks, pass a whole disk path
such as /dev/vda at ISO creation time:
./ze-host appliance iso --target /dev/vda lab
After the installer writes the disk in ISO mode, it powers off instead of rebooting. Remove the installer media, then power the target back on so the firmware boots from the written disk.
The ISO contains the full provisioned appliance image, including the embedded
ZeFS database. Handle the ISO with the same care as the .img file.
USB write method: the ISO can be written with dd, Etcher, or Rufus in DD
mode. Ventoy is also supported when the installer kernel includes loop device
and FAT/exFAT filesystem support (the hardware kernel profile has this). The
initrd detects the ISO file on the Ventoy data partition, loop-mounts it, and
proceeds with the installation. When using the qemu kernel profile, Ventoy
is not supported.
Remote operations (push, config-push)
Push a built image to a running gokrazy device via its HTTPS update endpoint:
./ze-host appliance push lab
./ze-host appliance push --image ze-20260427-143022.img lab
./ze-host appliance push --all
./ze-host appliance push --all --parallel 4
Push uses the update token (from secrets/update.token) for HTTP basic auth, and verifies the device TLS certificate against the stored cert.pem. That file holds the device certificate and the appliance CA root that issued it, so the anchor is the ISSUER: a certificate reissued with ./ze-host appliance replace-cert still verifies, and the device does not have to be pushed again first. No system CA pool is consulted.
When --image is set, the file name must resolve to a regular file inside the
appliance directory. Path traversal and symlinks escaping that directory are
rejected after the local update token is read, but before TLS setup or any
network request starts.
Preview the effective configuration (base + overlay merged) without building:
./ze-host appliance config lab --merged
Inspect the merged configuration without contacting the device:
./ze-host appliance config-push --dry-run lab
Apply the merged replacement to one device or all addressed devices:
./ze-host appliance config-push lab
./ze-host appliance config-push --all --parallel 4
The device address, SSH port, and username come from the named appliance configuration.
The command authenticates through the operator's ssh-agent and verifies the device
against ~/.ssh/known_hosts. Verify the device fingerprint through a trusted
channel before adding its key. A missing or changed key refuses the connection.
The authenticated account needs configuration commit permission. The daemon receives the complete replacement, validates it, and applies it through its normal configuration transaction. A successful push updates both the running configuration and its selected persistent source. The replacement must retain the device's credentials and required management listeners. Configuration text can contain secrets, so protect merged output and any retained copy.
A push succeeds only after the daemon returns its acceptance receipt and a zero SSH exit status. If the connection ends without that receipt, the outcome is uncertain. Inspect the live configuration through a fresh authenticated connection before retrying. The error prints the SSH command for that check.
Developers can run ./le test qemu appliance-config-push to check this command
against a newly built native appliance. It uses an owned SSH agent and a host
key verified from the stopped disk, checks live publication and retained
password/key login after reboot, and checks malformed-input, permission and
host-pin refusals. It requires native KVM or HVF and builds its own image;
unset ZE_INSTALL_IMAGE. Use ZE_INSTALL_KEEP=1 to retain its disk and serial
logs. This action proves configuration publication on a raw image; use the
installer actions for ISO installation evidence.
Device-side config behavior
At boot, unmanaged devices resolve the stored configuration through its active
version pointer. The legacy active entry is a fallback only when that pointer
is absent. An unresolved pointer fails rather than selecting the legacy entry.
If the legacy entry is missing, startup creates one from the seed template and
interface discovery, or from discovery alone. The stored instance name selects the
configuration name, with ze.conf as the default.
The separate boot inbox /perm/ze/config-pushed.conf validates its contents and
writes the legacy active entry. Invalid contents cause deletion of that inbox
file. The startup hash records this legacy entry in meta/config/active-hash
inside the live store. It does not establish the current canonical version.
Use authenticated show config running to inspect the running configuration.
Last-known-good hash: at build time, ./ze-host appliance build writes the SHA-256
of the assembled seed config to meta/config/last-known-good in the seed blob.
First boot imports this build-time integrity baseline into the live tree.
Boot-time pushed-file monitor: When a pushed config file is consumed at
boot, the monitor watches BGP peer close events for 30 seconds. A close event
writes the previous config back into the legacy active entry, falling back to
the seed if necessary. This store write neither promotes the active pointer nor
reloads the running daemon. Completing the window records that entry's hash in
meta/config/last-known-good-pushed. Live config-push has no automatic
health-window rollback.
Batch init
Initialize multiple appliances from a JSON manifest:
./ze-host appliance init --batch manifest.json
Manifest format (array of entries):
[
{"name": "edge-01", "hostname": "edge-01.lab", "password": "secret1", "device.address": "10.0.0.1"},
{"name": "edge-02", "hostname": "edge-02.lab", "password": "generate"}
]
Use "password": "generate" for per-device random passwords (printed to stdout once, never stored in plaintext). When an encryption passphrase is set, each encrypted secret write receives a fresh random salt and nonce.
Disaster recovery (export/import)
Export creates an encrypted archive of an appliance directory for offsite backup or bastion migration. Archives include config, secrets, and build metadata, but exclude images and ZeFS databases (both are rebuildable).
Export a single appliance:
./ze-host appliance export lab
# creates lab.ze.enc in the current directory
Export all appliances:
./ze-host appliance export --all
# creates appliances-YYYYMMDD-HHMMSS.ze.enc
Import restores from an archive:
./ze-host appliance import lab.ze.enc
Import to a different bastion (migration):
./ze-host appliance import lab.ze.enc --dir /path/to/new/bastion
Archives are always encrypted using the same Argon2id + XChaCha20-Poly1305 scheme as secrets at rest. The archive passphrase can differ when provided separately from the secrets passphrase. import --force overwrites files present in the archive, but it does not delete extra files already present in existing appliance directories.