Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Introduction

HexBox logo

nix-hex-box is a nix-darwin module that configures an Apple container machine backed aarch64-linux remote builder for Nix.

The module is designed for Darwin hosts that want to offload Linux derivations to a local virtualized builder while keeping the host configuration declarative.

Current design highlights:

  • installs Apple container from the official signed release package
  • pulls the published Alpine/Lix-based builder image by default, with an optional local custom image build
  • can optionally install Socktainer to expose a Docker-compatible local API socket
  • configures nix.buildMachines for ssh-ng://container-builder
  • manages durable state under ~/.local/state/hb
  • wakes the builder machine on demand through SSH ProxyCommand
  • keeps guest /nix persistent across machine stop/start cycles
  • supports guest-side idle shutdown and recovery-oriented health checks

This book documents the module itself. If you use nix-hex-box from another repo, that repo should only need a high-level integration guide.

Overview

nix-hex-box exports two identical module entry points:

  • darwinModules.default
  • darwinModules.container-builder

The main option namespace is services.container-builder.

When enabled, the module:

  • installs the Apple Container runtime package when needed
  • pulls the published Alpine/Lix-based builder image by default, with an optional local custom image build
  • creates a persistent Apple container machine for Linux builds
  • can optionally install Socktainer to expose a Docker-compatible local socket
  • writes helper scripts and SSH configuration under ~/.local/state/hb
  • configures host-side SSH aliases for nix-builder and container-builder, backed by a generated known_hosts file for builder host-key verification
  • configures nix.buildMachines so the host daemon can use the builder for Linux derivations
  • uses SSH ProxyCommand to auto-start the machine and avoid depending on a stable machine IP

The helper entrypoint is hb, which provides status, repair, logs, and inspection commands for the builder runtime.

If you enable Socktainer, the same helper also exposes Socktainer-specific status and log commands. See Socktainer.

Installation

Add the flake input and import the module into your Darwin host:

{
  inputs.hexbox.url = "github:RobertDeRose/nix-hex-box";

  outputs = inputs: {
    darwinConfigurations.my-host = inputs.darwin.lib.darwinSystem {
      system = "aarch64-darwin";
      modules = [
        inputs.hexbox.darwinModules.default
      ];
    };
  };
}

The module is intended for nix-darwin hosts running Apple Container capable macOS systems. The builder guest is always aarch64-linux.

Configuration

Minimal example:

services.container-builder = {
  enable = true;
  cpus = 4;
  memory = "8G";
  maxJobs = 4;
  # Optional Docker API compatibility layer:
  # socktainer.enable = true;
};

Common settings to review first:

  • hostAlias
  • cpus
  • memory
  • homeMount
  • maxJobs
  • protocol
  • idleShutdown.enable
  • idleShutdown.timeoutSeconds
  • imageRepository
  • nixVersion
  • imageContainerfile
  • imageBuildContext
  • socktainer.enable

The default image is pulled from this repository’s GitHub Container Registry package:

ghcr.io/robertderose/nix-hex-box/hexbox-builder:latest

Scheduled builds refresh latest for Alpine package updates, such as OpenSSH fixes. Builder image publishing is skipped until the configured Lix tag is at least seven days old. Image-definition changes on main also publish the versioned tag alpine-3.22-lix-2.95.2-2 for users who prefer a pinned image.

The image contains Alpine 3.22, OpenSSH, sudo, and Lix. Set imageContainerfile to build a local custom image instead. Set imageBuildContext to an absolute host path string when the custom image needs a build context. Custom images must provide socat, base64, getent, and a working /sbin/init, because HexBox uses them for bootstrap, the SSH proxy, and the machine boot path. Runtime bootstrap writes a minimal nix.conf that uses https://cache.nixos.org/ by default.

Current default behavior to keep in mind:

  • protocol = "ssh-ng"
  • hostAlias = "container-builder"
  • containerName = "nix-builder"
  • homeMount = "none"
  • idleShutdown.enable = true
  • idleShutdown.timeoutSeconds = 300
  • exposeHostContainerInternal = true
  • cli.completions.enable = false

The builder machine does not mount the host home directory by default. Set homeMount = "ro" or homeMount = "rw" only when the builder needs explicit access to host files.

If you want shell completions for hb, enable:

services.container-builder.cli.completions.enable = true;

This installs bash, zsh, and fish completion files through standard Nix completion directories. It does not detect your current shell or modify shell startup files.

Socktainer

nix-hex-box can optionally install and manage Socktainer, which exposes a Docker-compatible local API socket on top of Apple container.

Enable it with:

services.container-builder.socktainer = {
  enable = true;
};

The module installs the official Socktainer pkg, creates a user launch agent, and starts the daemon in the current primary user’s session.

Socktainer stores its socket and logs under:

$HOME/.socktainer

The Docker-compatible socket path is:

$HOME/.socktainer/container.sock

To point Docker-compatible clients at that socket manually:

export DOCKER_HOST=unix://$HOME/.socktainer/container.sock
docker ps

To export DOCKER_HOST automatically for user sessions:

services.container-builder.socktainer = {
  enable = true;
  setDockerHost = true;
};

This integration is optional and independent from the Nix remote builder path. The module still manages the builder machine directly through Apple container machine.

Runtime Model

Durable host state lives under:

~/.local/state/hb

The current runtime model is:

  • the builder is an Apple container machine, not an ordinary ephemeral container
  • activation rewrites helper scripts and SSH config
  • by default, the machine is created from the published HexBox GHCR image
  • custom imageContainerfile configurations build a local OCI image when the tag is missing
  • the SSH path uses ProxyCommand to auto-start the machine on demand
  • the same SSH path is usable by the root nix-daemon; it sudoes back to the runtime-owning macOS user before invoking Apple container
  • the guest SSH user is builder
  • the remote nix-daemon runs through a narrow passwordless sudo wrapper inside the guest
  • idle shutdown runs inside the guest and powers off the machine after the configured inactivity timeout
  • Socktainer, when enabled, runs as a separate user launch agent and exposes a Docker-compatible Unix socket under $HOME/.socktainer

The builder machine owns a persistent guest /nix store. Build outputs and substitutes survive machine stop/start cycles. They are deleted when the machine is removed, including explicit hb builder reset and automatic recreation after an image-contract generation change.

Socktainer is not part of the builder control path. It is an optional companion daemon for Docker-compatible local tooling on top of the same Apple container runtime.

Generated Files

Activation writes the operational helper files into ~/.local/state/hb.

Important files include:

  • bootstrap-keys.sh
  • bootstrap-machine.sh
  • builder-image/Containerfile when imageContainerfile is set
  • proxy.sh
  • start-container.sh
  • stop-container.sh
  • reset-container.sh
  • ssh-wrapper.sh
  • ssh_config
  • ssh_config_root
  • known_hosts
  • machine-generation
  • hexbox-readiness.log
  • hb

These files are the practical runtime interface to the builder. They are generated from the active Nix configuration and should not be edited manually.

By default, the builder image is pulled from GHCR. When services.container-builder.imageContainerfile is set, activation copies that Containerfile here and hb builder repair / start-container.sh build the custom image locally when its tag is missing. If imageBuildContext is set, it must be an absolute host path string and is passed directly to Apple container build.

machine-generation records the image contract used to create the current Apple container machine. If it differs from the active Nix configuration, start-container.sh recreates the machine, deleting the guest-local /nix store.

The repository copy of assets/hb.sh is also generated. Edit scripts/hb.sh and regenerate assets/hb.sh instead of changing the built helper directly.

The checked-in shell completion assets under assets/completions/ are part of the hb distribution path. The module installs them through Nix’s installShellCompletion hook into the standard shell completion directories instead of mutating user shell dotfiles.

Network And Access Paths

The module uses one SSH transport for both user access and Nix daemon access.

ProxyCommand path

The generated SSH config points nix-builder and container-builder at ~/.local/state/hb/proxy.sh as a ProxyCommand.

That proxy:

  • sudoes back to the configured runtime-owning macOS user when invoked by root
  • starts the Apple container system if needed
  • creates or updates the builder machine when needed
  • runs container machine run --root -i -n <machine> socat STDIO TCP:127.0.0.1:<ssh-port>
  • relays SSH directly into guest sshd and closes the relay when SSH exits

Because the SSH connection enters through container machine run, the host does not need to know the machine’s current IP address. A stopped machine is booted on demand before SSH reaches the guest.

Root daemon path

The root nix-daemon uses the same host SSH config for ${cfg.hostAlias}. The macOS root process cannot operate the user-owned Apple container runtime directly, so the proxy explicitly uses:

sudo -n -u <runtimeUser> -H ... container machine run ...

After SSH reaches the guest, Nix connects as the builder user. The remote program is the guest nix-daemon wrapper, which uses passwordless sudo inside the guest to run the real Lix daemon as root.

This removes the old localhost bridge and avoids direct published SSH ports.

Verification And Recovery

Main helper entrypoint:

hb builder

Recovery-aware verification path:

hb builder repair
hb builder test

Useful checks after activation:

hb builder
hb builder repair
ssh nix-builder true
nix store ping --store ssh-ng://container-builder
nix build --max-jobs 0 --rebuild nixpkgs#legacyPackages.aarch64-linux.hello

ssh nix-builder true uses the installed host alias and the generated ~/.local/state/hb/known_hosts file, so the builder host key is verified rather than accepted blindly.

hb builder repair attempts to recover the Apple container system before retrying the builder startup path. It also verifies:

  • container system health
  • builder image availability, building a configured custom image when missing
  • current builder machine status
  • SSH handshake success
  • outbound builder TCP reachability for common external domains
  • remote store reachability from the host side

After repair succeeds, hb builder test runs a timed trivial remote build through the builder. If repair fails, follow the reported recovery step for the runtime, machine, or network failure that was detected.

Other useful helper commands:

  • hb builder reset
  • hb builder ssh
  • hb builder inspect
  • hb builder gc
  • hb doctor
  • hb doctor runtime
  • hb doctor dns
  • hb doctor host
  • hb doctor host 22

If guest-side DNS looks wrong, first verify the Apple defaults. The default resolver should allow both normal external lookups and host.container.internal from inside the builder machine.

If Socktainer is enabled, useful checks include:

hb socktainer
hb socktainer status
hb socktainer logs
hb socktainer logs -f
DOCKER_HOST=unix://$HOME/.socktainer/container.sock docker ps

If services.container-builder.cli.completions.enable = true; is set, hb completions are installed for bash, zsh, and fish via standard Nix completion paths. No per-shell setup files are modified by the module.

Logs And Diagnostics

Runtime logs live in ~/.local/state/hb and inside the builder machine.

Common host log files:

  • hexbox-readiness.log

Common guest log files:

  • /var/log/hexbox-idle.log
  • /var/log/nix-daemon.log

Use the helper to read the most important logs:

hb builder logs readiness
hb builder logs boot
hb builder logs idle
hb socktainer logs
hb socktainer logs -f

These logs are usually the fastest way to determine whether a failure is in:

  • Apple container runtime startup
  • container machine boot
  • guest bootstrap
  • SSH readiness
  • idle shutdown behavior

Troubleshooting

Builder does not become SSH-ready

Start with:

hb builder repair
hb builder logs readiness
hb builder logs boot

Look for image build failures, container machine boot failures, guest bootstrap failures, SSH startup problems, or ProxyCommand errors.

Apple container runtime looks unhealthy

The Apple container runtime is still an external mutable subsystem. The module can reconcile configuration and machines, but it cannot guarantee the runtime substrate is always healthy.

hb doctor runtime checks the Apple container runtime and attempts recovery for known failure boundaries. hb builder repair uses the same runtime recovery path before retrying the builder. hb doctor dns also restarts the Apple container runtime and retries once if external reachability probes fail.

Cache resolution fails inside the guest

The guest writes a minimal nix.conf and depends on working DNS and network reachability to cache.nixos.org. If substitute downloads fail, check:

  • guest DNS settings
  • upstream cache availability
  • host networking state

The machine changed IP address

This is expected after stop/start. The Nix builder path does not use the machine IP directly; it connects through the generated SSH ProxyCommand, which runs container machine run and relays to guest sshd inside the machine.

Previous build outputs disappeared

The guest /nix store persists across normal machine stop/start cycles. If outputs disappeared, the machine was probably removed and recreated. hb builder reset is destructive for guest-local store contents.

Options

The main option namespace is services.container-builder.

Important options:

  • enable
  • hostAlias
  • sshUser
  • containerPort
  • workingDirectory
  • user
  • containerBinary
  • installer.url
  • installer.hash
  • installer.version
  • containerName
  • imageRepository
  • nixVersion
  • imageContainerfile
  • imageBuildContext
  • cpus
  • memory
  • homeMount
  • exposeHostContainerInternal
  • systems
  • supportedFeatures
  • mandatoryFeatures
  • maxJobs
  • speedFactor
  • protocol
  • readiness.timeoutSeconds
  • readiness.intervalSeconds
  • idleShutdown.enable
  • idleShutdown.timeoutSeconds
  • cli.completions.enable
  • socktainer.enable
  • socktainer.binary
  • socktainer.homeDirectory
  • socktainer.setDockerHost
  • socktainer.installer.url
  • socktainer.installer.hash
  • socktainer.installer.version

Machine notes:

  • containerName is the Apple container machine name.
  • imageRepository and nixVersion combine into the image reference used by container machine create.
  • The default image is published by this repository as ghcr.io/robertderose/nix-hex-box/hexbox-builder:latest.
  • Scheduled builds refresh latest; image-definition changes also publish alpine-3.22-lix-2.95.2-2 for pinned use. Builder image publishing is skipped until the configured Lix tag is at least seven days old.
  • Set imageContainerfile to build a local custom image instead of pulling the default image. imageBuildContext supplies an optional absolute host path to the build context; without it, HexBox builds with an empty generated context. Custom images must also provide socat, base64, getent, and a working /sbin/init, because HexBox uses them during bootstrap, SSH proxying, and machine boot.
  • homeMount defaults to none so the builder does not mount the host home directory.
  • idleShutdown.timeoutSeconds controls the guest watchdog that powers the machine off after no active SSH connections remain.

Host integration notes:

  • exposeHostContainerInternal defaults to true and ensures host.container.internal exists through container system dns.
  • The Nix builder path uses ssh-ng through generated SSH config and ProxyCommand; no stable machine IP or published host port is required.

Completion notes:

  • cli.completions.enable defaults to false.
  • When enabled, the module installs bash, zsh, and fish completion files into the standard Nix-managed completion directories.
  • The module does not try to detect the user’s shell or edit shell startup files.

See modules/container-builder.nix for the authoritative option defaults and types.

Design Notes

nix-hex-box currently follows these design choices:

  • published Alpine/Lix-based builder image in this repository’s GHCR package
  • optional local custom image builds from imageContainerfile
  • persistent Apple container machine with generation-aware recreation for image-contract changes
  • on-demand startup through SSH ProxyCommand
  • no localhost bridge or published SSH port in the default builder path
  • guest-side builder user with narrow passwordless sudo for the remote nix-daemon
  • guest-side idle shutdown using a lightweight watchdog
  • host-side container machine run executed as the runtime-owning macOS user, even when SSH is launched by root nix-daemon
  • optional Socktainer sidecar for Docker-compatible local tooling

Known constraints:

  • Apple container remains an external mutable runtime
  • custom local builder image builds depend on the user’s Containerfile inputs when configured
  • deleting the container machine deletes guest-local /nix store contents
  • host and guest behavior still depend on the health of Apple’s virtualization and networking layers

Historical design notes from earlier overlay-based and bridge-based experiments should no longer be treated as current behavior.