Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Introduction

HexBox logo

nix-hex-box is a nix-darwin module that configures an Apple Container based aarch64-linux remote builder for Nix.

The module is designed for Darwin hosts that want to offload Linux derivations to a local virtualized builder while keeping the host configuration declarative.

Current design highlights:

  • installs Apple container from the official signed release package
  • configures nix.buildMachines for ssh-ng://container-builder
  • uses a pinned upstream docker.io/nixos/nix:<version> builder image
  • manages durable state under ~/.local/state/hb
  • supports an optional bridge for the root nix-daemon
  • wakes the builder on demand for user-side SSH access through ProxyCommand
  • supports guest-side idle shutdown and recovery-oriented health checks

This book documents the module itself. If you use nix-hex-box from another repo, that repo should only need a high-level integration guide.

Overview

nix-hex-box exports two identical module entry points:

  • darwinModules.default
  • darwinModules.container-builder

The main option namespace is services.container-builder.

When enabled, the module:

  • installs the Apple Container runtime package when needed
  • writes helper scripts and SSH configuration under ~/.local/state/hb
  • configures host-side SSH aliases for nix-builder and container-builder
  • configures nix.buildMachines so the host daemon can use the builder for Linux derivations
  • optionally loads a launch agent that exposes the localhost bridge used by the root daemon path

The helper entrypoint is hb, which provides status, repair, logs, and inspection commands for the builder runtime.

Installation

Add the flake input and import the module into your Darwin host:

{
  inputs.hexbox.url = "github:RobertDeRose/nix-hex-box";

  outputs = inputs: {
    darwinConfigurations.my-host = inputs.darwin.lib.darwinSystem {
      system = "aarch64-darwin";
      modules = [
        inputs.hexbox.darwinModules.default
      ];
    };
  };
}

The module is intended for nix-darwin hosts running Apple Container capable macOS systems. The builder guest is always aarch64-linux.

Configuration

Minimal example:

services.container-builder = {
  enable = true;
  cpus = 4;
  memory = "8G";
  maxJobs = 4;
  bridge.enable = true;
};

Common settings to review first:

  • hostAlias
  • port
  • listenAddress
  • cpus
  • memory
  • maxJobs
  • bridge.enable
  • protocol
  • idleShutdown.enable
  • idleShutdown.timeoutSeconds
  • dns.*
  • imageRepository
  • nixVersion

The default image is the upstream pinned image:

docker.io/nixos/nix:2.34.6

The container guest writes a minimal nix.conf that uses https://cache.nixos.org/ by default.

Current default behavior to keep in mind:

  • bridge.enable = true
  • protocol = "ssh-ng"
  • hostAlias = "container-builder"
  • listenAddress = "127.0.0.1"
  • port = 2222

Runtime Model

Durable state lives under:

~/.local/state/hb

The current runtime model is:

  • the builder container is generation-aware
  • helper scripts are rewritten declaratively on activation
  • the user-side SSH path wakes the builder on demand with ProxyCommand
  • the root nix-daemon path can use the localhost bridge when enabled
  • idle shutdown runs inside the guest and stops sshd after inactivity

The builder uses the image’s built-in /nix directly. Build outputs live in the container’s writable layer, so recreating the container loses any guest local store writes that were not already available from substituters.

That design keeps the module simpler than the older overlay-based approach, but it also means the builder should be treated as a cache-backed disposable guest, not as a durable Linux machine image.

Generated Files

Activation writes the operational helper files into ~/.local/state/hb.

Important files include:

  • bootstrap-keys.sh
  • hexbox-bridge
  • init.sh
  • proxy.sh
  • start-container.sh
  • stop-container.sh
  • ssh-wrapper.sh
  • ssh_config
  • ssh_config_root
  • hexbox-readiness.log
  • hexbox-idle.log
  • init-debug.log
  • hb

These files are the practical runtime interface to the builder. They are generated from the active Nix configuration and should not be edited manually.

Network And Access Paths

The module uses two related access paths.

User-side SSH path

The user SSH config points at ~/.local/state/hb/proxy.sh as a ProxyCommand. That proxy:

  • starts the Apple container system if needed
  • starts the builder on demand
  • waits for guest sshd
  • resolves the current container IP
  • relays SSH directly into the guest

This path is used for helper access such as ssh nix-builder true.

Root daemon path

The root nix-daemon path uses the generated root SSH config for ${cfg.hostAlias}. There are two transport modes.

Bridge mode

This is the default with bridge.enable = true.

The bridge launch agent exposes the configured host socket:

127.0.0.1:2222

and forwards incoming connections into the wake-and-relay path.

Direct published-port mode

When bridge.enable = false, the module publishes the container SSH port directly with container run -p <listenAddress>:<port>:<containerPort>.

That avoids the bridge agent entirely, but it is a different host transport shape and should be chosen deliberately.

This split exists because the direct user path and the daemon-driven path have different compatibility constraints on macOS.

Verification And Recovery

Main helper entrypoint:

hb status

Recovery-aware verification path:

hb repair

Useful checks after activation:

hb status
hb repair
ssh nix-builder true
nix store ping --store ssh-ng://container-builder
nix build --max-jobs 0 --rebuild nixpkgs#legacyPackages.aarch64-linux.hello

hb repair attempts to recover the Apple container system before retrying the builder startup path. It also verifies:

  • container system health
  • bridge agent presence
  • current builder container status
  • SSH handshake success
  • cache reachability inside the guest
  • remote store reachability from the host side

Other useful helper commands:

  • hb reset
  • hb restart
  • hb ssh
  • hb inspect
  • hb gc

Logs And Diagnostics

Runtime logs live in ~/.local/state/hb.

Common log files:

  • hexbox-readiness.log
  • hexbox-idle.log
  • init-debug.log
  • hexbox-bridge.out.log
  • hexbox-bridge.err.log

Use the helper to read the most important logs:

hb logs readiness
hb logs bridge
hb logs bridge-out
hb logs boot
hb logs idle

These logs are usually the fastest way to determine whether a failure is in:

  • Apple container runtime startup
  • guest init/bootstrap
  • SSH readiness
  • bridge/proxy relay behavior

Troubleshooting

Builder does not become SSH-ready

Start with:

hb repair
hb logs readiness
hb logs boot

Look for guest init failures, SSH startup problems, or bridge/proxy timeouts.

Apple container runtime looks unhealthy

The Apple container runtime is still an external mutable subsystem. The module can reconcile configuration and containers, but it cannot guarantee the runtime substrate is always healthy.

hb repair attempts a recovery by starting the container system with kernel install enabled before retrying the builder.

Cache resolution fails inside the guest

The guest writes a minimal nix.conf and depends on working DNS and network reachability to cache.nixos.org. If substitute downloads fail, check:

  • guest DNS settings
  • upstream cache availability
  • host networking state

Container recreation lost previous build outputs

This is expected in the current runtime model. The guest store is not preserved across recreation; outputs are expected to come back from substituters.

Direct port mode behaves differently from bridge mode

If bridge.enable = false, the host connects through the directly published container port instead of the bridge agent. Troubleshooting should then focus on the published host socket and container port mapping rather than the launchd bridge path.

Options

The main option namespace is services.container-builder.

Important options:

  • enable
  • hostAlias
  • sshUser
  • listenAddress
  • port
  • containerPort
  • workingDirectory
  • user
  • containerBinary
  • installer.url
  • installer.hash
  • installer.version
  • containerName
  • imageRepository
  • nixVersion
  • cpus
  • memory
  • dns.servers
  • dns.search
  • dns.options
  • dns.domain
  • dns.disable
  • systems
  • supportedFeatures
  • mandatoryFeatures
  • maxJobs
  • speedFactor
  • protocol
  • autoStart
  • readiness.timeoutSeconds
  • readiness.intervalSeconds
  • idleShutdown.enable
  • idleShutdown.timeoutSeconds
  • bridge.enable

See modules/container-builder.nix for the authoritative option defaults and types.

Design Notes

nix-hex-box currently follows these design choices:

  • upstream pinned nixos/nix image instead of a custom prebuilt image
  • generation-aware container recreation
  • on-demand user-side startup through ProxyCommand
  • optional bridge for the root daemon path, with direct published-port mode available when disabled
  • guest-side idle shutdown using a lightweight watchdog
  • explicit DNS configuration support for builder cache resolution

Known constraints:

  • Apple container remains an external mutable runtime
  • the daemon path still depends on the localhost bridge in current deployments
  • recreating the container loses guest-local build outputs
  • host and guest behavior still depend on the health of Apple’s virtualization and networking layers

Historical design notes from earlier overlay-based experiments should no longer be treated as current behavior.