2  Why Immutable OS

Note

CentOS Stream bootc. Unattended boot. No drift. The host is a firmware layer for containers.

2.1 The problem with mutable hosts

  • Configuration drift over months of dnf install, manual edits, forgotten workarounds
  • Rebuilding from scratch is the only reliable remediation, but traditional installs make that expensive
  • The host should be reproducible from a single artifact

2.2 bootc and image-based Linux

  • bootc treats the OS as a container image: build with a Containerfile, deploy with bootc install to-disk, update with bootc upgrade
  • The running system is an immutable image; changes come through a new image, not package transactions
  • Upstream: containers/bootc under the Containers org, led by Red Hat

2.3 The three-tier image model

  • Tier 1: upstream CentOS Stream 10 bootc image, mirrored and signed in CI
  • Tier 2: base images built in immutable-base — adds ZFS modules, NVIDIA drivers, utility packages; build deps never appear in the final image
  • Tier 3: instance images (one per host) — adds users, SSH keys, container policy, machine-specific config

2.4 Kernel module management

  • ZFS and NVIDIA are out-of-tree modules compiled via DKMS at image build time
  • CentOS Stream’s kernel and kernel-devel can drift; CI detects the available kernel-devel and pins the build to match
  • Multi-stage builds: module builders compile .ko files, base images copy only the artifacts via manifest-driven COPY --from
  • No build tools (gcc, dkms, kernel-devel) remain in the final image

2.5 What lives on the host vs. in containers

  • On the host: ZFS, NVIDIA drivers, systemd, Envoy (the sole rootful container), Alloy (log/metric collection)
  • In rootless containers: everything else — every application is a pod owned by a dedicated service user
  • The host’s job is to boot, import ZFS pools, and run systemd; application logic belongs in containers

2.6 Container signature policy

  • /etc/containers/policy.json baked into the image: default deny, explicit allowlist of trusted registries
  • Registries allowed: GitLab registry (self-hosted images), Quay, Docker Hub, GHCR
  • Signature verification deferred pending cosign v3 bundle support in containers/image

2.7 Supply chain

  • All images signed with cosign keyless via GitLab OIDC
  • CycloneDX SBOMs generated with syft and attested to each image
  • Full rebuild on every CI run (no caching): intentional, ensures no stale layers