nixfredFLOXFIELD GUIDE · BY NIX UNOFFICIAL
LEARN / CONTAINERS AND KUBERNETES

Containers and Kubernetes

Flox treats packaging and isolation as separate jobs, so one environment can run on a laptop, be exported as an OCI image with flox containerize, or be referenced straight from a Kubernetes pod with no image at all. Containers stay the delivery format for teams that already run them, and Imageless Kubernetes is the optional route that skips the image pipeline.

This page is drawn from https://flox.dev/docs/man/flox-containerize, https://flox.dev/docs/concepts/flox-vs-containers, https://flox.dev/docs/imageless-kubernetes/intro, https://flox.dev/docs/imageless-kubernetes/tech, https://flox.dev/docs/imageless-kubernetes/config, https://flox.dev/docs/imageless-kubernetes/install/self-managed, https://flox.dev/docs/imageless-kubernetes/install/eks, https://flox.dev/docs/imageless-kubernetes/install/upgrading, https://flox.dev/blog/is-flox-an-alternative-to-containers/, https://flox.dev/blog/nix-and-containers-why-not-both/, https://flox.dev/blog/flox-and-containers-a-perfect-pattern-for-local-development/ and https://flox.dev/blog/with-kubernetes-uncontained-flox-unlocks-faster-more-reproducible-workloads-on-k8s/.

The framing: packaging is not isolation

The docs put it in one sentence: “Flox believes that software packaging should be distinct from the chosen type of isolation.” (https://flox.dev/docs/concepts/flox-vs-containers). With containers, packaging and isolation are mixed together; with Flox, they are separate, and Flox environments “work the same on bare metal, in VMs, in containers, in agent sandboxes” (https://flox.dev/docs/concepts/flox-vs-containers).

The engineering VP’s answer to “Flox or containers?” is that the question is the wrong shape: Flox environments for local, collaborative development, containers as the standard artifact for CI or production, and Flox inside containers for both (https://flox.dev/blog/is-flox-an-alternative-to-containers/).

Say: “You do not have to give up containers. Flox decides what is inside the box, and the box is your choice.” Ask: “Where do your containers get built today, and who owns the base images?”

flowchart LR
  A[Flox environment<br/>manifest.toml + lock] --> B[flox activate<br/>laptop, CI, VM]
  A --> C[flox containerize<br/>OCI image]
  A --> D[FloxHub push<br/>generation]
  C --> E[Registry and<br/>any container runtime]
  D --> F[Pod annotation<br/>runtimeClassName flox]
  F --> G[Flox containerd shim<br/>pulls env to node]
Whiteboard diagram (rendered from the source mermaid)

Flox versus containers, side by side

The concept page walks one project through both workflows. The differences it lists are the useful talking points (https://flox.dev/docs/concepts/flox-vs-containers):

TaskFloxContainer workflow
Startflox init creates a .flox directoryA Dockerfile that needs a FROM line, and the page notes the base image reference may change from moment to moment
Add packagesflox install curl yarn@1.22 python3@3.10RUN apt commands, plus a PPA for the older Python
Enterflox activate gives a subshell, with no separate build stepBuild the image, mount source, then exec into the container
ShareCheck .flox into Git; the lockfile gives every developer the same softwareEach developer builds locally, so results can differ, unless CI builds and pushes the image
Dev services[services.server] in the manifest, started with flox services start or flox activate --start-servicesA docker-compose.yml and mapped ports
Shipflox containerize produces an image with the same software used in dev and CIA multi-stage Dockerfile built in CI

The page also says Flox environments are not isolated from the host network, so no port mapping is needed (https://flox.dev/docs/concepts/flox-vs-containers).

The sibling blog posts add framing worth borrowing:

flox containerize

flox containerize exports an environment as a container image, and the image can go to a runtime registry, a file, or another process (https://flox.dev/docs/man/flox-containerize). The synopsis lists these flags (https://flox.dev/docs/man/flox-containerize):

FlagWhat it does
-f, --fileFile to write the image to; - writes to stdout; defaults to {name}-container.tar if no runtime is specified or detected
--runtimeRuntime that stores the image, or builds it on macOS; defaults to the first one found on PATH
--tagImage tag (the synopsis lists it; the example uses --tag 'v1')
--labelkey=value metadata, repeatable, merged with labels in [containerize.config]
-m, --mode (dev|run)Containerize in dev or run mode, overriding options.activate.mode
-d, -r, -DPick the environment: a directory, a FloxHub <owner>/<name> reference, or your default environment

Behavior that matters in a call:

  • The container acts like activation. docker run -it <container id> launches a bash subshell with the packages and variables set after the activation hook (akin to flox activate), and a non-interactive docker run <container id> hello is similar to flox activate -- (https://flox.dev/docs/man/flox-containerize).
  • Mode. Dev is the manifest default and run mode makes only the requested packages available in PATH; the manifest page explains this difference (https://flox.dev/docs/man/manifest.toml). For a lean production image, ask whether run mode fits [verify against your own manifest: the containerize page says only that --mode overrides the manifest setting].
  • macOS needs a runtime. “Exporting a container from macOS requires a supported runtime because a proxy container is used to build the environment and image.” Files are cached in a docker or podman volume named flox-nix, which can be removed when no containerize command is running (https://flox.dev/docs/man/flox-containerize).
  • Manifest config. A [containerize.config] table can set user, exposed-ports, cmd, volumes, working-dir, labels and stop-signal. The docs warn that containerize.config is experimental and its behavior is subject to change (https://flox.dev/docs/man/flox-containerize). Flox sets an entrypoint that activates the environment, and cmd then runs inside the activation, like flox activate -c cmd (https://flox.dev/docs/man/flox-containerize).
flox init
flox install hello
flox containerize -f - | docker load
docker run --rm -it <container id>

(https://flox.dev/docs/man/flox-containerize)

The image exports the same software used in dev and CI, down to the git revisions of the upstream source repositories (https://flox.dev/docs/concepts/flox-vs-containers). The docs state that the package list in the image is what the environment holds; they do not publish an image size figure [verify before promising smaller images].

Imageless Kubernetes: the idea

“Imageless Kubernetes is a new way to run Kubernetes backed by Flox environments rather than container images.” (https://flox.dev/docs/imageless-kubernetes/intro). A pod names a Flox environment in an annotation, and the environment must already be pushed to FloxHub, which “becomes the source of truth” (https://flox.dev/docs/imageless-kubernetes/intro). FloxHub then shows the installed packages and versions, the environment variables, the startup scripts, and an audit trail of who changed what (https://flox.dev/docs/imageless-kubernetes/intro).

The marketing blog names it “Kubernetes, Uncontained” and says: “Like Kata Containers and gVisor, Kubernetes, Uncontained uses CRI hooks to control pod startup behavior.” (https://flox.dev/blog/with-kubernetes-uncontained-flox-unlocks-faster-more-reproducible-workloads-on-k8s/). It stays standard Kubernetes: the only changes are a runtime class and one annotation (same post).

The pod, the RuntimeClass, the annotation, the shim

A sample pod from the intro page (https://flox.dev/docs/imageless-kubernetes/intro):

apiVersion: v1
kind: Pod
metadata:
  name: flox-containerd-demo
  annotations:
    flox.dev/environment: "flox/echoip"
spec:
  runtimeClassName: flox
  containers:
    - name: echoip
      image: flox/empty:1.0.0
      command: ["echoip"]

The update workflow has no image rebuild: an operator runs flox install -r myorg/myenv somepackage, then flox push -r myorg/myenv, and the next pod deploy pulls the latest generation (https://flox.dev/docs/imageless-kubernetes/intro).

Configuration knobs

Annotation or settingEffectSource
flox.dev/activate-modePods default to run mode; set dev for jobs such as CIhttps://flox.dev/docs/imageless-kubernetes/config
flox.dev/nix-mutable: "true"Pods are immutable by default, with /nix read-only and flox install not possible; this enables mutability, for debugginghttps://flox.dev/docs/imageless-kubernetes/config
flox.dev/skip-containersContainers left unmodified, for sidecars such as vault-agent or istiohttps://flox.dev/docs/imageless-kubernetes/config
flox.dev/skip-containers-execMain process runs in the environment, but kubectl exec runs outside ithttps://flox.dev/docs/imageless-kubernetes/config
flox.dev/disable-metrics: "true"Turns off telemetry for node pull and list, activation and kubectl exec; does not cover the node bootstrap commandhttps://flox.dev/docs/imageless-kubernetes/config

Private environments need a FloxHub token: create a Kubernetes secret from flox auth token and mount it at /var/run/secrets/flox.dev. Tokens from flox auth token expire one month after issue and need CLI 1.7.6 or newer; for a longer-lived organization token the docs point to service accounts (https://flox.dev/docs/imageless-kubernetes/config).

Installing it

Each node that hosts Flox pods needs Flox installed, the containerd shim installed, the shim registered with containerd, and the runtime registered with Kubernetes (https://flox.dev/docs/imageless-kubernetes/install/self-managed).

  1. Automatic install. As root on each node: flox activate -r flox/containerd-shim-flox-installer --trust. It detects the containerd version, installs the right shim, edits the containerd configuration and restarts containerd (https://flox.dev/docs/imageless-kubernetes/install/self-managed).
  2. Manual install. For distributions with their own containerd, such as K3s, the docs say the installer does not support them; install flox/containerd-shim-flox-2x for containerd 2.x or the -17 package for 1.7, then register the runtime in the containerd configuration or a K3s template (https://flox.dev/docs/imageless-kubernetes/install/self-managed).
  3. Label and apply. Label nodes with flox.dev/enabled=true, then apply the RuntimeClass (https://flox.dev/docs/imageless-kubernetes/install/self-managed).
  4. EKS. The docs recommend a new node group just for Imageless Kubernetes, bootstrapped via nodeadm user data, with Terraform and eksctl examples (https://flox.dev/docs/imageless-kubernetes/install/eks). A local kind demo is also documented (https://flox.dev/docs/imageless-kubernetes/examples/kind-demo).

Limits and honest caveats

Lead with: “You keep your manifests, Helm charts and admission control. You change one runtime class and one annotation.” Ask: “Is the slow part of your deploy the image build, the registry push, or the pull on the node?”

← Languages and tools · Agents and skills →