필사 모드: k0s Air-Gapped Install and k0sctl Automation — From Building the Image Bundle to Multi-Node Upgrades

• Introduction — Getting It Down to One Binary and One Bundle
• Figuring Out What to Import — the Files k0s Requires
• Building the Image Bundle — list-images and bundle-artifacts
  • Folding Your Own Workload Images Into the Same Bundle
• Manual Install — Single Node, and Controller Plus Worker
  • Flatly Blocking Any Attempt at an External Pull
• k0sctl Automation — Pushing Bundles and Binaries Over SSH
• The Upgrade Path and k0sctl's Limits
• Where This Gets Stuck — k0s Air-Gap Failure Modes, and a Comparison With k3s
  • An Honest Comparison With the k3s Flow
• Closing — It All Comes Down to the Automation Tool Using SSH
• References

Introduction — Getting It Down to One Binary and One Bundle

k0s's design philosophy is simple. A single binary is both the control plane and the worker, with no install script and no package repository needed. From an air-gap standpoint, this is a real advantage — the number of files that have to go through import review effectively drops to two. One k0s binary, and one image bundle.

That simplicity comes at a cost, though. k3s just copies the air-gap archive attached to a release and you're done, but k0s is closer to a structure where you, the user, explicitly decide what goes into the bundle. That makes folding in your own workload images actually easier, but in exchange, at the bundle-building stage you first have to get the k0s binary onto a machine with internet access.

Here's what this was checked against.

At k0sctl v0.32.0, the host connection and remote execution layer was swapped out entirely, from rig v0.x to rig v2.0.0. The release notes explicitly say to watch for regressions in the areas of connection, file transfer, and OS detection, so when adopting this in an air-gapped setting, verify the file upload path in staging first.

Figuring Out What to Import — the Files k0s Requires

The k0s release page already has the assets an air-gapped setup needs laid out. Here are the main assets as confirmed against v1.36.3+k0s.0.

If you use the bundle the release provides as-is, you can skip the bundle-building stage entirely. If it's a fresh install with no need to mix in your own images, this is the fastest path.

#!/usr/bin/env bash
# collect-k0s.sh — a staging device with internet access
set -euo pipefail

K0S_VERSION="v1.36.3+k0s.0"
ARCH="amd64"
URLVER="${K0S_VERSION/+/%2B}"
BASE="https://github.com/k0sproject/k0s/releases/download/${URLVER}"
OUT="./k0s-airgap-${K0S_VERSION}"

mkdir -p "${OUT}" && cd "${OUT}"

curl -fL -o k0s "${BASE}/k0s-${K0S_VERSION}-${ARCH}"
chmod +x k0s

curl -fL -o "k0s-airgap-bundle-${K0S_VERSION}-linux-${ARCH}.tar" \
  "${BASE}/k0s-airgap-bundle-${K0S_VERSION}-linux-${ARCH}.tar"

# Build the import manifest yourself — it's the only thing that lets you tell whether the media was damaged once you're inside.
sha256sum k0s "k0s-airgap-bundle-${K0S_VERSION}-linux-${ARCH}.tar" > MANIFEST.sha256
echo "${K0S_VERSION}" > VERSION
ls -la

Building the Image Bundle — list-images and bundle-artifacts

There are cases where you can't just use the release bundle and need to build your own. That happens when you want to fold in your own workload images, when you're swapping out the default CNI for something else, or when company policy requires every image to be re-tagged to an internal registry path before it's imported.

k0s can extract the list of images it needs on its own.

# Run on a machine with internet access, using the k0s binary
./k0s airgap list-images --all > airgap-images.txt
wc -l airgap-images.txt
cat airgap-images.txt

Once you have the list, bundle it up. The official documentation lays out three methods, and each has different preconditions.

# Method A — the k0s built-in tool (needs neither Docker nor a running k0s)
./k0s airgap bundle-artifacts -v -o image-bundle.tar < airgap-images.txt

# Method B — export from the containerd store of an already-running worker node
k0s ctr images export image-bundle.tar $(k0s airgap list-images | xargs)

# Method C — from a machine that has Docker
k0s airgap list-images --all > airgap-images.txt
xargs -I{} docker pull {} < airgap-images.txt
docker image save -o image-bundle.tar $(xargs < airgap-images.txt)

Here's the practical difference between the three methods.

Use Method A as your default, and if you already have a staging cluster, Method B is the safer bet for pulling "images that actually ran."

The official documentation states that because k0s uses loose platform matching, a multi-architecture bundle works across different platforms. That means in an air-gapped network with a mix of amd64 nodes and arm64 edge nodes, you can manage a single bundle for both, which is something you can use to shrink the number of separate import units.

Folding Your Own Workload Images Into the Same Bundle

The k0s documentation says you "can easily customize the bundle to include container images k0s doesn't use by default," and advises reviewing and editing the image list before bundling it up. In practice, all you have to do is add lines to the list file.

# 1) Extract the list of k0s system images
./k0s airgap list-images --all > airgap-images.txt

# 2) Append your in-house workload images to the same list
cat >> airgap-images.txt <<'EOF'
registry.internal.example:5000/apps/api:1.4.2
registry.internal.example:5000/apps/worker:1.4.2
registry.internal.example:5000/infra/postgres:16.4
registry.internal.example:5000/infra/prometheus:v3.1.0
EOF

# 3) Dedupe, then bundle it all in one shot
sort -u airgap-images.txt -o airgap-images.txt
./k0s airgap bundle-artifacts -v -o image-bundle.tar < airgap-images.txt

# 4) Update the manifest
sha256sum image-bundle.tar >> MANIFEST.sha256

I strongly recommend putting this list file into Git. An air-gap import is a recurring job, and you need to be able to reconstruct "what went into this particular import" later. Once the list is managed as code, all you need for the next import is the diff.

Manual Install — Single Node, and Controller Plus Worker

Once the import is done, the install itself is short. Put the bundle in the images folder under the data directory and start k0s, and k0s watches that folder and imports it automatically.

#!/usr/bin/env bash
# install-k0s-single.sh — single node, air-gapped
set -euo pipefail
STAGE=/opt/staging/k0s-airgap-v1.36.3+k0s.0

# 0) Integrity verification first
cd "${STAGE}" && sha256sum -c MANIFEST.sha256

# 1) Place the binary
sudo install -m 0755 "${STAGE}/k0s" /usr/local/bin/k0s
k0s version

# 2) Place the image bundle — needs to go in before startup
sudo mkdir -p /var/lib/k0s/images
sudo cp "${STAGE}"/k0s-airgap-bundle-*.tar /var/lib/k0s/images/image-bundle.tar

# 3) Single-node install
sudo k0s install controller --single
sudo k0s start

# 4) Check status (bundle import takes time, so it may not be Ready right away)
sudo k0s status
sudo k0s kubectl get nodes -o wide
sudo k0s kubectl -n kube-system get pods

For a layout that separates the controller and the worker, it looks like this. The image bundle is needed on the worker nodes. That's because the worker is the side actually pulling images and starting containers. When the controller doubles as a worker, the bundle needs to be on the controller too.

# controller node
sudo install -m 0755 /opt/staging/k0s /usr/local/bin/k0s
sudo k0s install controller --enable-worker --no-taints
sudo k0s start
sudo k0s status

# issue a worker join token (from the controller)
sudo k0s token create --role=worker > /opt/staging/worker.token

# worker node
sudo install -m 0755 /opt/staging/k0s /usr/local/bin/k0s
sudo mkdir -p /var/lib/k0s/images
sudo cp /opt/staging/image-bundle.tar /var/lib/k0s/images/image-bundle.tar
sudo k0s install worker --token-file /opt/staging/worker.token
sudo k0s start
sudo k0s status

Flatly Blocking Any Attempt at an External Pull

The most reliable way to confirm the bundle actually landed correctly is to block off the path for pulling images externally, and see what happens. k0s supports this through cluster configuration.

# k0s.yaml
apiVersion: k0s.k0sproject.io/v1beta1
kind: ClusterConfig
spec:
  images:
    default_pull_policy: Never

The official documentation explains that this setting "makes the imagePullPolicy for Pods Never, guaranteeing that images are not pulled from the internet." In an air-gapped setting, it's better to turn this value on from the start. Do that, and if the bundle is missing an image, you get an immediate failure instead of a timeout several minutes later, so you catch a gap in your import list right on the spot.

That said, turning this value on can result in a configuration where images pulled from your in-house registry are affected too, so whether it applies across the board to application pods needs to be confirmed with an actual deployment while this cluster setting is active. Whether the scope is limited to system components or extends to workloads as a whole was something I could not settle from the documentation body alone within this check's scope, so verify it directly in staging.

k0sctl Automation — Pushing Bundles and Binaries Over SSH

Past three nodes or so, a manual procedure quickly falls apart. Someone forgets to copy the bundle on one node, and that one node just behaves strangely. k0sctl replaces this repetition with a single YAML file.

The key is that k0sctl runs from a jump host inside the air-gapped network. k0sctl connects to target nodes over SSH, not the internet, so as long as the jump host has the k0s binary and image bundle, k0sctl handles shipping the rest to each node.

# Place the k0sctl binary on the air-gapped jump host (this needs to be imported too)
sudo install -m 0755 /opt/staging/k0sctl /usr/local/bin/k0sctl
k0sctl version

# Generate the initial config file
k0sctl init > k0sctl.yaml

# k0sctl.yaml — a 3-node air-gapped configuration
apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
  name: onprem-airgap
spec:
  k0s:
    version: v1.36.3+k0s.0
    config:
      apiVersion: k0s.k0sproject.io/v1beta1
      kind: ClusterConfig
      spec:
        images:
          default_pull_policy: Never
  hosts:
    - role: controller
      uploadBinary: true
      k0sBinaryPath: /opt/staging/k0s
      ssh:
        address: 10.10.20.11
        user: k0sadmin
        keyPath: /home/k0sadmin/.ssh/id_ed25519
    - role: worker
      uploadBinary: true
      k0sBinaryPath: /opt/staging/k0s
      ssh:
        address: 10.10.20.21
        user: k0sadmin
        keyPath: /home/k0sadmin/.ssh/id_ed25519
      files:
        - src: /opt/staging/image-bundle.tar
          dstDir: /var/lib/k0s/images
          perm: 0755
    - role: worker
      uploadBinary: true
      k0sBinaryPath: /opt/staging/k0s
      ssh:
        address: 10.10.20.22
        user: k0sadmin
        keyPath: /home/k0sadmin/.ssh/id_ed25519
      files:
        - src: /opt/staging/image-bundle.tar
          dstDir: /var/lib/k0s/images
          perm: 0755

Two fields here are decisive for an air-gapped setting.

• uploadBinary: true — instead of each node downloading k0s on its own, k0sctl uploads the binary from the execution host. Without this, nodes attempt to reach the release server and fail on the spot in an air-gapped setting. The official air-gap documentation's own example uses this field.
• files — sends arbitrary files to a specified path on the node. The air-gap documentation's own example uses exactly this mechanism to place the image bundle on the workers.

Specifying the local binary path to upload with k0sBinaryPath is widely used in practice, but as of this check, I could not directly confirm this exact field name in the body of the official air-gap documentation. Since uploadBinary alone is sufficient to make it work, if the field name is uncertain, set only uploadBinary: true first, run k0sctl apply --debug to confirm which binary is being uploaded from where, and add the rest after. For the same reason, since the kind value in the air-gap documentation's example renders inconsistently depending on how it's viewed, this post uses kind: Cluster, which is the value from the k0sctl install documentation.

Applying and connecting look like this.

# Apply the configuration — binary upload, bundle transfer, and cluster setup all in one shot
k0sctl apply --config k0sctl.yaml

# Retrieve kubeconfig
k0sctl kubeconfig --config k0sctl.yaml > kubeconfig
kubectl --kubeconfig kubeconfig get nodes -o wide
kubectl --kubeconfig kubeconfig -n kube-system get pods

SSH access requirements can actually be the tricky part in an air-gapped setting. Port 22 has to be open on each node from the jump host, key-based authentication has to work, and the target user has to be able to use sudo with no password. If even one of these three conflicts with your company's security policy, give up on the k0sctl path and go with manual install instead. This is something to confirm before adoption.

The Upgrade Path and k0sctl's Limits

k0sctl's biggest practical value isn't installation — it's upgrades. The procedure is exactly the same as install.

# 1) Import the new version's binary and bundle to the jump host
sha256sum -c /opt/staging/k0s-v1.36.3/MANIFEST.sha256

# 2) Update only the version string and file paths in k0sctl.yaml
#    spec.k0s.version: v1.36.3+k0s.0
#    hosts[].files[].src: /opt/staging/k0s-v1.36.3/image-bundle.tar

# 3) Apply with the same command — k0sctl processes the nodes in order
k0sctl apply --config k0sctl.yaml

# 4) Verify
kubectl --kubeconfig kubeconfig get nodes -o wide
kubectl --kubeconfig kubeconfig get pods -A --field-selector=status.phase!=Running

Needing to upload the new image bundle first is a sequencing detail specific to air-gapped settings. Because the files entry comes first, k0sctl transfers the bundle and then swaps in k0s, but if the bundle filename matches the previous one it gets overwritten, and if it doesn't, you end up with two sitting side by side. On an edge node tight on disk, cleaning up the old bundle needs to be a separate step.

There's a clear limit too. The official documentation states plainly that k0sctl can add nodes, but it cannot remove an existing one. When replacing hardware or scaling down in an air-gapped setting, deleting it from the k0sctl config doesn't remove it from the cluster, so a separate procedure is needed.

# Node removal is manual
kubectl --kubeconfig kubeconfig drain worker-03 --ignore-daemonsets --delete-emptydir-data
kubectl --kubeconfig kubeconfig delete node worker-03

# Clean up k0s on that node
sudo k0s stop
sudo k0s reset

Where This Gets Stuck — k0s Air-Gap Failure Modes, and a Comparison With k3s

A word on import time. The official documentation only says k0s watches the images folder and auto-imports it — it doesn't say how long it takes. In practice, since it's the work of unpacking a bundle of several hundred MB into the containerd store, it can take several minutes on slow-disk edge hardware. Don't immediately treat a node showing NotReady right after startup as a failure — check the service logs first to confirm the import is progressing.

sudo journalctl -u k0scontroller -f    # controller
sudo journalctl -u k0sworker -f        # worker
sudo k0s ctr images ls | wc -l         # is the count of imported images going up?

An Honest Comparison With the k3s Flow

To sum up, k0s is better at building bundles and multi-node automation, while k3s is better at private registry mirroring and troubleshooting resources. If you're already running an in-house registry and need image-path rewriting, k3s's registries.yaml gives you a real edge. On the other hand, in an environment where you need to repeatedly build and upgrade multiple nodes using only a bundle, with no registry, a single k0sctl file gives you a real edge.

Closing — It All Comes Down to the Automation Tool Using SSH

k0sctl works in an air-gapped setting not because its feature set is flashy, but because it uses SSH instead of the internet. From the moment a file that passed import review lands on the jump host, building the cluster becomes a purely internal operation. Whether a tool has this property or not is the first criterion for choosing tools in an air-gapped setting.

And put the list file in Git. An import that doesn't record what went into the bundle always forces the same investigation to start over from scratch at the next import.

References

• k0s Air-Gapped Installation
• k0s Manual Install
• k0s Install using k0sctl
• k0s releases
• k0sctl releases
• k3s Air-Gap Install (for comparison)