필사 모드: The Complete Guide to Installing k3s in an Air-Gapped Network — From Image Tarball Transfer to a Private Registry and Agent Joins

• Opening — you build a cluster out of three files that cleared review
• Working out what to transfer — what you download on the connected side
  • Transfer checklist
  • Medium transfer and integrity verification
• Installing the server node — placing images and SKIP_DOWNLOAD
• Agent joins and an embedded etcd HA setup
• The private registry path — registries.yaml and a self-signed CA
  • Redistributing images between nodes — the embedded registry mirror
• Here is where you get stuck — air-gapped k3s failure modes
  • Failure mode 1 — an image that is not in the archive
  • Failure mode 2 — the false lead the default endpoint fallback creates
  • Failure mode 3 — CoreDNS cannot find anything
• Version drift and post-install verification
  • When the archive and the binary do not line up
  • A post-install verification script
• Closing — the transfer list is the design document
• References

Opening — you build a cluster out of three files that cleared review

Air-gapped installation is hard not because Kubernetes is hard. It is hard because most install tooling is designed around the assumption of "we will download it when we need it," and in an air-gapped network that "when" never arrives. Run the get.k3s.io script as-is and curl times out on the very first line, and that is the end of it.

So air-gapped installation runs in the opposite order. First you finalize the list of what has to be brought in, fill that list exactly on the connected side, attach checksums and submit it for review, and only after the medium is inside do you begin installing. If a single item is missing from the list there is no way to undo that from the inside, and you wait anywhere from days to weeks for the next transfer review.

This article is that list and that procedure, written out command by command for k3s. The versions it was verified against are below.

Every command below is written against v1.36.2+k3s1. If you use a different version you only change the version string, but the archive and the binary must be the same version. The reason comes later.

Working out what to transfer — what you download on the connected side

An air-gapped k3s installation needs at least three files. On top of those come your workload images and your deployment manifests.

#!/usr/bin/env bash
# collect-k3s.sh — run this on a staging machine that has internet access
set -euo pipefail

K3S_VERSION="v1.36.2+k3s1"
ARCH="amd64"
# The + in a URL path has to be encoded as %2B
URLVER="${K3S_VERSION/+/%2B}"
BASE="https://github.com/k3s-io/k3s/releases/download/${URLVER}"
OUT="./k3s-airgap-${K3S_VERSION}"

mkdir -p "${OUT}"
cd "${OUT}"

# 1) System image archive (zstd recommended, saves space on the medium)
curl -fL -o "k3s-airgap-images-${ARCH}.tar.zst" \
  "${BASE}/k3s-airgap-images-${ARCH}.tar.zst"

# 2) k3s binary
curl -fL -o k3s "${BASE}/k3s"

# 3) Install script (use it with SKIP_DOWNLOAD so it touches no network at run time)
curl -fL -o install.sh https://get.k3s.io

# 4) Checksum file (when the release publishes one)
curl -fL -o "sha256sum-${ARCH}.txt" "${BASE}/sha256sum-${ARCH}.txt" || \
  echo "WARN: the release has no sha256sum-${ARCH}.txt. Substitute your own computed values."

ls -la

For the sha256sum-amd64.txt asset, I could not establish from the release page asset list whether it exists as of this check. If it is there, use it; if it is not, it is safer to treat a manifest computed by the person doing the transfer as the authoritative copy, as shown below. Air-gap review is going to want your own manifest anyway, the one that proves "who fetched this file, when, and from where."

# Build your own checksum manifest (connected segment)
cd "./k3s-airgap-v1.36.2+k3s1"
sha256sum k3s k3s-airgap-images-amd64.tar.zst install.sh > MANIFEST.sha256
cat MANIFEST.sha256

On RHEL-family nodes with SELinux enabled you also have to bring in the k3s-selinux RPM. The official docs state that this RPM must be installed manually before k3s on an SELinux-enabled node. Leave this file out and the install itself still succeeds, then blows up much later in the form of containers that cannot read their volumes.

Transfer checklist

The last two rows are the heart of this table. The k3s archive contains only the images k3s needs to bring itself up. That is roughly CoreDNS, Traefik, local-path-provisioner, metrics-server and pause. Your internal application images are obviously not in there, and neither is Prometheus, nor a replacement Ingress controller. The failure modes section comes back to this.

Medium transfer and integrity verification

Once the medium is inside, verification always comes before installation. If a compressed archive arrives quietly corrupted, k3s lets the import failure slide past as a single log line and starts up anyway — and then you find out several minutes later, when you look at Pod status.

# On the air-gapped node
cd /opt/staging/k3s-airgap-v1.36.2+k3s1

# 1) Checksum verification — if this fails, go no further
sha256sum -c MANIFEST.sha256

# 2) Confirm the archive itself opens (needs zstd)
zstd -t k3s-airgap-images-amd64.tar.zst && echo "archive OK"

# 3) Check the list of images the archive holds
zstd -dc k3s-airgap-images-amd64.tar.zst | tar -tf - | grep -E 'manifest|repositories' | head

The zstd binary may not be on the node. Air-gapped Linux images are often minimal installs, and on some distributions zstd is not a default package. You have two options here. Install zstd from an OS package mirror first, or fetch the .tar.gz or the uncompressed .tar asset on the connected side to begin with. The release publishes all three forms: k3s-airgap-images-amd64.tar, .tar.gz and .tar.zst. Even if the size hurts, .tar.gz has one fewer failure point for a first installation.

Installing the server node — placing images and SKIP_DOWNLOAD

Now the actual install. Order matters. Place the images first and run the install script second. Do it the other way around and k3s starts up, fails to find the images and drops into a retry loop.

#!/usr/bin/env bash
# install-k3s-server.sh — air-gapped server node
set -euo pipefail

STAGE=/opt/staging/k3s-airgap-v1.36.2+k3s1

# 1) Place the system image archive
sudo mkdir -p /var/lib/rancher/k3s/agent/images/
sudo cp "${STAGE}/k3s-airgap-images-amd64.tar.zst" /var/lib/rancher/k3s/agent/images/

# 2) Enable the conditional import cache (v1.33.1+k3s1 and later)
#    With this file present, a restart skips re-import unless the archive changed
sudo touch /var/lib/rancher/k3s/agent/images/.cache.json

# 3) Place the binary
sudo cp "${STAGE}/k3s" /usr/local/bin/k3s
sudo chmod +x /usr/local/bin/k3s

# 4) Run the install script — tell it to skip the download
sudo chmod +x "${STAGE}/install.sh"
sudo INSTALL_K3S_SKIP_DOWNLOAD=true "${STAGE}/install.sh"

It is barely an exaggeration to say INSTALL_K3S_SKIP_DOWNLOAD=true is the whole procedure. Without it the script tries to reach the release server. To pass server options along, use INSTALL_K3S_EXEC.

sudo INSTALL_K3S_SKIP_DOWNLOAD=true \
  INSTALL_K3S_EXEC="server --cluster-init --tls-san 10.10.20.10 --tls-san k8s-api.internal.example --write-kubeconfig-mode 0644 --disable traefik" \
  /opt/staging/k3s-airgap-v1.36.2+k3s1/install.sh

Managing options in a config file beats stringing them out on the command line. In an air-gapped network reinstalls and node additions are frequent, and every time, somebody drops one option.

# /etc/rancher/k3s/config.yaml
cluster-init: true
tls-san:
  - 10.10.20.10
  - k8s-api.internal.example
write-kubeconfig-mode: '0644'
disable:
  - traefik
node-label:
  - topology.kubernetes.io/zone=dc-a

Confirm startup like this.

sudo systemctl status k3s --no-pager
sudo k3s kubectl get nodes -o wide
sudo k3s kubectl -n kube-system get pods

# Check the containerd store directly to see whether the images really imported
sudo k3s ctr images ls | awk '{print $1}' | sort -u | head -20

If k3s ctr images ls comes back empty, the import failed. At that point look at the containerd log rather than journalctl -u k3s -n 200. The path is /var/lib/rancher/k3s/agent/containerd/containerd.log.

Agent joins and an embedded etcd HA setup

Agent nodes follow the same order. Place the image archive and the binary, then run the install script with the join details passed as environment variables.

# Read the token on the server node
sudo cat /var/lib/rancher/k3s/server/node-token

#!/usr/bin/env bash
# install-k3s-agent.sh — air-gapped agent node
set -euo pipefail

STAGE=/opt/staging/k3s-airgap-v1.36.2+k3s1
SERVER_URL="https://10.10.20.10:6443"
JOIN_TOKEN="K10xxxxxxxx::server:xxxxxxxx"

sudo mkdir -p /var/lib/rancher/k3s/agent/images/
sudo cp "${STAGE}/k3s-airgap-images-amd64.tar.zst" /var/lib/rancher/k3s/agent/images/
sudo touch /var/lib/rancher/k3s/agent/images/.cache.json
sudo cp "${STAGE}/k3s" /usr/local/bin/k3s
sudo chmod +x /usr/local/bin/k3s

sudo INSTALL_K3S_SKIP_DOWNLOAD=true \
  K3S_URL="${SERVER_URL}" \
  K3S_TOKEN="${JOIN_TOKEN}" \
  "${STAGE}/install.sh"

If you need HA, you have to pick the datastore up front. The k3s default datastore is SQLite, and SQLite cannot scale to more than one server node. To go with embedded etcd, bring the first server up with --cluster-init and attach the rest with --server.

# Server 1 — initialize the cluster (identical to putting cluster-init: true in config.yaml)
sudo INSTALL_K3S_SKIP_DOWNLOAD=true \
  K3S_TOKEN="shared-secret" \
  INSTALL_K3S_EXEC="server --cluster-init --tls-san 10.10.20.9" \
  ./install.sh

# Servers 2 and 3 — join the existing server
sudo INSTALL_K3S_SKIP_DOWNLOAD=true \
  K3S_TOKEN="shared-secret" \
  INSTALL_K3S_EXEC="server --server https://10.10.20.10:6443 --tls-san 10.10.20.9" \
  ./install.sh

The official docs state that an embedded etcd cluster needs an odd number of server nodes to maintain quorum. Quorum for n servers is (n/2)+1. A two-node setup tolerates exactly as many failures as a one-node setup while adding operational complexity, so it buys you nothing. Network flags such as --cluster-dns, --cluster-domain, --cluster-cidr and --service-cidr also have to be identical on every server node. When you add nodes one at a time in an air-gapped network, these values drift apart easily.

Even if you already brought up a single server on SQLite, there is a way out. According to the official docs, restarting the existing server with the --cluster-init flag switches it to etcd. That said, this is not the kind of operation you attempt in production with no backup.

The private registry path — registries.yaml and a self-signed CA

The standard way to distribute images in an air-gapped network is to stand up an internal registry and make the nodes look only at it. In k3s you configure that through /etc/rancher/k3s/registries.yaml.

# /etc/rancher/k3s/registries.yaml
mirrors:
  docker.io:
    endpoint:
      - 'https://registry.internal.example:5000'
  registry.k8s.io:
    endpoint:
      - 'https://registry.internal.example:5000'
  ghcr.io:
    endpoint:
      - 'https://registry.internal.example:5000'
    rewrite:
      '^(.*)': 'mirror/ghcr/$1'

configs:
  'registry.internal.example:5000':
    auth:
      username: k3s-puller
      password: 'replace-at-transfer-time'
    tls:
      ca_file: /etc/rancher/k3s/certs/internal-ca.crt

If you want to funnel every registry into a single endpoint, use a wildcard entry. The official docs state that an asterisk entry can be used as the default configuration on both the mirrors and the configs side, and that the asterisk has to be wrapped in quotes.

# Send every registry to the internal registry (wildcard)
mirrors:
  '*':
    endpoint:
      - 'https://registry.internal.example:5000'

configs:
  'registry.internal.example:5000':
    tls:
      ca_file: /etc/rancher/k3s/certs/internal-ca.crt

rewrite changes the leading part of the path. On a registry like Harbor, where namespacing by project is mandatory, you cannot use the original paths as-is without it. The example above sends ghcr.io/foo/bar to registry.internal.example:5000/mirror/ghcr/foo/bar.

If you use a self-signed CA, trust has to be configured in two places. About half of all attempts trip here.

# 1) For containerd — the location that ca_file in registries.yaml points at
sudo mkdir -p /etc/rancher/k3s/certs
sudo cp /opt/staging/internal-ca.crt /etc/rancher/k3s/certs/internal-ca.crt
sudo chmod 644 /etc/rancher/k3s/certs/internal-ca.crt

# 2) OS trust store — for other clients such as helm, skopeo, crictl and curl
#    RHEL family
sudo cp /opt/staging/internal-ca.crt /etc/pki/ca-trust/source/anchors/
sudo update-ca-trust extract
#    Debian family
sudo cp /opt/staging/internal-ca.crt /usr/local/share/ca-certificates/internal-ca.crt
sudo update-ca-certificates

# 3) registries.yaml changes take effect only after a restart — on every node
sudo systemctl restart k3s        # server node
sudo systemctl restart k3s-agent  # agent node

The official docs call out that third step explicitly: "k3s must be restarted on each node for configuration changes to take effect." And this file has to be distributed to every node that pulls images. Putting it on the servers and forgetting the agents is a common mistake.

Verification looks like this.

# Confirm the registry configuration reached containerd
sudo k3s ctr --namespace k8s.io images pull \
  registry.internal.example:5000/library/busybox:1.36

# If it fails, read the containerd log (not the kubelet message)
sudo tail -n 100 /var/lib/rancher/k3s/agent/containerd/containerd.log

Redistributing images between nodes — the embedded registry mirror

k3s ships an embedded distributed registry mirror built on Spegel. It lets one node hand an image it already has to another node without an external registry. That is useful before you have stood up an internal registry, or when edge nodes cannot even reach the internal registry.

# /etc/rancher/k3s/config.yaml (every server node)
embedded-registry: true

# /etc/rancher/k3s/registries.yaml (every node)
mirrors:
  '*':

Per the official docs, nodes must be able to reach each other on their internal IPs over TCP 5001 (the P2P network that shares the list of available images) and 6443 (the local OCI registry each node hosts). In an air-gapped network with tight firewalls 5001 is often blocked, so open it in advance.

One misconception is worth correcting. This feature only redistributes images that already exist on some node. There is no push capability for injecting images into it. The initial entry still has to come from the airgap archive or from k3s ctr images import.

Here is where you get stuck — air-gapped k3s failure modes

A document that only writes down the happy path is useless in an air-gapped network. Here are the points where you actually get stranded.

Failure mode 1 — an image that is not in the archive

The most common and the most deflating failure. The k3s airgap archive holds only the images k3s itself needs. Your internal applications, the Ingress controller you swapped in, the monitoring stack — all of those have to be transferred separately.

If you do not have an internal registry yet, you can import directly onto the node as a stopgap.

# On the connected segment: bundle the images you need into a single archive
docker pull myapp/api:1.4.2
docker pull myapp/worker:1.4.2
docker save -o myapp-images.tar myapp/api:1.4.2 myapp/worker:1.4.2
sha256sum myapp-images.tar >> MANIFEST.sha256

# On the air-gapped node: option A — drop it in the image directory and restart
sudo cp myapp-images.tar /var/lib/rancher/k3s/agent/images/
sudo systemctl restart k3s

# Option B — import immediately, without a restart
sudo k3s ctr --namespace k8s.io images import myapp-images.tar
sudo k3s ctr --namespace k8s.io images ls | grep myapp

When you use an image loaded via option B, check the imagePullPolicy in the Pod spec. If the tag is latest, the default policy becomes Always, so the node tries to pull even though the image is right there, and fails. In an air-gapped network always use explicit version tags, and pin imagePullPolicy: IfNotPresent where you need to.

Failure mode 2 — the false lead the default endpoint fallback creates

containerd has a behavior where, as a last attempt, it connects to the original registry regardless of the mirror configuration in registries.yaml. In an air-gapped network that last attempt always fails, and the error kubelet shows the user is the result of that last attempt. So you cannot tell whether the mirror is misconfigured, whether the mirror does not have the image, or whether the mirror is not being used at all.

There are two responses.

# Response A — the real cause is in the containerd log
sudo grep -iE 'failed|error' /var/lib/rancher/k3s/agent/containerd/containerd.log | tail -40

# Response B — /etc/rancher/k3s/config.yaml
# Turns off the default endpoint fallback for registries that have a mirror configured
disable-default-registry-endpoint: true

The official docs describe this option as disabling containerd default registry endpoint fallback when a mirror is configured for that registry. It was introduced as an experimental feature in a January 2024 release, and it applies only to registries that have a mirror entry in registries.yaml. Registries with no mirror configured still fall back. Turning this on makes the error message point at the actual failure point, which cuts debugging time dramatically.

Failure mode 3 — CoreDNS cannot find anything

The symptom is that in-cluster names (kubernetes.default.svc.cluster.local) resolve fine, but internal or external domain lookups stall for 5 seconds and then fail. The default CoreDNS Corefile forwards queries outside the cluster domain to the node resolver, and if that resolver points at a public DNS server that is unreachable from the air-gapped network (8.8.8.8, say), every query waits out a timeout.

Start by checking the actual configuration. Do not guess — read it from the cluster.

# Check what CoreDNS is using as its forward target
sudo k3s kubectl -n kube-system get configmap coredns -o yaml

# Check the node resolver
cat /etc/resolv.conf
sudo resolvectl status 2>/dev/null | head -30

# Check from the node itself whether internal DNS actually answers
dig @10.10.10.53 registry.internal.example +short

There are two branches to the response. First, clean up the node /etc/resolv.conf so it points only at internal DNS. On a node managed by systemd-resolved a symlink means editing the file directly gets reverted, so you have to change the resolved configuration instead. Second, if touching the node file is awkward, give k3s a separate resolver file for kubelet.

# Keep a separate resolver file just for the air-gapped network
sudo tee /etc/rancher/k3s/resolv.conf > /dev/null <<'EOF'
nameserver 10.10.10.53
nameserver 10.10.10.54
search internal.example
options timeout:1 attempts:2
EOF

# /etc/rancher/k3s/config.yaml
resolv-conf: /etc/rancher/k3s/resolv.conf

The --resolv-conf flag is documented in the official CLI reference as "Kubelet resolv.conf file" and can also be set through the K3S_RESOLV_CONF environment variable. In an environment with no internal DNS at all, you are better off pinning the forward target in the CoreDNS Corefile to an internal resolver, or nailing down just the names you need with the hosts plugin. options timeout:1 at least caps the delay of the queries that still leak out to one second.

Version drift and post-install verification

When the archive and the binary do not line up

This problem is guaranteed to appear once several people have been doing transfers over several months in an air-gapped network. The v1.35.6 archive somebody brought in last month is still sitting in the image directory, while the binary transferred this time is v1.36.2.

Here is what happens. k3s v1.36.2 wants the CoreDNS, pause and local-path-provisioner tags that match it, but the images imported on the node carry v1.35.6 tags. The tag is not in the containerd store, so it tries to pull, and since this is an air-gapped network it fails. All the log says is "image not found," and the clue about a version mismatch appears nowhere.

Diagnosis and cleanup go like this.

# 1) Binary version
k3s -v

# 2) What has piled up in the image directory
ls -la /var/lib/rancher/k3s/agent/images/

# 3) System image tags in the containerd store
sudo k3s ctr --namespace k8s.io images ls | grep -E 'coredns|pause|local-path|metrics-server'

# Cleanup — delete the old archive, leave only the new one, then restart
sudo systemctl stop k3s
sudo rm -f /var/lib/rancher/k3s/agent/images/k3s-airgap-images-amd64-v1.35.6.tar.zst
sudo rm -f /var/lib/rancher/k3s/agent/images/.cache.json
sudo cp /opt/staging/k3s-airgap-images-amd64.tar.zst /var/lib/rancher/k3s/agent/images/
sudo systemctl start k3s

The reason you delete .cache.json alongside it is that from v1.33.1+k3s1 onward this cache file is what drives the judgement "this archive has already been imported." If you swap the archive file and the cache is still there, the import can be skipped. The upgrade procedure in the official docs likewise says to add the new archive and delete the existing one.

The way to prevent drift in the first place is to bind the transfer unit to a version.

# Pin the version into both the transfer directory name and the file names
/opt/staging/k3s-v1.36.2+k3s1/
├── MANIFEST.sha256
├── VERSION            # one line: v1.36.2+k3s1
├── install.sh
├── k3s
└── k3s-airgap-images-amd64.tar.zst

# Put a version gate in the install script
EXPECTED="$(cat "${STAGE}/VERSION")"
ACTUAL="$(/usr/local/bin/k3s -v | awk '/^k3s version/{print $3}')"
if [ "${ACTUAL}" != "${EXPECTED}" ]; then
  echo "FATAL: binary ${ACTUAL} differs from transfer bundle ${EXPECTED}" >&2
  exit 1
fi

A post-install verification script

Finishing the installation is not the end. In an air-gapped network the gap between "it looks like it is up" and "it is actually usable" is wide. Running the script below right after installation cuts incidents a few days later by a lot.

#!/usr/bin/env bash
# verify-airgap-k3s.sh
set -uo pipefail
ERR=0
K="sudo k3s kubectl"

echo "== 1. Node status"
${K} get nodes -o wide
NR=$(${K} get nodes --no-headers | grep -cv " Ready ") || true
[ "${NR}" -gt 0 ] && { echo "FAIL: ${NR} NotReady node(s)"; ERR=$((ERR+1)); }

echo "== 2. System Pods"
BAD=$(${K} -n kube-system get pods --no-headers | grep -cvE "Running|Completed") || true
[ "${BAD}" -gt 0 ] && { ${K} -n kube-system get pods | grep -vE "Running|Completed"; ERR=$((ERR+1)); }

echo "== 3. Any attempt to reach out to an external registry"
if sudo grep -qiE 'docker\.io|registry\.k8s\.io|ghcr\.io' \
     /var/lib/rancher/k3s/agent/containerd/containerd.log 2>/dev/null; then
  echo "WARN: the containerd log shows traces of external registry access"
  echo "      check the registries.yaml mirror settings and disable-default-registry-endpoint"
fi

echo "== 4. DNS"
${K} run dnscheck --rm -i --restart=Never --image=registry.internal.example:5000/library/busybox:1.36 -- \
  nslookup kubernetes.default.svc.cluster.local || { echo "FAIL: cluster DNS"; ERR=$((ERR+1)); }

echo "== 5. Certificate expiry"
sudo k3s certificate check --output table

echo "== 6. Datastore"
if [ -d /var/lib/rancher/k3s/server/db/etcd ]; then
  echo "datastore: embedded etcd"
  ls -la /var/lib/rancher/k3s/server/db/snapshots/ 2>/dev/null | tail -5
else
  echo "datastore: sqlite (cannot add server nodes)"
fi

echo "== Result: ${ERR} error(s)"
exit "${ERR}"

Item 5, k3s certificate check --output table, matters especially in air-gapped operations. k3s client and server certificates are valid for 365 days from issue, and on a service restart they renew automatically if they are within 120 days of expiry. But no restart means no renewal. That is exactly the path by which an air-gapped cluster nobody has touched for months dies of certificate expiry. The Day 2 operations article covers the response in detail.

Closing — the transfer list is the design document

In an air-gapped k3s installation the real work is not the commands, it is the list. Archive, binary, install script, checksums, CA, workload images, charts, SELinux RPM — counting every byte you will need on the inside while you are still on the outside. The commands themselves come to fewer than ten lines.

And that list is not something you build once and are done with. When the version goes up the archive and the binary have to go up together, and the moment only one of the two moves, the cluster quietly goes wrong. Five lines — pinning the version into the transfer unit and putting a gate in the install script — save you the three weeks of waiting for the next transfer review.

References

• k3s Air-Gap Install
• k3s Private Registry Configuration
• k3s Embedded Registry Mirror
• k3s High Availability Embedded etcd
• k3s Server CLI Reference
• k3s Certificate Management
• k3s Releases