필사 모드: Air-Gapped Kubernetes Day 2 Operations — How to Stop a Cluster From Dying of Certificate Expiry

• Opening — air-gapped clusters mostly die of certificates
• Certificates — detection comes first, renewal comes second
  • Detection
  • Renewal on a kubeadm cluster
  • Renewal on k3s
• Time and names — NTP, internal DNS, OS package mirrors
• etcd backups and a rehearsed restore
• Upgrades with no upstream — staging bundles and skew rules
• Observability without SaaS and an exportable support bundle
• Daily, weekly and quarterly checklist
• Closing — air-gapped operations happen on a calendar
• References

Opening — air-gapped clusters mostly die of certificates

Handle incidents on air-gapped clusters a few times and a pattern emerges. The spectacular failures are actually rare. What dominates looks like this.

A cluster built last year that has been running fine stops answering kubectl one day. The error is a single line about certificates. You go to search for it and there is no internet. The internal wiki has the build procedure but not the renewal procedure. The person who built it has left the company. The API server is not up, so you cannot read logs as pod logs either.

The cause of this scenario is always the same. The certificates had a one-year lifetime, and over that year nobody restarted the cluster. In a connected environment this problem still resolves itself with a few searches in the end, but in an air-gapped network recovery takes days.

This article is about the day after installation. The verification baseline is below.

Certificates — detection comes first, renewal comes second

More important than memorizing the renewal command is knowing in advance that an expiry is approaching. Knowing the renewal command on the day of expiry does not help much. By then the service has already stopped.

Detection

On a kubeadm cluster there is a dedicated command.

# On a control plane node
sudo kubeadm certs check-expiration

The output shows the expiry date and remaining lifetime per certificate, the CA that signed it, and whether the certificate is externally managed. The important part is that the kubelet client certificate does not appear in this output. The official docs explain that the kubelet client certificate is managed by automatic rotation and lives separately under the kubelet directory. You have to look at it on its own.

# Check the kubelet certificate separately
sudo ls -la /var/lib/kubelet/pki/
sudo openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -enddate

k3s has a dedicated subcommand. It lets you pick the output format, which makes it easy to drop into a script.

sudo k3s certificate check --output table

And k3s emits a Kubernetes event as expiry approaches. Per the official docs the reason is CertificateExpirationWarning, and it is attached to the node in question. Catch that event as an alert and detection becomes automatic.

# Query the expiry warning events
sudo k3s kubectl get events -A --field-selector reason=CertificateExpirationWarning

I recommend keeping one detection script that works regardless of distribution and putting it on cron. In an air-gapped network there is no external monitoring SaaS, so this script is effectively your only early warning.

#!/usr/bin/env bash
# cert-watch.sh — run daily. Signals with a nonzero exit if expiry is within 60 days.
set -uo pipefail
THRESHOLD_DAYS=60
NOW=$(date +%s)
WARN=0

check_pem() {
  local f="$1"
  [ -f "${f}" ] || return 0
  local end epoch left
  end=$(openssl x509 -in "${f}" -noout -enddate 2>/dev/null | cut -d= -f2) || return 0
  epoch=$(date -d "${end}" +%s 2>/dev/null) || return 0
  left=$(( (epoch - NOW) / 86400 ))
  if [ "${left}" -lt "${THRESHOLD_DAYS}" ]; then
    echo "WARN  ${left} day(s) left  ${f}  (${end})"
    WARN=$((WARN+1))
  else
    echo "ok    ${left} day(s) left  ${f}"
  fi
}

# kubeadm family
for f in /etc/kubernetes/pki/*.crt /etc/kubernetes/pki/etcd/*.crt; do
  check_pem "${f}"
done
# kubelet
check_pem /var/lib/kubelet/pki/kubelet-client-current.pem
# k3s family
for f in /var/lib/rancher/k3s/server/tls/*.crt; do
  check_pem "${f}"
done

echo "${WARN} warning(s)"
exit "${WARN}"

Renewal on a kubeadm cluster

kubeadm has one property that matters a great deal in practice. Running kubeadm upgrade renews every certificate automatically. Which means a cluster that performs regular upgrades barely ever hits certificate problems. That is also exactly why air-gapped clusters die of certificates. Transfer review means no upgrades, and no upgrades means no renewal opportunity either.

Manual renewal goes like this.

# Renew everything
sudo kubeadm certs renew all

# Individual renewal is possible too
sudo kubeadm certs renew apiserver
sudo kubeadm certs renew admin.conf

There is an important caveat the official docs spell out. After a manual renewal you have to restart the control plane components yourself. kubeadm does not restart them for you. Renew and forget the restart and you end up in a state where the command succeeded but the services are still holding the old certificates.

# The simplest way to restart the static pods — move the manifests away briefly and put them back
sudo mkdir -p /tmp/kube-manifests
sudo mv /etc/kubernetes/manifests/*.yaml /tmp/kube-manifests/
sleep 20
sudo mv /tmp/kube-manifests/*.yaml /etc/kubernetes/manifests/

# Pick up the new admin.conf
sudo cp /etc/kubernetes/admin.conf "${HOME}/.kube/config"
sudo chown "$(id -u):$(id -g)" "${HOME}/.kube/config"

# Verify
sudo kubeadm certs check-expiration
kubectl get nodes

If you want to extend the lifetime itself, you specify it in the cluster configuration. In an environment like an air-gapped network where restart opportunities are rare, it is worth considering.

apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
certificateValidityPeriod: 8760h # default 365 days
caCertificateValidityPeriod: 87600h # default 3650 days

The values use Go duration format, and the largest supported unit is the hour. That said, extending the lifetime means slowing down the rotation cycle, so check first that it does not conflict with your security policy. And when you use an external CA, kubeadm cannot manage the certificates. In that case you have to produce CSRs with kubeadm certs generate-csr and get them signed externally, and renewal is entirely manual.

Renewal on k3s

k3s automatically renews certificates that are within 120 days of expiry when the service restarts (90 days before the May 2025 release). This works by reusing the existing key and extending only the validity period.

# The simplest way to trigger automatic renewal — a restart
sudo systemctl restart k3s

# Explicit rotation
sudo systemctl stop k3s
sudo k3s certificate rotate
sudo systemctl start k3s

# Rotate specific services only
sudo k3s certificate rotate --service api-server,etcd,kubelet

The rotatable service names are listed in the official docs as admin, api-server, controller-manager, scheduler, k3s-controller, k3s-server, cloud-controller, etcd, auth-proxy, kubelet and kube-proxy.

The CA is a separate matter. The self-signed CA k3s generates on first startup is valid for 10 years and is not renewed automatically. Rotation has its own dedicated command.

# Rotate the self-signed CA — cross-signing keeps the trust chain intact
# (The script has to be brought in with the transfer. You cannot fetch it with curl in an air-gapped network.)
sudo bash /opt/staging/rotate-default-ca-certs.sh
sudo k3s certificate rotate-ca --path=/var/lib/rancher/k3s/server/rotate-ca

# Replacing it with an internal CA — prepare it in a separate directory, then apply
sudo k3s certificate rotate-ca --path=/opt/k3s/server
# After applying, restart k3s on every node.
# Changing the root CA itself requires --force, and the tokens on joined nodes must be reset first.

rotate-default-ca-certs.sh is a script that lives in the k3s repository. You cannot fetch it from the internet in an air-gapped network, so it has to go on the initial transfer list. It is a textbook example of an item you discover is missing exactly when you need it.

Time and names — NTP, internal DNS, OS package mirrors

After certificates, the next thing that quietly takes down an air-gapped cluster is time. And the symptoms of a time problem never look like a time problem.

TLS verification compares the certificate notBefore and notAfter against the current time. Let a node clock drift by a few minutes and you get errors saying the certificate is not valid yet, or that it has already expired. etcd is more sensitive still. Leases and leader election are time-based, so as the skew between nodes grows the leader flaps repeatedly, write latency spikes, and in the worst case quorum judgement itself wobbles.

The reason this problem is especially common in an air-gapped network is clear. The default NTP configuration points at a public NTP pool, and that is unreachable from an air-gapped network. At install time the hardware clock is accurate so nobody notices, and it drifts apart gradually over months.

# Check the current sync status — this is item number one on an air-gap checklist
timedatectl status
chronyc sources -v 2>/dev/null || ntpq -p 2>/dev/null
chronyc tracking 2>/dev/null

# Pin to internal NTP servers
sudo tee /etc/chrony.d/internal.conf > /dev/null <<'EOF'
server 10.10.10.11 iburst prefer
server 10.10.10.12 iburst
makestep 1.0 3
EOF

# You have to remove the default configuration pointing at the public pool (the path varies by distribution)
sudo grep -rn "pool\|ntp.org" /etc/chrony.conf /etc/chrony.d/ 2>/dev/null

sudo systemctl restart chronyd
sleep 5
chronyc sources -v

# See the clock skew across every node in the cluster at once
for n in $(kubectl get nodes -o jsonpath='{.items[*].metadata.name}'); do
  printf "%-20s " "${n}"
  ssh "${n}" "chronyc tracking 2>/dev/null | awk '/System time/{print \$4, \$5, \$6}'" || echo "unreachable"
done

Internal DNS is a problem of the same character. CoreDNS forwards queries outside the cluster domain to the node resolver, and if that resolver points at an unreachable public DNS server, every query waits out a timeout. The symptom presents as "it is slow," and it takes a while to work out that DNS is the cause.

# Check what CoreDNS is looking at
kubectl -n kube-system get configmap coredns -o yaml

# The node resolver
cat /etc/resolv.conf

# Confirm the internal DNS answers
dig @10.10.10.53 registry.internal.example +short

An OS package mirror is not as urgent as certificates or NTP, but without one you will suddenly be unable to do anything at all one day. Kernel security patches, container runtime dependency packages, and installing debugging tools (tcpdump, strace, zstd) all get blocked. When you build the air-gapped environment, stand up an internal package mirror alongside it and configure the cluster nodes to look only at that. Synchronizing the mirror itself has to become part of the regular transfer schedule.

etcd backups and a rehearsed restore

Most teams do backups. Most teams have never done a restore. In an air-gapped network that difference is fatal. If the first time you run the restore procedure is in the middle of an incident, you can neither search the documentation nor ask anyone outside.

k3s has snapshot functionality built in. Per the official docs the defaults are a schedule at 12-hour intervals, retention of 5, and a storage location of db/snapshots under the data directory.

# /etc/rancher/k3s/config.yaml — recommended settings for an air-gapped network
etcd-snapshot-schedule-cron: '0 */6 * * *'
etcd-snapshot-retention: 12
etcd-snapshot-compress: true
etcd-snapshot-dir: /var/backups/k3s-snapshots

# On-demand snapshot — mandatory right before an upgrade or a risky change
sudo k3s etcd-snapshot save

# Listing and pruning
sudo k3s etcd-snapshot ls
sudo k3s etcd-snapshot prune --snapshot-retention 12
sudo k3s etcd-snapshot delete on-demand-node1-1753900000

There is one more file you absolutely have to back up alongside them in an air-gapped network. It is an item the official docs warn about explicitly. It is the server token. The token is used to encrypt confidential data, so without the same token value at restore time the snapshot is unusable.

# Keep the snapshot and the token together as one set
sudo install -m 0600 /var/lib/rancher/k3s/server/token \
  /var/backups/k3s-snapshots/token-$(date +%Y%m%d)

# Push the backup set to storage outside the cluster (a separate NAS inside the air-gapped network, say)
sudo rsync -a --delete /var/backups/k3s-snapshots/ backup-nas:/k3s/prod/

Here is the restore procedure. Actually run this procedure on staging every quarter. Having read it and having done it are different things.

# Single server
sudo systemctl stop k3s
sudo k3s server \
  --cluster-reset \
  --cluster-reset-restore-path=/var/backups/k3s-snapshots/etcd-snapshot-node1-1753900000
# Once the command above prints its completion message, exit with Ctrl-C, then
sudo systemctl start k3s

# Multiple servers — the order matters
# 1) Stop k3s on every server
sudo systemctl stop k3s

# 2) Perform the cluster-reset restore on the first server only (the command above)

# 3) Start the first server
sudo systemctl start k3s

# 4) Delete the data directory on the remaining servers — skip this step and they will not rejoin
sudo rm -rf /var/lib/rancher/k3s/server/db/

# 5) Start the remaining servers
sudo systemctl start k3s

On a kubeadm cluster you use etcdctl directly.

# Snapshot
sudo ETCDCTL_API=3 etcdctl snapshot save /var/backups/etcd-$(date +%Y%m%d%H%M).db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# Verify snapshot integrity — a great many teams save without ever verifying
sudo ETCDCTL_API=3 etcdctl snapshot status \
  /var/backups/etcd-*.db --write-out=table

Put restore rehearsals on the quarterly calendar and record the results in a document. What you need to record is not whether it succeeded but how long it took and where you got stuck. That becomes your recovery time objective for a real incident.

Upgrades with no upstream — staging bundles and skew rules

An air-gapped upgrade is "transfer new artifacts and swap them in." The procedure itself is not hard; what is hard is the planning.

The first thing to check is the version skew policy. The permitted version difference between the control plane and kubelet and kubectl is revised with each release, so before you build the transfer plan, confirm the permitted range for your target version in the official version skew policy document and write that value explicitly into the transfer plan. Write this value from memory and you will get it wrong, guaranteed.

And the control plane cannot be raised by skipping minor versions. If transfer review happens once a quarter and the minor version went up twice in that window, a single transfer has to carry all the artifacts for a two-step upgrade. Fail to do this arithmetic in advance and you end up holding the transfer medium able to complete only half the climb.

# Working out the transfer plan — confirm the steps between the current and target versions
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.nodeInfo.kubeletVersion}{"\n"}{end}'
kubectl version -o json | jq -r '.serverVersion.gitVersion'

Introduce the concept of a staging bundle. Put everything needed for one upgrade round on a single transfer medium, and pin things with a manifest so that versions cannot drift within it.

/opt/staging/upgrade-2026Q3/
├── VERSION                       # one line: the target version
├── MANIFEST.sha256
├── step1-v1.35.7/
│   ├── k3s
│   └── k3s-airgap-images-amd64.tar.zst
├── step2-v1.36.2/
│   ├── k3s
│   └── k3s-airgap-images-amd64.tar.zst
├── addons/                       # add-on component images going up in this round
└── ROLLBACK.md                   # rollback criteria and commands for each step

A pre-upgrade checklist only gets followed if it is short.

#!/usr/bin/env bash
# pre-upgrade.sh
set -uo pipefail
ERR=0

echo "[1] Snapshot"
sudo k3s etcd-snapshot save || ERR=$((ERR+1))

echo "[2] Token backup"
sudo cp /var/lib/rancher/k3s/server/token "/var/backups/token-$(date +%Y%m%d%H%M)" || ERR=$((ERR+1))

echo "[3] Bundle integrity"
(cd /opt/staging/upgrade-2026Q3 && sha256sum -c MANIFEST.sha256) || ERR=$((ERR+1))

echo "[4] Node status"
kubectl get nodes --no-headers | grep -v " Ready " && ERR=$((ERR+1))

echo "[5] PodDisruptionBudget"
kubectl get pdb -A

echo "[6] Remaining certificate lifetime"
sudo k3s certificate check --output table

echo "${ERR} error(s)"
exit "${ERR}"

The upgrade itself is swapping the archive and swapping the binary. At that point you must delete the old archive. Leave it and two versions coexist in the image directory, producing a state that is hard to predict.

sudo systemctl stop k3s
sudo rm -f /var/lib/rancher/k3s/agent/images/k3s-airgap-images-amd64.tar.zst
sudo rm -f /var/lib/rancher/k3s/agent/images/.cache.json
sudo cp /opt/staging/upgrade-2026Q3/step2-v1.36.2/k3s-airgap-images-amd64.tar.zst \
  /var/lib/rancher/k3s/agent/images/
sudo install -m 0755 /opt/staging/upgrade-2026Q3/step2-v1.36.2/k3s /usr/local/bin/k3s
sudo systemctl start k3s
sudo k3s kubectl get nodes -o wide

Observability without SaaS and an exportable support bundle

In an air-gapped network the observability stack has to live entirely on the inside too. There is no remote write target, no dashboard marketplace, no alert destination. Three things have to be decided at design time.

First, dashboards and rules are transfer items. The path where you import a Grafana dashboard from the UI does not work in an air-gapped network. Download the JSON during the collection stage, transfer it, and deploy it through provisioning.

# DMZ collection stage — secure the dashboard JSON as a file
mkdir -p dashboards
curl -fL -o dashboards/node-exporter.json \
  "https://grafana.com/api/dashboards/1860/revisions/latest/download"
sha256sum dashboards/*.json >> MANIFEST.sha256

# Inside the air-gapped network — deploy through provisioning (ConfigMap mount approach)
apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-dashboard-provider
  namespace: monitoring
data:
  provider.yaml: |
    apiVersion: 1
    providers:
      - name: airgap
        orgId: 1
        folder: 'Airgap'
        type: file
        disableDeletion: true
        options:
          path: /var/lib/grafana/dashboards

Second, make the alert destination an internal system. You cannot reach an external messenger or a paging SaaS. What realistically remains is internal SMTP, a webhook endpoint on an internal messenger, or an internal ticketing system API. If you have none of the three, the last resort is writing local logs on the node and having a person check them periodically, but at that point it is an inspection, not an alert. An observability stack with no alert path is a dashboard, not monitoring.

# Alertmanager — a configuration that only goes out through internal SMTP
route:
  receiver: internal-mail
  group_wait: 30s
  repeat_interval: 4h
receivers:
  - name: internal-mail
    email_configs:
      - to: 'k8s-ops@internal.example'
        from: 'alertmanager@internal.example'
        smarthost: 'smtp.internal.example:25'
        require_tls: false

Third, you have to be able to produce a support bundle. The only way to get help from a vendor or an outside expert in an air-gapped network is to package the information needed for diagnosis into an archive and carry it out through export review. But building it with no preparation creates two problems. Either the information you need is missing, or information that must not leave is included and export review rejects it.

#!/usr/bin/env bash
# support-bundle.sh — a diagnostic bundle designed with export review in mind
set -uo pipefail
TS="$(date +%Y%m%d-%H%M)"
DIR="/var/tmp/support-${TS}"
mkdir -p "${DIR}"/{cluster,nodes,logs}
K="sudo k3s kubectl"

echo "== Cluster resources (secrets excluded)"
${K} get nodes -o wide            > "${DIR}/cluster/nodes.txt" 2>&1
${K} get pods -A -o wide          > "${DIR}/cluster/pods.txt" 2>&1
${K} get events -A --sort-by=.lastTimestamp > "${DIR}/cluster/events.txt" 2>&1
${K} describe nodes               > "${DIR}/cluster/nodes-describe.txt" 2>&1
${K} api-resources                > "${DIR}/cluster/api-resources.txt" 2>&1
${K} version -o json              > "${DIR}/cluster/version.json" 2>&1
# Secrets by name only — never include the values
${K} get secrets -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,TYPE:.type \
                                  > "${DIR}/cluster/secrets-list.txt" 2>&1

echo "== Logs from failing pods only"
${K} get pods -A --field-selector=status.phase!=Running --no-headers 2>/dev/null \
| while read -r ns name _; do
    ${K} -n "${ns}" logs "${name}" --all-containers --tail=500 \
      > "${DIR}/logs/${ns}_${name}.log" 2>&1 || true
    ${K} -n "${ns}" describe pod "${name}" \
      > "${DIR}/logs/${ns}_${name}.describe" 2>&1 || true
  done

echo "== Node information"
uname -a                          > "${DIR}/nodes/uname.txt" 2>&1
free -h                           > "${DIR}/nodes/mem.txt" 2>&1
df -h                             > "${DIR}/nodes/disk.txt" 2>&1
timedatectl status                > "${DIR}/nodes/time.txt" 2>&1
chronyc tracking                  > "${DIR}/nodes/chrony.txt" 2>&1
sudo journalctl -u k3s -n 5000 --no-pager   > "${DIR}/logs/k3s.log" 2>&1
sudo tail -n 5000 /var/lib/rancher/k3s/agent/containerd/containerd.log \
                                  > "${DIR}/logs/containerd.log" 2>&1
sudo k3s certificate check --output table   > "${DIR}/cluster/certs.txt" 2>&1

echo "== Pre-export check — detect token and key patterns"
grep -rIlE 'BEGIN (RSA |EC )?PRIVATE KEY|K10[0-9a-f]{16,}|password:\s*\S+' "${DIR}" \
  > "${DIR}/REDACTION-REVIEW.txt" 2>/dev/null || true
echo "Review the following files before exporting, without exception:"
cat "${DIR}/REDACTION-REVIEW.txt"

tar -czf "/var/tmp/support-${TS}.tar.gz" -C /var/tmp "support-${TS}"
sha256sum "/var/tmp/support-${TS}.tar.gz"
echo "Created: /var/tmp/support-${TS}.tar.gz"

Make sure you include the sensitive-data detection in the last step. If a server token or a private key slips out inside a support bundle, being rejected at export review is the good outcome; in the worst case it gets handled as a security incident. Have a human confirm with their own eyes that the detection result is empty before filing the export request.

Daily, weekly and quarterly checklist

The things you cannot automate in air-gapped operations ultimately have to be looked at by a person on a schedule. The criterion for splitting up the intervals is "how long until the service stops if this is left alone."

The item skipped most often in this table is the quarterly restore rehearsal. And that is the one that comes back most expensively during a real incident. The rehearsal can be done by building a fresh cluster, and even just restoring a snapshot on a separate node confirms a substantial part of it.

Closing — air-gapped operations happen on a calendar

There is not much special technique in Day 2 operations for an air-gapped cluster. The certificate renewal commands, the snapshot restore procedure, the upgrade order — all of it is written down in the official documentation. And yet clusters still die, because those commands are not executed at the moment they are needed.

In a connected environment this problem does not show itself much. Something goes wrong, you search, you fetch a tool, and you recover within days. In an air-gapped network the same problem becomes weeks. So skill in air-gapped operations shows up not as incident response ability but as the ability to keep a calendar.

Put the checklist on the calendar, assign an owner, and record the rehearsal results. Those three things are what separates a cluster that dies of certificate expiry from a cluster that runs quietly for five years.

References

• kubeadm certificate management
• k3s Certificate management
• k3s etcd-snapshot CLI
• k3s Backup and Restore
• Kubernetes version skew policy
• kubeadm upgrade
• k3s Air-Gap Install