필사 모드: Choosing an On-Prem Lightweight Kubernetes — Splitting k3s, k0s, and RKE2 by Datastore and Regulation

• Introduction — "Lightweight" Is Not a Selection Criterion
• Axis 1: Datastore — Half of the Decision Splits Right Here
• Axis 2: What Gets Bundled — the Cost of Stripping It Back Out
• Axis 3: Process Placement — Single Binary vs. Static Pods
• Axis 4: Regulatory Compliance — Who Actually Needs RKE2's CIS and FIPS
  • FIPS Is Narrower Than You'd Think
• Axis 5: Upgrades, Support Lifespan, and the Footprint on Small Nodes
  • Summary Table by Axis
• Three Scenarios and Recommendations
  • Scenario 1: Edge Nodes on the Factory Floor
  • Scenario 2: An Internal Service Cluster on a Financial or Government Air-Gapped Network
  • Scenario 3: Developer Laptops and CI
• Closing — Choose the Axis That's Hardest to Reverse First
• References

Introduction — "Lightweight" Is Not a Selection Criterion

The question that comes up most often when standing up a new on-prem cluster is "which is better, k3s or RKE2." Answer that question with a feature comparison table, though, and you're usually wrong. All three distributions are certified Kubernetes, all three ship close to a single binary, and all three officially support an air-gapped install path. A feature list alone doesn't distinguish them.

What actually decides the choice is about five axes, and the first of them — the datastore — is the hardest one to change later. Six months after installation, a request comes in for "we need one more server node," and that's when you find out you started with SQLite.

This post lays those axes out one by one, and closes by answering three concrete scenarios. Here's the verification basis.

Axis 1: Datastore — Half of the Decision Splits Right Here

k3s's default datastore is SQLite. The official CLI documentation describes the --datastore-endpoint option as specifying "the data source name for etcd, NATS, MySQL, Postgres, or SQLite (the default)." And with a SQLite configuration, you cannot add server nodes. If you need high availability, you have to move to embedded etcd or attach an external datastore.

# k3s — start with embedded etcd (first server)
k3s server --cluster-init

# k3s — external datastore (PostgreSQL example)
k3s server --datastore-endpoint="postgres://k3s:PASSWORD@10.10.30.5:5432/k3s?sslmode=require"

There is a migration path even for a cluster already stood up on SQLite. The official documentation explains that restarting an existing server with the --cluster-init flag converts it to etcd. That said, it's not something to attempt in production without a backup.

k0s specifies this via spec.storage.type, with etcd and kine as the valid values. The official configuration documentation's example shows type: etcd; using kine, you connect a SQLite/MySQL/PostgreSQL-family backend through the connection string in spec.storage.kine.dataSource. To attach an externally managed etcd, you specify endpoints and TLS file paths under spec.storage.etcd.externalCluster.

# k0s — embedded etcd (the basic form from the documentation's example)
apiVersion: k0s.k0sproject.io/v1beta1
kind: ClusterConfig
spec:
  storage:
    type: etcd

# k0s — attaching an externally managed etcd cluster
apiVersion: k0s.k0sproject.io/v1beta1
kind: ClusterConfig
spec:
  storage:
    type: etcd
    etcd:
      externalCluster:
        endpoints:
          - https://10.10.30.11:2379
          - https://10.10.30.12:2379
        etcdPrefix: onprem-cluster
        caFile: /etc/pki/etcd/ca.crt
        clientCertFile: /etc/pki/etcd/client.crt
        clientKeyFile: /etc/pki/etcd/client.key

RKE2's overview documentation doesn't state a datastore explicitly. Within this check's scope, I could not confirm from the official documentation body whether RKE2 offers a SQLite datastore, so if a single-node, ultra-lightweight setup is your goal, confirm this item directly before adopting it. That said, given RKE2's character, a situation where you'd want to run this distribution as a SQLite single node is itself rare. RKE2 runs its control plane as static pods, and the documentation states its own goal as regulatory compliance.

From an air-gap standpoint, the conclusion on this axis is this. If you can set the server node count to 3 from the start, start with embedded etcd. In an air-gapped network, adding a node later means starting the import review process all over again, so the cost of migrating later is far higher than in a normal environment. And embedded etcd requires an odd number of nodes. A 2-node configuration has the same fault tolerance as 1 node, while only adding more to operate.

Axis 2: What Gets Bundled — the Cost of Stripping It Back Out

Lightweight distributions bundle components in advance so you can get going "right out of the box." That's an advantage on day one, and a disadvantage on the day, six months later, when you want to swap out the Ingress controller.

k3s is the most explicit about this. Per the official CLI documentation, the bundled components you can turn off with the --disable flag are coredns, servicelb, traefik, local-storage, metrics-server, and runtimes. The fact that this list is written out plainly in the documentation matters a lot in practice.

# /etc/rancher/k3s/config.yaml — when using a different ingress instead of Traefik
disable:
  - traefik
  - servicelb

# On an already-installed cluster, turning it off can leave the manifest around and let it come back.
# k3s watches /var/lib/rancher/k3s/server/manifests, so clean that up too.
sudo ls /var/lib/rancher/k3s/server/manifests/
sudo systemctl restart k3s
sudo k3s kubectl -n kube-system get pods

k0s specifies its network provider via spec.network.provider; per the official documentation, the valid values are calico, kuberouter, and custom, with kuberouter as the default. The documentation states that choosing custom means "the user takes responsibility for all CNI configuration and host-level configuration." The decision to use a different CNI here becomes, in effect, a declaration that "I'll handle everything myself" — the choice is clear, but there's no middle ground.

RKE2 goes as far as splitting CNI out by image archive entirely. This is where the air-gap import list actually differs.

This split actually helps in an air-gapped setting. There's no reason to put a CNI image you're not going to use onto media and submit it for review. Conversely, get the import list wrong and you bring in core but leave out CNI, and nodes sit stuck at NotReady.

Axis 3: Process Placement — Single Binary vs. Static Pods

k3s and k0s run control-plane components inside their own process. Run ps on a node and you won't see a separate kube-apiserver process. RKE2 is different. The official overview documentation states plainly that "RKE2 runs control-plane components as static pods managed by the kubelet."

This difference produces three practical consequences.

1. The debugging path is different. On RKE2, you can read the API server's logs as pod logs and edit the manifest directly. On k3s and k0s, you have to dig through a single service log (journalctl). In an air-gapped setting with no external support available, this difference is felt keenly.
2. How well upstream documentation applies varies. RKE2's own documentation states that it "inherited tight alignment with upstream Kubernetes from RKE1." Whether the Kubernetes reference book in your hand applies as-is in an environment where internet search is blocked matters more than you'd think.
3. The resource footprint is different. Split the processes and memory splits too. On a 2 GB RAM industrial PC, this difference is decisive.

Axis 4: Regulatory Compliance — Who Actually Needs RKE2's CIS and FIPS

This axis is RKE2's whole reason for existing. Its overview documentation states its target area as "security and compliance for U.S. federal government sectors," and says it enables FIPS 140-2 compliance and lets a cluster pass the "CIS Kubernetes Benchmark v1.7 or v1.8."

The CIS profile is turned on with a single line of configuration.

# /etc/rancher/rke2/config.yaml
profile: 'cis'

# API server argument that may also be needed together on RKE2 v1.29 and above
kube-apiserver-arg:
  - 'service-account-extend-token-expiration=false'

The official hardening guide explains that this generic profile automatically applies the benchmark controls appropriate to the RKE2 version, so you don't need to change setting values across upgrades. That said, host preparation has to happen first, and if it isn't done, RKE2 aborts startup with a fatal error.

# 1) Create the etcd user and group
sudo useradd -r -c "etcd user" -s /sbin/nologin -M etcd -U

# 2) Apply kernel parameters — RPM family
sudo cp -f /usr/share/rke2/rke2-cis-sysctl.conf /etc/sysctl.d/60-rke2-cis.conf
sudo systemctl restart systemd-sysctl

# 2') non-RPM family (Ubuntu, etc.)
sudo cp -f /usr/local/share/rke2/rke2-cis-sysctl.conf /etc/sysctl.d/60-rke2-cis.conf
sudo systemctl restart systemd-sysctl

The official documentation warns to apply this sysctl change only on a fresh install, before Kubernetes is deployed. Restarting sysctl on an already-running cluster produces unexpected side effects.

What changes when you turn the CIS profile on is also laid out in the documentation. Pod Security admission gets enforced cluster-wide as restricted (excluding kube-system, compliance-operator-system, and tigera-operator), a network policy restricting traffic to within the same namespace gets deployed, agent manifest and config file permissions get tightened from 644 to 600, and the etcd static pod runs as the etcd user.

Here's the item most teams miss: the audit log.

# RKE2's default audit policy logs nothing.
# You have to change level from None to Metadata or above for logs to be recorded.
sudo vi /etc/rancher/rke2/audit-policy.yaml
sudo systemctl restart rke2-server.service
sudo tail -f /var/lib/rancher/rke2/server/logs/audit.log

In an air-gapped audit, answering the question "did you turn on the audit log?" with "we turned on the profile" won't pass. You have to edit the policy file directly.

FIPS Is Narrower Than You'd Think

There's one sentence you absolutely must confirm when reading the FIPS documentation, and it's about CNI. The official documentation states that among the supported CNIs, only Canal, the default, is rebuilt for FIPS compliance. Cilium, Calico, and Multus are not.

So if "we need FIPS" and "we want to use an eBPF-based CNI" are both required at the same time, you have to give up one of them. Miss making this decision at the design stage, and you end up ripping out the CNI right before an audit.

And there's a part worth saying plainly. If FIPS and CIS aren't required by a contract or an audit checklist, there's no need to choose RKE2 for that reason. In domestic financial and government air-gapped networks, what's actually required is usually CIS-family hardening and audit logging, and often not the FIPS 140 validated module itself. Start by checking exactly what's spelled out in the requirements document, and separate what RKE2 does automatically for you from what you'd have to do by hand regardless of which distribution you pick.

Axis 5: Upgrades, Support Lifespan, and the Footprint on Small Nodes

In an air-gapped network, upgrading means "bring in a new binary and a new image bundle over media, and swap them in." All three distributions share this same skeleton, but the character of their automation tooling differs.

k3s's automatic upgrade takes extra work in an air-gapped setting. The official documentation states that using automatic upgrade requires the rancher/k3s-upgrade, rancher/system-upgrade-controller, and rancher/kubectl images to be in your private registry. Leave these three images off your import list and automatic upgrade stalls on the very first attempt.

All three distributions follow the upstream Kubernetes minor-version cadence for support lifespan. What matters in an air-gapped setting isn't the lifespan itself so much as the alignment between your import cadence and the support cadence. If import review happens once a quarter but patches ship monthly, in practice you end up rolling up accumulated patches all at once, once a quarter. In this case, pinning to one minor version lower to use a stabilized patch line results in fewer incidents.

The resource footprint already diverges based on Axis 3's process placement. The exact figures vary a lot with CNI choice, node count, and workload, so I won't cite numbers here. Instead, here's just the decision criterion. If you have an industrial node with 4 GB of RAM or less, take the static-pod approach out of the running. A structure where control-plane components each come up as separate containers has no headroom on that hardware.

Summary Table by Axis

Cells left as "not confirmed" in this table weren't filled in with a guess. In an air-gapped adoption decision, treating an unconfirmed item as if it were confirmed is the most expensive mistake you can make. If a given cell would affect your decision, confirm it directly in staging before adopting.

Three Scenarios and Recommendations

Scenario 1: Edge Nodes on the Factory Floor

One industrial PC per line, 4 GB of RAM, remote access only opened during inspections, node count in the tens to hundreds. If one goes down, only that one line stops; the others are unaffected.

Recommendation: k3s. If the node count is high and you're building the same setup repeatedly, k0s plus k0sctl.

There are three reasons. First, since this structure doesn't need high availability, a SQLite single node is actually the appropriate choice. Managing etcd quorum in this scenario is pure cost. Second, at 4 GB of RAM, static-pod placement has no headroom. Third, k3s's embedded registry mirror lets nodes that can't reach an in-house registry share images with each other.

# Minimal edge-node configuration — /etc/rancher/k3s/config.yaml
disable:
  - traefik
  - servicelb
  - metrics-server
write-kubeconfig-mode: '0600'
kubelet-arg:
  - 'image-gc-high-threshold=70'
  - 'image-gc-low-threshold=50'

The reason to lower the image GC thresholds is that edge nodes have small disks. In an air-gapped edge setting, if the disk fills up, there's no way to re-fetch images, and the node becomes entirely unusable.

If there are hundreds of nodes and you need to repeat the same build, lean toward k0sctl. A structure where a single YAML file manages the node list, and the same command carries all the way through to upgrades, reduces human error the most when it's going to be repeated hundreds of times.

Scenario 2: An Internal Service Cluster on a Financial or Government Air-Gapped Network

A cluster hosting internal business systems. 10-30 nodes, subject to audit, with a security checklist in place, and business operations stop if there's an outage.

Recommendation: RKE2 — but only if the requirements document actually contains CIS-family items.

There are two reasons. First, a single line, profile: 'cis', adjusts Pod Security admission, network policy, file permissions, and the etcd execution user all together. Do this by hand and every item needs its own verification, and you'd have to explain "why did we configure it this way" every single audit. Having the distribution provide the rationale through official documentation is far more advantageous. Second, because of the static-pod structure, upstream Kubernetes knowledge and tooling apply as-is, which is an advantage for carrying this forward long-term with in-house staff and no external support.

# /etc/rancher/rke2/config.yaml — a baseline for an air-gapped, regulated cluster
profile: 'cis'
system-default-registry: 'registry.internal.example:5000'
tls-san:
  - k8s-api.internal.example
  - 10.10.20.10
kube-apiserver-arg:
  - 'service-account-extend-token-expiration=false'

system-default-registry is especially useful in an air-gapped setting. The official documentation states that this value only accepts an RFC 3986 URI authority. That means you can only use a host and optional port — you can't add a path. It's a common failure case to try to append a project path here too.

Conversely, if the requirements document has neither CIS nor FIPS and just says "enhanced security" with nothing more specific, the case for choosing RKE2 is weak. In that case, applying Pod Security Standards and network policies directly to k3s carries a lighter operational burden.

Scenario 3: Developer Laptops and CI

For feature development and integration testing. Created and torn down several times a day, with a lifespan of minutes to hours.

Recommendation: k3s or k0s single node. Look only at startup speed and ease of disposal.

The datastore discussion is meaningless in this scenario. SQLite is the best choice, and you strip out as many components as possible to cut startup time.

# k3s single node on a CI runner — turn off everything unnecessary
sudo INSTALL_K3S_SKIP_DOWNLOAD=true \
  INSTALL_K3S_EXEC="server --disable traefik --disable servicelb --disable metrics-server --disable local-storage --write-kubeconfig-mode 0644" \
  ./install.sh

# k0s single node — using only the subcommand, with no install script
sudo k0s install controller --single
sudo k0s start
sudo k0s kubectl get nodes

# Tear down
sudo k0s stop && sudo k0s reset
# or, for k3s
sudo /usr/local/bin/k3s-uninstall.sh

One thing to watch in air-gapped CI. If a CI runner builds a fresh cluster every single time, image import happens every single time too. The cost of unpacking a bundle of several hundred MB over and over ends up dominating build time, so it's better to redesign toward using a runner image with a pre-populated containerd storage snapshot, or reusing the cluster instead.

Closing — Choose the Axis That's Hardest to Reverse First

The feature differences between the three distributions shrink as time passes. What doesn't shrink are the constraints created by your initial decision. Adding a server node to a cluster started on SQLite; having already built on Cilium in an environment that turns out to require FIPS; deploying a static-pod distribution to a 4 GB RAM node. In all three cases, fixing it later means rebuilding the cluster, and in an air-gapped network, rebuilding a cluster means starting import review over again from scratch.

So here's the order to follow. Decide the datastore first, read the regulatory requirements document, check the node's actual RAM capacity, and then choose whichever distribution satisfies all three. The rest of the differences can be absorbed as you operate.

References

• k3s Server CLI Reference
• k3s High Availability Embedded etcd
• k0s Configuration Reference
• k0s Air-Gapped Installation
• RKE2 Overview
• RKE2 CIS Hardening Guide
• RKE2 FIPS 140-2 Enablement
• RKE2 Air-Gap Install