Skip to content
Published on

Why Installing RHEL Packages in an Air-Gapped Network Is Genuinely Hard — Dependencies Are a Graph, and You Cannot Solve It from the Inside

Share
Authors

Opening — "could you just grab one rpm file for me?"

Anyone who has run an air-gapped network has fielded this request. Someone needs a tool installed on an inside server, there is no network, so could you please download one rpm on a connected machine and put it on a USB stick.

Then they carry that file in, run the install, and see this.

# What actually happens when you bring in a single rpm and try to install it
sudo dnf install ./htop-3.2.2-1.el9.x86_64.rpm
# Error:
#  Problem: conflicting requests
#   - nothing provides libnl-3.so.200()(64bit) needed by htop-3.2.2-1.el9.x86_64

So you go fetch the missing library, and then that library needs something else. Three or four round trips later half a day is gone, and the transfer review board only meets once a day.

This series is about eliminating those round trips. This first post is about structure rather than commands. Once you understand why this is not a file-copying problem, every command in the remaining six posts becomes obvious.

Dependencies are a graph, not a list

The most common misconception is a flat list: package A needs B and C. RPM dependencies are not shaped like that.

The unit of dependency in RPM is not a package name but a capability. A package declares what it offers with Provides and what it needs with Requires. Those capabilities include not only package names but shared library sonames, file paths, and even versioned symbols. That string in the error above is exactly one of those.

Looking at the real size makes it concrete. On a connected machine, ask.

# The capabilities httpd requires directly — a mix of symbols and files, not package names
dnf repoquery --requires httpd

# Turn those capabilities into the packages that actually provide them, and follow it all the way down
dnf repoquery --requires --resolve --recursive httpd | sort -u | wc -l

--requires shows the required capabilities, --resolve turns each capability into the package that provides it, and --recursive keeps going instead of stopping at the first level. All three options are in the official DNF command reference. The line count varies with system state, but the conclusion is the same everywhere: it is not a number you follow by hand.

There is one more trap here. dnf repoquery --tree and the depsolver both compute with what is already installed excluded. A library that happens to be present on your connected machine is treated as "not needed" and vanishes from the result. Carry that result into the air-gapped network and the inside server, which does not have it, blocks again. Post 2 tackles this head-on with --installroot.

The side that solves the graph and the side that installs are on different networks

This is the heart of it.

The thing that resolves dependencies is the depsolver, and the depsolver's input is the entire repository metadata. It has to know which package provides which capability before it can search for a valid combination. Inside the air-gapped network that metadata does not exist, so the computation cannot even start.

Outside, the metadata exists, but you do not know the state of the system you are installing onto. The side that can compute and the side that needs the answer are separated. That separation is the whole problem.

Which leaves exactly two approaches.

  • Move the resolved result. Solve dependencies on the outside, produce a bundle of rpm files, and transfer only that bundle. Use this when the requirement is well defined and small.
  • Move the repository itself. Mirror the repository, transfer it, and run the depsolver normally on the inside. Use this when you do not yet know what you will install, and when you will be running many servers for a long time.

Post 2 is the first approach, post 3 is the second. Most organizations end up at the second.

How Red Hat content is actually delivered

The official RHEL 9 documentation separates content repositories like this. BaseOS holds the core operating system functionality that forms the foundation for all installations, and AppStream holds additional user-space applications, runtime languages, and databases. The documentation states that both content sets are required by RHEL and available in all RHEL subscriptions. CodeReady Linux Builder ships with every subscription as well, but it is for developers and Red Hat does not support the packages in it.

Reaching that content requires a registered system with the right repositories enabled. On a connected machine the flow looks like this.

# Check registration and subscription status
sudo subscription-manager status

# List the repository IDs currently enabled on this system
sudo subscription-manager repos --list-enabled | grep "^Repo ID"

# Enable the repositories you need (IDs differ by architecture and version)
sudo subscription-manager repos \
  --enable rhel-9-for-x86_64-baseos-rpms \
  --enable rhel-9-for-x86_64-appstream-rpms

One thing has to be said plainly here. Redistributing Red Hat content without a subscription may breach your agreement, so confirm your organization's license terms first. This series does not cover working around entitlement. Where a step requires entitlement, it says so and stops there. Whether building an internal mirror for internal systems is permitted by your contract is a question for your legal team or your Red Hat account contact, not a question that technology answers.

If you have no subscription and cannot follow along, CentOS Stream, Rocky Linux, AlmaLinux, and Fedora are the practical stand-ins. The behaviour of dnf, createrepo_c, and rpm reproduces almost entirely. Repository IDs, GPG keys, and the way minor versions are pinned all differ per distribution, though, so re-verify those parts against your own environment when you adapt a procedure.

The repository is a moving target

Run the same command today and two weeks from now and you get different results. Security updates land, minor releases advance, packages get obsoleted.

This matters because in an air-gapped network a transfer is an event. A bundle clears review once, installs several servers, and then months later you add one more server and reuse the same bundle. At that point "just download it again" does not work. Downloading again brings in something different, and the new server drifts from the existing ones.

So every transfer unit has to carry three things: the date the repository snapshot was taken, the release version used, and the exact NEVRA and checksum of every package. Post 4 builds that manifest concretely.

Modularity adds one more layer

RHEL 8 introduced modules inside AppStream: several versions of the same component split into streams, held in one repository, with only one stream active at a time. The consequence is that dependency resolution now happens at the RPM level and the module level simultaneously. In the words of the RHEL 9 documentation, modular dependencies are an additional layer on top of regular RPM dependencies and behave like dependencies between repositories.

Post 5 covers why this hurts in an air-gapped network. What matters right now is that the situation is completely different per version. Against the official documentation it lines up like this.

ItemRHEL 8RHEL 9RHEL 10
Primary command in the docsyum (an alias for dnf)dnfdnf (dnf5 based)
Default module streamsPresent, and auto-enabledNo default streams predefinedNot applicable
Modules offeredThe core AppStream mechanismFrom 9.1, as extra shorter-life-cycle versionsNo modularity chapter in the official guide
Switching streamsdistro-sync then module reset then module enable then distro-syncdnf module switch-to, one lineNot applicable
Adding a repositoryyum-config-manager --add-repodnf config-manager --add-repodnf config-manager --add-repo

The official RHEL 10 "Managing software with the DNF tool" guide has no modularity chapter at all, and lists only RPM and Software Collections as the formats Application Streams come in. Take a RHEL 8 runbook straight to RHEL 10 and you will be running commands that do not exist.

What this series covers

The goal is a transfer procedure you build once that still produces the same result six months later.

  1. Why it is hard (this post)
  2. Downloading on the outside — the difference between dnf download, reposync, and yumdownloader, plus --installroot
  3. Building a local repository — createrepo_c, repodata, .repo files, and GPG keys
  4. Transfer and integrity — checksums, signature verification, a reproducible bundle manifest
  5. Modules and version pinning — locking the minor version and reproducing state
  6. Container images — podman save and skopeo, and an internal registry
  7. Operations — patch cadence, rollback, managing CVE response lag

Closing — redefine the problem and the procedure falls out

Air-gapped installation is not a problem of moving files. It is a problem of finishing the computation on the side that can compute, and moving the result in a reproducible form.

Define it that way and everything else follows. To finish the computation you have to resolve against a clean root. To reproduce the result you have to pin the snapshot date and release version. To trust what you moved you have to verify signatures. That is the remaining six posts.

Commands and options were verified against the official documentation on 2026-08-15. Behaviour differs by RHEL version, so re-check against the documentation for the version you are running.

Try it yourself

Next in the series

References