Skip to content
Published on

Modules and Version Pinning in an Air-Gapped Network — RHEL 8, 9, and 10 Tell Different Stories

Share
Authors

Opening — it is in the repository, but dnf says it is not

The most bewildering moment in air-gap transfer work is this one.

# The rpm file is clearly sitting in the repository directory
ls /srv/repo/rhel9-appstream/ | grep nodejs
# nodejs-18.20.4-1.module+el9.4.0+21212+d9e3c1f2.x86_64.rpm

# And yet dnf says it does not exist
dnf list available nodejs
# Error: No matching Packages to list

The file is there. The metadata was generated. And still it reports nothing.

This is not a broken repository, it is modular filtering doing exactly what it is supposed to do. And whether the symptom shows up at all depends on the RHEL version. This post is about that structure, and the way out.

Check your version first

This is the post in the series where the version differences matter most. Laid out against the official documentation, it looks like this.

ItemRHEL 8RHEL 9RHEL 10
How modules are offeredThe core mechanism of AppStreamFrom 9.1, as additional versions with a shorter life cycleNo modules chapter in the official DNF documentation
Default streamsPresent. Documentation: default streams do not change throughout the major releaseDocumentation: "no default module streams are predefined"Not applicable
Installing without naming a streamThe default stream is enabled automaticallyYou have to name a streamNot applicable
Switching streamsdistro-syncmodule resetmodule enabledistro-synca single dnf module switch-toNot applicable
AppStream delivery formatsRPM, modules, Software CollectionsRPM, modules, Software CollectionsPer the documentation, RPM and Software Collections

The official RHEL 10 "Managing software with the DNF tool" document has no chapter about modules, and the command list in its appendix has no dnf module family either. If your environment is RHEL 10 only, skip the first half of this post and start reading at "Pin the version".

Why modular packages disappear

The RHEL 9 documentation describes module dependencies as "an additional layer on top of regular RPM dependencies" that "behave similarly to hypothetical dependencies between repositories". For that additional layer to work, the module metadata has to be present.

And the dnf modulesync documentation carries the decisive sentence: DNF requires modular metadata when it installs modular packages.

That explains the symptom above.

  1. You synced AppStream with dnf reposync but did not add --download-metadata
  2. All the package files arrived, but the module metadata did not
  3. createrepo_c generates only ordinary metadata out of rpm files. The module data is not inside the rpms, so it cannot be generated
  4. The dnf on the inside sees the modular RPMs but has no idea which stream they belong to
  5. Filtering kicks in, and those packages never show up in queries

There are three routes out, and the ones higher on the list are the more correct ones.

Route 1 — pull the module metadata down along with the packages

# Always add --download-metadata
sudo dnf reposync \
  --repoid=rhel-9-for-x86_64-appstream-rpms \
  --download-path=/var/tmp/airgap-bundle/repos \
  --download-metadata \
  --gpgcheck \
  --arch=x86_64 --arch=noarch

--download-metadata pulls the repository metadata down as-is, so the result can be used as a repository immediately. In that case there is no need to run createrepo_c again after transfer, and running it can actually cost you the module data, so do not run it.

Route 2 — build a repository that includes the modules with dnf modulesync

This is the cleanest option when you only need particular modules. The documentation describes the command as one that "downloads packages from modules according to provided arguments and creates a repository with modular data in working directory". It does the download and the repository creation in one step.

# Download the whole nodejs module and build a repository containing the module data
dnf modulesync nodejs

# A specific stream and profile, dependencies included, into a chosen location
dnf --destdir=/var/tmp/airgap-bundle/nodejs modulesync nodejs:18/minimal --resolve

# Newest modules only
dnf modulesync --newest-only nodejs

--resolve is documented as "Resolve and download needed dependencies", and -n, --newest-only as "Download only packages from the newest modules".

The two-step approach the documentation introduces also fits an air-gapped network well. Settle what you actually need on a connected machine with dnf module install, then run dnf modulesync --destdir=..., and you end up with a repository holding only the packages that system actually asked for.

Route 3 — turn the filtering off with module_hotfixes

This is the last resort. The dnf configuration documentation defines module_hotfixes as "Set this to True to disable module RPM filtering and make all RPMs from the repository available. The default is False".

# /etc/yum.repos.d/airgap-appstream.repo
[airgap-appstream]
name=RHEL 9 AppStream (airgap, no modular metadata)
baseurl=file:///srv/repo/rhel9-appstream
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
module_hotfixes=1
metadata_expire=-1

The packages become visible immediately. What you give up is the guarantee the modules were providing. Packages from different streams can end up installed side by side, and that combination is not one Red Hat ever tested. If route 1 or route 2 is available to you, use those.

The trap where modules drop out entirely under an empty installroot

The --installroot technique from part 2 carries one more module-related trap. Here is the warning from the official dnf documentation, reproduced as-is.

On a modular system you may also want to use the
--setopt=module_platform_id=<module_platform_name:stream> command-line option
when creating the installroot, otherwise the module_platform_id value will be
taken from the /etc/os-release file within the installroot (and thus it will be
empty at the time of creation, the modular dependency could be unsatisfied and
modules content could be excluded).

An empty root has no release information file, so the platform ID comes out empty, and the download then succeeds with the module content excluded. Because it drops out silently and without an error, it is the hardest kind of failure to catch.

# For RHEL 9 — specify the platform ID as well
sudo dnf download \
  --installroot=/var/tmp/airgap-root \
  --releasever=9.4 \
  --setopt=module_platform_id=platform:el9 \
  --setopt=reposdir=/etc/yum.repos.d \
  --resolve --alldeps \
  --destdir=/var/tmp/airgap-bundle/rpms \
  nodejs

You can read the platform ID value straight off the system.

# Read the platform ID of the current system as-is
grep PLATFORM_ID /etc/os-release
# PLATFORM_ID="platform:el9"

Pin the version

From here on, everything applies unchanged to RHEL 10, which has no modules.

The most common cause of reproduction breaking in an air-gapped network is the minor version moving. The release version variable is derived from the rpmdb by default, so if you leave it alone it climbs along with the system. There are three ways to pin it, and they differ in scope.

# 1. Applies to one command only — use this for transfer downloads
dnf download --releasever=9.4 --resolve --alldeps httpd

# 2. Pin for the whole system (when using a subscription) — as spelled in the Red Hat docs
sudo subscription-manager release --set 9.4

# Check the current pinned value
subscription-manager release

# 3. Pin with a dnf variable file — for environments that do not use a subscription
echo "9.4" | sudo tee /etc/dnf/vars/releasever

The third one is the method Red Hat's upgrade documentation gives for setting the release version manually in a RHUI environment. It works on the same principle in configurations where subscription tooling is not involved at all, such as an internal air-gapped mirror. If you run CentOS Stream, Rocky, or Alma, this is generally the one you use.

To pin at the level of individual packages, use versionlock.

sudo dnf install python3-dnf-plugin-versionlock

# Lock to the currently installed version
sudo dnf versionlock add httpd

# Check the lock list
dnf versionlock list

# Specify directly with a glob (used as-is, with no NEVRA resolution)
sudo dnf versionlock add --raw 'httpd-2.4.57-*'

# Remove one entry
sudo dnf versionlock delete httpd

# Remove them all
sudo dnf versionlock clear

Per the documentation, add is "Add a versionlock for all available packages matching the spec", --raw is "Do not resolve <package-name-spec> to NEVRAs to find specific version to lock to", and the configuration file is /etc/dnf/plugins/versionlock.conf.

One thing to watch out for. Red Hat's upgrade documentation tells you to release the locks with dnf versionlock clear before a major upgrade, because leftover locks make dependency resolution fail. In an air-gapped network too, it is worth building in a step that checks the lock state before any large transfer.

Record the state and reproduce it

Even with an identical bundle, a different install order can produce a different result. It is safer to record the state you settled on the first server and apply that same state to the rest.

# 1. Enabled module streams and installed profiles (RHEL 8 / 9)
dnf module list --installed > state-modules.txt

# 2. Record every installed package as a NEVRA
rpm -qa --queryformat '%{NAME}\t%|EPOCH?{%{EPOCH}}:{0}|\t%{VERSION}\t%{RELEASE}\t%{ARCH}\n' \
  | sort > state-packages.tsv

# 3. Only what the user installed explicitly (dependencies excluded)
dnf history userinstalled > state-userinstalled.txt

# 4. Transaction history
dnf history list > state-history.txt

Number 3 is especially useful. dnf history userinstalled shows only the packages a person installed directly, so if you install just that list on another server, the dependencies follow on their own. That is far safer than shoving the full package list in as-is.

If you have to change a stream on RHEL 8, follow the order in the documentation exactly. It says to first confirm that yum distro-sync finishes with "Nothing to do. Complete!", then to go through yum module reset and yum module enable and run yum distro-sync again. If a dependency conflict comes up along the way you need --allowerasing, and the documentation states that for the Perl module this option is always required, because some packages in a default RHEL 8 installation depend on Perl 5.26.

On RHEL 9 it is one line.

# RHEL 9 only — switch streams with a single command
sudo dnf module switch-to nodejs:20

Redistributing Red Hat content without a subscription may violate your agreement, so check your organization's license terms first. Module metadata is part of Red Hat content too.

Closing — check the version first, and do not lose the metadata

This post comes down to three things.

Modular RPMs are invisible without the metadata. Add --download-metadata to reposync, or use dnf modulesync. module_hotfixes is a last resort.

Specify the platform ID under an empty installroot. Leave it out and the module content is excluded with no error at all.

Pin the release version. That one thing removes a large share of reproduction failures, and it holds just as well on RHEL 10, which has no modules.

The commands and options were verified against the official documentation on 2026-08-15. They vary by RHEL version, so re-check against the documentation for the version you are running.

Try it yourself

Previous / next in the series

References