Skip to content
Published on

Bringing Container Images into an Air-Gapped Network — podman save and skopeo Are Different Tools

Share
Authors

Opening — images always follow RPMs

Once the package transfer procedure is in place, the very next request arrives. Bring the container images in on the same medium, please.

There are two families of tools here, and they get confused constantly.

  • podman save / podman loadthey go through the local image store. You pull first so the image lands locally, export that to a file, and load it back into a local store on the inside
  • skopeo copy / skopeo syncthey do not go through a local store. They move the image straight from source to destination

With one or two images, either family works. Once it is dozens of images and it has become a recurring job, the difference is obvious.

Understand transports first

To use skopeo you have to know the transport notation. Going by the containers-transports documentation, these are the ones you use in an air-gapped network.

NotationMeaning per the documentation
docker://"An image in a registry implementing the 'Docker Registry HTTP API V2'."
dir:"An existing local directory path storing the manifest, layer tarballs and signatures as individual files."
docker-archive:"An image is stored in the docker-save(1) formatted file."
oci:"An image in a directory structure compliant with the 'Open Container Image Layout Specification' at path."
oci-archive:"a tar(1) archive with contents compliant with the 'Open Container Image Layout Specification'."
containers-storage:"An image located in a local containers storage."

docker-archive: carries a trap the documentation spells out: "a write to a docker-archive: destination completely overwrites path, replacing it with the single provided image." This is exactly where the accident of adding images to the same tar file one at a time — and wiping out the previous one — comes from. To hold several images in one file, use dir: or oci:, or use the multi-image option of podman save.

The name resolution rule is worth committing to memory as well. The documentation defines it as "If name does not contain a slash, it is treated as docker.io/library/name". In an air-gapped network that means a short name points at an external host that does not exist for you.

podman save and load — the simple, dependable path

For moving a handful of images, this side is more convenient.

# Connected machine: pull the image (state the target architecture)
podman pull --arch amd64 registry.access.redhat.com/ubi9/ubi:9.4

# Export a single image as an archive
podman save --quiet -o ubi9.tar registry.access.redhat.com/ubi9/ubi:9.4

# Put several images into one archive (only supported by the docker-archive format)
podman save --multi-image-archive -o bundle.tar \
  registry.access.redhat.com/ubi9/ubi:9.4 \
  registry.access.redhat.com/ubi9/nginx-124:latest

# Export in OCI format
podman save -o ubi9-oci.tar --format oci-archive registry.access.redhat.com/ubi9/ubi:9.4

The documentation states four accepted values for --format. docker-archive is "A tar archive interoperable with docker load(1)", oci-archive is "A tar archive using the OCI Image Format", oci-dir is "A directory using the OCI Image Format", and docker-dir is "dir transport with v2s2 manifest type".

--multi-image-archive, -m is "Allow for creating archives with more than one image. Additional names are interpreted as images instead of tags. Only supported for --format=docker-archive". It cannot be combined with the OCI formats.

The loading side on the inside is simpler still.

# Air-gapped side: load from the archive into the local store
podman load -i bundle.tar

# Check the loaded images
podman images

The options for podman load are --input, -i and --quiet, -q, plus the help flag. That is all. The documentation describes the command as one that "restores an archive created by podman save as the same image, preserving its layers, history and tags".

There is one operational caveat the documentation adds. "Use the environment variable TMPDIR to change the temporary storage location of container images. Podman defaults to use /var/tmp." Loading a large image sometimes fails because the temporary space runs out, so depending on how the partitions on your air-gapped server are laid out, you may have to set this variable.

Note: podman save has no signing options at all. The official option list contains exactly seven entries — --compress, --format, --help, --multi-image-archive, --output, --quiet, --uncompressed. To attach a signature to an image you have to use the skopeo side, covered below.

skopeo copy — moving images without going through local storage

The documentation for skopeo copy contains one sentence that matters a great deal in an air-gapped network. "Uses the system's trust policy to validate images, rejects images not trusted by the policy." The copy itself goes through a policy check, which means that if you configure the policy properly, the transfer path becomes the verification path.

# From the registry to a directory on the transfer medium
skopeo copy \
  docker://registry.access.redhat.com/ubi9/ubi:9.4 \
  dir:/media/transfer/images/ubi9

# From the transfer medium to the internal registry
skopeo copy \
  dir:/media/transfer/images/ubi9 \
  docker://registry.internal.example.com/base/ubi9:9.4

# Registry to registry directly (when you can reach both sides)
skopeo copy \
  docker://quay.io/skopeo/stable:latest \
  docker://registry.internal.example.com/skopeo:latest

Two options are easy to miss in an air-gapped setting.

# Copy every image in a multi-architecture list
skopeo copy --all \
  docker://registry.access.redhat.com/ubi9/ubi:9.4 \
  dir:/media/transfer/images/ubi9

# Copy only the architectures you actually need
skopeo copy --multi-arch=linux/amd64,linux/arm64 \
  docker://quay.io/skopeo/stable:latest \
  docker://registry.internal.example.com/skopeo:latest

# Turn retries on (the default is no retries)
skopeo copy --retry-times 3 \
  docker://registry.access.redhat.com/ubi9/ubi:9.4 \
  dir:/media/transfer/images/ubi9

The documentation for --all, -a reads "If source-image refers to a list of images, instead of copying just the image which matches the current OS and architecture ... attempt to copy all of the images in the list, and the list itself". The key point is that the default behaviour is to copy only the one image matching the current system. Pull on an x86_64 laptop and transfer to an ARM server and the wrong thing goes in without a single error.

--multi-arch defaults to system and accepts all, index-only, and a comma-separated list of platforms. The documentation warns that index-only and platform lists produce "sparse manifest lists" and "usually fail unless the referenced per-architecture images are already present in the destination, or the target registry supports sparse indexes".

--retry-times is "The number of times to retry. By default, no retries are attempted". That the default is no retries at all is something you must handle deliberately in any transfer script that deals with large images.

skopeo sync — the right answer for air-gapped mirroring

If you move many images on a regular schedule, this is the tool. The description sentence in the official documentation is the theme of this whole series, word for word.

"Synchronize images between registry repositories and local directories. Synchronization is achieved by copying all the images found at source to destination - useful when synchronizing a local container registry mirror or for populating registries running inside of air-gapped environments."

What differs from copy is that the source and destination transports are given as separate flags. The source accepts docker, dir, and yaml; the destination accepts docker and dir.

# Connected side: every tag in the repository into a directory on the transfer medium
skopeo sync --src docker --dest dir \
  registry.access.redhat.com/ubi9/ubi /media/transfer/images

# Air-gapped side: from the medium into the internal registry
skopeo sync --src dir --dest docker \
  /media/transfer/images/ubi:9.4 registry.internal.example.com/base/

# Prevent path collisions when images with the same name come from several sources
skopeo sync --src docker --dest dir --scoped \
  registry.access.redhat.com/ubi9/ubi /media/transfer/images

# See what would move without actually moving anything
skopeo sync --src docker --dest dir --dry-run \
  registry.access.redhat.com/ubi9/ubi /media/transfer/images

For a docker source the documentation states that "If no image tag is specified, skopeo sync copies all the tags found in that repository". Leave the tag off and you get everything, so be careful when you are budgeting the size of a transfer.

--scoped is "Prefix images with the source image path, so that multiple images with the same name can be stored at destination". According to the example in the documentation, with this option the medium ends up with paths shaped like registry.example.com/busybox:1-glibc, and without it you get busybox:1-glibc. It is mandatory when you pull the same name from several registries.

--dry-run ("Run the sync without actually copying data to the destination") and --keep-going ("If any errors occur during copying of images, those errors are logged and the process continues syncing rest of the images and finally fails at the end") are especially useful when you automate transfer jobs.

Once the transfer list grows, managing it in YAML is the better option. The format the documentation defines looks like this.

# sync.yml — the list of images to bring in
registry.access.redhat.com:
    images:
        ubi9/ubi:
            - "9.4"
            - "latest"
        ubi9/nginx-124: []
    images-by-tag-regex:
        ubi9/python-311: ^3\.11-[0-9]+$
    tls-verify: true
quay.io:
    images:
        skopeo/stable:
            - latest
# Sync everything in one pass, following the YAML list
skopeo sync --src yaml --dest docker sync.yml registry.internal.example.com/mirror/

According to the documentation, an empty list means all tags, a digest can be written where a tag would go, images-by-tag-regex selects tags with a regular expression, and images-by-semver selects them with a semver constraint. Keep this file itself as a transfer record and it plays the same role the manifest from post 4 does.

skopeo sync has no --multi-arch. When you need multiple architectures, you use --all, -a.

Actually enforcing signatures

For image signing to mean anything in an air-gapped network, verification has to be enforced. That configuration lives in policy.json.

According to the documentation, the policy is read from $HOME/.config/containers/policy.json by default and from /etc/containers/policy.json when that does not exist. The structure has two parts, a global default and a per-transport transports section, and "The global default set of policy requirements is mandatory".

{
    "default": [{"type": "reject"}],
    "transports": {
        "docker": {
            "registry.internal.example.com": [
                {
                    "type": "signedBy",
                    "keyType": "GPGKeys",
                    "keyPath": "/etc/pki/containers/internal-signing-key.gpg"
                }
            ]
        },
        "dir": {
            "": [{"type": "insecureAcceptAnything"}]
        }
    }
}

The documentation defines the meaning of each requirement type this way. reject is "This requirement rejects every image, and every signature", insecureAcceptAnything is "This requirement accepts any image (but note that other requirements in the array still apply)", and signedBy is "This requirement requires an image to be signed using "simple signing" with an expected identity, or accepts a signature if it is using an expected identity and key". Per the documentation keyType currently supports only GPGKeys, and exactly one of keyPath, keyPaths, or keyData must be present.

The matching rule is worth knowing too. "If multiple policy requirements match a given image, only the requirements from the most specific match apply, the more general policy requirements definitions are ignored." Hang a strict policy on a broad scope and then carve out an exception on a narrow scope, and inside that narrow scope the broader policy does not apply.

Attaching the signature is the skopeo side of the job.

# Sign while pushing into the internal registry
skopeo copy --sign-by security@example.com \
  dir:/media/transfer/images/ubi9 \
  docker://registry.internal.example.com/base/ubi9:9.4

--sign-by is "Add a "simple signing" signature using that key ID for an image name corresponding to destination-image". If you need the sigstore approach, --sign-by-sigstore-private-key is there.

If your registry does not support storing signatures, you use lookaside signature storage. The registries.d documentation defines lookaside as "URL of the signature storage. This URL is used for reading existing signatures, and if lookaside-staging does not exist, also for adding or removing them", and lookaside-staging as "URL of the signature storage, used for editing it (adding or deleting signatures)".

# /etc/containers/registries.d/internal.yaml
docker:
    registry.internal.example.com:
        lookaside: http://sigstore.internal.example.com/signatures
        lookaside-staging: file:///srv/signatures-staging

The source comments in the documentation state that the sigstore and sigstore-staging keys are old names that are no longer documented, so when you write a new file, use the lookaside spelling.

Always specify the target architecture

This section is short but it accounts for a lot of failures.

# Pull with the architecture stated explicitly
podman pull --arch arm64 registry.access.redhat.com/ubi9/ubi:9.4

# OS and architecture in one go (cannot be used together with --arch or --os)
podman pull --platform linux/arm64 registry.access.redhat.com/ubi9/ubi:9.4

--arch is "Override the architecture, defaults to the host, of the image to be pulled", and --platform is "Specify the platform for selecting the image. (Conflicts with --arch and --os)". That the default is the host is where the whole problem starts.

--all-tags, -a carries a warning of its own. "IMPORTANT: When using the all-tags flag, Podman does not iterate over the search registries in the containers-registries.conf(5) but always uses docker.io for unqualified image names." Use this option with a short name while preparing an air-gapped transfer and you end up pointed at a registry you never intended.

It continues into Kubernetes

Everything above gets you as far as putting images into the air-gapped registry. Making an actual cluster use those images is a separate subject, and there are posts on this blog that already cover it.

The comparison of the package managers themselves, npm, uv, rpm, and brew compared, also helps put rpm in context.

Redistributing Red Hat content without a subscription may violate your agreement, so check your organization's licensing terms first. Red Hat container images are covered by their own separate terms of use as well, so confirm those before you build an internal registry mirror.

Closing — pick the tool to match the scale

For two or three images, podman save and podman load are enough. For dozens of images on a regular schedule, skopeo sync is the right answer, and skopeo copy sits between the two.

Whichever you choose, do not skip these two things: stating the architecture explicitly, and enforcing signature verification through policy. Leave out either one and the command still succeeds — the problem surfaces weeks later.

Commands and options were verified against the official documentation on 2026-08-15. How podman and skopeo are shipped differs across RHEL versions, so confirm the installation method against the documentation for the version you are running.

Try it yourself

Previous / next in the series

References