필사 모드: You Added Registry Instances and Availability Did Not Move — What zot Scale-Out Actually Sells

• Introduction — the illusion on the day you scaled to three instances
• The essence of a registry is not storage but naming
• The choice zot made — the disk is the OCI image layout
• Scale-out is sharding, not HA
• What each of the two modes gives up
• boltdb effectively pins the instance count to one
• The conditions under which digest pinning breaks in mirroring
• The actual size of the "lightweight single binary"
• References

Introduction — the illusion on the day you scaled to three instances

Suppose you were nervous about a structure where every deployment hangs off a single internal registry, so you scaled it to three instances. You put a load balancer in front, attached health checks, and three green lights come up on the dashboard.

Then the moment you take one down, docker pull fails entirely for certain repositories. Even though the other two are perfectly fine.

This is not a bug; it is the design. What you turned on is not high availability but sharding. And this distinction matters far more than performance numbers when choosing a registry, yet in most documentation it is lumped into the single phrase "scale out."

This post draws that distinction using the official documentation of zot as evidence. It is not an argument for or against choosing zot, but about what you should confirm in the documentation before moving on.

The essence of a registry is not storage but naming

Understanding a registry as an "image file server" leads you to misread its scaling design. What a registry actually manages is two kinds of name.

The first is the digest. In the form sha256:abc..., it is the hash of the blob content. Same content means same name, and same name means same content. There are no collisions here and no cache invalidation is needed.

The second is the tag. Something like myapp:latest, attached by a person and changed at any time to point at a different digest. This is the state, and every hard part of a distributed system lives here.

Once you understand this structure, the constraints of the scaling design become visible naturally. Blobs can be replicated anywhere in any number and it does not matter. But on what myapp:latest currently points to, the instances have to give the same answer. So registry clustering ultimately converges on the question of how to serialize tag updates.

The choice zot made — the disk is the OCI image layout

The design premise of zot appears in the first line of its documentation. On disk it uses the OCI Image Layout Specification as-is, and on the network it uses the OCI Distribution Specification as-is. In the documentation wording, any client that conforms to the Distribution Specification can read from and write to a zot registry.

Why this matters in practice is that it means copying the disk directory as-is gives you a valid image store. The character of migration and backup changes completely. With a registry that keeps the image structure in its own metadata database, you have to move that database along with it, and if versions differ you cannot move it at all.

Storage backends supported are the local filesystem (including remote filesystems mounted over NFS or FUSE), AWS S3, GCS, and Azure Blob Storage. The one exception spelled out in the documentation is that, because of the path separator, S3 is not supported on Windows.

Scale-out is sharding, not HA

Now to the main point. Read the zot scale-out documentation as it is written and it goes like this.

When a request comes in, the instance hashes the repository path, looks it up in a hash table, and finds the instance responsible for that repository. The hash function is SipHash, chosen for collision and preimage attack resistance. If it is not the one responsible, it forwards the request to the responsible instance and returns the response back in a proxy role.

The configuration looks like this.

"cluster": {
  "members": [
    "zot-server1:9000",
    "zot-server2:9000",
    "zot-server3:9000"
  ],
  "hashKey": "loremipsumdolors",
  "tls": {
    "cacert": "test/data/ca.crt"
  }
}

Note that members is a static list. There is no consensus protocol, no leader election, no membership gossip. Because every instance holds the same list and the same hashKey, they perform the same computation and arrive at the same answer. That is the elegant part of this design. It substitutes per-repository ownership for the tag-update serialization problem and removes consensus entirely.

The price is written right there in the documentation. This configuration is not self-healing, and if one instance dies, the repositories mapped to that instance are affected until the cluster is restarted. Contrasting it with high availability, the original says that an instance or storage going offline affects service availability.

In other words, three instances are throughput divided three ways, not fault tolerance multiplied three times.

What each of the two modes gives up

The zot documentation splits scale-out into two forms.

Compute-only scaling has every instance share a single S3-compatible storage and a distributed cache (Redis or DynamoDB). Local cache is not used. Because storage is shared, the instances come close to being stateless compute nodes.

Compute-plus-storage scaling gives each instance its own local storage. In exchange, the documentation states that the UI is not supported in this form.

And common to both forms, in a scale-out deployment CVE scanning is disabled and the trust extension is not supported because it requires a shared directory.

Why this list matters is that the reason an organization stands up its own registry is usually exactly "so we can see vulnerability scan results right in the registry." If turning on scaling turns that feature off, scanning has to move to the CI pipeline side. It is a fact you need to know before deciding to scale, and an awkward one to learn after an incident.

boltdb effectively pins the instance count to one

Turn on deduplication and zot keeps only one copy of a blob with the same content on disk and has multiple manifests reference it. On a local filesystem it is implemented with hard links. It enforces the configured state at startup: when enabled it deduplicates existing blobs, and when disabled it restores cloud storage blobs to their original state.

This deduplication requires metadata that remembers "which digest physically lives where," and what holds that is the cache driver.

• boltdb — the default for local storage. It is embedded as a file in the zot root directory. The documentation states explicitly that it does not provide concurrent write access, and therefore multiple zot instances cannot share it.
• DynamoDB — for remote storage. Requires AWS credentials and IAM permissions.
• Redis — the alternative for remote storage. Supports single-instance and cluster configurations and lets you set a key prefix.

Here is the quiet trap. Leave the defaults, bring up two instances, and mount the same NFS volume, and each will carry its own boltdb while touching the same directory. It is a combination the documentation forbids, but you cannot tell from looking at the config file alone. Remember that scaling out onto shared storage means changing the cache driver first.

The conditions under which digest pinning breaks in mirroring

zot can mirror an upstream registry, and there are two modes. Periodic polling sweeps upstream every pollInterval and copies matching images locally; on-demand fetches and caches when a request comes in. The two modes can also be used together.

{
  "urls": ["https://registry1:5000"],
  "onDemand": false,
  "pollInterval": "6h",
  "content": [{ "prefix": "/repo", "destination": "/local" }]
}

The constraints the documentation points out hurt particularly in practice.

First, Docker Hub does not support catalog listing, so you have to use on-demand only. Periodic polling needs to receive a list of "what is there" in order to work, and that is unavailable.

Second, converting Docker images to OCI format breaks digest-pinned pulls and signature verification. This is because conversion changes the manifest bytes and therefore the hash. To preserve the original digest you have to turn on preserveDigest, which requires putting docker2s2 into http.compat.

If your deployment pipeline pins images by digest rather than by tag — and it should — you have to check this condition the moment you introduce a mirror.

The actual size of the "lightweight single binary"

The phrase "a single binary, statically built with no dependencies" always accompanies zot introductions. The first part is true. No separate runtime or database process is needed, and it runs without privilege escalation.

That said, the word "lightweight" is better checked before use. The Linux asset sizes for GitHub release v2.1.20 (published 4 August 2026) are as follows.

The full build is large because it puts every feature, including vulnerability scanning and the web UI, into one binary. The documentation also announces that it provides two variants, full and minimal, describing minimal as a build with features stripped out for security.

What you have to judge here is not the size itself but what is inside it. Put the scanner inside the registry binary and you have to restart the registry to update the scanner. If you do not want that coupling, you are better off using minimal and pulling scanning outside. Either way, it should be a conscious choice.

References

• zot official documentation v2.1.20 — scale-out clustering
• zot official documentation v2.1.20 — storage and deduplication
• zot official documentation v2.1.20 — mirroring
• project-zot/zot — GitHub, Apache-2.0 (source for the v2.1.20 release asset sizes)
• OCI Image Layout Specification