Scanning the production image with Trivy — 768 findings, and the real work was three packages and one Dockerfile line

Why scan

One Python backend runs this whole site — login, blog, language learning, lab grading. Dependencies are pinned with hashes in requirements.lock, so what is inside is clear; whether any of it has known vulnerabilities had never been measured. I ran Trivy over two targets.

• The repository — Python dependencies in requirements.txt, Kubernetes manifests, secret strings
• The production image — the filesystem of the container that is actually running (OS packages + installed Python packages)

The second one matters most. The repository alone misses OS packages; the image alone does not show where to fix things.

Scanning inside the production image

The registry (Harbor) requires authentication and the pods carry no pull secret (the nodes hold the credentials). So instead of pulling the image out of the registry, I ran the production image itself as a container, copied the Trivy binary in with an init container, and scanned the root filesystem from inside.

initContainers:
  - name: get-trivy
    image: docker.io/aquasec/trivy:0.58.0
    command: ["cp", "/usr/local/bin/trivy", "/tools/trivy"]
    volumeMounts: [{ name: tools, mountPath: /tools }]
containers:
  - name: scan
    image: 192.168.219.202/labhub/backend:build-634      # same image as production
    command: ["sh", "-c", "/tools/trivy rootfs --scanners vuln --format json /"]

No credentials needed, and the node already has the image so it is fast. My first attempt set runAsUser: 0, which collided with the CronJob template's runAsNonRoot policy and left the pod Pending — removing it fixed that. The root filesystem is readable as an ordinary user.

First result: 935 findings — 167 of them were the scanner

{'LOW': 207, 'HIGH': 348, 'MEDIUM': 340, 'CRITICAL': 12, 'UNKNOWN': 28}
by target: debian 13.6 → 720 · Python → 48 · tools/trivy → 167

Look at the third line. tools/trivy, 167 findings — the dependencies of the Trivy binary (a Go program) I had copied in with the init container. The scanner was inside the scan target, so it counted itself. Four of the twelve CRITICALs (go-git, x/crypto, grpc, the Go stdlib) were exactly that. Excluding it:

without the scanner: {'LOW': 197, 'HIGH': 275, 'MEDIUM': 264, 'CRITICAL': 8, 'UNKNOWN': 24}  = 768
with a fixed version available: {'CRITICAL': 1, 'HIGH': 53, 'MEDIUM': 55, 'LOW': 32}          = 141

768 is a frightening number, but action only becomes clear once you split by "is there a fixed version?" 141 can be acted on; the other 627 have no fix in Debian yet (including 7 CRITICALs — glib, mbedtls, libxml2, perl-base).

The 141 fixable ones come down to three groups

authlib — signature verification bypass in login

The single CRITICAL comes first. CVE-2026-27962, JWK header injection in JWS. If a token is verified with key=None, the library takes the key from the token's own jwk header to check the signature. An attacker signs with their own key, puts that key in the header, and it passes. Fixed in 1.6.9.

I checked whether this site takes that path. authlib is called in exactly one place:

token = await oauth.google.authorize_access_token(request)

The starlette client verifies the ID token against keys fetched from Google's JWKS, so it does not hit the key=None path directly. There was still no reason to keep the library at that version, so it was bumped.

OS packages — the other side of digest pinning

util-linux and openssl had fixes in Debian. Why were they missing from the image? Because the Dockerfile started like this:

FROM python:3.12-slim@sha256:2c941e86…      # pinned by digest
RUN apt-get update && apt-get install -y --no-install-recommends fonts-nanum ffmpeg

Pinning by digest makes builds reproducible, but OS security fixes never arrive. apt-get install adds new packages; it does not upgrade the ones already there. Two lines changed — the digest was bumped to the current one and apt-get upgrade -y was added, so every build picks up the fixes available at that moment.

Measuring again after the fix

I built the same Dockerfile locally and scanned it the same way.

The remaining 3 HIGH are fixed only in starlette 1.x and are handled separately together with a fastapi upgrade. The remaining 7 CRITICALs have no Debian fix yet, so this image cannot reach them — that has to be recorded as "waiting", not "fixed".

How do you know the upgrades are harmless

Bumping three packages and regenerating the lock can move other things too. Regenerated with the same python:3.12 + pip-tools 7.5.1 as the production build, only those three packages changed.

I installed that lock with --require-hashes in a container and ran the whole suite of 2,014 tests. Four failed. Stopping there would read as "the upgrade broke something". So I installed the original lock in the same container and ran the same four again — they fail identically. They are tests that need the local stack (running services, hooks, grading tools). Without a control group, "unrelated" is a guess.

Pillow skips a major version, so its call sites were checked separately: Image.open, ImageDraw, ImageFont.truetype, ImageCms, ImageOps — all still present in 12.x.

Secret detection: 12 hits, 0 real leaks

Trivy's secret scanner reported 12 hits. I opened every one.

All 12 are teaching material or examples. But "it's lab material, so it's fine" is a judgment you can only make after opening the file. Skim the list and a real one could hide among them.

Takeaways

• Scan both the repository and the production image. The repo misses the OS; the image hides where to fix.
• If you put the scanner inside the container, subtract the scanner itself from the results. That was 167 findings.
• Do not be scared by the count; split by "is there a fixed version?" 768 became 141, and the real work was three packages and two Dockerfile lines.
• A digest-pinned base image never receives OS fixes unless the build runs apt-get upgrade.
• When tests fail after an upgrade, run the same tests on the original version as a control.
• Never judge a secret-detection hit without opening the file.