Secrets Management — Why a .env File Is Not Enough, the Paths Environment Variables Leak Through, and How to Design Rotation

• Introduction — .env Is in .gitignore, So Is It Safe?
• The Paths Environment Variables Actually Leak Through
• If the Secret Is Already Committed — Revocation First, History Second
• Comparing Storage Options — and the base64 in a Kubernetes Secret Is Not Encryption
• A Structure That Removes Static Keys — Short-Lived Credentials and OIDC Federation
• The Design That Makes Rotation Possible — Two Valid Keys
• Detection — What Pre-commit Hooks and Repository Scanning Cannot Do
• Wrapping Up — Not Whether You Hid It, but How Many Minutes It Takes to Change It

Introduction — .env Is in .gitignore, So Is It Safe?

The standard advice about secrets management is short. Do not hardcode them, move them into environment variables, and put the .env file in .gitignore. That is where most documents stop.

The problem is that this advice blocks exactly one threat: plaintext landing in the source code repository. Read actual incident reports, though, and the leak paths are far more varied. Process memory, crash reports, CI logs, container image layers, APM dashboards, and misconfigured debug endpoints.

The more important problem comes next. When you learn a secret has leaked, can you invalidate that key and swap in a new one within minutes? In most teams, the answer to this question is "we do not know." Nobody knows how many copies are sitting where, and nobody knows what breaks when the value changes.

This post covers two things: how far environment variables actually leak, and how to build a structure that lets you respond within five minutes when they do.

The Paths Environment Variables Actually Leak Through

An environment variable is a process attribute. It is not a file — it is a string array the kernel holds per process, and on Linux you can read it like a file.

ps -eo pid,user,comm | grep -E 'node|python'

   2841 app      node
   3102 app      python3

# With the same UID or as root, read the whole process environment
tr '\0' '\n' < /proc/2841/environ | grep -iE 'key|token|secret|password'

DATABASE_URL=postgres://app:pr0d-Db-Pass@db.internal:5432/app
STRIPE_SECRET_KEY=sk_live_51NxbQ2Lk9vHc0pMz7RtYaWq
JWT_SIGNING_KEY=8f2a1c9d4e7b6a305f18c2d9e0b7a4f1

The important fact here is that this file stays readable for as long as the process is alive. Deleting the .env file after loading it changes nothing. The values are already in the kernel.

Inside a container it is no different. A sidecar in the same pod, anyone with node access, and a debug container all see the same thing.

kubectl debug -it deploy/payments --image=busybox --target=app -- sh
# From inside the debug container
tr '\0' '\n' < /proc/1/environ

The second path is child processes. Environment variables are inherited. When the application runs an external tool such as an image converter or a backup script, the moment that tool dumps the whole environment into its own log, the secret flows into the log collector.

The third is crash reports and debug pages. Here is an antipattern you see in the wild all the time.

// Code you must never write
process.on('uncaughtException', (err) => {
  logger.error({ err, env: process.env }, 'fatal error, dumping context')
  process.exit(1)
})

That single line stores every secret permanently in the log index. Log collection systems usually have broader access than the application does. Framework debug pages carry the same risk. Django debug mode renders configuration values on the exception page, and the Werkzeug debugger even opens an interactive console. Cases of these being left on in staging and exposed to the internet keep recurring.

The fourth is CI logs. Even with secret masking, a transformed value gets through.

# Masking only hides strings that match exactly
set -x
curl -H "Authorization: Bearer $API_TOKEN" https://api.example.com/deploy
echo "$API_TOKEN" | base64        # base64 encoding walks straight past masking

+ curl -H 'Authorization: Bearer ***' https://api.example.com/deploy
+ base64
c2tfbGl2ZV81MU54YlEyTGs5dkhjMHBNejdSdFlhV3E=

The last is container images. A value passed as a build argument stays in the image history verbatim.

docker build --build-arg NPM_TOKEN=npm_9f3aQ2v8Lx -t myapp:1.4.2 .
docker history --no-trunc myapp:1.4.2 | grep -o 'NPM_TOKEN=[A-Za-z0-9_]*'

NPM_TOKEN=npm_9f3aQ2v8Lx

Deleting the file in a RUN step leaves it in the earlier layer. Once pushed to a registry, everyone who pulls that image can read it. Build-time secrets must be passed by mounting.

# syntax=docker/dockerfile:1
FROM node:22-slim
RUN --mount=type=secret,id=npmrc,target=/root/.npmrc \
    npm ci --omit=dev

docker build --secret id=npmrc,src=$HOME/.npmrc -t myapp:1.4.2 .

If the Secret Is Already Committed — Revocation First, History Second

The most common reaction to finding a key in git history is to rewrite the history. That order is wrong. The correct order is revoke, investigate the impact, then clean up the history.

The reason is simple. A credential pushed to a public repository is collected by automated scanners within seconds to minutes. No matter how cleanly you scrub the history, a value that has already been copied cannot be taken back. And rewriting history alone leaves all of the following behind.

Forked repositories, the local clones of developers who already pulled, the pull request references the platform retains (often still reachable if you know the commit hash), caches held by search engines and code search services, and CI caches and artifacts. In other words, rewriting history is not an action that undoes the leak; it is hygiene work that reduces recurrence.

The first priority is invalidation.

# Create and deploy the new key first, then deactivate and delete the old one
aws iam create-access-key --user-name ci-deployer
aws iam update-access-key --access-key-id AKIAIOSFODNN7EXAMPLE \
  --status Inactive --user-name ci-deployer
aws iam delete-access-key --access-key-id AKIAIOSFODNN7EXAMPLE --user-name ci-deployer

The second priority is investigating the impact. Confirm since when and from where that key was used.

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=AKIAIOSFODNN7EXAMPLE \
  --start-time 2026-07-01T00:00:00Z \
  --query 'Events[].[EventTime,EventName,Username,CloudTrailEvent]' --output text | head -20

If you see unfamiliar IPs, regions you never use, or unexpected API calls, you have to switch to breach response.

The third priority is cleaning up the history. First find which commits it went into.

git log --all --oneline -S 'AKIAIOSFODNN7EXAMPLE'

7c19ae4 chore: add deploy script
2f80b13 fix: correct region in deploy script

For the cleanup, git filter-repo is the standard. filter-branch is slow and error-prone, and is no longer recommended.

cat > replacements.txt <<'EOF'
AKIAIOSFODNN7EXAMPLE==>REDACTED
pr0d-Db-Pass==>REDACTED
EOF

git filter-repo --replace-text replacements.txt --force
git push --force-with-lease --all
git push --force-with-lease --tags

After that you have to tell every collaborator to re-clone. Pushing from an existing clone brings the deleted commits back.

Comparing Storage Options — and the base64 in a Kubernetes Secret Is Not Encryption

The criterion for choosing a secrets management tool is not "is it encrypted." It is how many exposure paths there are, how many minutes revocation and replacement take, and whether you can tell who read it and when.

It is not that lower is better; it is that the further down you go, the shorter your response time during an incident. The row practitioners misread most often in this table is the Kubernetes Secret, so let us start there.

Look at a Secret manifest and the values are base64 encoded, which makes them look encrypted. base64 is an encoding, not encryption. There is no key, and reversing it takes one command.

kubectl get secret app-db -o jsonpath='{.data.password}' | base64 -d; echo

pr0d-Db-Pass

With the default configuration, Secrets are stored in etcd in plaintext. Anyone who obtains an etcd backup file, a snapshot, or a disk image reads every Secret. Let us confirm it.

ETCDCTL_API=3 etcdctl \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets/default/app-db | hexdump -C | head -4

00000000  2f 72 65 67 69 73 74 72  79 2f 73 65 63 72 65 74  |/registry/secret|
00000010  73 2f 64 65 66 61 75 6c  74 2f 61 70 70 2d 64 62  |s/default/app-db|
00000020  0a 6b 38 73 00 0a 0c 0a  02 76 31 12 06 53 65 63  |.k8s.....v1..Sec|
00000030  72 65 74 12 8e 01 0a 6f  0a 06 61 70 70 2d 64 62  |ret....o..app-db|

The plaintext is right there. Turn on encryption at rest and the value gets a provider prefix while the body becomes unreadable.

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - kms:
          apiVersion: v2
          name: cloud-kms
          endpoint: unix:///var/run/kmsplugin/socket.sock
      - identity: {}

00000000  2f 72 65 67 69 73 74 72  79 2f 73 65 63 72 65 74  |/registry/secret|
00000020  0a 6b 38 73 3a 65 6e 63  3a 6b 6d 73 3a 76 32 3a  |.k8s:enc:kms:v2:|

identity must always come last in the list. Put it first and you are back to plaintext storage. Changing the configuration does not touch existing Secrets, which stay in plaintext until they are rewritten, so you have to refresh all of them once.

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

The second misconception is RBAC. Anyone with get secrets permission in a namespace reads every Secret in that namespace. It is extremely common for edit permission handed to a developer as a convenience to be, in practice, permission to read production credentials.

kubectl auth can-i get secrets --namespace payments --as dev@example.com

yes

A compromise used often in practice is an external secrets operator. The real value lives in the secrets manager, and only a reference is committed to the cluster.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: app-db
  namespace: payments
spec:
  refreshInterval: 15m
  secretStoreRef:
    name: aws-secretsmanager
    kind: ClusterSecretStore
  target:
    name: app-db
  data:
    - secretKey: password
      remoteRef:
        key: prod/payments/db
        property: password

This manifest is safe to commit to git, because it contains no value. That said, the operator materializes the value into a cluster Secret, so the RBAC and encryption-at-rest problems above remain exactly as they were. Using a secrets manager does not exempt you from cluster-side hygiene.

A Structure That Removes Static Keys — Short-Lived Credentials and OIDC Federation

Everything discussed so far has been about "where to put the static key." A better answer is not to create static keys at all.

The old way to access the cloud from CI was to create an access key and paste it into a CI variable. That key never expires, you cannot tell who copied it, and rotation depends on a human remembering. Today the runner exchanges an OIDC token it was issued for cloud credentials.

permissions:
  id-token: write
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/gha-deploy
          aws-region: ap-northeast-2
      - run: aws sts get-caller-identity

{
    "UserId": "AROAY3EXAMPLEID:GitHubActions",
    "Account": "123456789012",
    "Arn": "arn:aws:sts::123456789012:assumed-role/gha-deploy/GitHubActions"
}

There is no stored key. The credentials are valid only for the duration of the job run.

The configuration mistake that comes up most often here is the condition clause in the trust policy. The following is dangerous.

{
  "Condition": {
    "StringLike": {
      "token.actions.githubusercontent.com:sub": "repo:acme/*"
    }
  }
}

This condition lets any repository, any branch, any pull request in the organization assume the role. It creates a path for a PR workflow raised from a fork to obtain the production deploy role. You have to pin it down to the branch or the environment.

{
  "Condition": {
    "StringEquals": {
      "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
      "token.actions.githubusercontent.com:sub": "repo:acme/payments:environment:production"
    }
  }
}

Kubernetes pods use the same structure. A projected service account token can specify an audience and an expiration, and the cloud STS verifies it.

spec:
  serviceAccountName: payments
  volumes:
    - name: aws-token
      projected:
        sources:
          - serviceAccountToken:
              audience: sts.amazonaws.com
              expirationSeconds: 3600
              path: token

Database credentials can be created dynamically too. The Vault database secrets engine creates a user at request time and deletes it when the lease ends.

vault read database/creds/payments-readonly

Key                Value
---                -----
lease_id           database/creds/payments-readonly/9tKq2xL0
lease_duration     1h
lease_renewable    true
password           A1a-3mQx9Zr7Kt2Vb0Ns
username           v-token-payments-readonly-8Xk1qP

This credential disappears on its own an hour later. Even if it leaks the window is narrow, and the username stays in the logs so you can trace which request it was issued for. Shrinking the lifetime to one hour is almost always more effective than straining to hide a static key perfectly.

The Design That Makes Rotation Possible — Two Valid Keys

Rotation usually fails for reasons that have nothing to do with storage. It fails because the code assumes "there is exactly one valid key." Under that assumption, swapping a key necessarily creates a momentary window of mismatch, and so nobody ever rotates.

The fix is to separate verification from signing. Always sign with a single current key, and verify against the entire set of valid keys.

Take webhook signature verification as an example. The fragile implementation looks like this.

import hmac, hashlib, os

SECRET = os.environ["WEBHOOK_SECRET"]

def verify(body: bytes, signature: str) -> bool:
    expected = hmac.new(SECRET.encode(), body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, signature)

The moment you change the key, every request signed with the previous key is rejected. Since there is no way to change the sender and the receiver atomically at the same instant, replacement is effectively impossible.

Turn it into a set and replacement becomes possible.

import hmac, hashlib, os

# Comma-separated list of valid keys. The first one is the current signing key.
KEYS = [k.strip() for k in os.environ["WEBHOOK_SECRETS"].split(",") if k.strip()]

def sign(body: bytes) -> str:
    return hmac.new(KEYS[0].encode(), body, hashlib.sha256).hexdigest()

def verify(body: bytes, signature: str) -> bool:
    for key in KEYS:
        expected = hmac.new(key.encode(), body, hashlib.sha256).hexdigest()
        if hmac.compare_digest(expected, signature):
            return True
    return False

Now the replacement procedure works with zero downtime.

Step 1  Append the new key to the valid list and deploy   → sign with old key, verify both
Step 2  Move the new key to the front and deploy          → sign with new key, verify both
Step 3  Confirm requests signed with the old key are gone → confirm via metrics
Step 4  Remove the old key from the list and deploy       → only the new key is valid
Step 5  Revoke the old key at the issuer

To see step 3 with your own eyes, you have to record which key matched as a metric.

def verify(body: bytes, signature: str) -> bool:
    for index, key in enumerate(KEYS):
        expected = hmac.new(key.encode(), body, hashlib.sha256).hexdigest()
        if hmac.compare_digest(expected, signature):
            metrics.increment("webhook.signature.verified", tags=[f"key_index:{index}"])
            return True
    metrics.increment("webhook.signature.failed")
    return False

When the counter for key_index:1 reaches zero, it is safe to delete the old key. Rotation becomes something you drive by observation rather than guesswork.

For JWT the same principle is implemented with the kid header and JWKS. The issuer publishes the new key in JWKS first, and only changes the signing key after verifiers have picked it up. For database accounts, alternating between two users is the standard approach. Refresh app_a on even rounds and app_b on odd rounds, and have the application read the currently active user from the secrets manager. At no moment is there fewer than one valid credential, so there is no outage.

The core point is this. Rotation is not an operational procedure but a design property. If the code assumes a single key, no secrets manager, however good, will make actual replacement happen.

Detection — What Pre-commit Hooks and Repository Scanning Cannot Do

The last layer is detection. The tooling is mature.

# Scan the working tree and the whole history
gitleaks detect --source . --redact --report-format json --report-path leaks.json

# Check only the staged changes right before a commit
gitleaks protect --staged --redact

Finding:     STRIPE_SECRET_KEY=sk_live_REDACTED
Secret:      REDACTED
RuleID:      stripe-access-token
File:        deploy/staging.env
Line:        14
Commit:      7c19ae4a2f3b18c0d95e7f4a6b2c1d80e3f9a5b6

Wire it up as a pre-commit hook and it gets caught on the developer machine.

repos:
  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.18.4
    hooks:
      - id: gitleaks

Scanners with verification cut the noise dramatically. They call the real API with the value they found and report only the keys that are alive.

trufflehog git file://. --only-verified --json | jq -r '.DetectorName + " " + .Verified'

That is what the tools do for you. Here are the limits.

A pre-commit hook is local configuration, so it gets bypassed. One git commit --no-verify is all it takes, and a new joiner spends days without the hook installed. Real enforcement comes from server-side push protection. Treat the hook as a convenience and the server check as the control.

Detection rules only work well on values with a distinctive shape. An access key beginning with AKIA or a token with the sk_live_ prefix is caught exactly, but a shapeless value like a database password or the API key of a private service depends on entropy estimation. Entropy rules produce many false positives, which pushes you to raise the threshold, and then you miss the real passwords.

And scanners only look at code. Copies pasted into wikis, issue attachments, chat logs, spreadsheets and local notes are out of scope. In real incidents, the leak point is quite often not the repository at all.

So there is one thing to check before investing in detection: assuming this secret is exposed, how many minutes do revocation and replacement take? If that number is large, no amount of dense detection will reduce the loss during an incident. Conversely, if that number is small, late detection still means limited damage. The priority is always to shorten response time first.

Wrapping Up — Not Whether You Hid It, but How Many Minutes It Takes to Change It

Three sentences to summarize.

An environment variable is a delivery mechanism, not a storage location. The process environment is readable from the same host, and it keeps leaking into logs, crash reports and image layers. .gitignore blocks exactly one of those paths.

When a secret is exposed, the order is revoke, investigate, clean up the history. Do it the other way around and you spend your time on cleanup while the already copied value stays live.

And the surest improvement is to reduce static keys. Move CI to OIDC federation, move databases to dynamic credentials, and make whatever static keys remain replaceable at any time through a valid-key-set design.

One question is enough to measure a team's secrets management maturity. Assuming this credential were published on the internet right now, how many minutes would it take to revoke it, replace it with a new value, and bring the service back to normal? If the answer is measured in hours, you should build the rotation path before buying more tools.