Chaos and Order

💡 왼쪽 원문을 읽으면서 오른쪽에 따라 써보세요. Tab 키로 힌트를 받을 수 있습니다.

Introduction — the browser shows a padlock but curl fails

You deployed a newly issued certificate. Opening it in a browser shows a healthy padlock and the certificate details look fine. Yet calling that endpoint from the backend fails like this.

curl -sS https://api.example.com/v1/health

curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

The most common conclusion at this point is "curl does not know the latest CA". So people update ca-certificates, and when that does not help they add -k or turn validation off in the application. That is a measure that covers the cause, and it usually pushes a misconfigured server into production.

The real cause is almost always one thing. The server is not sending the intermediate certificate, and the browser succeeded by filling that gap on its own. This post explains that mechanism precisely and organises things so you can name the cause from the error message alone.

Reading the TLS 1.3 handshake as a sequence of messages

To debug, you need to know which messages travel in which order. TLS 1.3 completes in a single round trip.

Client                                                  Server
  |                                                      |
  |-- ClientHello ------------------------------------->  |
  |     supported_versions: TLS 1.3                       |
  |     key_share: x25519 public key                      |
  |     server_name: api.example.com   (SNI, plaintext)   |
  |     application_layer_protocol_negotiation: h2, http/1.1
  |                                                      |
  |  <------------------------------------ ServerHello -- |
  |     key_share: x25519 public key                      |
  |     ---- everything from here on is encrypted ----    |
  |  <------------------------------- EncryptedExtensions |
  |  <----------------------------- CertificateRequest(*) |
  |  <------------------------------------- Certificate   |
  |  <------------------------------- CertificateVerify   |
  |  <------------------------------------------ Finished |
  |                                                      |
  |-- Certificate(*) ---------------------------------->  |
  |-- CertificateVerify(*) ---------------------------->  |
  |-- Finished ---------------------------------------->  |
  |-- Application Data (first HTTP request) ----------->  |
  |                                                      |
  (*) appears only when mTLS is required

Three facts with direct operational impact come out of this.

First, every message after ServerHello is encrypted. In TLS 1.2 you could pull the server certificate straight out of a packet capture, but in TLS 1.3 you cannot. If Wireshark does not show you the certificate, that does not mean the server failed to send one. To inspect the certificate you must do it from the client side, with a tool such as openssl s_client.

Second, the SNI in ClientHello is still plaintext. Even with everything else encrypted, the hostname you are connecting to is exposed. This is part of why Encrypted Client Hello came along. From a debugging perspective it is actually useful: you can always read the SNI out of a packet capture.

Third, the client sends Finished and the first request back to back. The client considers the handshake finished before the server has validated the client certificate. That is why an mTLS authentication failure appears not as a handshake error but as a connection drop immediately after the first request goes out. This difference is the core of what makes mTLS debugging hard.

Without SNI the server hands you whatever certificate it likes

Hundreds of sites sitting on a single IP and port is the standard arrangement these days. From the TCP connection alone the server cannot know which certificate to present. The SNI extension in ClientHello is what tells it.

Connect without SNI and the server returns the certificate of the default virtual host.

# Connect straight to the IP with SNI omitted
openssl s_client -connect 203.0.113.40:443 -noservername </dev/null 2>/dev/null \
  | openssl x509 -noout -subject

subject=CN = default.vhost.example.net

# Specify SNI and the correct certificate comes back
openssl s_client -connect 203.0.113.40:443 -servername api.example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -subject

subject=CN = api.example.com

The most frequent mistake with openssl s_client lives right here. Use only -connect and omit -servername, and no SNI is transmitted, so you end up looking at a different certificate from the real service and drawing the wrong conclusion. Recent OpenSSL sets the hostname from -connect as SNI automatically, but not when you connect by IP or use an older version. Making a habit of always specifying -servername is the safe path.

Clients that do not send SNI really do exist. Very old Java clients, some embedded devices and certain monitoring agents behave this way. If only such clients produce certificate errors, the cause is not the certificate but the missing SNI. What you choose as the default virtual host on the server side becomes the countermeasure.

It is also worth knowing how to check SNI in a capture.

sudo tcpdump -nn -i any -A 'tcp port 443 and tcp[((tcp[12:1] & 0xf0) >> 2)] = 0x16' | head -40

Chain validation and the missing intermediate — where the browser and curl diverge

Validation starts at the leaf certificate and climbs the issuer links, succeeding when it reaches a root held in the local trust store. The responsibility of the server is to send every intermediate certificate except the root.

A healthy chain looks like this.

openssl s_client -connect api.example.com:443 -servername api.example.com </dev/null 2>/dev/null \
  | sed -n '/Certificate chain/,/---/p'

Certificate chain
 0 s:CN = api.example.com
   i:C = US, O = Let's Encrypt, CN = R11
 1 s:C = US, O = Let's Encrypt, CN = R11
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
---

A server missing the intermediate certificate comes out like this.

Certificate chain
 0 s:CN = api.example.com
   i:C = US, O = Let's Encrypt, CN = R11
---
...
Verify return code: 20 (unable to get local issuer certificate)

There is a 0 but no 1. The client has to find the issuer named R11 in the trust store, but the trust store holds only the root, ISRG Root X1, and not the intermediate R11. So the chain breaks.

Then why does the browser succeed? This is the crux.

Inside the leaf certificate is a URL from which the issuer certificate can be downloaded.

openssl s_client -connect api.example.com:443 -servername api.example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -text | grep -A3 'Authority Information Access'

            Authority Information Access:
                OCSP - URI:http://r11.o.lencr.org
                CA Issuers - URI:http://r11.i.lencr.org/

Chrome, Safari and Edge, when they cannot find the issuer during validation, send an HTTP request to that CA Issuers URL and download the intermediate certificate themselves. This is called AIA fetching, or AIA chasing. Firefox does it slightly differently: it preloads known intermediate certificates and caches any intermediate it has seen once.

OpenSSL, by contrast, does not follow AIA by default. curl, wget, Python requests, Go and most of Node fall into this category. Java also does not follow it by default and requires the com.sun.security.enableAIAcaIssuers property to be turned on.

Which gives us the following proposition. If it works in the browser but fails in curl, it is the server that is wrong, not the client. The browser is merely papering over the server mistake. On top of that, AIA fetching is a network request, so every first connection spends an extra tens to hundreds of milliseconds. That turns into user-visible latency too.

Two commands are all it takes to confirm.

# Check whether the chain the server sends validates on its own (system default trust store)
openssl s_client -connect api.example.com:443 -servername api.example.com \
  -verify_return_error </dev/null 2>&1 | tail -3

# Dump the chain to see how many certificates there are
openssl s_client -connect api.example.com:443 -servername api.example.com -showcerts </dev/null 2>/dev/null \
  | grep -c 'BEGIN CERTIFICATE'

Verify return code: 0 (ok)
2

The fix is server configuration. The nginx ssl_certificate directive must point at a file (fullchain) with the leaf certificate and the intermediate certificate concatenated in order. Apache takes fullchain in SSLCertificateFile in recent versions, while older versions need SSLCertificateChainFile specified separately. On a load balancer, check that you did not leave the chain field empty on the certificate registration screen.

An error message dictionary

Reading the message accurately determines most of the cause on its own.

Error message	Actual cause	Command to confirm	Action
unable to get local issuer certificate	The server did not send the intermediate, or no root locally	Count the s_client chain, check the trust store exists	Deploy fullchain on the server, or install ca-certificates
certificate has expired	notAfter has passed, or the client clock is off	openssl x509 -noout -dates, check date	Renew, or synchronise NTP
certificate is not yet valid	Before notBefore, effectively a client clock problem	date, timedatectl	Synchronise the clock
hostname mismatch, no alternative subject name	The name you connected to is not in the certificate SAN list	openssl x509 -noout -ext subjectAltName	Add the name to SAN, or connect with the right SNI
self signed certificate in certificate chain	An internal CA or an intercepting proxy sits in the middle	Check the issuer at the top of the chain	Register the internal root in the trust store
self signed certificate	The leaf certificate itself is self-signed	Check whether issuer and subject are the same	Get a properly issued certificate
tlsv1 alert protocol version	Client and server minimum protocol versions do not overlap	Compare with -tls1_2 and -tls1_3 given to s_client separately	Adjust server configuration or update the client
sslv3 alert handshake failure	No common cipher suite, or no client certificate under mTLS	openssl ciphers, give -cert and -key to s_client	Align cipher suites or present a client certificate
certificate required (alert 116)	Under TLS 1.3 mTLS the client sent an empty certificate	Server logs, give -cert and -key to s_client	Configure the client certificate

Two misunderstandings need correcting.

On hostname mismatch, modern clients do not look at the CN field at all and check only subjectAltName. Concluding that things are fine because the domain is in CN has been wrong since 2017. Chrome removed the CN fallback in version 58, and OpenSSL and Go did the same. You must check SAN.

openssl s_client -connect api.example.com:443 -servername api.example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -ext subjectAltName

X509v3 Subject Alternative Name:
    DNS:api.example.com, DNS:api-internal.example.com

certificate has expired is frequently misdiagnosed too. Quite often it is not the server certificate but the client clock that is wrong. Embedded devices without a clock, VMs booted after a long shutdown, and containers whose time drifted from the host are the classic cases. Making a habit of checking the expiry date and the current time together saves time.

openssl s_client -connect api.example.com:443 -servername api.example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -dates
date -u

notBefore=Jun  2 08:14:31 2026 GMT
notAfter=Aug 31 08:14:30 2026 GMT
Sun Jul 26 05:11:20 UTC 2026

The order to check things with openssl s_client

It helps to memorise one command that shows everything at once.

HOST=api.example.com
PORT=443

openssl s_client -connect "${HOST}:${PORT}" -servername "${HOST}" \
  -showcerts -status -alpn h2,http/1.1 </dev/null 2>&1 | head -60

Read the key output like this.

CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R11
verify return:1
depth=0 CN = api.example.com
verify return:1
---
Certificate chain
 0 s:CN = api.example.com
   i:C = US, O = Let's Encrypt, CN = R11
 1 s:C = US, O = Let's Encrypt, CN = R11
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
---
ALPN protocol: h2
OCSP response: no response sent
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Verify return code: 0 (ok)

Verify return code: 0 (ok) is the final verdict. The verify lines that come down from the highest depth tell you at which step the chain broke.

Splitting the test by protocol is useful too.

openssl s_client -connect api.example.com:443 -servername api.example.com -tls1_3 </dev/null 2>&1 | grep -E 'New,|Cipher'
openssl s_client -connect api.example.com:443 -servername api.example.com -tls1_2 </dev/null 2>&1 | grep -E 'New,|Cipher'

New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
New, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256

Protocols that use STARTTLS, such as mail or databases, need an option added.

openssl s_client -connect smtp.example.com:587 -starttls smtp -servername smtp.example.com </dev/null
openssl s_client -connect db.example.com:5432 -starttls postgres </dev/null

When looking with curl, the following combination carries the most information.

curl -v --resolve api.example.com:443:203.0.113.40 https://api.example.com/v1/health 2>&1 | grep -E '^\*'

* Server certificate:
*  subject: CN=api.example.com
*  start date: Jun  2 08:14:31 2026 GMT
*  expire date: Aug 31 08:14:30 2026 GMT
*  subjectAltName: host "api.example.com" matched cert's "api.example.com"
*  issuer: C=US; O=Let's Encrypt; CN=R11
*  SSL certificate verify ok.

--resolve bypasses DNS and connects to a specific IP while keeping SNI and the Host header on the original name. It is essential when verifying individual nodes behind a load balancer one by one.

Trust stores, containers, and where mTLS fails

The other axis on which validation fails is the client-side trust store. Paths and refresh commands differ per distribution.

# Debian, Ubuntu
ls -l /etc/ssl/certs/ca-certificates.crt
cp corp-root.crt /usr/local/share/ca-certificates/corp-root.crt
update-ca-certificates

# RHEL, Rocky, Fedora
ls -l /etc/pki/tls/certs/ca-bundle.crt
cp corp-root.crt /etc/pki/ca-trust/source/anchors/
update-ca-trust extract

# Alpine
apk add --no-cache ca-certificates
cp corp-root.crt /usr/local/share/ca-certificates/
update-ca-certificates

The most frequent container accident is a minimal image with no trust store at all. Put a Go binary on top of scratch or alpine and it fails like this.

x509: certificate signed by unknown authority

Nothing is wrong with the certificate; there is simply not a single root inside the image to compare against. On Alpine, install the ca-certificates package; for a static binary, use a distroless base image or copy the certificate bundle in from a multi-stage build.

FROM alpine:3.20 AS certs
RUN apk add --no-cache ca-certificates

FROM scratch
COPY --from=certs /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY app /app
ENTRYPOINT ["/app"]

It is also worth remembering that the environment variable pointing at the trust store differs per runtime.

export SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt   # OpenSSL, Go, curl
export REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt  # Python requests
export NODE_EXTRA_CA_CERTS=/usr/local/share/ca-certificates/corp-root.crt  # Node
# Java uses a separate keystore
keytool -importcert -alias corp-root -file corp-root.crt \
  -keystore "$JAVA_HOME/lib/security/cacerts" -storepass changeit -noprompt

If self signed certificate in certificate chain only appears on the corporate network, the cause is usually an intercepting proxy. When a company runs TLS inspection appliances, every certificate comes out re-signed by an internal CA. Looking at the issuer at the top of the chain reveals it immediately.

mTLS doubles the number of failure points. The order to check is as follows.

# Check whether the server requests a client certificate, and which CAs it accepts
openssl s_client -connect mtls.example.com:8443 -servername mtls.example.com </dev/null 2>&1 \
  | sed -n '/Acceptable client certificate CA names/,/^---/p'

Acceptable client certificate CA names
C = KR, O = Example Internal CA, CN = Example Issuing CA
Client Certificate Types: RSA sign, ECDSA sign

This list is decisive. If the issuer of your client certificate is not in it, that certificate will never get through. Check the issuer.

openssl x509 -in client.crt -noout -issuer -subject -dates
openssl verify -CAfile ca.crt client.crt

issuer=C = KR, O = Example Internal CA, CN = Example Issuing CA
subject=CN = payments-service
notBefore=Jul  1 00:00:00 2026 GMT
notAfter=Jul  1 00:00:00 2027 GMT
client.crt: OK

Now present the client certificate and connect.

openssl s_client -connect mtls.example.com:8443 -servername mtls.example.com \
  -cert client.crt -key client.key -CAfile ca.crt </dev/null 2>&1 | tail -5

curl -sS --cert client.crt --key client.key --cacert ca.crt \
  https://mtls.example.com:8443/v1/health

The TLS 1.3 property mentioned earlier shows itself here. Even when the client certificate is rejected, the handshake looks like it succeeded, and the error appears only after the first request has been sent.

curl: (56) OpenSSL SSL_read: error:0A000412:SSL routines::sslv3 alert bad certificate, errno 0

Looking at client logs alone makes it look like the connection dropped mid-request, so an mTLS failure must always be examined alongside server logs. Whether the extended key usage is right is another frequently missed item. A server certificate needs serverAuth and a client certificate needs clientAuth.

openssl x509 -in client.crt -noout -ext extendedKeyUsage

X509v3 Extended Key Usage:
    TLS Web Client Authentication

Closing — count the chain before you fix the client

The order of judgement for certificate problems summarises briefly.

First, connect with openssl s_client and always add -servername, then count how many entries appear under Certificate chain in the output. If only the leaf certificate shows up, the cause is settled at that moment. The server is not sending the intermediate certificate, and the browser was filling that gap through AIA. Instead of touching the client, deploy fullchain on the server and you are done.

If it fails even with a complete chain, that is when it is time to look at the trust store. Check whether ca-certificates exists in the container, whether the internal CA is registered, and whether the per-runtime environment variables are right.

And the option of turning validation off should be kept until the very last. -k and verify=False do not solve the problem; they only make it invisible, and shipping in that state leaves you fully exposed to interception attacks. Most certificate errors are resolved precisely by one line of server configuration.