Split View: 국내 개발 블로그 명글 큐레이션 2 — 장애 회고와 트러블슈팅, 직접 열어 확인한 12편

Korean Dev Blog Curation 2 — Incident Retrospectives and Troubleshooting, 12 Posts I Opened and Checked

• The Genre Korean Developers Write Best
• When the Cause Was Somewhere Else Entirely
  • 사내 서비스 장애 해결 일대기 - p6spy 사용시 DB Routing 안됨, HikariCP 커넥션 누수
  • Timeout이 작동하지 않은 이유
  • 문제 현상 이면에 숨겨진 원인 찾기: TCP 통신 트러블슈팅
  • 쿠버네티스 네트워크 문제 트러블슈팅
• Getting Hands Dirty with Memory and CPU
  • Java의 heap dump를 이용한 OOM 원인 분석 with Eclipse MAT
  • 3. [400분이 걸리는 10만 개의 알림 요청 시간을 줄여보자] 왜 CPU 스파이크가 발생할까? JVM WarmUP으로 해결해보자
  • [Java] 에러 스택 트레이스 최적화를 도와주는 -XX:+OmitStackTraceInFastThrow(사라진 예외 스택 트레이스)
• Incidents Made by Databases and Connections
  • [트러블슈팅 - DB] 외래키(Foreign Key)와 데드락(DeadLock) 그리고 쿼리 지연 실행
  • HikariCP Connection 고갈 문제
  • MySQL 트러블슈팅 회고 (feat. 대소문자 구분)
  • JPA가 Fetch Join에 대한 On절을 지원하지 않는 이유
• An Incident Made by People, and the Nerve to Publish It
  • 주니어 개발자의 첫 장애 회고 (새벽에 문자발송이라니..)
• Related Posts and Tools on This Blog
• Other Posts in This Series

The Genre Korean Developers Write Best

Read Korean developer writing for long enough and one impression settles in. Conceptual explainers overlap heavily with what exists in English; incident retrospectives do not. They are about a specific thing that happened in the author's own service, which makes them irreplaceable, and the ones that do not hide the failure carry the highest information density.

The selection method is the same across the series. I found candidates by searching and then opened each post directly to check it, keeping the ones whose explanations were concrete and reproducible. For this post I added one more criterion: I preferred posts where the symptom and the cause turned out to be in different places — where the record of the wrong first suspicion survives. Those are closest to what real investigation feels like.

This list is an editorial pick, not a ranking. I did not measure views or popularity and have no way to. If someone else had chosen, the list would look different.

I gave priority to personal blogs. velog is heavily represented here, which reflects where this genre concentrates rather than any judgment that velog is better.

One note for readers outside Korea: every linked post is written in Korean. You will need Korean to read them, though the stack traces, packet captures, and configuration snippets that carry the argument read the same in any language.

All links were opened and checked directly on 2026-08-12. Personal blog posts can disappear or change addresses.

When the Cause Was Somewhere Else Entirely

There is one mark of a good troubleshooting post: the distance between the first suspicion and the actual cause is still visible.

사내 서비스 장애 해결 일대기 - p6spy 사용시 DB Routing 안됨, HikariCP 커넥션 누수

• Blog · Author: Just Do It · uHan2 (velog)
• One-line summary: Tracing a silent failure of read/write datasource routing back to a SQL logging library.
• Read this if: You have attached a read replica and felt traffic was not splitting the way you expected.

This sits first because the cause was in a place nobody would think to suspect. A library added purely to print SQL nicely wrapped the connection in a way that bypassed the lazy-acquisition proxy, and routing quietly stopped working. It is the classic case of a developer-convenience tool changing runtime behavior, and that class of bug does not get caught in code review. The author also reported the finding upstream, which shows what the end of an investigation ought to look like.

Timeout이 작동하지 않은 이유

• Blog · Author: sejinkim.log · sejin (velog)
• One-line summary: Why a call stalled for fifteen seconds despite a three-second timeout — the answer is DNS resolution.
• Read this if: You believe you have blocked the path by which an external API outage propagates into your service.

Assuming a configured timeout means safety is a common misconception, and this post pinpoints exactly where it breaks. Name resolution typically happens synchronously and outside the HTTP client's timeout, leaving a window your setting never covers. The post compares how different clients behave, so you finish knowing what to check in your own stack. It is short, but it changes how you read a timeout configuration afterward.

문제 현상 이면에 숨겨진 원인 찾기: TCP 통신 트러블슈팅

• Blog · Author: JooSing (velog)
• One-line summary: A symptom that looked like an application-level exception turns out, through packet analysis, to be a TCP half-close.
• Read this if: You want to keep open the possibility that an apparent state-management bug lives in the network layer.

The title carries the thesis exactly. The exception message pointed at duplicate handling, but in reality a connection was left half-closed and blocking the next attempt. Looking only at application logs, you would never find this; capturing packets is the turning point. The cross-layer investigation is preserved as written, which is where you pick up the instinct for when to widen the scope of a search.

쿠버네티스 네트워크 문제 트러블슈팅

• Blog · Author: hyeongjun-hub.log · hyeongjun Jo (velog)
• One-line summary: A two-week hunt that traced recurring latency spikes in a production cluster to a single port blocked by a Helm chart.
• Read this if: Unexplained intermittent latency has you staring at dashboards.

The scale of this one is striking. Three engineers worked it for two weeks, and the hypotheses they discarded along the way are still in the text. The final cause was that the port load balancer components use to recognize each other was blocked, so several nodes advertised the same virtual IP at once and the ARP tables destabilized. Since a chart default created the problem, this applies to any team running managed components. The investigation tools and their order are written down concretely, which makes it reproducible.

Getting Hands Dirty with Memory and CPU

Slow and dying are different problems, and the investigative tools differ too. The posts in this section actually use those tools.

Java의 heap dump를 이용한 OOM 원인 분석 with Eclipse MAT

• Blog · Author: kyle's devlog (velog)
• One-line summary: Tracing a dead grading server through a heap dump, then automating dump collection for the next incident.
• Read this if: When an OOM hits, you do not know what to look at besides logs.

The cause itself is interesting: a user submitted an infinite loop, which produced an enormous output file, and code that read the whole thing into memory filled the heap. But the bigger value is the preparation for next time. The author built a mechanism that ships any dump straight to durable storage, so the next investigation does not have to wait for a reproduction. The order in which to read a histogram and a dominator tree is spelled out too, so it doubles as tool documentation.

3. [400분이 걸리는 10만 개의 알림 요청 시간을 줄여보자] 왜 CPU 스파이크가 발생할까? JVM WarmUP으로 해결해보자

• Blog · Author: rawfish의 개발공부.log · 이건회 (velog)
• One-line summary: CPU hitting 99 percent only right after deploy, explained down to the JIT compilation stage and solved with warm-up.
• Read this if: You have accepted a few minutes of slow responses after every deploy as unavoidable.

Post-deploy slowness usually gets waved away as cold caches, but this post goes down to what the JIT compiler does when it has no cached profile. Thread dumps, monitoring screenshots, and load test results are all attached, so the claims are backed by observation. It is the third entry in a series, so it has context on both sides, and a concrete target of a hundred thousand notifications means the improvement comes out as a number. It also helps you tell apart the cases where warm-up is warranted from the ones where it is not.

[Java] 에러 스택 트레이스 최적화를 도와주는 -XX:+OmitStackTraceInFastThrow(사라진 예외 스택 트레이스)

• Blog · Author: MangKyu's Diary (망나니개발자)
• One-line summary: Why the stack trace of a repeatedly thrown exception disappears at some point, explained via a JVM optimization flag.
• Read this if: You have suspected a logging misconfiguration when only exception names showed up with no stack trace.

This behavior blocks investigators precisely where it hurts: the more frequently an exception fires, the more likely its stack trace vanishes. Without knowing the cause is JVM optimization rather than logging, you can spend hours in the wrong place. A reproduction example is included, showing traces disappearing after repeated calls, so you can confirm it in your own environment. It is common knowledge to those who know and undiscoverable to those who do not, and having it written up in Korean is valuable.

Incidents Made by Databases and Connections

Plenty of outages leave the application code blameless and collapse at the data layer. This is especially true when the cause is a framework default.

[트러블슈팅 - DB] 외래키(Foreign Key)와 데드락(DeadLock) 그리고 쿼리 지연 실행

• Blog · Author: hocaron.log · Hocaron (velog)
• One-line summary: A deadlock with no explicit locking anywhere, traced to foreign key constraints combined with write-behind ordering.
• Read this if: You have seen a deadlock log and found no locking code anywhere to explain it.

The post connects two pieces of knowledge: that changing a child row propagates a lock to the parent row, and that the ORM defers writes and then flushes them in a fixed order. Each is known on its own; the contribution here is showing that when they overlap the deadlock becomes deterministically reproducible. The chosen fix was removing the foreign key constraint, and the reasoning behind that choice is written down. Whether or not you agree, the reasoning is exposed, which makes it usable as material for an argument.

HikariCP Connection 고갈 문제

• Blog · Author: zini9188.log (velog)
• One-line summary: A server death after adding notifications, traced to pool exhaustion, and the setting sitting behind it.
• Read this if: You have ever papered over a problem by raising the pool size.

A pool that runs dry is usually about paths where connections are not returned, not about pool size. This post identifies the structure where a long-lived connection overlaps the transaction scope and pins a database connection, and shows that one framework setting was creating that structure. The fix splits the code so connection-holding and data access are separated, which gets closer to the root than simply disabling the setting. Worth reading before you add any real-time feature.

MySQL 트러블슈팅 회고 (feat. 대소문자 구분)

• Blog · Author: 숑숑의 삽질로그 · 숑숑 (velog)
• One-line summary: Two strings the application considered different and the database considered identical, resulting in the same record being both updated and deleted.
• Read this if: Your code compares strings in both the language and the database.

Everyone knows comparison semantics can differ per layer, but seeing that turn into an actual incident is rare. This post shows the scene directly. The cause is a single collation setting; the symptom is the entire sync logic going wrong. Both fixes — making the comparison explicit and changing the schema — are offered, so you can pick to fit your situation. If your service handles multiple languages or email addresses, this is not somebody else's problem.

JPA가 Fetch Join에 대한 On절을 지원하지 않는 이유

• Blog · Author: Tecoble (Woowacourse) · 5기_베베
• One-line summary: Working around a restriction on filtering a fetch join can cause rows to be genuinely deleted from the database.
• Read this if: You have wondered why an ORM forbids certain syntax.

Forbidden syntax usually has a reason behind it, and rarely one this alarming. Filter to fetch only part of a collection and the persistence context interprets the remainder as deleted, and that interpretation can go out as an actual delete statement. Examples let you follow why, and you come away understanding the ORM's behavioral model one level deeper. It is a company-affiliated blog, but I included it because among Korean writeups on this topic the causal chain is the clearest here.

An Incident Made by People, and the Nerve to Publish It

Procedural accidents happen about as often as technical ones. Publishing them takes a different kind of nerve.

주니어 개발자의 첫 장애 회고 (새벽에 문자발송이라니..)

• Blog · Author: 작심삼일을 꾸준히 · Jeonghwa (velog)
• One-line summary: A load test that reached the real SMS path and sent hundreds of thousands of messages overnight — and the deploy meant to stop it that brought the problem back.
• Read this if: You cannot state precisely how separated your staging environment is from production.

The structure of this incident is textbook. The test environment was wired to the real sending path, and the deploy intended to roll it back auto-reverted on a health check timeout, restoring a version where the offending code was still live. It grew not from one mistake but from two safeguards each failing in their own way, and that chain is laid out in chronological order. Publishing a junior-era incident in this much detail is uncommon, and what the reader takes away is which seam in their own team to go look at. I included it not as a cautionary tale but because the form of the retrospective itself is good.

Related Posts and Tools on This Blog

• Debugging the Linux OOM killer
• Deadlock debugging and prevention
• Sizing a connection pool
• Connection refused versus timeout
• Kubernetes OOMKilled and exit code 137
• Tools: HTTP status code reference · SQL playground

Other Posts in This Series

• Previous: Korean Dev Blog Curation 1 — Backend and Infrastructure
• Next: Korean Dev Blog Curation 3 — AI and ML in Practice
• Full list: 4 Frontend · 5 Career and growth