💡 왼쪽 원문을 읽으면서 오른쪽에 따라 써보세요. Tab 키로 힌트를 받을 수 있습니다.

원문 렌더가 준비되기 전까지 텍스트 가이드로 표시합니다.

> "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." — Brian Kernighan

What a software engineer actually does for a living is not write code; it is **debug**. Various studies put the share of time at 25-50%, and the ratio tends to grow as you become more senior — complex distributed systems, subtle memory corruption, non-deterministic concurrency bugs, mysteries that only reproduce in production. In 2026, debugging is no longer a matter of sprinkling `print` statements. It is a discipline that combines **scientific method, an enormous tooling ecosystem, and AI assistance**.

This article covers it end to end: native debuggers (GDB 15, LLDB 19, rr, Pernosco, WinDbg TTD), Chrome DevTools 2025 with AI assistance, the Bun/Deno/Node inspectors, the Python debugger lineup (pdb/pdbpp/ipdb/pudb), language-specific debuggers for Rust/Go/Java, eBPF-based observation, Valgrind and the Sanitizers, the DAP (Debug Adapter Protocol) standard — and, most importantly, the **debugging mindset** that ties them together.

1. The Debugging Mindset — Hypothesis-Driven Scientific Debugging

Good debugging starts not with a tool but with a way of thinking. Andreas Zeller's *Why Programs Fail* (2009; second edition 2023) frames debugging as the scientific method: form a hypothesis, design an experiment, try to falsify it. The core loop:

1. **Observe the symptom**: under which inputs, in which environment, with what result?

2. **Form hypotheses**: N candidate causes, ranked by likelihood.

3. **Predict**: if this hypothesis is true, changing A should change B.

4. **Experiment**: the smallest change that tests the prediction.

5. **Falsify or refine**: drop the hypothesis or narrow it based on the result.

Engineers who run this loop fast are good debuggers. The most common rookie trap is **changing code at random while hoping for the best** — Zeller calls this "shotgun debugging," and it wastes time and introduces new bugs. Another trap is **confirmation bias**: looking only for evidence that supports your hypothesis and ignoring counter-evidence. A skilled debugger actively tries to *falsify* their own hypothesis.

A principle senior engineers in 2026 emphasize is **"a bug is information."** The most expensive resource when a bug appears is the exact conditions that reproduce it; capturing a **minimal reproducer** before those conditions evaporate is half the battle. Zeller's delta debugging algorithm (2002) automates this; modern shrinking in Hypothesis, QuickCheck, and fast-check applies the same idea.

2. GDB 15 — The De Facto Standard Native Debugger

GDB (the GNU Debugger, `gnu.org/software/gdb`) is the de facto standard native debugger and has evolved for 40 years since Richard Stallman started it in 1986. **GDB 15**, released January 2025, brought Python 3.12 integration, complete DWARF 5 support, and improved pretty-printing for languages like Rust, Swift, and Modula-2.

The basic workflow:

Build with -g (debug symbols) and -O0 or -Og (limited optimization)

gcc -g -Og main.c -o app

Enter the debugger

gdb ./app

Or attach to a running process

gdb -p 12345

Or analyze a core dump

gdb ./app core.12345

The core commands are `break`, `run`, `next`, `step`, `print`, `backtrace` (alias `bt`), `info`, `watch`, and `continue`.

(gdb) break main.c:42 # line-based breakpoint

(gdb) break parse_request # by function name

(gdb) break parse_request if errno == 5 # conditional

(gdb) watch *ptr # stop when memory changes

(gdb) catch throw # stop on C++ exceptions

(gdb) bt full # full stack with locals

(gdb) frame 3 # jump to frame 3

(gdb) info locals # list local variables

(gdb) print/x *(struct request *)0x7ffe8000 # cast and print as hex

The most powerful feature in GDB 15 is **Python scripting**. With `gdb.execute`, `gdb.parse_and_eval`, and custom pretty-printers, you can render compiled domain objects in a human-readable form.

In ~/.gdbinit or a separate .py file

class RequestPrinter:

def __init__(self, val):

self.val = val

def to_string(self):

path = self.val['path'].string()

method = self.val['method'].string()

return f"Request {method} {path}"

def lookup_request(val):

if str(val.type) == 'struct request':

return RequestPrinter(val)

return None

gdb.pretty_printers.append(lookup_request)

TUI (Text User Interface) mode, `gdb -tui` or `Ctrl+X A`, splits the terminal into four panes showing source, registers, assembly, and command input. In 2025, `gdb-dashboard` (`github.com/cyrus-and/gdb-dashboard`) has become the de facto standard UI extension.

3. LLDB 19 — The Apple/LLVM Standard

LLDB (`lldb.llvm.org`) is the LLVM project's debugger and evolves with the rest of LLVM (LLVM 19 shipped in September 2025). Xcode uses it internally on all Apple platforms (macOS, iOS, watchOS, visionOS), and it is the default debugger for `rust-lldb` and Swift.

Commands resemble GDB but use a slightly different vocabulary.

(lldb) breakpoint set --file main.c --line 42 # alias: b main.c:42

(lldb) breakpoint set --name parse_request

(lldb) run

(lldb) thread step-over # n

(lldb) thread step-in # s

(lldb) frame variable # info locals

(lldb) frame select 3

(lldb) thread backtrace # bt

(lldb) memory read --size 4 --format x 0x100000 # hex dump

(lldb) expression -- (*request).path # p (*request).path

LLDB's strength is that its **Python script bridge** is integrated more deeply than GDB's. The `script` command opens a Python REPL inside LLDB, and objects like `SBValue`, `SBFrame`, and `SBProcess` make debug sessions programmable.

(lldb) script

>>> import lldb

>>> frame = lldb.thread.GetSelectedFrame()

>>> req = frame.FindVariable('request')

>>> print(req.GetChildMemberWithName('path').GetSummary())

For Swift debugging, `po` (print object) invokes the Swift object's `description`, and `v` (variable) is optimized for faster variable display. Swift 6.1 (2025) integrated Swift Macros debugging into LLDB, so macro expansion can be inspected directly.

In Rust, the `rust-lldb` (and `rust-gdb`) wrappers automatically load pretty-printers for standard library types (`Vec`, `String`, `HashMap`).

4. rr — Deterministic Record and Replay Debugging

rr (`rr-project.org`) is a record-and-replay debugger Mozilla introduced in 2014. It makes **non-deterministic bugs deterministically reproducible**: a recorded execution replays with the same syscalls, thread schedule, and signal order, so the same bug repeats indefinitely.

Record

rr record ./app --input data.json

Replay (drops into a gdb-style session)

rr replay

Or auto-replay the latest recording

rr replay -p 0

Where recordings live

ls ~/.local/share/rr/

rr's real magic is **reverse execution**.

(rr) reverse-continue # back to the previous breakpoint

(rr) reverse-step # one step back

(rr) reverse-next # one line back

(rr) watch -l x # find backwards where x was written

The classic scenario "the pointer is NULL but I have no idea where it became NULL" is solved by `reverse-continue` + `watch *ptr`. rr runs backwards until the memory write that set NULL, immediately revealing the culprit.

There are three caveats. rr only runs on Intel CPUs (AMD Zen 5 partial support landed in 2026 but is still beta), it uses CPU PMCs (Performance Monitoring Counters), which makes virtualization tricky, and recording serializes multi-core parallel execution (1.2x to 5x overhead). The payoff for non-deterministic bugs is so dramatic that the Firefox, Chromium, and Microsoft Edge teams all have rr-based workflows.

5. Pernosco — Cloud-Based Omniscient Debugger

Pernosco (`pernos.co`) is a cloud debugger spun out of Mozilla in 2018 by rr's creators Robert O'Callahan and Kyle Huey. The core idea: **upload an rr recording to the cloud, and Pernosco pre-analyzes the entire execution to give you an "omniscient" debug session in a web UI**.

Record locally with rr

rr record ./app

Upload via the Pernosco CLI (account required)

pernosco-submit upload ~/.local/share/rr/latest-trace

After upload, Pernosco indexes the full trace in the backend for 1-10 minutes, then provides:

- **Arbitrary point jumps**: click anywhere on the timeline and see the full memory, registers, and call stack at that moment.

- **Dataflow tracking**: trace where a value came from, backwards.

- **Automatic root-cause hints**: "this NULL was freed at line 42 thirty seconds ago and reused."

- **Shareable URLs**: send a colleague the debug session itself.

Mozilla, Cloudflare, Sourcegraph, and Discord use Pernosco for production-grade bug hunting. The 2026 pricing is on the expensive side (roughly 10-50 USD per trace), but resolving a mystery bug that would have eaten a week in an hour is a clear win.

6. WinDbg + TTD — Windows Time Travel Debugging

The standard Windows debugger is **WinDbg** (`learn.microsoft.com/windows-hardware/drivers/debugger/`), and since 2018 **WinDbg Preview** (now the WinDbg "engine 2", consolidated with a modern UI in 2025) has been the main client. WinDbg handles user mode, **kernel mode**, **crash dumps**, and **TTD (Time Travel Debugging)**.

TTD is the Windows counterpart to rr/Pernosco. `TTTracer.exe` records an execution into a `.run` file.

In an administrator prompt

TTTracer.exe -out C:\traces\ -launch myapp.exe

Open the .run in WinDbg for reverse debugging

WinDbg command window

0:000> g- # reverse go

0:000> t- # reverse step

0:000> wt- # reverse watch

WinDbg shines for **kernel debugging**. Open a BSOD (Blue Screen of Death) dump file (`.dmp`) and trace which driver caused an IRQL violation or which one caused a deadlock. Windows security teams, anti-cheat engines, and EDR vendors live in WinDbg every day.

7. Chrome DevTools 2025 — AI Assistance and MCP

Chrome DevTools has been one of the most rapidly evolving debuggers since its 2008 debut, and 2025 marked a substantive shift with **AI assistance (Insights, Recorder)** and **DevTools MCP**.

Key panels:

- **Sources**: breakpoints, conditionals, logpoints, async stack traces, automatic source map resolution.

- **Performance**: trace recording, flame charts, AI-generated insights ("the LCP delay is caused by main thread blocking").

- **Memory**: heap snapshots, allocation timeline, detached DOM trees, retainer trees.

- **Network**: per-request timing waterfalls, initiator chains, throttling simulation.

- **Coverage**: unused JS and CSS measurement.

- **Recorder**: capture user scenarios and auto-generate Playwright/Puppeteer scripts.

**Chrome DevTools MCP** (`github.com/ChromeDevTools/chrome-devtools-mcp`), released in April 2025, lets AI coding agents like Claude, Cursor, and Codex operate an actual browser for debugging and testing. Ask Claude "find and fix the button that's throwing a console error," and the agent uses MCP to read the console, find selectors, capture traces, and analyze the cause.

Chrome DevTools is also the standard for Node.js debugging. A process started with `node --inspect` shows up in `chrome://inspect` or `about:inspect` and connects over the V8 inspector protocol.

Node.js: open the inspector at startup

node --inspect index.js

Break on the first line and wait for attach

node --inspect-brk index.js

Specify a port (useful in Docker)

node --inspect=0.0.0.0:9229 index.js

8. Firefox DevTools / Safari Web Inspector

Firefox DevTools (`firefox-source-docs.mozilla.org/devtools-user/`) offers near-parity with Chrome but a few distinctive features:

- **CSS Grid Inspector**: overlays grid lines, gaps, and areas (Chrome caught up later, but Firefox pioneered it).

- **Accessibility Inspector**: detects WCAG violations automatically.

- **Network Throttling**: WebRTC and WebSocket traffic analysis is richer than Chrome's.

- **about:debugging**: attach to other tabs, service workers, and extensions from a single page.

Safari Web Inspector is essential for iOS/macOS device debugging. On a Mac, choose Safari to the Develop menu to your iPhone simulator or device. Safari 18.4 (2025) added official WebGPU debugging, and Safari 19 (2026) ships native React DevTools integration.

The three browsers expose different wire protocols — WebKit Inspector Protocol (Safari), CDP / Chrome DevTools Protocol (Chrome), Firefox Remote Debugging Protocol — but Playwright and Puppeteer abstract the differences.

9. Python Debugging — pdb, pdbpp, ipdb, pudb, debugpy

Python has surprisingly many debugger options. The baseline is `pdb` (`docs.python.org/3/library/pdb.html`).

def buggy(x, y):

pdb.set_trace() # or the Python 3.7+ breakpoint() builtin

z = x / y

return z

buggy(10, 0)

The `breakpoint()` builtin reads `PYTHONBREAKPOINT` to choose a debugger (e.g. `PYTHONBREAKPOINT=ipdb.set_trace`).

**pdbpp** (`github.com/pdbpp/pdbpp`) is a drop-in replacement for `pdb` that adds syntax highlighting, sticky mode (always show the current function at the top), and tab completion. After installation, plain `import pdb; pdb.set_trace()` is automatically enhanced.

**ipdb** (`github.com/gotcha/ipdb`) is IPython-based, bringing `?`, `??`, and magic commands into the debug session.

**pudb** (`github.com/inducer/pudb`) is a terminal GUI debugger with a curses-based four-pane layout (source, stack, variables, command). It is the favorite over SSH where X11 is not available.

**rpdb** (`github.com/tamentis/rpdb`) is a remote debugger that exposes the debug session over a network socket, useful when you temporarily need to attach to a production server.

**debugpy** (`github.com/microsoft/debugpy`) is the VS Code default Python debugger backend and speaks DAP.

Wait for an attach in production

debugpy.listen(("0.0.0.0", 5678))

debugpy.wait_for_client() # blocks until a client attaches

Comparison:

| Tool | When to use | Strength |

|---|---|---|

| pdb | Fast entry, no extra deps | Builtin, `breakpoint()` |

| pdbpp | Local standalone use | Auto-enhanced, sticky mode |

| ipdb | Data / notebook work | IPython integration |

| pudb | SSH and terminal | Full-screen GUI |

| rpdb | Production attach | Remote socket |

| debugpy | IDE (VS Code) | DAP, multi-client |

10. Bun 1.3 / Deno 2 / Node.js — JavaScript Runtime Debugging

**Bun** (`bun.sh`) stabilized its **built-in inspector** in version 1.2 (September 2024), and Bun 1.3 (January 2026) added direct VS Code DAP support on top of Chrome DevTools.

Bun: start the built-in inspector

bun --inspect index.ts

Break on the first line

bun --inspect-brk index.ts

Explicit port

bun --inspect=ws://0.0.0.0:6499 index.ts

Bun's inspector is V8-inspector-compatible, but because Bun is built on JavaScriptCore some trace shapes differ. Bun 1.3.2 (April 2026) brought major source map improvements.

**Deno 2** (`deno.com`, the Node-compatible 1.0 GA was October 2024) is V8-based, so its inspector matches Node closely.

deno run --inspect=0.0.0.0:9229 server.ts

deno run --inspect-brk server.ts

**Node.js** is the most mature.

node --inspect index.js # default 9229

node --inspect-brk index.js # break on first line

node --inspect-port=9230 index.js # explicit port

node --inspect=0.0.0.0:9229 index.js # allow external attach

`chrome://inspect` with "Discover targets" enabled picks the process up automatically. In VS Code, a `launch.json` with `"type": "node"` is enough.

Example `launch.json`:

// launch.json (JSON; not evaluated externally)

{

"version": "0.2.0",

"configurations": [

{

"type": "node",

"request": "launch",

"name": "Launch Program",

"program": "${workspaceFolder}/index.js"

}

]

}

`${workspaceFolder}` is a VS Code variable, not a JavaScript template literal.

11. Rust Debugging — dbg!, cargo-flamegraph, tokio-console

The fastest Rust debugging tool is the `dbg!` macro.

fn calculate(x: i64, y: i64) -> i64 {

let intermediate = dbg!(x * 2); // prints file, line, and value to stderr

let result = dbg!(intermediate + y);

result

}

The output is of the form `[src/main.rs:2] x * 2 = 20` — much more context than `println!("{:?}", expr)`.

For full-featured debugging, use `rust-gdb` (the GDB wrapper) or `rust-lldb`. Both load pretty-printers for the Rust standard library, so `Vec<i32>` and `HashMap<K, V>` render readably.

cargo build --profile=dev # default debug build

rust-gdb target/debug/myapp

rust-lldb target/debug/myapp

**tokio-console** (`github.com/tokio-rs/console`) is a dedicated debugger for the Tokio runtime that visualizes running async tasks, resources, and deadlocks in real time.

// In Cargo.toml: tokio with the "tracing" feature, plus console-subscriber

fn main() {

console_subscriber::init();

// ...

}

In another terminal, run `tokio-console` and you immediately see busy tasks, polling frequency, and slow tasks. Indispensable for deadlock and starvation debugging in Tokio-based servers.

**cargo-flamegraph** (`github.com/flamegraph-rs/flamegraph`) wraps perf/dtrace and produces a flame graph in one command.

cargo flamegraph --bin myapp -- --some-arg

generates flamegraph.svg

**pprof-rs** (`github.com/tikv/pprof-rs`) is an in-process CPU profiler that exports results in the same format as Go's pprof, suitable for production.

12. Go Debugging — Delve (`dlv`)

Go's standard debugger is **Delve** (`github.com/go-delve/delve`); GDB technically works, but it does not understand goroutines and channels well and is hardly used. Delve 1.24 shipped alongside Go 1.23 and 1.24 in 2025, stabilizing generics debugging.

Run in debug mode

dlv debug ./cmd/server

Run a compiled binary (must include debug symbols)

dlv exec ./server

Attach to a running process

dlv attach 12345

Analyze a core dump

dlv core ./server core.12345

Remote (headless) mode

dlv --listen=:2345 --headless --api-version=2 debug

Delve commands:

(dlv) break main.go:42

(dlv) break main.processRequest

(dlv) condition 1 r.UserID == 42 # conditional breakpoint

(dlv) continue

(dlv) next

(dlv) step

(dlv) goroutines # list all goroutines

(dlv) goroutine 3 # select goroutine 3

(dlv) bt

(dlv) print req.Body

(dlv) call myFunc(42) # dangerous but possible

The Go extension in VS Code uses Delve as a backend, so GUI debugging is natural. GoLand embeds Delve directly as of 2025.

13. Java Debugging — JFR + JMC + IDE Debuggers

Java has the JVM-standard **JDWP (Java Debug Wire Protocol)**, which standardizes the wire between IDE and debugger. The **JDB** CLI is deprecated; in practice, **IntelliJ IDEA**, **Eclipse**, and the **VS Code Java Pack** debug over JDWP.

Start the JVM in debug mode

java -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005 -jar app.jar

suspend=y waits for an IDE to attach

For performance and memory debugging, **JFR (Java Flight Recorder)** + **JMC (JDK Mission Control)** is the standard. JFR is a low-overhead (under 2%) profiler built into the JVM, and it is increasingly run continuously in production.

Start a JFR recording

jcmd <pid> JFR.start name=profile duration=60s filename=profile.jfr

Or as a JVM start argument

java -XX:StartFlightRecording=duration=60s,filename=profile.jfr -jar app.jar

Analyze with JMC

jmc profile.jfr

JFR/JMC together expose GC pauses, allocation hot paths, lock contention, and IO bottlenecks in one view, making them the de facto standard for production-grade Java performance debugging.

Java 11 introduced **Async Stack Traces**, which stitches asynchronous callbacks across `CompletableFuture` and reactive chains so a stack trace appears as a single chain. Project Loom's **Virtual Threads** (Java 21 and later) pose new challenges, and IntelliJ 2025.1 introduced a dedicated panel for virtual thread visualization.

14. Database Debugging — EXPLAIN ANALYZE and Friends

Half of all service performance bugs are in the database, and half of those are caused by bad query plans. PostgreSQL's `EXPLAIN ANALYZE` is the most powerful debugging command around because it ships actual execution statistics alongside the plan.

EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT)

SELECT u.id, u.email, COUNT(o.id) AS orders

FROM users u

LEFT JOIN orders o ON o.user_id = u.id

WHERE u.created_at > NOW() - INTERVAL '30 days'

GROUP BY u.id, u.email

ORDER BY orders DESC

LIMIT 100;

Watch for **Seq Scan vs Index Scan**, **Rows planned vs actual** (a big gap means stale statistics), **Buffers: shared hit / read** (IO cost), and **Sort method: external merge** (sort spilling to disk because of too little memory).

**pgBadger** (`pgbadger.darold.net`) turns PostgreSQL slow-query logs into HTML reports, and the **pg_stat_statements** extension provides live cumulative statistics.

MySQL has `EXPLAIN FORMAT=JSON` and `EXPLAIN ANALYZE` in 8.0+, with **MySQL Performance Schema** for cumulative statistics.

For ORM users, logging the actual SQL the ORM emits is the first debugging tool: Hibernate's `show_sql`, Prisma's `log: ['query']`, Drizzle's `logger: true`, and ActiveRecord's `ActiveRecord::Base.logger`.

15. Distributed Systems Debugging — OpenTelemetry and Structured Logs

In microservices, bugs do not stay inside a single process. A request hops through 5-50 services, and a single-node debugger cannot see where it went wrong.

**OpenTelemetry** (`opentelemetry.io`) is the de facto standard SDK for distributed tracing, logs, and metrics. Once your code is instrumented, every call carries the same trace ID through Jaeger, Tempo, Honeycomb, Datadog, or New Relic.

const tracer = trace.getTracer('my-service')

async function handleRequest(req: Request) {

return tracer.startActiveSpan('handleRequest', async (span) => {

span.setAttribute('user_id', req.userId)

try {

const result = await processRequest(req)

span.setStatus({ code: 1 }) // OK

return result

} catch (err) {

span.recordException(err as Error)

span.setStatus({ code: 2, message: (err as Error).message }) // ERROR

throw err

} finally {

span.end()

}

})

}

**Structured logging** lets you join logs across services on the same trace ID. JSON line format is the standard (`pino`, `winston`, `zap`, `logrus`, `slog`), with Datadog Logs, Grafana Loki, or Elastic Cloud as the backend.

The biggest pitfall in distributed debugging is **clock skew**. If two nodes are 50 ms apart, you can see traces where causality looks reversed. That is why OpenTelemetry uses trace context (parent-child span IDs) rather than wall-clock time.

16. eBPF Debugging — bpftrace, BCC, Pixie

**eBPF** (`ebpf.io`) lets you safely inject code into the Linux kernel and is becoming the new standard for debugging, observability, and security. The appeal of eBPF-based tools: **near-zero overhead on production systems with no code changes**.

**bpftrace** (`bpftrace.org`) lets you write eBPF programs as awk-style one-liners.

Trace every openat syscall

bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args->filename)); }'

Which files does Node.js open?

bpftrace -e 'tracepoint:syscalls:sys_enter_openat /comm == "node"/ { printf("%s\n", str(args->filename)); }'

TCP connect latency distribution

bpftrace -e 'kprobe:tcp_v4_connect { @start[tid] = nsecs; } kretprobe:tcp_v4_connect /@start[tid]/ { @dur = hist(nsecs - @start[tid]); delete(@start[tid]); }'

**BCC** (`github.com/iovisor/bcc`) is a framework for writing more complex eBPF tools in Python/C, shipping ready-made tools like `opensnoop`, `execsnoop`, and `tcpconnect`.

**Pixie** (`pixielabs.ai`, CNCF Sandbox) brings eBPF-driven automatic observability and debugging to Kubernetes. Installing the agent auto-captures all HTTP, gRPC, MySQL, and Postgres traffic without any code changes.

**Coroot** (`coroot.com`) combines eBPF with OpenTelemetry to automatically map call relationships and latencies across microservices.

17. Memory Bugs — Valgrind, ASan, MSan, TSan, UBSan

The eternal enemy of C/C++ and Rust unsafe code is **memory safety bugs**: use-after-free, double-free, buffer overflow, uninitialized read, data race. The 2026 standard is compile-time Sanitizers plus runtime Valgrind.

**Valgrind Memcheck** (`valgrind.org`) is a dynamic analyzer with nearly 30 years of history; it intercepts every memory access for validation.

valgrind --tool=memcheck --leak-check=full --show-leak-kinds=all ./app

Overhead is 10-50x, but the overwhelming advantage is that it works on any binary without code changes.

**ASan (AddressSanitizer)** is a compile-time option (`-fsanitize=address`) that catches use-after-free and buffer overflow with about 2x runtime overhead.

clang -fsanitize=address -g -O1 main.c -o app

./app

**MSan (MemorySanitizer)**: detects uninitialized memory reads (`-fsanitize=memory`).

**TSan (ThreadSanitizer)**: detects data races (`-fsanitize=thread`); essential for multi-threaded debugging.

**UBSan (UndefinedBehaviorSanitizer)**: integer overflow, null dereference, misaligned pointers (`-fsanitize=undefined`).

Keeping these four sanitizers turned on in CI alongside a fast test suite is the standard 2026 practice. Chromium, Firefox, and LLVM itself run daily ASan and TSan builds.

18. Performance Profilers — perf, Instruments, VTune, pprof

Performance bugs are debugging too. The fastest path to fast code is to **measure the slow parts**.

**Linux perf** (`perf.wiki.kernel.org`) is the kernel's built-in profiler; it reaches hardware PMCs to measure CPU, cache, and branch misses.

Per-function CPU time

perf record -g ./app

perf report

Render as a flame graph

perf record -F 99 -g ./app -- sleep 30

perf script | stackcollapse-perf.pl | flamegraph.pl > out.svg

**Instruments** (macOS, part of Xcode) is Apple's standard profiler, with templates like Time Profiler, Allocations, Leaks, Network, and System Trace.

**Intel VTune** (`intel.com/vtune`) specializes in Intel CPUs and analyzes micro-architectural metrics (cache miss, branch misprediction, vectorization opportunities).

**pprof** (`github.com/google/pprof`) is Google's profile format and analysis tool; Go's built-in `runtime/pprof`, Rust's `pprof-rs`, and C++'s gperftools are all compatible.

// In another terminal:

// go tool pprof -http=:8080 http://localhost:6060/debug/pprof/profile?seconds=30

19. DAP — Debug Adapter Protocol

The secret behind a single IDE driving many debuggers is the **Debug Adapter Protocol (DAP)** (`microsoft.github.io/debug-adapter-protocol/`). Microsoft published it in 2017; it is a JSON-RPC based protocol that separates IDEs from debugger backends.

Standard messages include `initialize`, `launch`, `attach`, `setBreakpoints`, `stackTrace`, `scopes`, `variables`, `evaluate`, `continue`, `next`, `stepIn`, `stepOut`, and `terminate`.

Clients that speak DAP include VS Code, Vim's `vimspector`, Neovim's `nvim-dap`, Emacs's `dap-mode`, JetBrains IDEs, and Sublime Text's `Debugger` — essentially every modern editor. Backends include `debugpy` (Python), the `node` inspector, `delve` (Go), `lldb-vscode` (C/C++), and `js-debug` (Node/Chrome).

VS Code's `launch.json` is essentially the argument set for a DAP `launch` message. The separation of clients (IDEs) and backends (language-specific debuggers) over a standard protocol is DAP's core value.

20. AI Debugging — Claude Code, Cursor, Copilot Workspace

The biggest shift of 2025-2026 is **debugging capability in AI coding agents**.

**Claude Code** (`claude.com/code`) is a terminal-based agent: paste a stack trace, log, or test output, and it forms hypotheses, reads code, and proposes fixes. Since Claude 4.7 (2025), it can drive real browser debugging through Chrome DevTools MCP.

**Cursor** (`cursor.com`) is a VS Code fork with a "Fix with AI" button on inline error messages. Cursor 1.0 (2026) integrates with the debugger session itself, so the LLM can inspect variables at a paused breakpoint and form hypotheses.

**GitHub Copilot Workspace** (`githubnext.com/projects/copilot-workspace`) is an experimental product that automates the issue to code change to PR flow; for debugging, it can "write a test that reproduces this issue and then fix it."

The limits of AI debugging are also clear. **Non-reproducible bugs** (timing, environment-dependent), **debugging that requires understanding an entire huge codebase**, and **integration with external systems** (databases, third-party APIs) still need a human in the lead. AI is excellent at hypothesis generation, code reading, and repetitive automation, but root cause decisions ultimately rest with people.

21. Reverse Debugging — UDB, rr, Pernosco, TTD Compared

A summary of time-travel and reverse debugging tools:

|---|---|---|---|---|

**UDB** (`undo.io`) is the enterprise reverse debugger from Undo (Cambridge, UK); EA, Riot Games, and Bloomberg use it. It supports more CPUs and platforms than rr and has stronger IDE integration.

Reverse debugging shines in these scenarios:

1. **NULL pointer tracking**: `watch *ptr` + `reverse-continue` to find who wrote NULL.

2. **State corruption**: a value looks wrong, so step back to find who set it.

3. **Non-deterministic deadlocks**: record once, replay forever.

4. **Test flakiness**: reproduce a CI test that fails only occasionally.

22. Print Debugging, Done Well

The oldest, most-used, and most under-appreciated debugging tool is still `print` — `printf`, `console.log`, `eprintln!`, `dbg!`, `fmt.Println`. Do it well:

1. **Write the full context**: variable name, value, function name, and line. Rust's `dbg!` and Go's `log.Printf("[%s] x=%v", funcName, x)` are exemplars.

2. **JSON Lines**: human readable and re-analyzable with grep or jq. `console.log(JSON.stringify({ event: 'request', userId, latency }))`.

3. **Use log levels**: trace/debug/info/warn/error, switchable in production.

4. **Correlation IDs**: every log line on the same request shares an ID.

5. **`console.dir` vs `console.log`**: to print an object deeply, use `console.dir(obj, { depth: null })`.

6. **Rust `tracing` crate**: build spans and events via macros and auto-export them through OpenTelemetry.

The trap of print debugging is **log floods**. Leave it on by accident and you fill disks in production, and the log bill grows larger than the EC2 bill. Control levels with environment variables, and watch out for one-line logs that explode cardinality (too many distinct values).

23. Field Reports — Coupang, Naver, Kakao (Korea)

**Coupang's post-mortem culture** is well known in the Korean industry. Following the partial outage caused by the 2021 AWS Tokyo region incident, Coupang made writing English-language post-mortems for every P0/P1 incident and sharing them internally a policy. The debugging-side emphasis is on **timeline accuracy** and **5 Whys root cause analysis**.

**Naver D2 / SRE blogs** (`d2.naver.com`, `tech-blog.naver.com`) have published Java, JVM, and MySQL debugging stories for years. The 2024 series "Tracing microservice failures with Pinpoint" walks through real-world use of Naver's open-source tracer Pinpoint (`github.com/pinpoint-apm/pinpoint`).

**Kakao's October 2022 data center fire** remains Korea's largest debugging and post-mortem case. The fire took KakaoTalk, Daum, and KakaoT offline for between 5 and 127 hours. Kakao subsequently moved to multi-DC active-active architecture, completely overhauled its incident response playbook, and mandated distributed tracing. The full story and lessons have been published in a multi-year series on the Kakao tech blog (`tech.kakao.com`).

What the three companies have in common is a **culture that turns post-mortems into assets**. Debugging is not a personal skill but a learning loop for the organization — that is the consistent message from Korean big tech.

24. Field Reports — Cookpad, Mercari, Hatena, HSAS (Japan)

**Cookpad's engineering blog** (`techlife.cookpad.com`) publishes more Ruby and Rails debugging case studies than anywhere else in Japan. The 2024 series "Debugging production Ruby processes with rbtrace" covers non-disruptive diagnosis using `rbtrace` (`github.com/tmm1/rbtrace`) and `rbspy` (`github.com/rbspy/rbspy`).

**Mercari Engineering Blog** (`engineering.mercari.com/en/blog/`) has a rich library of Go and Kubernetes debugging cases. The 2025 post "Debugging Goroutine Leaks at Mercari Scale" combines `pprof` with `goleak` for automatic leak detection.

**Hatena Engineer Seminar** is the longest-running engineering conference series in Japan, and Hatena's Perl, Go, and TypeScript debugging know-how is published every year.

**HSAS and the Japanese security community** stand out in reverse engineering and exploit debugging with tools like GDB, radare2, and Ghidra. The debugging workflows used by SECCON and DEF CON CTF teams from Japan represent the frontier of system and binary debugging.

25. References

- *Why Programs Fail* (Zeller, 2009; 2023 second edition) — the canon of hypothesis-driven debugging

- GDB documentation — `https://sourceware.org/gdb/current/onlinedocs/gdb/`

- GDB Python API — `https://sourceware.org/gdb/current/onlinedocs/gdb/Python.html`

- gdb-dashboard — `https://github.com/cyrus-and/gdb-dashboard`

- LLDB documentation — `https://lldb.llvm.org/`

- LLDB Python reference — `https://lldb.llvm.org/python_reference/index.html`

- rr project — `https://rr-project.org/`

- Pernosco — `https://pernos.co/`

- WinDbg TTD — `https://learn.microsoft.com/windows-hardware/drivers/debugger/time-travel-debugging-overview`

- Chrome DevTools — `https://developer.chrome.com/docs/devtools/`

- Chrome DevTools MCP — `https://github.com/ChromeDevTools/chrome-devtools-mcp`

- Firefox DevTools — `https://firefox-source-docs.mozilla.org/devtools-user/`

- Safari Web Inspector — `https://webkit.org/web-inspector/`

- Python pdb — `https://docs.python.org/3/library/pdb.html`

- pdbpp — `https://github.com/pdbpp/pdbpp`

- ipdb — `https://github.com/gotcha/ipdb`

- pudb — `https://github.com/inducer/pudb`

- debugpy — `https://github.com/microsoft/debugpy`

- Bun docs — `https://bun.sh/docs/runtime/debugger`

- Deno docs — `https://docs.deno.com/runtime/manual/basics/debugging_your_code`

- Node.js inspector — `https://nodejs.org/api/debugger.html`

- Delve — `https://github.com/go-delve/delve`

- tokio-console — `https://github.com/tokio-rs/console`

- cargo-flamegraph — `https://github.com/flamegraph-rs/flamegraph`

- pprof-rs — `https://github.com/tikv/pprof-rs`

- JDK Mission Control — `https://www.oracle.com/java/technologies/jdk-mission-control.html`

- Java Flight Recorder — `https://docs.oracle.com/en/java/javase/21/jfapi/`

- OpenTelemetry — `https://opentelemetry.io/`

- bpftrace — `https://bpftrace.org/`

- BCC — `https://github.com/iovisor/bcc`

- Pixie — `https://github.com/pixie-io/pixie`

- Coroot — `https://coroot.com/`

- Valgrind — `https://valgrind.org/`

- AddressSanitizer — `https://github.com/google/sanitizers/wiki/AddressSanitizer`

- ThreadSanitizer — `https://github.com/google/sanitizers/wiki/ThreadSanitizerCppManual`

- Linux perf — `https://perf.wiki.kernel.org/`

- pprof — `https://github.com/google/pprof`

- Debug Adapter Protocol — `https://microsoft.github.io/debug-adapter-protocol/`

- Pinpoint APM — `https://github.com/pinpoint-apm/pinpoint`

- Cookpad Techlife — `https://techlife.cookpad.com/`

- Mercari Engineering — `https://engineering.mercari.com/en/blog/`

- Kakao Tech Blog — `https://tech.kakao.com/`