필사 모드: The Complete Guide to Concurrency: The Order in Which You Shrink Shared State

• Introduction
• 1. Why Concurrency Bugs Are Different — Failures That Do Not Reproduce
  • 1-1. The failure is a function of timing, not input
  • 1-2. Observing changes the phenomenon
  • 1-3. A passing test is not evidence of absence
  • 1-4. Failures stay behind quietly
• 2. Step Zero: Can the Shared State Be Removed?
  • 2-1. Immutable data and scope reduction
  • 2-2. Key-based partitioning
  • 2-3. Contested: message passing vs shared memory with locks
• 3. What a Data Race Is, and What the Memory Model Gives You
  • 3-1. The definition and DRF-SC
  • 3-2. Why "just reading one integer" is dangerous
  • 3-3. Atomic operations do not give you atomicity
• 4. Setting the Atomicity Boundary
  • 4-1. Write the invariant as a sentence first
  • 4-2. The cost of widening or narrowing the boundary
  • 4-3. What must never happen inside a critical section
• 5. Locks — Pessimistic vs Optimistic, and Lock Ordering
  • 5-1. Pessimistic and optimistic locking
  • 5-2. The same problem at the HTTP boundary
  • 5-3. Lock ordering — the only practical deadlock prevention
  • 5-4. The trap in read/write locks
• 6. Beyond a Single Process: The Boundary Between DB Transactions and Distributed Locks
  • 6-1. The order: transaction → conditional update → distributed lock
  • 6-2. What a distributed lock does not guarantee
  • 6-3. Idempotency is part of concurrency design
• 7. Queues and Backpressure — Where to Put the Waiting Line
  • 7-1. Queues already exist, several layers deep
  • 7-2. An unbounded queue converts an outage into latency
  • 7-3. Bounded queues and rejection policies
  • 7-4. Contested: async/await vs threads
• 8. How to Test Concurrent Code
  • 8-1. Separate out the part you can make deterministic
  • 8-2. Widen the race window on purpose
  • 8-3. Repetition, race detectors, property-based tests
  • 8-4. Do not ignore flaky tests
• 9. Failure Mode Catalogue
• Quiz: Check Your Understanding
• Closing
• References
• Further reading

Introduction

This blog already has two posts on concurrency. Concurrency vs Parallelism explains how the two concepts differ, and Comparing Distributed Lock Patterns compares the infrastructure you might build a lock on. This post fills the gap between them. It covers the working order an application designer actually follows: remove shared state where you can, narrow it where you cannot, then defend and test whatever is left.

The internals of database isolation levels (MVCC, snapshot isolation) are deliberately out of scope. This post goes only as far as the boundary question — "when should the storage layer own this instead of the application?" — and leaves everything inside that boundary to the database category.

Compressed into one line: most of concurrency design is not the skill of taking locks well, it is the skill of having less to lock.

1. Why Concurrency Bugs Are Different — Failures That Do Not Reproduce

1-1. The failure is a function of timing, not input

An ordinary bug is a function of its input. Same input, same failure — which is why you can build a reproduction case and a regression test. A concurrency bug is a function of scheduling. Same input, same code, same server, and yet the outcome depends on exactly when a thread got preempted. The reproduction case exists only probabilistically.

1-2. Observing changes the phenomenon

Adding one log line and watching the problem disappear is a common experience. Logging is I/O, and I/O yields the thread, which changes the width of the race window. Debuggers are worse: a breakpoint effectively behaves like a global lock. So concurrency problems are hard to narrow down by observation, and static reasoning — reading the code and checking invariants — carries far more of the weight.

1-3. A passing test is not evidence of absence

If the race window is 100 nanoseconds, a thousand local runs are unlikely to land inside it. Production, called thousands of times per second, will land inside it within a day. In concurrency, the design argument comes before the test result. You must be able to write "why is this code safe" as a sentence; the tests are the net that catches the moment that argument collapses.

1-4. Failures stay behind quietly

Concurrency bugs do not throw an exception and stop. The balance is right but one transaction record is missing; stock is zero but two orders got through. Discovery happens days later during reconciliation, by which time the logs from the moment of failure are gone. There is one practical check question: "if this function runs twice at exactly the same instant, what breaks?"

2. Step Zero: Can the Shared State Be Removed?

The first step is not choosing a lock, it is reducing what has to be locked. Here is the ladder, ordered from cheapest.

Step 0  Do not share       immutable data / copy by value / pure computation
Step 1  Move ownership     exactly one owner at a time, hand off by message
Step 2  Narrow the scope   request scope · thread local · task local
Step 3  Split by key       the same key always goes to the same worker/partition
Step 4  Lock what is left  define atomicity boundary → lock → test

2-1. Immutable data and scope reduction

Read-only data is safe no matter how many threads touch it. Values that change occasionally and are read constantly — configuration, routing tables — should not be edited in place; build a new object and swap the reference wholesale. The swap is a single reference write, and readers keep seeing their old snapshot. Note that an immutable collection whose elements are mutable guarantees nothing.

• Request scope: an object that lives only for the duration of a request is not shared. The moment you park it in a singleton cache for convenience, it becomes shared state.
• Thread local: good for trace IDs and transaction contexts, but values from a previous request leak when a pooled thread is reused, so a cleanup path is mandatory.
• Ownership transfer: fill a buffer, hand it off, and never touch it again. This is enforced only by discipline, so it has to be checked explicitly in review.

2-2. Key-based partitioning

The most practical reduction technique. Do not keep state globally; split it by key and always route the same key to the same worker. It fits domains with a natural key — per-account balances, per-order state machines. Within a key you get sequential processing without locks, and across keys there is no contention.

The cost is equally clear. A hot key makes that key's throughput the whole system's bottleneck, and during a rebalance that changes the partition count there is a window where the same key is attached to two workers at once. Lease checks and idempotent handling are needed for that window.

2-3. Contested: message passing vs shared memory with locks

This is where practitioners genuinely disagree. Be suspicious of any article that declares one side correct.

• The message-passing side (actors, CSP): one owner monopolizes the state and everyone else asks by message, so data races disappear structurally. A single owner makes reasoning easy.
• The shared-memory-with-locks side: message passing does not remove races, it only changes their shape. Reordered requests, unbounded mailbox growth, and deadlock while awaiting a reply all remain. Invariants spanning multiple actors end up forcing you to hand-implement a coordination protocol.
• The real axes: is the invariant contained within one entity or spread across several; which model does your runtime make cheap; does your team debug from stack traces or from message logs.

If an invariant is contained in one entity, message passing wins. If several entities must move together, a store with transaction boundaries wins. This is not taste — it depends on the shape of the invariant.

3. What a Data Race Is, and What the Memory Model Gives You

3-1. The definition and DRF-SC

The Go memory model defines a data race as "a write to a memory location happening concurrently with another read or write to that same location, unless all the accesses involved are atomic data accesses," and states the programmer's obligation plainly: "Programs that modify data being simultaneously accessed by multiple goroutines must serialize such access."

The reward for meeting that obligation matters. The same document says "In the absence of data races, Go programs behave as if all the goroutines were multiplexed onto a single processor." This property is usually called DRF-SC. If you leave zero races, you never have to reason about instruction reordering or cache visibility. Leave even one, and no intuition about that program's behavior is guaranteed.

3-2. Why "just reading one integer" is dangerous

An unsynchronized read is often excused as "the value is just slightly stale." The actual range of outcomes is wider. If the compiler hoists a read out of a loop into a register, you never see the update at all. Reordering can make another thread observe the flag being set before the data was written. Values wider than a machine word can be read half-updated.

// Example — an unsynchronized flag does not guarantee termination
var done bool          // touched by several goroutines, protected by nothing
go func() { work(); done = true }()
for !done {            // the compiler may hoist done into a register
}

3-3. Atomic operations do not give you atomicity

The most common misunderstanding. An atomic type guarantees the atomicity of one operation. String operations together and the gap between them is open.

Safe        counter.Add(1)                 a single atomic operation
Dangerous   if counter.Load() < limit {    the gap between read and write is open
                counter.Add(1)
            }

This is a check-then-act race, and it produces an important distinction. You can have no data race and still have a race condition. Every access above is atomic, so there is no data race, yet the invariant "we never exceed the limit" is broken. A race detector catches the former and not the latter.

A race detector only catches races that actually occurred on an executed path, so it earns its keep when run in CI alongside integration tests with wide coverage.

4. Setting the Atomicity Boundary

4-1. Write the invariant as a sentence first

A critical section is not measured in lines of code, it is defined per invariant. Write down the sentence you have to protect: "the balance can never go negative," "the sum of reserved seats never exceeds total seats," "an order cannot reach shipping without passing payment complete." Then find the span in which that sentence is momentarily false — that whole span is one boundary.

Example — the atomicity boundary of a balance debit

  ├─ read the current balance   ┐
  ├─ compare to the amount      │ if another party reads or changes the balance
  ├─ subtract from the balance  │ between these four steps, the invariant breaks
  └─ write the new balance      ┘

4-2. The cost of widening or narrowing the boundary

Too narrow and the invariant breaks. That is a correctness problem and it is not negotiable. Too wide and throughput drops. That is a performance problem, so it can be measured and negotiated. Therefore take the wide boundary first to secure correctness, and narrow it only after measurement confirms a bottleneck. Doing it the other way plants correctness defects under the name of optimization.

4-3. What must never happen inside a critical section

The length of a critical section is the waiting time of everyone else.

• Network calls and disk I/O: the latency is not under your control. One slow external API stalls every thread queued on that lock.
• Acquiring another lock: the moment you take a second lock while holding the first, deadlock becomes possible. If it is unavoidable, the ordering rule in 5-3 is mandatory.
• Invoking user callbacks: you cannot know what happens inside, and reentering to take the same lock again is a common accident.

Reduced to a pattern: inside the lock touch only memory, do computation and I/O outside. Copy the data you need out under the lock, process it outside, and take a short lock again if you must apply the result. Because state may have changed by that second lock, revalidation is required.

5. Locks — Pessimistic vs Optimistic, and Lock Ordering

5-1. Pessimistic and optimistic locking

Pessimistic locking takes the lock first and then works. It fits stock deduction or seat assignment, where contention is genuinely frequent and a retry would be visible to the user. The costs are waiting, deadlock risk, and a throughput ceiling set by lock hold time. An acquisition timeout is not optional — unbounded waiting turns an outage into a deadlock and makes diagnosis much harder.

Optimistic locking proceeds without a lock and, at write time, checks "has this changed since I read it?"

-- Example — optimistic locking with a version column
UPDATE accounts
   SET balance = balance - 1000,
       version = version + 1
 WHERE id = 42
   AND version = 7;      -- zero rows affected means conflict → retry

It wins when conflicts are rare and reads dominate, but as conflicts grow, retries explode and it becomes slower than a lock. You need a retry cap and backoff, and on retry you must re-read the data. Retrying from the old snapshot fails forever.

5-2. The same problem at the HTTP boundary

When a client reads a value, the user edits it, and another user has already changed it in the meantime, that is the lost update problem. RFC 6585 defines 428 Precondition Required, stating that the code means "the origin server requires the request to be conditional" and that its purpose is to prevent the lost update problem. The server rejects unconditional updates with 428, and the client resends with the validator in a conditional header.

PUT /accounts/42 HTTP/1.1
If-Match: "v7"

# If the server is already at v8 → 412 Precondition Failed
# If the conditional header is missing entirely → 428 Precondition Required

5-3. Lock ordering — the only practical deadlock prevention

If the thread holding A waits for B while the thread holding B waits for A, both stop forever. There is exactly one preventive measure that works in practice: give every lock a global order and always acquire in that order.

Lock hierarchy example (acquire top to bottom only)

  1. tenant lock   2. account lock   3. order lock   4. cache shard lock

  Rule: you may not take 2 while holding 3.
        Within one level, acquire in sorted key order.

When you must take two locks at the same level (transfers between accounts are the classic case), sort the keys and take the smaller one first. That single rule makes circular waiting structurally impossible. An acquisition timeout surfaces deadlock as a failure instead of a permanent stall, but it is a detection tool, not a preventive one.

5-4. The trap in read/write locks

They look attractive when reads dominate, but adopting them without measurement is often a loss. When critical sections are short, the extra bookkeeping makes them slower than a mutex, and if reads never stop, writes starve. Code that holds a read lock and tries to upgrade to a write lock is a textbook deadlock pattern.

6. Beyond a Single Process: The Boundary Between DB Transactions and Distributed Locks

The moment you go from one process to two, the lock your language gives you guarantees nothing. Each instance takes its own mutex and updates the same row simultaneously. It is the classic incident that surfaces on the day autoscaling is first enabled.

6-1. The order: transaction → conditional update → distributed lock

1. Does it fit inside one database transaction? If so, finish there. Proven concurrency control already lives inside the store.
2. Can it be expressed as a constraint or a conditional update? Unique constraints and conditional updates of the WHERE version = ? form produce correctness without a lock. "Prevent duplicate creation" is usually a unique-constraint problem, not a lock problem.
3. Does it span multiple stores or external systems? Only then does a distributed lock enter.

Designs that jump straight to 3 are common, but most cases end at 1 or 2.

6-2. What a distributed lock does not guarantee

A distributed lock does not guarantee mutual exclusion. It only lowers the probability. A lock is usually a TTL lease, so if the work outlasts the TTL, the lease expires, another instance takes the same lock, and the original instance keeps writing in the belief that it still holds it. A few seconds of GC pause or container throttling produces the same situation, and application code has no way to detect it. If TTL arithmetic depends on per-node clocks, clock drift becomes a correctness problem directly.

So a distributed lock is a contention reducer, not the last line of defense for correctness. Put the last line in the store: a conditional update that checks a version at write time, or an idempotent design where a second arrival yields the same result.

6-3. Idempotency is part of concurrency design

Any system with retries eventually runs the same work twice. RFC 9110 defines idempotency as "the intended effect on the server of multiple identical requests is the same as for a single request" and lists GET, HEAD, PUT, DELETE, OPTIONS and TRACE as idempotent methods. POST is not idempotent. Resources created by POST therefore need a separate idempotency key.

• The client generates a unique idempotency key per request.
• The server stores the result under that key and returns the stored result when the same key arrives again.
• The key must be stored inside the same transaction as the real work. Otherwise a failure in between leaves a duplicate.
• The same key can arrive twice concurrently, so the unique constraint on the key table is the real line of defense.

The procedure is covered in more detail in Idempotency and Retries.

7. Queues and Backpressure — Where to Put the Waiting Line

7-1. Queues already exist, several layers deep

Before debating whether to add a queue, admit something: the socket receive buffer, the thread pool work queue, the connection pool wait list and the database lock wait list are all queues. The design question is not "should we have a queue" but which queue hits its limit first and what happens then.

request → [socket buffer] → [thread pool queue] → [pool wait] → [DB lock wait] → work

  Every stage needs a bound and a rejection policy.
  A single unbounded stage becomes the hole where latency and memory drain away.

7-2. An unbounded queue converts an outage into latency

Without a bound, overload shows up not as failure but as rising latency. The problem is that the latency attaches even to requests that no longer matter. The client timed out and retried 30 seconds ago, and the server is still processing work nobody is waiting for.

The cascading failure chapter of the Google SRE book defines this structure as "a failure that grows over time as a result of positive feedback." Overload causes retries and retries increase overload. The same chapter points out the multiplicative effect of layered retries: three layers retrying four times each turn one user action into 64 attempts.

7-3. Bounded queues and rejection policies

• Set a bound: the queue-length bound is roughly "acceptable wait time ÷ average service time." With a 200 ms target, 20 ms of work and ten workers, the bound is about 100.
• Reject early: the SRE book recommends shedding early (for example returning 503) instead of queueing indefinitely. Rejection is not failure, it is protection.
• Drop stale items: record the enqueue time and discard anything past the client timeout the moment you dequeue it.
• Control retries: the SRE book states "Always use randomized exponential backoff when scheduling retries," gives a retry budget example such as 60 retries per minute per process, and specifies "Don't retry a given request indefinitely." Permanent errors and retriable errors must be distinguished by error code, and permanent errors are never retried.
• Signal upstream: real backpressure is not rejection, it is slowing the producer down. Blocking writes on a bounded buffer and credit-based flow control belong here.

You can simulate how a retry policy changes success probability with the Retry Probability Calculator.

7-4. Contested: async/await vs threads

The choice of execution model has no settled answer either.

• The async side: it does not pin a thread per connection, so on I/O-bound workloads the memory and context-switch cost is far lower. If tens of thousands of concurrent connections are the goal, the options narrow quickly.
• The thread side: stack traces stay intact, debuggers work as usual, and blocking libraries can be used directly. Async creates the function-color problem and spreads through the whole codebase, and a single CPU-heavy line on an event loop stalls everything.
• The real axes: is the workload I/O-bound or CPU-bound; does the runtime offer lightweight threads; which side is the dependency ecosystem on; which side is your team's debugging tooling ready for.

What matters is that neither model removes the shared-state problem. Even on a single-threaded event loop another task can interleave at an await point, so a check-then-act straddling an await still breaks. Changing the execution model leaves the work in sections 2 through 5 exactly where it was.

8. How to Test Concurrent Code

8-1. Separate out the part you can make deterministic

The most effective technique is to reduce the number of tests that "run things concurrently and hope." Extract state-transition logic into pure functions and that part becomes deterministically testable, thinning the remaining concurrency surface.

8-2. Widen the race window on purpose

Instead of relying on random execution, force the interleaving you want. Plant a barrier or latch as a test-only hook so two threads meet at exactly that point.

Example — a test that forces two updates to overlap

  thread A: read ──────┐(wait on barrier)──────→ write
  thread B: read ──────┘(release barrier)──────→ write

  Expected: the second write is rejected as a conflict.
  This reproduces lost update structurally, not probabilistically.

8-3. Repetition, race detectors, property-based tests

Run concurrency tests with the repetition count turned up. Once in CI and several hundred times in a nightly job is a realistic split. Turn the race detector on in a separate job and run it alongside integration tests. If there is randomness, log the seed. It is the only clue you will have for reproduction.

The property that a concurrent result must be explainable by some sequential order expresses well as a property test. Running random operation sequences concurrently and comparing against a sequential model sweeps far more interleavings than hand-written cases (Property-Based Testing in Practice).

8-4. Do not ignore flaky tests

Papering over intermittent failures with a re-run is the most common way concurrency defects get hidden. A flaky test is usually one of two things: badly written, or the discovery of a real race. You need a rule against covering it with retry settings before you know which.

9. Failure Mode Catalogue

A pattern repeats through this table. Most of these entries would not exist at all had the shared scope been smaller. Going one more rung down the ladder in section 2 is worth more than memorizing this table. If the symptom is a latency spike, Connection Pool Sizing and the Circuit Breaker Pattern are worth checking as well.

Quiz: Check Your Understanding

Closing

The most-used tool in concurrency design is not the mutex, it is deletion. Stop sharing what does not need sharing, stop making global what does not need to be global, split what can be split by key — and what actually has to be locked shrinks dramatically.

For whatever remains, follow the order. Write the invariant as a sentence, make the span where it breaks your atomicity boundary, touch only memory inside that boundary, put storage constraints and idempotency as the last line of defense once you cross a process boundary, and attach a bound and a rejection policy to every queue. "It's probably fine" is not a design. If you cannot write down why it is safe in one paragraph, that code is not finished.

References

• The Go Memory Model — go.dev — quoted for the definition of a data race, the obligation to serialize concurrent access, and the sequential-consistency (DRF-SC) sentence. Checked 2026-08-15.
• RFC 6585: Additional HTTP Status Codes — IETF — quoted for 428 Precondition Required requiring conditional requests to prevent the lost update problem. Checked 2026-08-15.
• RFC 9110: HTTP Semantics — IETF — quoted for the definition of idempotency, the list of idempotent methods, and the fact that POST is not idempotent. Checked 2026-08-15.
• Google SRE Book — Addressing Cascading Failures — quoted for the definition of cascading failure, the randomized exponential backoff recommendation, the retry budget example, the multiplicative effect of layered retries, and the recommendation to shed early. Checked 2026-08-15.
• The shared-state reduction ladder, the lock hierarchy example, and the failure mode table in section 9 do not appear in the sources above — they are procedures assembled in this post.

Further reading

• Related post on this blog: Concurrency vs Parallelism
• Related post on this blog: Comparing Distributed Lock Patterns
• Related post on this blog: Idempotency and Retries: APIs You Can Trust
• Related post on this blog: Why a Bigger Connection Pool Costs You
• Related tool: Retry Probability Calculator
• Related tool: Rate Limit Simulator

Complete Guide Series

• Previous: The Complete Guide to Error Handling: Designing Failure as a Contract
• Next: The Complete Guide to Test Strategy: Decision Criteria Instead of the Pyramid Debate