Split View: Prometheus TSDB 내부 구조: WAL, Chunks, Blocks, Compaction

data/
  |-- wal/
  |     |-- 00000001
  |     |-- 00000002
  |     +-- 00000003
  |
  |-- chunks_head/
  |     |-- 000001
  |     +-- 000002
  |
  |-- 01BKGV7JBM69T2G1BGBGM6KB12/   (Block ULID)
  |     |-- meta.json
  |     |-- index
  |     |-- chunks/
  |     |     |-- 000001
  |     |     +-- 000002
  |     +-- tombstones
  |
  |-- 01BKGTZQ1SYQJTR4PB43C8PD98/   (Block ULID)
  |     |-- meta.json
  |     |-- index
  |     |-- chunks/
  |     +-- tombstones
  |
  |-- lock
  +-- queries.active

Segment File Internal Structure:
+----------+----------+----------+-----+
| Record 1 | Record 2 | Record 3 | ... |
+----------+----------+----------+-----+

Each Record:
+--------+--------+---------+------+
| Type   | Length | CRC32   | Data |
| 1 byte | varint | 4 bytes | ...  |
+--------+--------+---------+------+

Series Record:
+----------+----------------------------+
| Series ID| Labels (name/value pairs)  |
+----------+----------------------------+

Samples Record:
+----------+-----------+-------+
| Series ID| Timestamp | Value |
+----------+-----------+-------+

WAL Checkpoint Process:
1. Collect current active series list
2. Create checkpoint directory (checkpoint.NNNNN)
3. Write Series Records for active series
4. Delete previous checkpoint and corresponding WAL segments

Head Block:
+------------------------------------------+
| memSeries Map                             |
|   series_id_1 -> memSeries_1             |
|   series_id_2 -> memSeries_2             |
|   ...                                     |
+------------------------------------------+
| Posting Lists (in-memory index)          |
+------------------------------------------+
| Stripe Lock Pool                         |
+------------------------------------------+

memSeries:
+------------+
| ref (ID)   |
| labels     |
| chunks     | --> [chunk_0] -> [chunk_1] -> [chunk_current]
| headChunk  | --> currently active chunk (writable)
| firstTs    |
| lastTs     |
| lastValue  |
+------------+

Sample 1: t1 (raw value stored)
Sample 2: d1 = t2 - t1 (first delta)
Sample 3: dd = (t3 - t2) - (t2 - t1) (delta-of-delta)

With regular scraping, dd is mostly near 0
-> Can be represented with very few bits

Bit encoding:
dd == 0: 1 bit ('0')
-63 <= dd <= 64: 2 + 7 = 9 bits
-255 <= dd <= 256: 2 + 9 = 11 bits
-2047 <= dd <= 2048: 2 + 12 = 14 bits
Otherwise: 4 + 32 = 36 bits

Sample 1: v1 (raw float64 stored, 64 bits)
Sample 2: xor = v2 XOR v1
Sample 3: xor = v3 XOR v2

Consecutive similar values produce XOR results mostly zeros
-> Compression using leading zeros and trailing zeros

Bit encoding:
xor == 0: 1 bit ('0')
Same leading/trailing as previous: 2 + significant bits
Otherwise: 2 + 5(leading) + 6(significant_length) + significant bits

1. New sample arrives
2. Append to headChunk
3. headChunk reaches 120 samples or chunkRange (2 hours)
4. headChunk completion (becomes immutable)
5. Written to chunks_head/ directory as mmap file
6. New headChunk created
7. Previous chunks accessed via mmap (can be freed from memory)

Block Directory:
01BKGV7JBM69T2G1BGBGM6KB12/
  |-- meta.json      (block metadata)
  |-- index          (series index)
  |-- chunks/
  |     |-- 000001   (chunk data)
  |     +-- 000002
  +-- tombstones      (deletion markers)

{
  "ulid": "01BKGV7JBM69T2G1BGBGM6KB12",
  "minTime": 1602547200000,
  "maxTime": 1602554400000,
  "stats": {
    "numSamples": 1234567,
    "numSeries": 5678,
    "numChunks": 9012
  },
  "compaction": {
    "level": 1,
    "sources": ["01BKGV7JBM69T2G1BGBGM6KB12"]
  },
  "version": 1
}

Index File Structure:
+------------------+
| Symbol Table     |  (dictionary of all label names/values)
+------------------+
| Series           |  (per-series labels and chunk references)
+------------------+
| Label Index      |  (label name -> possible values)
+------------------+
| Postings         |  (label pair -> series ID list)
+------------------+
| Postings Offset  |  (offset table for posting lists)
+------------------+
| TOC              |  (Table of Contents)
+------------------+

Example:
job="prometheus" -> [1, 3, 5, 7, 9]
job="node"       -> [2, 4, 6, 8, 10]
instance="localhost:9090" -> [1, 2]
instance="localhost:9100" -> [3, 4]

Query: job="prometheus" AND instance="localhost:9090"
  -> [1, 3, 5, 7, 9] INTERSECT [1, 2]
  -> [1]

Chunk File Format:
+--------+--------+--------+-----+
| Chunk 1| Chunk 2| Chunk 3| ... |
+--------+--------+--------+-----+

Each Chunk:
+----------+----------+--------+------+
| Length   | Encoding | Data   | CRC  |
| uvarint  | 1 byte   | ...    | 4B   |
+----------+----------+--------+------+

Level 0: [2h] [2h] [2h] [2h] [2h] [2h]
                     |
                     v (compaction)
Level 1: [  6h  ] [  6h  ] [  6h  ]
                     |
                     v (compaction)
Level 2: [      18h       ] [  6h  ]

Compaction decision criteria:
1. When 3 or more blocks at the same level exist
2. When merged result time range does not exceed 10% of retention
3. When merged result does not exceed max-block-duration

Compaction process:
1. Select blocks to merge
2. Create new block directory (temporary)
3. Merge all series from source blocks
4. Apply tombstones (remove deleted data)
5. Generate new index
6. Generate new chunk files
7. Update meta.json (increment level, record sources)
8. Atomically activate new block
9. Delete previous blocks (deferred)

Occurs when:
- Backfilling past data
- Out-of-order sample setting enabled
- Duplicate blocks after restoration

Processing:
1. Detect overlapping blocks
2. Merge samples of same series (sort by timestamp)
3. Remove duplicate samples
4. Consolidate into single block

Stage 1 - Deletion request (API call):
  - Record deletion range in tombstones file
  - Actual data still exists

Stage 2 - During compaction:
  - Apply tombstones to exclude data in deleted ranges
  - New block does not contain deleted data

--storage.tsdb.retention.time=15d (default)

Behavior:
1. Blocks with maxTime older than current time minus retention are deletion candidates
2. Checked on each compaction cycle
3. Deleted at block granularity (no partial deletion)

--storage.tsdb.retention.size=10GB

Behavior:
1. Calculate total disk usage of all blocks
2. Delete oldest blocks first when limit exceeded
3. WAL and chunks_head excluded from size calculation

--storage.tsdb.retention.time=30d
--storage.tsdb.retention.size=50GB

- Blocks older than 30 days deleted regardless of size
- If exceeding 50GB, oldest blocks deleted even within 30 days

mmap usage areas:
1. Persistent block index files
2. Persistent block chunk files
3. Head Block completed chunks (chunks_head/)

mmap benefits:
- OS automatically manages memory via page cache
- Only needed portions loaded into memory (lazy loading)
- OS automatically frees pages under memory pressure
- Reduces Go process heap memory burden

Major memory consumers:

1. Head Block memSeries (largest proportion):
   - Approximately 500 bytes to 1KB per series
   - Proportional to active series count

2. Head Block headChunks:
   - One active chunk per series
   - Approximately 100-200 bytes per chunk

3. Posting Lists (in-memory index):
   - Proportional to label cardinality

4. WAL replay buffer:
   - Temporarily high memory usage at startup
   - Released after replay completes

5. Query execution memory:
   - Proportional to concurrent queries and result sizes

--storage.tsdb.out-of-order-time-window=30m

Behavior:
1. Past sample arrives with timestamp older than Head Block's latest
2. If within OOO window, written to separate WBL (Write-Behind Log)
3. Stored as out-of-order chunks in Head Block
4. Merged with in-order chunks during compaction

WBL vs WAL:
- WAL: for in-order samples only
- WBL: for out-of-order samples only
- Both logs used for crash recovery
- WBL only active when OOO window is configured

Write path:
WAL Write (sequential I/O) -> Head Block Update (memory)

Characteristics:
- WAL performs only sequential writes for disk I/O optimization
- Head Block updates are memory operations, extremely fast
- Capable of ingesting millions of samples per second

Read path:
1. Query parsing and optimization
2. Posting list lookup for relevant series
3. Load chunks for the time range
4. Chunk decoding and result computation

Optimizations:
- Fast series filtering via posting list intersection
- Load only necessary chunks via mmap
- Skip irrelevant blocks by time range