Skip to content
Published on

Iceberg v3 Row Lineage: Row IDs Aren't Stored in the File

🇰🇷KO🇺🇸EN🇯🇵JA🇨🇳ZH
Split View필사 모드
Share
Authors

Introduction — In v3, Row Lineage Can't Be Turned Off

The "Format Versioning" section of the Apache Iceberg spec itself opens like this — versions 1, 2, and 3 are complete and adopted by the community, while version 4 is under active development and not yet officially adopted. Which means v3 is no longer future tense. The Java library already has its constants set to read and write up through v4 (SUPPORTED_TABLE_FORMAT_VERSION = 4).

As a list, what v3 adds looks like this — nanosecond timestamps, the unknown/variant/geometry/geography types, column defaults, multi-argument partition transforms, row lineage tracking, binary deletion vectors, and table encryption keys. Deletion vectors have gotten a lot of coverage. This post looks at row lineage, which has been covered relatively less, and which can bite your pipeline far more quietly.

One fact to know up front: row lineage is not a table property you switch on or off. Here is the entirety of Iceberg core's determination logic.

public static boolean supportsRowLineage(Table table) {
  Preconditions.checkArgument(null != table, "Invalid table: null");
  if (table instanceof BaseMetadataTable) {
    return false;
  }

  return formatVersion(table) >= TableMetadata.MIN_FORMAT_VERSION_ROW_LINEAGE;
}

MIN_FORMAT_VERSION_ROW_LINEAGE is 3. In other words, if the format version is 3 or higher, row lineage is on. The early design had a metadata field to toggle it, but that field was removed ("Core: Enable row lineage for all v3 tables", PR #12593), and the spec is categorical about it — at v3 and above, an Iceberg table must track the row lineage fields for every newly created row. The moment you upgrade to v3, every engine writing to that table takes on the obligation to maintain lineage. Whether you wanted that feature or not.

One reassuring fact in the other direction: in that same file, DEFAULT_TABLE_FORMAT_VERSION is still 2 — true on the main branch as of July 2026 as well. Creating a new table does not make it v3 — v3 is a door you have to walk through explicitly.

What Row Lineage Gives You

v3 defines two reserved metadata columns.

Field IDNameTypeMeaning
2147483540_row_idlongA unique identifier for each row within the table
2147483539_last_updated_sequence_numberlongThe sequence number of the commit that last modified that row

The key point is that _row_id is the row's identity. It has to stay the same even when the row is UPDATEd, even when compaction rewrites it into a different file, even when the partition changes. This separates physical location from logical identity — without it, you cannot answer "when did this row first appear, and how many times has it changed since" without a separate ledger outside the table.

Add _last_updated_sequence_number to that, and "give me only the rows that changed after sequence number N" becomes expressible purely with table metadata. This is the foundation for building incremental processing and CDC without external tooling.

The Core Design — Inherited, Not Written

This is the whole of the feature. _row_id is not stored in the data file.

When writing a new row, the writer leaves _row_id and _last_updated_sequence_number as null. It's even fine to omit the columns entirely — the spec says that if the column is absent, the reader should treat it "as if a column that is null for all rows" existed. The actual values are computed at read time.

As for why, the spec writes the answer itself — the commit sequence number and the starting row ID aren't determined until the snapshot commits successfully. Iceberg's commits use optimistic concurrency control. When two writers try to commit at the same time, one loses, and the loser retries on top of the latest snapshot. If row IDs were baked into the data file, every retried commit would have to rewrite every data file and manifest file already written. The bigger the write, the bigger the cost of a single conflict. In the spec's own words, inheritance exists to let files be written before the values are determined, so that optimistic commits don't have to rewrite files when they retry.

The tradeoff is clear. Because there's no stored value, the chain that produces the value has to be exact. The chain flows down like this.

table metadata     next-row-id           (starting point for the next IDs to hand out)
   └─ snapshot      first-row-id         = table's next-row-id at commit time
        └─ manifest   first_row_id       = shifted by the row count of preceding manifests
             └─ data file  first_row_id  = shifted by the record_count of preceding files
                  └─ row      _row_id    = data file's first_row_id + _pos

The bottom line is the conclusion — a row's ID is the first_row_id of the data file it lives in, plus its position within the file (_pos). Instead of storing the ID, the design reserves and hands out ranges of ID space, then offsets by position.

Walking Through the Spec's Example

The example in the spec shows this chain well. Start with a table whose next-row-id is 1000. An append snapshot receives the table's current next-row-id, 1000, as its first-row-id.

When the manifest list is written, each manifest is assigned a first_row_id.

manifest_pathadded_rows_countexisting_rows_countfirst_row_id
existing750925
added1100251000
added201001125
added3125251225

added1 receives the same 1000 as the snapshot, and from there each subsequent manifest is shifted by the preceding manifest's row count (added rows + existing rows): 1000 + 125 = 1125, 1125 + 100 = 1225.

There's a trap the spec flags here — added2 has zero added data files, yet it still shifts the next manifest's first_row_id by 100. A data file without a first_row_id can still be assigned an ID even in the EXISTING state, so that much space is reserved for it.

The data files inside the first manifest, added1, look like this.

statusfile_pathrecord_countfirst_row_id
EXISTINGdata125800
ADDEDdata250null → 1000
ADDEDdata350null → 1050

data1 was already assigned 800 previously, so that value is simply copied over. data2 and data3 are written as null, and at read time they get 1000 and 1050 respectively, starting from the manifest's first_row_id (1000) and shifting by the preceding files' record_count. So the third row in data3 has a _row_id of 1050 + 2 = 1052.

Once the commit finishes, the table's next-row-id is updated. The spec's mandatory clause says to shift by at least the number of row IDs newly assigned in that snapshot, and the recommended calculation is to sum the added_rows_count and existing_rows_count of every manifest that was assigned a first_row_id. In the example that sum is 375 (added1's 125 + added2's 100 + added3's 150), so next-row-id becomes 1000 + 375 = 1375.

Notice that EXISTING rows consume ID space here too. Even though they already have their own IDs and don't get new ones, a range is still reserved for them. Since the mandatory rule only sets a floor and the recommended calculation runs generously above it, gaps in the ID space are normal. _row_id is unique but not dense — don't use it to count rows or assume continuity.

The Rules at Read Time

The rules on the reader side are nailed down in Appendix E of the spec.

  • If a data file's first_row_id is non-null, the reader fills any null or missing _row_id with first_row_id + _pos.
  • Likewise, it fills any null or missing _last_updated_sequence_number with that data file's data_sequence_number.
  • A value that's already non-null is read as-is, untouched.
  • If a data file's first_row_id is null, the reader outputs _row_id and _last_updated_sequence_number as null.

The third rule matters. If a value already exists, that means it's a "row that was carried over," and its identity must not be overwritten.

Where This Gets Real — The Places Lineage Breaks

Compaction Kills Lineage

When a row is moved to a different data file (compaction, rewrite, a partition change — whatever the reason), the rules a writer has to follow are these.

  1. A row's existing non-null _row_id must be copied to the new data file.
  2. If the write modified the row, _last_updated_sequence_number is left null (so the sequence number at modification time is inherited).
  3. If it wasn't modified, the existing non-null _last_updated_sequence_number is copied over as-is.

That sounds simple, but it's nasty from an implementation standpoint. Compaction used to be a simple job — "read the data and rewrite it into bigger files." In v3, though, the compaction job has to read and write two hidden metadata columns alongside the data. Just rewrite naively, and every _row_id gets freshly reissued, turning every row in the table into a "row that just appeared for the first time." A CDC pipeline built on top of lineage would end up doing a full reprocess. Silently, without a single error.

The proof that this isn't a theoretical worry is all in the commit logs. We'll get to it below.

Equality Deletes Don't Track Lineage

The spec carves out an explicit exception — row lineage is not tracked for rows modified via equality delete. The reason is in the spec too. An engine using equality delete writes the change without reading the existing data in the first place, so it has no way to put the original row ID on the new row. Such an update is therefore treated as if the existing row were entirely removed and a brand-new row were added.

This is a big deal. For a workload that relies on equality delete — like Flink's upsert streaming — row identity does not survive an UPDATE even after upgrading to v3. A plan to build CDC on top of row lineage can fall apart roughly halfway right here.

An Upgraded Table's Past Is Permanently Null

When a v2 table is upgraded to v3, next-row-id is initialized to 0, and existing snapshots are not modified. In those snapshots that have no first-row-id, the first_row_id on the data files and manifests is null, so every row's _row_id reads as null. In the spec's own words — snapshots created before the upgrade have no row IDs.

The first commit after the upgrade does assign IDs to existing files too. But that means "those files just received IDs right now," not that their past history has been restored. Row lineage is not retroactive.

ID Ranges Diverge Across Branches

There's one footnote the spec leaves quietly. After an upgrade, new snapshots on different branches assign non-overlapping ID ranges to existing data files, based on the next-row-id at commit time. For a data file that lives on multiple branches, a writer may reuse another branch's first_row_id, or it may assign a new first_row_id to avoid a large metadata rewrite.

In other words, in a table that uses branches, _row_id is not guaranteed to point at the same row across branches. If you were planning to use _row_id as a join key, this is where that plan should stop.

Engine Reality Check — The Gap Between the Spec's "Must" and the Implementation

This is the reason this post exists. Vendor blogs draw "v3 support" as a checkbox. The actual code tells a different story. What follows is what I directly verified against each repository's main/master branch and merged PRs, as of July 16, 2026.

Spark 4.0 is the de facto reference. Row lineage support landed via PR #13310 on July 14, 2025. According to the PR description, it uses a conditional nullification mechanism that's new in Spark 4.0; that mechanism doesn't exist in Spark 3.5, so Iceberg had to implement its own separate rewrite rules there (PR #12736, April 28, 2025). The rules that PR lays out are exactly the spec we saw earlier — the existing row ID is always carried forward regardless of the operation type, it's null on INSERT, and the sequence number is preserved for unmodified rows and set to null for modified ones.

Compaction preservation was a separate piece of work. This is exactly why I said earlier that "compaction kills lineage." Compaction lineage preservation for Spark 4.0 only landed via PR #13555 on July 22, 2025, and was backported to 3.5 and 3.4 the very next day (PR #13637, #13641). The implementation approach is interesting — in the PR author's own words, for a rewrite it works by "hijacking" SparkTable's schema to add the metadata columns. The author is well aware this isn't elegant.

Trino is further along than I expected. This one nearly fooled me via search results — several secondary sources state that "Trino wasn't v3-ready as of June 2026" or that "row lineage is still pending," and that isn't true once you look at the repository directly. Deletion vectors merged to master via PR #27788 on January 16, 2026, and row lineage via PR #27836 on March 19, 2026. The latter covers not just reads but preserving the original row ID on UPDATE/MERGE, and re-enabling OPTIMIZE on v3 tables while keeping lineage intact.

What Trino did before that is also worth a look. PR #27786 (January 10, 2026) opened up v3 table creation, while making sure that every v3 feature not yet implemented is explicitly rejected with NOT_SUPPORTED. DELETE/UPDATE/MERGE, OPTIMIZE, add_files, deletion vectors, column defaults, and encryption on v3 tables were all caught by this. As the PR description puts it, "to avoid violating the spec," they chose to fail fast and predictably instead of quietly returning a wrong answer. That's the right call for a partial implementation, and other engines could learn from it.

Column names differ per engine. Iceberg core and Spark use _row_id / _last_updated_sequence_number, while Trino, following its own metadata-column convention, exposes them as $row_id / $last_updated_sequence_number. The field IDs are the same, so the data is compatible, but queries don't port over as-is.

Flink is forked into two paths. This is the most confusing part. The Flink connector on the Iceberg main branch has two compaction paths, and they behave in opposite ways regarding row lineage.

The old actions API (org.apache.iceberg.flink.actions.RewriteDataFilesAction) still simply rejects v3 tables. This code is present as-is across v1.20, v2.0, and v2.1 on the main branch.

Preconditions.checkArgument(
    !TableUtil.supportsRowLineage(table),
    "Flink does not support compaction on row lineage enabled tables (V3+)");

The new maintenance API (org.apache.iceberg.flink.maintenance.operator.DataFileRewriteRunner), on the other hand, preserves lineage. It landed via PR #14149 on November 6, 2025, and the code branches like this.

boolean preserveRowId = TableUtil.supportsRowLineage(value.table());

try (TaskWriter<RowData> writer = writerFor(value, preserveRowId)) {
  try (DataIterator<RowData> iterator = readerFor(value, preserveRowId)) {

To sum up — compacting a v3 table with Flink requires the maintenance API, and if you're on the old actions API, the moment you upgrade to v3, compaction stops by throwing an exception. That sounds like bad news, but it's actually good design. Failing loudly beats silently destroying lineage. For what it's worth, that rejection logic itself was added deliberately, via PR #13646 ("Flink: fail file rewrite for V3 tables as row lineage not supported", July 23, 2025).

PyIceberg still can't write v3 tables. pyiceberg/table/metadata.py on the main branch still reads like this.

SUPPORTED_TABLE_FORMAT_VERSION = 2

PR #3070, which set out to add v3 manifest/manifest-list writing and row lineage snapshot commits, was closed without merging, and its successor, PR #3551 ("Writing v3 Table Metadata"), is still open, last updated on July 8, 2026. The v3 upgrade support PR #3623 is a draft waiting on #3551 to merge. If you have a pipeline that writes to Iceberg with Python, the accurate move is to leave it out of your v3 plans for now.

And the inheritance logic is still being fixed. A week before writing this post, on July 9, 2026, PR #17039 ("Core: Fix row lineage last updated sequence inheritance") was merged. Here's the bug — Java was using fileSequenceNumber() instead of data_sequence_number when filling in the _last_updated_sequence_number metadata column.

The difference between the two is usually invisible. But if compaction back in v2 days created a replacement file carrying an older data sequence number, and that table is later upgraded to v3, the two diverge. data_sequence_number is tied to the file's content, so it survives a rewrite, while file_sequence_number is when that physical file was added. So the buggy version, when asked "when was this row last modified," answered with when compaction happened to run instead of the original modification time.

This shows exactly what inheritance-based design costs. Since the value isn't in the file, get one inheritance rule wrong and the data is fine but the answer is wrong — and that wrongness never throws an exception. The fact that a bug like this still surfaces 21 months after the spec first introduced row lineage in October 2024 (PR #11130) tells you this feature's difficulty runs far ahead of how much space the spec document gives it. There were similar-natured fixes on the Avro reader side too, in February and March 2026 (PR #15187, #15508).

When Not to Use It

Laid out honestly, here's where things stand.

Don't upgrade to v3 yet if

  • You have even one PyIceberg writer in your pipeline. It can't write v3 tables right now.
  • You run compaction with Flink's old actions API and have no plan to migrate to the maintenance API.
  • Your workload is centered on equality delete (typically Flink upsert streaming). You'd upgrade for lineage, and row identity still wouldn't survive an UPDATE.
  • Multiple engines write to the same table, and even one of them hasn't had its lineage preservation verified. A single writer that doesn't honor lineage breaks lineage for the entire table.

It's fine to upgrade if

  • Your writers are limited to Spark 4.0 (or the backported 3.5/3.4) or a recent Trino.
  • You've actually verified that your compaction path preserves lineage — by comparing _row_id before and after compaction yourself, not by reading documentation.
  • The benefit of deletion vectors alone already pays for the upgrade. Row lineage comes along as a bonus with v3; the actual reason most organizations move to v3 is deletion vectors.

And before using row lineage for CDC, be sure to confirm this — _row_id is unique but not dense, it isn't stable across branches, it doesn't exist for pre-upgrade history, and it doesn't survive equality delete. Check first whether these four facts conflict with your design.

Closing

Row lineage's design is clever. Instead of storing the row ID, it reserves and hands out ranges of ID space and offsets by position, so that data files don't need to be rewritten even when an optimistic commit retries. Anyone who knows Iceberg's commit model will immediately understand why it's shaped this way.

The cost is a shift in responsibility. Because the value isn't in the file, producing and preserving it correctly depends entirely on the writer and reader getting it right. The spec writes "must," but the spec doesn't execute code. So the picture as of July 2026 looks like this — Spark 4.0 works, Trino has worked since March, Flink splits depending on which API you use, PyIceberg still doesn't, and core's inheritance logic was still being fixed as recently as last week.

This isn't an argument against v3. Deletion vectors alone already make v3 worth going to. But you need to go in knowing that the moment you upgrade to v3, row lineage switches on with no way to turn it off, and every tool that touches that table signs onto an implicit contract. If even one tool that can't honor that contract is mixed in, what you get isn't lineage — it's the illusion that you have lineage. That's much worse.

Iceberg's overall structure and operations are covered separately in Apache Iceberg Data Lakehouse Table Format Guide.

References

Discuss on Twitter • View on GitHub