A Practical Procedure for Moving a Codebase with AI — Stand Up the Judge First, and Measure Review Rate Instead of Lines

Introduction — Get the Order Wrong and the Loss Grows With Scale
Step 0 — Stand Up the Judge Before Translating, and Debug the Judge First
Step 1 — The Rulebook, the Dependency Map, and a 3-File Dress Rehearsal
Step 2 — Cut Along Verifiable Boundaries and Delegate the Queue to Disk
Step 3 — What to Measure: Lines Converted Is Not Progress
Step 4 — Three Failure Modes and How to Detect Each
Closing — The Value of This Procedure Is Not Speed but Reversibility
References

Introduction — Get the Order Wrong and the Loss Grows With Scale

In the first half of 2026 a run of large-scale LLM migrations was published back to back. Bun moved 535,000 lines of Zig to Rust in 11 days (the announcement), and Anthropic's Mike Krieger moved 165,000 lines of Python to TypeScript over a single weekend (a case carried in a methodology writeup). The two cases differ in language, in scale, and in team composition, yet the procedure is startlingly similar.

And what matters most in that procedure is not any individual technique but the order. Start translating first and bolt verification on afterward, and the loss compounds exponentially with scale. Port 100 files wrong and then discover the rules were wrong, and you can just rerun 100; discover it after 1,400, and by that point the next stage of work is already stacked on top.

This post takes only the reproducible parts of the published primary sources (Anthropic's migration kit and the methodology document, plus the Bun announcement) and lays them out in order. I kept only the parts that are not tied to a particular model or tool, and where something is unverified I said so. The analysis of the Bun case itself is covered in a separate post, so here we look only at the procedure.

Step 0 — Stand Up the Judge Before Translating, and Debug the Judge First

The most common mistake is skipping this step. A migration buys speed by replacing human review with machine adjudication. Without an adjudicator you have not bought speed; you have simply omitted verification.

Build the behavioral oracle from at least one of three things.

A portable test suite is the best option. There is one condition — the tests must touch only the public surface. Tests that call internal functions directly or inspect private state have to be thrown away along with the language, so they cannot serve as an oracle. This is where Bun's 11 days came from. The tests were written in TypeScript and therefore independent of the runtime's implementation language, and zero tests were deleted or skipped during the port.

Golden outputs are the second choice on legacy where the tests are thin. Fix a bundle of real inputs, freeze the old implementation's output to a file, and compare the new implementation's output byte for byte. It works well for systems with deterministic input and output — batch jobs, report generators, parsers.

Differential runs are the strongest of the three. You push the same input through both implementations at once and compare the results. Krieger's case took this form: he built a parity harness out of 7 real scenarios and set the rule that any behavioral change counts as a bug. Refusing to allow an "intended improvement" exception is the crux. The moment you allow an exception, the oracle becomes negotiable.

And there is one more step here that most people miss — debug the judge first. You have to run the oracle against the original code and confirm everything passes, then run it against a deliberately broken original and confirm it properly fails. A judge that fails everything is usually not right; it is broken.

# Judge verification: does it pass when it should pass, and fail when it should fail?
git stash list >/dev/null 2>&1 || exit 1

# 1) On the intact original, everything must be green
./run-oracle.sh --target=legacy || echo "FAIL: the judge cannot even pass the original"

# 2) On a deliberately broken original, a red light must appear
#    (a minimal mutation such as flipping one comparison is enough)
git apply mutations/flip-one-comparison.patch
./run-oracle.sh --target=legacy && echo "FAIL: the judge does not catch the mutation. fix the oracle first"
git apply -R mutations/flip-one-comparison.patch

The time it costs to run those two lines is always cheaper than the time you would later spend debugging "all the tests pass but production is broken."

Step 1 — The Rulebook, the Dependency Map, and a 3-File Dress Rehearsal

The rulebook is a document whose purpose is to make each translation judgment exactly once. The meta-rule the kit attaches works fine as the criterion for what belongs in it — if two agents could answer a question differently, that question is a rulebook entry. Mapping error-handling idioms, representing null, integer overflow policy, logging format, naming conventions, and concurrency primitive mappings are the typical entries. Bun's PORTING.md was around 600 lines, and it took roughly 3 hours before any coding began.

The dependency map is built with a deterministic script. Do not ask the model "what does this file depend on." A 30-line script that parses import statements is more accurate and reproducible. What this map produces is the work order (leaves toward the root) and the list of cycles. You need to know the cycles in advance — a circular dependency the source language tolerated blowing up all at once as thousands of module errors in the target language is the classic accident of this kind of work.

The gap inventory is the list of information the target language newly demands. Going from Zig to Rust, ownership and lifetimes go here; from Python to TypeScript, explicit interface contracts. This is information that does not exist in the source, so it is not translation but decision. Humans make the decisions and write them into the rulebook.

Then comes the dress rehearsal. Pick just 3 files and run the whole pipeline. One agent translates by following the rulebook, another translates without it as a senior in that language would, and a third audits the difference between the two results. Throw away every translation produced here and fix only the rules. Bun caught two issues at this stage that would have spread across all 1,448 files had they gone forward. If there is one place to pour human time into, it is here.

Step 2 — Cut Along Verifiable Boundaries and Delegate the Queue to Disk

When choosing the unit of work, the criterion is not size but verifiability. After porting one unit, a machine has to be able to adjudicate whether it "worked." The reason file-level units fit well is that the adjudication is over as soon as you check whether the target file exists and whether it compiles.

Here you have to split structure-preserving porting from redesign. A structure-preserving port that maintains file-to-file correspondence keeps the dress rehearsal, line-by-line comparison, and the file-level queue all intact. Decide instead to redraw module boundaries while you move, and those devices collapse simultaneously — there are no source lines to compare against, the unit swells to a module, and the revert radius swells with it. The temptation to move and redraw in a single pass is the most common cause of failure in this procedure. Separate the order. Move first, then redraw with the oracle green.

Delegate the done-check to disk state. This one decision gives you resumability and revertibility at the same time.

# Queue = entries in the manifest that do not yet have an output file. Keep no state beyond that.
awk -F'\t' '{ if (system("[ -f " $2 " ]") != 0) print }' migration/manifest.tsv \
  > migration/queue.tsv

wc -l < migration/queue.tsv        # work remaining
git worktree list                  # give each worktree its own queue slice

The moment you ask the agent for progress, progress itself becomes model output and therefore subject to hallucination. File existence does not hallucinate.

Maintain revertibility three ways. Split commits per unit so the revert radius stays at a single file, separate the translation stage from the fix-up stage into different branches or worktrees, and record the point at which the rulebook was revised in the commit message. That last item matters because when the rules change, you have to be able to pin down the scope that must be regenerated among artifacts produced before that point.

Where the compiler sits in the loop is also decided at this stage. When type checking is fast, as in TypeScript or Go, put it inside the translation loop for immediate feedback; when it is slow, as with Cargo, run it once after translation finishes and use the error list as the next queue. Bun took the latter route and cleared roughly 1,600 compile errors in 12 hours.

Step 3 — What to Measure: Lines Converted Is Not Progress

The number that shows up most often in migration reports and is the most useless is "lines converted." That number is output volume, not progress. The metrics that actually change decisions are these five.

Metric	Definition	What it tells you
Judge pass rate	Share of oracle cases the new implementation passes	The only definition of done. Bun went from 972 failing files to 23, then to 0
Rule recidivism	Share of reviewer findings that were already covered by the rulebook	If it does not fall, the problem is the process, not the rules
Diff review rate	Lines a human actually read divided by lines generated	The true size of the risk. At a million lines this is honestly a single-digit percentage
Rework per unit	Average number of times a single file re-entered the queue	Past 2 rounds, the rulebook is not covering that area
Residual markers	Remaining count of TODO(port) and BUG(port) comments	The real debt that survives the merge. It keeps you from mistaking merge day for done day

Of these I strongly recommend calculating and recording the diff review rate. Compute it honestly at the million-line scale and it is usually under 5 percent. If that number is uncomfortable, the discomfort is the correct reaction. It is the size of the risk this approach accepts, and it is a value you owe to the decision-makers. Do not hide it; instead, write alongside it what adjudicated the other 95 percent — the compiler, the oracle, the adversarial reviewer.

The economics of review come out like this. What a human has to read is not the generated code but the generation rules and the representative samples those rules produced. Six hundred lines of rulebook, the diffs of the 3 dress-rehearsal files, and the cases where the reviewers disagreed. A human can read all three of those, and reading them actually changes the outcome. A review that skims 1,448 files, by contrast, spends the time without changing the outcome.

Step 4 — Three Failure Modes and How to Detect Each

Silent semantic drift. It compiles and the tests pass, but the behavior is subtly different. It mainly shows up in floating-point rounding, sort stability, the distinction between null and empty, the order of error propagation, time zone handling, and integer overflow behavior. Because it occurs on paths the tests do not cover, tests will not catch it. There is exactly one detection method — differential runs. Push a sample of production traffic or a log of real inputs through both implementations and compare the outputs. This is why Krieger set the rule that "any behavioral change is a bug." The moment you start an exception list, your means of stopping this failure mode disappears.

Hallucinated APIs. Code that calls nonexistent functions, wrong signatures, or the argument order of an old version. When you are moving to a statically typed language the compiler catches all of it, so in practice it is not a big problem. What is dangerous is when the target is a dynamic language. Move to Python or Ruby and the bad call survives to runtime, and if there is no test on that path it first blows up in production. If the target is dynamic, you must add import resolution and signature checking as a separate static analysis stage, and put that stage inside the translation loop.

Tests that pass for the wrong reason. The most dangerous of the three. There are three representative forms. First, the same agent that wrote the new code also fixed the tests — it ends up serving as both judge and defendant. Keep oracle files write-protected on the migration branch, and route any needed change through a human approval step. Second, tests that swallow exceptions — the failure path quietly turns into a pass. Third, tests that skip the assertion section via an early return.

# Check every round that the oracle was not mutated during the migration
git diff --stat migration-base..HEAD -- tests/ | tail -1

# Look not at the pass count but at the "number of assertions actually executed" alongside it.
# If the pass count held steady while the assertion count dropped, a test quietly went empty.
./run-oracle.sh --report=assertions | tee migration/assertions-$(date +%s).txt

There is one response principle common to all three failure modes. If a reviewer makes the same finding three times, that is not an individual bug but a systematic error, so do not fix the file — fix the rulebook and regenerate the affected batch. Start patching individual files by hand and the rules and the code diverge, and from that point on regeneration becomes impossible.

Closing — The Value of This Procedure Is Not Speed but Reversibility

To sum up, the order is this.

Build a behavioral oracle, and first confirm that the oracle fails on deliberately broken code.
Build the rulebook, the dependency map, and the gap inventory, and hammer on the rules with a 3-file dress rehearsal. Throw the translations away and keep only the rules.
Cut along verifiable boundaries into a queue, and delegate the done-check to disk state so that stopping and resuming are free.
Instead of lines converted, watch judge pass rate, rule recidivism, diff review rate, rework per unit, and residual marker count.
Reflect a reviewer's repeated findings in the rules rather than in the files, and regenerate the affected scope.

The real benefit this procedure gives you is not speed. It is that you can throw it all away and run it again. With the rulebook, the queue, and the oracle kept separate, even in the worst case you delete the branch, fix the rules, and rerun. The reason a multi-year migration used to be a project you bet your career on was that it was irreversible, not that it was hard.

If you keep only one thing, keep this — if you have not decided what will adjudicate the end before you begin translating, that migration is not yet ready to start.