Skip to content
Published on

What to Learn Deeply, What to Skim — A Learning Strategy for the AI Era

🇰🇷KO🇺🇸EN🇯🇵JA🇨🇳ZH
Split View필사 모드
Share
Authors

Introduction — When Answers Became Free

If any question resolves in three seconds, what is still worth putting in your head?

The simplest answer — "only what the AI doesn't know" — is useless, because the AI knows almost all of it. So this piece uses a different axis. What you learn deeply is not defined by what AI doesn't know. It's defined by what you must be able to catch AI getting wrong.

An earlier post, Does AI Actually Make Developers Faster?, asked whether you are faster today. This one asks the long-horizon version: are you getting better? And the answer learning science gives is not pleasant for anyone who uses these tools daily.

Retrieval, Not Lookup, Is What Builds Skill

Start with the experiment. Roediger and Karpicke, Psychological Science, 2006 (vol. 17, no. 3).

Washington University undergraduates studied short science passages. In Experiment 2 there were three conditions:

  • SSSS — read the passage four times. No test.
  • SSST — read it three times, take one recall test.
  • STTT — read it once, take three recall tests. No feedback given.

On a final test five minutes later, the result is common sense. SSSS scored 83%, SSST 78%, STTT 71%. More reading wins.

One week later the ranking inverts completely. STTT held 61%, SSST 56%, and SSSS — the group that dutifully read it four times — collapsed to 40%. The most striking number is exposure. The SSSS group read the passage an average of 14.2 times. The STTT group read it 3.4 times. Four times the reading, and 21 percentage points worse a week later. Measured as forgetting over that week, SSSS lost 52% of what it had; STTT lost 14%.

And the real bomb in the paper is the questionnaire (Table 2). Right after studying, participants rated how well they thought they would remember the passage in a week, on a 7-point scale:

  • SSSS: 4.8
  • SSST: 4.2
  • STTT: 4.0

The most confident group remembered the least. In the authors' words, "repeated studying inflated students' confidence." Their conclusion: "Testing is a powerful means of improving learning, not just assessing it."

Fix the vocabulary. Retrieval is pulling something out of your own head. Lookup is having it handed to you. The first builds memory. The second builds the feeling of having built memory.

Desirable Difficulties — and Undesirable Ones

The mechanism comes from Bjork and Bjork's 2011 chapter, "Making Things Hard on Yourself, But in a Good Way."

The core idea is the split between learning and performance. Performance is what you can observe during training; learning is the durable change. As they put it, "current performance can be a highly unreliable index of whether learning has occurred."

Hence the paradox: the conditions that raise performance fastest are not the conditions that produce learning. Rereading generates perceptual fluency, and we mistake that fluency for comprehension. In Bjork's theoretical terms, we misread retrieval strength as storage strength — and as a result we actively prefer worse learning conditions to better ones.

So Bjork calls the conditions that hurt now and pay later desirable difficulties. His list: varying the conditions of learning, interleaving topics rather than blocking them, spacing rather than massing, and using tests rather than presentations as study events. The Bjork lab adds generation — producing the answer instead of reading it — an effect traced to Slamecka and Graf's 1978 study.

One number, from Rohrer and Taylor (2007) as cited in the chapter: learners who practiced problems interleaved solved 63% of new problems a week later; those who practiced them blocked by type solved 20%. During training, of course, the blocked group looked better. It always does.

Now the caveat that holds this whole essay together. Bjork warns hard: "Many difficulties are undesirable during instruction and forever after." Not all struggle helps. A difficulty is only desirable if the learner has the background knowledge to meet it successfully. Without that foothold, it is simply an undesirable difficulty — wasted effort. Hold on to that: in a moment it becomes the test for when handing the work to AI is exactly right.

The AI-Era Trap: Outsourcing Retrieval

The problem compresses to one sentence.

Retrieval is what builds skill, and AI is a retrieval machine. Let the model pull the answer out, and you skip the precise act that creates expertise. The tool is best at exactly the thing you cannot afford to delegate. And this is not an analogy — it has been measured.

Bastani et al. (PNAS, 2025). A field experiment with nearly a thousand Turkish high school math students. Three arms: GPT Base (a standard ChatGPT-style interface), GPT Tutor (prompts safeguarded to give teacher-designed hints instead of answers), and a control group with no AI.

  • During practice, AI wins overwhelmingly. Grades improved 48% with GPT Base and 127% with GPT Tutor.
  • Then the AI was taken away and students sat an unassisted exam. The GPT Base group scored 17% lower than the control group that had never touched AI at all.
  • The safeguarded GPT Tutor largely eliminated that harm — but note carefully: it eliminated the harm; it did not beat the control group. That 127% practice gain did not follow anyone into the exam room.

The authors' diagnosis is one word: students used GPT-4 as a "crutch." This is Bjork's learning/performance dissociation reproduced outside the lab. Performance during practice soared; learning did not stick.

Lee et al. (CHI 2025, Microsoft Research and CMU). A survey of 319 knowledge workers covering 936 real-world examples of generative AI use at work. The key finding is about the direction of trust: "higher confidence in GenAI is associated with less critical thinking." The more people trusted the model, the less they thought; the more they trusted themselves, the more they thought. The authors also report that AI shifts the character of critical thinking away from producing answers and toward verifying, integrating, and supervising them.

The paper also invokes Bainbridge's classic irony of automation: hand the routine work to the machine and you also lose the reps that keep you capable of handling the exceptions. The exceptions still come.

Both studies' limits, stated up front. Bastani studied high schoolers doing math, not senior engineers. Lee et al. is a self-reported survey and correlational — the title itself says "Self-Reported." And as the earlier post showed, self-report about how AI affects your own performance is the least trustworthy instrument we own. The mechanism transfers plausibly. The effect sizes do not.

What to Learn Deeply, What to Skim — Sorting by Half-Life

The strategy is not a reading list. It is allocation. You have a finite budget of desirable difficulty, and the only question is where to spend it. Two criteria: half-life, and whether you could catch the error.

Long half-life — own it, deeply, through retrieval. These explain why, and they last decades.

  • Operating systems — processes, memory, file descriptors, scheduling
  • Networking — TCP, TLS, HTTP semantics, timeouts and retries
  • Databases — transactions, isolation levels, indexes, query plans
  • Distributed systems — consistency, partitions, idempotency, failure modes
  • Debugging method — hypothesize, observe, bisect
  • Data structures and algorithms — not memorized, but at the level of reasoning about cost
  • Writing — the craft of making a thought checkable

TCP has outlived every framework you have ever learned. And these are precisely the instruments you reach for when judging what the model just handed you.

Short half-life — skim it, let AI carry it. Framework APIs, syntax trivia, config incantations, CLI flags, YAML schemas. These belong to AI for three reasons: they are well documented, so the model is good at them; they are verifiable in seconds (it runs or it doesn't); and they expire anyway. Learning them deeply is investing in a depreciating asset.

The test that sorts the two buckets is single: would you notice if the model were wrong?

  • A bad CLI flag announces itself — the command fails. → Skim it.
  • A subtly wrong isolation level, a missing idempotency key, a race hiding in a retry path: nothing announces those. The bill arrives months later. → You must own these.

AI's signature failure mode is "almost right," and only someone who knows why can catch the "almost." The grid for sorting tasks by familiarity and verification cost is in the earlier post.

The Fix: Attempt First, Then Check

The prescription is not "don't use AI." It is reordering. Put the retrieval attempt before the lookup.

The evidence is Kornell, Hays and Bjork (2009, JEP: LMC, vol. 35, no. 4), and the design is almost cruel. The researchers engineered materials so the attempt was guaranteed to fail: fictional general-knowledge questions ("What peace treaty ended the Calumet War?" — there was no such war), and word-association items where the rare correct guesses were thrown out of the analysis.

  • Test condition — try to answer, then see the answer. (The attempt fails, 100% of the time.)
  • Read-only condition — see the question and the answer together.

The result: "Unsuccessful retrieval attempts enhanced learning with both types of materials." Even a failed attempt beat going straight to the answer. The authors' conclusion is that taking hard tests, rather than avoiding errors, may be a key to learning. That is the entire prescription. In practice, five moves:

1. Spend five minutes writing your answer before you prompt. "This bug is connection-pool exhaustion." "This query won't hit the index." Being wrong is fine. Kornell's result says the wrong attempt is what installs the right answer.

2. Use AI as a critic, not an autocomplete. Not "write this for me" but "here is my design — where does it break?" Same tool, opposite cognitive position. You generate; the model tries to falsify.

3. Be the one who gets questioned. Ask the model to quiz you on the code you just merged. That is retrieval practice on your actual codebase.

4. In long-half-life territory, deliberately switch the tool off. Not asceticism — training. The Bastani study is precisely a measurement of what happens when the tool goes away, and someday it will: an incident at 3am, an air-gapped network, or just a problem the model gets confidently wrong.

5. In short-half-life territory, use it without guilt. That is the entire point of the split. Straining to recall a CLI flag is exactly Bjork's undesirable difficulty — nothing accumulates.

A preprint points the same way ("Struggle First, Prompt Later," arXiv 2025): attempting first and prompting afterward helped most on complex, higher-order tasks. But it is not peer-reviewed, and I could not verify its sample size. Treat it as a signpost, not as evidence.

Honest Limits

So this doesn't get oversold, here are the weak points, written by me.

Lab-to-workplace transfer is an analogy. Roediger and Karpicke measured undergraduates recalling prose over one week. Bjork's difficulties were established largely on word lists, motor tasks, and classroom material. None of that is "an engineer learning distributed systems over three years." The direction replicates well; the magnitude in your job has never been measured. Discount accordingly.

And now the most inconvenient number. As covered in The Elite Athlete Mindset, Ericsson never coined the "10,000-hour rule" — in his words, "there is nothing special or magical about ten thousand hours" — and Macnamara, Hambrick and Oswald's 2014 meta-analysis found deliberate practice explains about 26% of performance variance in games, 21% in music, 18% in sports, 4% in education, and under 1% in professions. The domain closest to ours is the weakest one. Their own conclusion is exactly that measured: deliberate practice matters, but not as much as has been claimed. That number cuts against my thesis, which is why I am stating it plainly — "practice deliberately" has a low ceiling in professional work, and this essay is only valid underneath that ceiling.

Sometimes not learning is correct. If you will touch a tool once in your life, hand it to the AI entirely. That is not laziness — that is the strategy working.

Closing

Compressed to one sentence: AI is extraordinarily good at precisely the thing that, if you let it do that thing for you, guarantees you never get good at it.

That is not an argument for abstinence. It is an argument for allocation. Your budget of desirable difficulty is finite. Spend it on what has a long half-life and on what you must be able to catch AI getting wrong. Spend nothing on the rest. The sticky-note version is three lines:

  • Attempt first, then check.
  • What lasts, learn yourself; what rots, let the tool carry.
  • Ask not "what did I learn" but "would I catch it if the model were wrong?"

Mercifully, these habits are small — five minutes before a prompt is the whole thing. Why small is what actually sticks is covered in Habits Aren't Built in 21 Days.

If you take one thing from this, take Roediger's table. The group that read the passage fourteen times and rated themselves most confident is the group that remembered the least a week later. Fluent, fast, and certain. AI puts you in that exact chair every single day. The feeling of understanding and the fact of understanding come apart — and only one of them shows up when the tool is gone.

References

Discuss on Twitter • View on GitHub