Skip to content
Published on

The Technique of Hallucinating Instead of Classifying, and How to Validate It — Why a Fake Label Beats the Raw Query

🇰🇷KO🇺🇸EN🇯🇵JA🇨🇳ZH
Split View필사 모드
Share
Authors

This post is based on items I read directly from the Hacker News API and the GeekNews feed on 2026-08-15. Scores and rankings keep moving.

What was up there

An item read from the Hacker News API. The title is Don't classify, hallucinate, the item number is 49249523, and as of 2026-08-15 it stood at 212 points with 82 comments. The link points to a post by softwaredoug.

What the technique is

The setup is common. You have to place products or documents into a fixed taxonomy, but the taxonomy has hundreds of entries, so it does not fit in a prompt and it runs into schema size limits on structured outputs.

The usual response is constrained decoding — preventing the model from producing anything outside the valid list. But when the list is large, the list itself is the cost and the limit.

What the post proposes runs the other way: do not constrain the model, just let it invent. The procedure is three steps.

  1. Ask a small, cheap model to write freely what this item's classification would be. It does not matter if the path does not exist in the real taxonomy.
  2. Embed both the invented string and all the real taxonomy entries.
  3. Take the dot product and pick the closest real entry.

The example given is concrete. For a query, the model invents a path like Furniture / Living Room / Tables / Coffee, which resolves to the real taxonomy's Furniture / Living Room Furniture / Coffee Tables & End Tables / Coffee Tables. The post states that every real classification was embedded with MiniLM and matched by dot product.

The post connects the idea to the same family as generating hypothetical documents to improve matching in retrieval.

Why it can work

The benefit the post emphasizes is cost: a small model suffices, and it sidesteps schema size limits.

But there is a more interesting question. Is going through an invented string better than simply embedding the raw query? If there is no gain, the technique has only added a step.

The explanation for that is embedding asymmetry. The raw query is a short product description. The target, though, is a hierarchical path string. Even when the two mean the same thing, the shape of the sentence is different, and distance in an embedding space is influenced by that shape as well as by meaning.

Ask a model to invent a classification and its output naturally becomes a string that looks like a classification. The invented result therefore lands in the same shaped region as the target, which favors nearest-neighbor search. This is the same principle that makes hypothetical documents work in retrieval: turn the question into the shape of a document, then compare it against documents.

Put plainly, what this technique transfers is not knowledge but form. The threads on embedding models and retrieval quality are in debugging RAG retrieval quality and state-of-the-art text embedding models.

What the comments raised

The post reports no measured results — it reads as a conceptual writeup — and the comments went straight at that.

The sharpest comment named the assumption the technique rests on: is it not presupposing that the invented classification will be more selective with respect to the real schema than the query itself? That question tests the asymmetry logic above head-on, and only measurement answers it.

Another asked whether an A/B test was run, and along with it whether classification returned correctly into the known categories, and whether errors from the model's invention and from the embedding search compounded. Those two questions define exactly the experiment you must run before adopting this.

And two cheaper alternatives came up.

  • Retrieve then classify: embed the query, compare against the real taxonomy entries, and put only the few closest into the prompt for a small model to choose from. It solves the schema size problem directly while removing a generation step.
  • Structured output walking down the hierarchy: decide at the top level, then at each level below. More round trips, but fewer options per step.

One comment also shared a procedure for when no taxonomy exists at all: embed everything, cluster, sample from each cluster, and have the model name that cluster — with the caveat that it is sensitive to thresholds. Another noted doing something similar with word embeddings ten years ago.

How to validate it

If you are considering this technique, what to measure before adoption is clear. The experiment the comments demanded is the design.

Example: minimum comparison experiment
Shared setup - 500 to 1,000 human-labeled items

  Option A (baseline)    : embed the query as-is -> nearest classification
  Option B (proposed)    : query -> model invents a classification -> embed -> nearest
  Option C (alternative) : embed query -> top 10 candidates into prompt -> model picks

Values to record
  - accuracy (top 1, top 3)
  - cost and latency per item
  - in Option B, how far the invented string sat from the real taxonomy

Option A is the crux of this experiment. Measure B without A and you cannot tell whether the technique contributed anything. That comparison is exactly what the post does not answer.

And to see error compounding, split Option B's failures into two groups: cases where the invented classification was itself wrong, and cases where the invention was plausible but the embedding attached it to the wrong entry. If the first dominates, fix the generation prompt; if the second does, fix the embedding model or how the entries are represented. Without that split, overall accuracy tells you nothing about where to work.

Who this does not apply to

If your taxonomy has a few dozen entries or fewer, you do not need this. Putting the list in the prompt is more accurate, cheaper, and easier to debug. This technique exists for when the list is too large to include.

It also does not suit classification where accuracy is a regulatory requirement. Where misclassification is expensive — tax codes, medical codes — routing through a plausible intermediate artifact is itself an audit finding. Deterministic rules and human review remain the answer there.

In environments where the taxonomy changes frequently it is actually favorable, since only the embeddings need recomputing and no prompt has to be edited. Conversely, if the taxonomy is fixed and you have enough labeled data, training a dedicated classifier may be cheaper and faster than all of this.

Summary

The value of this technique is not in the provocative framing about the model making things up but in the observation underneath it: in embedding search, the shape of what you say matters as much as what you say, and you can insert one intermediate artifact to align that shape with the target. But the post did not measure the gain, so this is not yet a validated method — it is a hypothesis worth validating. If you adopt it, start by measuring it beside the baseline of simply embedding the raw query.

Sources and related reading

The explanation of asymmetry and the comparison experiment design are my own, built on the post and the comments.

Discuss on Twitter • View on GitHub