Skip to content
←

필사 모드: Simulated Customers Never Walk Away — Where LLM User Simulators Inflate Agent Scores

English
0%
정확도 0%
💡 왼쪽 원문을 읽으면서 오른쪽에 따라 써보세요. Tab 키로 힌트를 받을 수 있습니다.

Introduction — Who Is the User Scoring Your Agent?

Let's pull apart the structure of a conversational agent benchmark. τ-bench (Yao et al., arXiv:2406.12045) hands an agent API tools and policy documents in a domain like airlines or retail, has it talk to a user across multiple turns, and then scores the conversation by comparing the database state at the end against a target goal state. Scoring by state comparison is what makes this benchmark honest — it looks at whether a reservation actually changed, not at an LLM judge's taste.

The problem sits in the seat next door. The conversation partner — the user — is also played by a language model. And that fact usually gets one line in a methodology footnote.

Here's the thing worth pausing on: the user simulator is not the thing being measured — it is the measuring instrument. While the agent takes the exam, the simulator is the proctor who reads the questions aloud and reacts. In a physics experiment, nobody measures a length with an uncalibrated ruler and then reports the third decimal place in a paper. Yet for nearly two years, that is essentially what we have been doing. Nobody had verified the ruler.

In 2026, that verification arrived from three directions, and three independent studies point the same way. Simulated users are nicer than real users. And that niceness is not uniformly spread noise — it is a bias concentrated exactly where it matters most.

The Two Jobs the Simulator Carries

First, let's lay out how much load this tool is actually carrying. Per the framing in the Sim2Real gap paper (Zhou et al., arXiv:2603.11245), the simulator performs two roles at once.

  1. Generating user turns — producing the utterances the agent has to respond to.
  2. Supplying the evaluation signal — providing feedback on how well the conversation went.

So if the simulator is biased, that bias leaks through both channels. The problem the agent faces gets easier, and the judgment of whether it solved that problem well gets more lenient too. And this output flows into leaderboard numbers, and worse, into reinforcement-learning reward signals.

For reference, τ-bench's originally reported numbers were these — top function-calling agents of the time, like gpt-4o, solved less than half the tasks, and pass^8 — which checks whether the same task succeeds every time across repeated runs — came in under 25% on the retail domain. That alone is a humbling number. The point of this post is that even this number may be biased toward leniency.

First Piece of Evidence — Running τ-bench on 451 Real People

Zhou et al. (arXiv:2603.11245, March 2026) introduce their work as the first study to run the τ-bench protocol on real people as-is. With 451 participants and 165 tasks, they put 31 LLM simulators — spanning commercial, open-source, and specialized families — through the same protocol for comparison. To measure how closely a simulator resembles a real user's interaction behavior and feedback, they propose a new metric called USI (User-Sim Index).

The behavioral conclusion is blunt. LLM simulators are excessively cooperative, stylistically uniform, and missing realistic frustration or ambiguity. The paper reports that this produces a kind of "easy mode," which inflates agent success rates above the human baseline. What matters here is that the comparison point is not another simulator but an actual human baseline — so there is grounding for both the direction and the existence of the inflation.

The same held on the evaluator side. Real people render distinct judgments across 8 quality dimensions, whereas simulated users consistently produced more positive feedback. Rule-based rewards failed to capture the rich feedback signal real human users generate.

And here is the finding most uncomfortable for practitioners — higher general model capability did not translate into more faithful user simulation. This runs directly against the hope that "frontier models will just fix this over time." Plugging a smarter model into the simulator slot is not an automatic fix.

Second Piece of Evidence — The Simulator Doesn't Proxy Every User Equally

Lost in Simulation (Seshadri et al., arXiv:2601.17087, January 2026) ran a user study on τ-bench retail tasks with participants from the US, India, Kenya, and Nigeria. Let's walk through the results one at a time.

No robustness. Which user LLM you pick swings agent success rate by up to 9 percentage points. It's worth pausing on the size of that number — given that today's leaderboards fight over gaps of a few percentage points between models, this means which model you seat in the user role moves the score about as much as which agent you're testing does. If two teams each report "62% on τ-bench retail" using different user models, those two numbers were never comparable quantities to begin with.

Calibration is systematically off. Evaluations run with simulated users underestimated agent performance on hard tasks and overestimated it on medium-difficulty tasks. Because the direction flips with difficulty, a correction like "it's inflated by X% overall, so just subtract that" doesn't work.

Proxy quality differs by demographic group. Speakers of AAVE (African American Vernacular English) had consistently worse success rates and calibration error than Standard American English speakers, and the gap widened significantly with age. The simulator was the worst proxy for AAVE speakers and Indian English speakers. In other words, simulator-based evaluation structurally under-shows failures for specific user populations. Before it's a fairness problem, it's a measurement coverage problem.

On top of this, simulated users produced conversational artifacts and revealed failure patterns different from those of real people.

Third Piece of Evidence — Where Real Money Is on the Line

The two studies above are strong but share a common limit: the "real people" serving as the baseline are paid participants performing an assigned goal. Simulated Customers Never Walk Away (Liang Chen, arXiv:2606.20708, June 16, 2026) calls exactly this a structural blind spot.

The logic runs like this. Once a goal is assigned, the user's willingness becomes an exogenous variable. A participant instructed to "cancel this flight" was never being asked, within that conversation, whether they actually wanted to cancel it. So existing frameworks can measure whether the simulator talks like a human (communicative fidelity), but cannot, in principle, measure whether the simulator decides like a human (decision fidelity). A real user's motivation is endogenous, latent, and decays over time.

So the paper establishes the new concept of decision fidelity, and uses an unusual testbed to measure it.

The Testbed — ZhenaiSales

2,790 production conversations between an LLM sales agent actually deployed on a Chinese matchmaking platform and real customers (parents consulting on behalf of their adult children). Of these, 793 converted — the parent actually paid, and that payment is verified against order records. The remaining 1,997 did not convert. 98% carry a structured demographic profile, and conversation length has a median of 16 turns (range 6–80).

One detail shows the authors paying close attention. Converted conversations are truncated at the moment of first payment. If post-payment service conversation were left in, the outcome would leak straight back in (an average of 19 turns per buyer were trimmed away). What's left is only the pre-decision dynamics.

This is exactly what separates it from role-play corpora. Here, every label is the outcome of a real person actually spending, or not spending, real money.

Method — Teacher-Forced Probes

The measurement design is the heart of this paper, so it's worth looking at closely.

Real conversation prefix h_t  (up through turn t, between a real person and a real agent)
        |
        +---> real user's next turn u          --\
        |                                        >--- scored by the same instrument Φ
        +---> simulator's next turn û           --/
                                                 |
                       depth bias D(h) = δ(Φ(û,h)) - δ(Φ(u,h))
                                                 |
                       averaged per conversation -> D̄
                                                 |
       endpoint Δ = E[D̄ | non-buyer] - E[D̄ | buyer]

Both branches start from the exact same real prefix and are scored by the exact same instrument. The instrument Φ is an LLM perceiver that maps a user turn to a structured state — engagement stage (exploring / engaging / considering / deciding / resisting), emotion, and blockers (price, trust, capability, timing, external factors, past failure). It outputs strict JSON, and when scoring turn t it only sees context up through t (causal application) — a safeguard against future information leaking in.

This design has a nice property, which the paper formalizes as Proposition 1. Because both branches share the same history and are scored by the same instrument, (i) any additive bias the instrument has toward context cancels out within the probe, and (ii) any general stylistic bias of the simulator unrelated to the outcome loads onto both layers equally and cancels out in Δ. Only differences correlated with the outcome survive. This design cancels out, by construction, the two confounders that prior literature treated as the whole problem — judge miscalibration and simulator style.

The test is a conversation-level permutation test (20,000 permutations, one-sided), reported alongside Cohen's d.

Results — The Disengagement Deficit

The main experiment uses a profile-conditioned persona simulator (Claude backbone), 374 conversations (181 non-converted, 193 converted), and 1,109 matched probes.

Simulator              E[D̄|non-buyer]  E[D̄|buyer]      Δ         d (p)
------------------------------------------------------------------------
Claude persona          +0.396         +0.092        +0.304    0.38 (0.0002)
  (n=374)             [0.30, 0.49]  [-0.04, 0.22]  [0.14, 0.46]
Claude, instructed      +0.098         -0.187        +0.285    0.34 (0.008)
  (n=200)             [-0.05, 0.24] [-0.36, -0.01] [0.06, 0.51]
DeepSeek persona        +0.428         +0.098        +0.330    0.41 (0.002)
  (n=200)             [0.27, 0.59]  [-0.06, 0.26]  [0.11, 0.56]

Brackets are bootstrapped 95% confidence intervals. Under perfect decision fidelity every term would be 0.

Two things stand out.

First, the simulator reproduces eventual buyers almost exactly. The bias is +0.09 — "a tenth of a stage," in the paper's own phrase. So this simulator is not globally biased. That matters, because it rules out explaining the problem away as generic sycophancy or verbosity — either of those would have pushed both layers up equally.

Second, eventual walk-aways get inflated four times as much (+0.40). The result is an outcome-conditioned contrast of Δ = +0.304, d = 0.38, p = 0.0002. The paper names this pattern the disengagement deficit.

Breaking the mechanism down by stage distribution:

Non-buyer layer (real -> sim)
  resisting                25.1%  ->  13.5%    -11.6%p   resistance roughly halved
  considering               21.9%  ->  40.1%    +18.2%p   considering nearly doubled
  deciding                   4.3%  ->   4.3%      0.0%p   purchases are not invented

Buyer layer (real -> sim)
  resisting                16.0%  ->  13.6%     -2.4%p
  considering               33.7%  ->  46.6%    +12.9%p

The deciding row here is decisive. The simulator does not invent purchases that never happened. So this is not the crude failure mode of "the simulator makes everyone say yes and scores get inflated." Instead, the simulator manufactures people who are, in reality, backing away, into interested deliberators. That is a far harder failure to catch.

A qualitative example compresses this difference (Table 3 in the paper, translated):

Real user (stage)               Simulator (stage)                  How to read it
--------------------------------------------------------------------------
"I'm busy" (resisting)          "How much is it?" (considering)    avoidance -> interest
"I'm fine" (resisting)          "Go ahead, tell me" (engaging)     refusal -> engagement
"I have a headache" (resisting) "Guess I'll look into it" (considering)  disengaging -> deliberating
"Thank you" (considering)       "That's a bit pricey, give me a discount" (considering)  polite exit -> haggling

The paper's one-line summary is exact — real non-buyers say "not right now" and stop, while simulated non-buyers ask about price.

Generality — Not One Model's Quirk

Swapping the simulator backbone to DeepSeek-V4-Flash (a different family, different training pipeline, different alignment recipe) reproduces the deficit at the same strength — Δ = +0.330, d = 0.41, p = 0.002. The mechanism is also the same (non-buyer resistance −9.4%p, considering +16.8%p).

They also checked whether the instrument itself was the culprit. Keeping the simulator as Claude but swapping only the judge to a DeepSeek-family model and rescoring a 150-conversation subset, the deficit did not disappear — it got stronger: Δ = +0.351, d = 0.42, p = 0.015. A separate check for whether the judge peeks into the future (a format control that varied only the visibility window — prefix-only versus full conversation) showed no outcome-correlated shift (d = −0.04, p = 0.62).

Why Prompting Doesn't Fix It — The Marginal-Realism Trap

This is where the paper has the most to teach.

The obvious first move for practitioners is: "just tell the simulator it's allowed to not be interested." The paper runs exactly that experiment. The instructed variant adds explicit behavioral permission to the prompt — real parents aren't always cooperative or interested; if they were lukewarm earlier, they're allowed to stay that way ("yeah," "I'm busy," "I'll think about it," "no"); disinterest, stalling, and irritation are all permitted.

Here's what happened.

  • It works at the margin. Overall depth bias dropped from +0.24 to −0.04 — a fivefold reduction. Non-buyer resistance suppression nearly vanished too, from −11.6%p to −1.7%p.
  • The outcome-conditioned contrast barely moves. Δ = +0.285 (d = 0.34, p = 0.008) — essentially unchanged from the prior +0.304.

What happened here? The instruction made the simulator uniformly more resistant. Instead of getting non-buyers roughly right, it now overshoots resistance for buyers too (E[D̄|buyer] = −0.187 — reading them as less engaged than they actually are). The simulator learned how to perform disengagement but still doesn't know who is supposed to disengage. The instruction didn't remove the error — it moved it from non-buyers onto buyers.

The paper calls this the marginal-realism trap, and I think this is the paper's real contribution. Averaged across the whole population, a simulator that over-engages non-buyers while faithfully tracking buyers can still match the marginal stage distribution quite well — and that is precisely the statistic population-level alignment metrics reward. The error is only visible once you condition on the outcome, and that requires real outcome data. No matter how well-built, a communicative-fidelity benchmark is structurally blind to this.

In one sentence: the quantity that's easy to measure and easy to optimize (marginal realism) is not the quantity that governs transfer (conditional decision fidelity).

Interestingly, this observation shows up independently from another team, too. RealUserSim (arXiv:2605.20204) reports that hand-written behavioral instructions cause Directive Amplification — the model over-interprets the instruction and swings to unnatural behavioral extremes, and the degree varies widely by simulator model. Two papers, two different attempts to sculpt user behavior via prompting, and two different ways it fails.

When This Feeds a Training Loop — The Simulator Misprices Pressure Tactics

If this were only an evaluation problem, we could file it under "reported success rates are an upper bound" and move on. The problem is what happens when this signal feeds a training loop.

The paper labels the agent's action right before each probe (rapport / probe / pitch / handle-objection / push-close / reassure) and compares real versus simulated reactions broken down by action. 595 probes, DeepSeek as the instrument, two-sided Fisher's exact test.

Preceding agent action      n     resisting (real->sim)    considering (real->sim)
---------------------------------------------------------------------------
probe (neutral question)    85    10.6% ->  2.4%            3.5% ->  4.7%   (p=1.0)
rapport (building rapport)  19    21.1% -> 15.8%           10.5% -> 10.5%
pitch (the offer)          161    11.8% ->  5.6%           45.3% -> 75.2%   (+29.9%p)
push-close (pressure)      169    18.9% -> 10.7%           41.4% -> 52.1%
handle-objection            94    31.9% -> 29.8%           26.6% -> 43.6%
reassure                    67    40.3% -> 29.9%           13.4% -> 14.9%

The distortion is concentrated exactly on pressure tactics. After a pitch, real users move to considering 45.3% of the time; simulated users, 75.2% (+29.9%p). After a push-close, simulated resistance is suppressed from 18.9% to 10.7% (−8.2%p, p = 0.046). Meanwhile the shift toward considering after a neutral probe is negligible, 3.5% to 4.7% (p = 1.0).

The implication is uncomfortably clear. An agent optimized against this simulator would observe that pitching and pushing raises interest — when in reality, that behavior invites resistance or moves nothing at all. In the paper's words, the simulator over-rewards the exact tactics that lose real customers. This isn't noise; it's a directional bias, which makes it not merely inaccurate but an actively misleading training signal.

An agent's sales skill is largely the skill of handling the hard cases — recognizing a customer who's about to walk and either winning them back or gracefully letting go. A simulator with no willingness decay wipes those cases out of the test set entirely.

Why the Simulator Can't Walk Away

The paper's explanation is compelling. The deficit has a sign (toward engagement), a location (eventual walk-aways), and a magnitude (near zero for buyers). This pattern isn't explained by generic sycophancy — it follows naturally from what instruction tuning optimizes for.

Alignment data rewards a conversational agent for being helpful, responsive, and for keeping the conversation moving forward. And that data contains almost none of what real disengagement actually looks like — curt avoidance ("I'm busy"), stalling ("I'll think about it"), and silence. A simulator instructed to play a parent therefore defaults back to the cooperative conversationalist it was trained to be, and when it needs to express resistance, it can only render it as considering — its nearest in-distribution neighbor.

Here's the decisive observation — the simulator sees the exact same prefix as the real user. What's missing isn't information, it's behavioral repertoire. That's why the failure is conditional. For buyers, the cooperative extension is roughly right, so it looks faithful; for non-buyers it's wrong, and what's missing is exactly the willingness decay.

Honest Limitations — How Far to Trust This

One reason this paper is good is that the authors don't blur their limitations. I'll carry them over as-is.

Single domain, single language. ZhenaiSales is Chinese, parent-mediated matchmaking. It has a distinctive social dynamic — parents negotiating on behalf of adult children, and face-culture norms that favor indirect refusal. The observed disengagement signals ("I'm busy," "I'm fine") are a culturally specific form of polite resistance.

The authors explicitly split their evidence into two tiers here, and this distinction matters.

  • (a) The claim that LLM simulators are excessively cooperative now has multi-study, cross-language grounding. Zhou et al. (arXiv:2603.11245) reported the same "easy mode" gap in English customer-service conversations (451 participants). This phenomenon looks rooted in instruction tuning, not in a particular language or culture.
  • (b) The claim that the deficit is outcome-conditioned (Δ is greater than 0, non-buyers are inflated more than buyers) rests on single-domain evidence and awaits replication. The direction of the conditional deficit follows logically from the instruction-tuning explanation and so seems likely to generalize, but the magnitude may vary by domain.

This distinction should not be blurred. The third piece of evidence in this post comes from a single domain, a single language.

The instrument is an LLM. Decision-state labels were assigned by an LLM. This is mitigated by the Proposition 1 cancellation argument, the causal-labeling check, and the cross-family instrument swap, but the authors themselves note that human-verified state labels (Fleiss' kappa on a subset) remain an important item for future work.

Teacher forcing, not free rollout. This study measures next-turn decision fidelity on top of real history. In a free rollout, the deficit could compound, and the authors expect that this would strengthen the conclusion — but that was not measured here. An expectation is an expectation.

A profile-conditioned prompted simulator. Two prompted simulators were tested, standard and instructed. Retrieval-based or fine-tuned simulators may behave differently.

The outcome is a proxy for willingness. Payment is a coarse, endpoint-only signal for an underlying trajectory. Still, it is clearly a real, outcome-backed ground truth that prior work lacked.

The same humility applies to the other numbers cited in this post. Seshadri et al.'s 9-percentage-point figure is specific to τ-bench retail tasks, and Zhou et al.'s USI is a newly proposed metric that hasn't yet been vetted by the community.

So What Should You Actually Do

Practically, here's where this leaves us.

1. Report the user simulator model as part of your experimental setup. The way you'd report the agent model, scaffold, and number of attempts. If the user model can swing scores by up to 9 percentage points, that's a condition, not a footnote. A τ-bench number without a disclosed simulator isn't a comparable quantity.

2. Read reported success rates as an upper bound. And remember that the bound is loosest exactly on the hardest subgroups — users about to leave, irritated users, users who don't speak the standard dialect. A correction that subtracts a global constant won't work, since, as Seshadri et al. showed, the sign of the bias flips with difficulty.

3. Don't validate a simulator on marginal realism. A simulator that looks human at population-level statistics can still be systematically wrong once you condition on outcomes. And a one-line prompt permission to "not be interested" isn't a verified fix — it's moving the error to a different layer.

4. Look toward structural solutions. A few directions exist.

  • Constrain the simulator with the environment. τ²-bench (arXiv:2506.07982) binds the user simulator tightly to the environment, restricting its behavior to tools and observable state. Rather than trusting the simulator's prose, this narrows what the simulator is even able to do.
  • Ground it in real behavioral data. RealUserSim (arXiv:2605.20204) grounds its simulator by extracting 7,275 executable behavior profiles from roughly 14,000 real human-LLM conversations in WildChat. On its own fidelity benchmark (PT3, 600 conversations across 71-plus domains) it reports raising agreement across five behavioral dimensions from 24.2% to 45.3%. Interestingly, applying it to τ-bench surfaced three failure mechanisms invisible to the cooperative simulator and dropped task success rates by an average of 3.2–3.5 percentage points. The absolute numbers aren't large, but the direction is downward, as expected.
  • Gate with a willingness model. The disengagement-deficit paper's proposal is that decision fidelity can't be injected via prompting — it has to be learned from real disengagement trajectories, or the simulator's cooperativeness has to be governed by an external willingness model. And it names a target metric — drive Δ to zero while keeping the buyer layer intact.

5. Verify against real people at least once. This is the shared conclusion across all three papers. For a simulator to hold value over a full lifecycle, it has to be anchored to a human baseline at least once. And if it's your own product, you likely already have that anchor — real logs, real outcomes.

Closing

The lesson here is not "don't use user simulators." Running multi-turn evaluation with humans is slow and expensive, and without simulation the iteration loop doesn't work at all. In fact, none of the three papers argues for abandoning simulation.

The lesson is narrower and more practical. A user simulator is not benchmark scenery — it is a measuring instrument, and an instrument needs its own paper and its own error bars. Three studies in 2026 have started measuring that error for the first time, and the answer points one way — simulated users are nicer than real users, that niceness isn't uniform, and it happens to concentrate exactly where it's most expensive. What an agent whose whole purpose is changing decisions needs to be faithful to isn't how a user talks — it's how a user decides.

The principle laid out in Separating Signal from Noise in Coding Evals holds here too — evaluate on your own tasks and your own users, not on a leaderboard. This post adds one thing to that: if you're going to stand an LLM in for your users, you owe it to yourself to measure, at least once, where that stand-in diverges from the real thing. And that divergence doesn't show up in the average.

References

현재 단락 (1/127)

Let's pull apart the structure of a conversational agent benchmark. [τ-bench](https://arxiv.org/abs/...

작성 글자: 0자원문 글자: 23,246자작성 단락: 0/127