https://www.youngju.dev/blog 천천히 올바르게. AI Researcher & DevOps Engineer Youngju's tech blog. GPU/CUDA, LLM, MLOps, Kubernetes AI workloads, distributed training, and data engineering. ko fjvbn2003@gmail.com (Youngju Kim) fjvbn2003@gmail.com (Youngju Kim) Sat, 11 Jul 2026 00:00:00 GMT https://www.youngju.dev/blog/2026-07-11-signal-noise-coding-evals.en https://www.youngju.dev/blog/2026-07-11-signal-noise-coding-evals.en OpenAI's evals team argues that SWE-bench Verified, the most widely used coding benchmark, no longer gives meaningful signal because of contamination and design flaws. Look at benchmarks through two axes — signal (the power to separate better models from worse ones) and noise (the run-to-run wobble in scores) — and much of the leaderboard turns out to be standing on noise. This post lays out how contamination, harness differences, unscoreable tasks, and overfitting inflate scores, and why a team picking a coding assistant should evaluate on its own tasks rather than the leaderboard — the eval-first principle. Sat, 11 Jul 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) llm-evaluationcoding-benchmarksswe-benchbenchmarksai-codingsignal-vs-noise https://www.youngju.dev/blog/2026-07-11-signal-noise-coding-evals.ja https://www.youngju.dev/blog/2026-07-11-signal-noise-coding-evals.ja OpenAI の評価チームは、最も広く使われるコーディングベンチマークである SWE-bench Verified が、汚染と設計上の欠陥のためにもはや意味のある信号を与えないと指摘します。信号（モデルを見分ける力）とノイズ（実行ごとに揺れる変動）という二つの軸で見直すと、リーダーボードの順位の多くがノイズの上に立っていることが分かります。汚染・ハーネスの差・採点不能な課題・過学習がどのようにスコアを膨らませるかを整理し、チームがコーディングアシスタントを選ぶ際にリーダーボードではなく自分の課題で評価すべき理由を、eval-first の原則とともに説明します。 Sat, 11 Jul 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) llm-evaluationcoding-benchmarksswe-benchbenchmarksai-codingsignal-vs-noise https://www.youngju.dev/blog/2026-07-11-signal-noise-coding-evals https://www.youngju.dev/blog/2026-07-11-signal-noise-coding-evals OpenAI 평가팀은 가장 널리 쓰이는 코딩 벤치마크인 SWE-bench Verified가 오염과 설계 결함 탓에 더 이상 의미 있는 신호를 주지 못한다고 밝혔습니다. 신호(모델을 가려내는 힘)와 잡음(실행마다 흔들리는 변동)이라는 두 축으로 다시 보면, 리더보드 순위의 상당수가 잡음 위에 서 있습니다. 오염·하네스 차이·채점 불가능한 과제·과적합이 어떻게 점수를 부풀리는지 정리하고, 팀이 코딩 어시스턴트를 고를 때 리더보드가 아니라 자기 과제로 평가해야 하는 이유를 eval-first 원칙과 함께 설명합니다. Sat, 11 Jul 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) llm-evaluationcoding-benchmarksswe-benchbenchmarksai-codingsignal-vs-noise