Chaos and Order

💡 왼쪽 원문을 읽으면서 오른쪽에 따라 써보세요. Tab 키로 힌트를 받을 수 있습니다.

What RFT is, in plain language
When to use RFT instead of prompt tuning
How custom graders help
The five-step RFT loop
Data prep that actually works
Evaluation discipline for training and rollout
A realistic decision checklist
Official links

What RFT is, in plain language

OpenAI reinforcement fine-tuning, or RFT, adapts a reasoning model using a feedback signal that you define through a programmable grader. The goal is not just to make the model sound better. The goal is to push it toward expert-level performance inside a specific domain.

That makes RFT different from prompt tuning alone. Prompt work can improve instruction following, tone, or formatting. RFT is stronger when the task has a clear success condition and the model can be trained against a repeatable signal.

The current public docs describe RFT as supported on o-series reasoning models, and the guide names o4-mini and o4-mini-2025-04-16. OpenAI's October 6, 2025 AgentKit launch post also says RFT reached general availability on o4-mini, with private beta support for GPT-5 and custom graders in beta.

When to use RFT instead of prompt tuning

Use prompt tuning first when the problem is mostly about clarity, style, or a small behavior shift. Reach for RFT when the model must get better at a task that is hard to express in a prompt but easy to score.

RFT is a strong fit when:

the task is unambiguous
success criteria are specific and stable
domain experts can agree on what a good answer looks like
you need better reasoning in a narrow workflow
failures are expensive enough that a stronger training signal is worth the setup cost

That is why RFT is a good fit for things like policy review, security analysis, legal passage selection, structured decision support, and other workflows where the answer can be judged with a tight rubric.

Prompt tuning is still valuable for shaping voice and instructions. RFT becomes the better option when prompt changes stop producing durable gains and you need the model to internalize the behavior.

How custom graders help

Custom graders are the core of RFT. They turn your business rules into a training signal.

For product teams, that means you can encode what actually matters in the product:

did the model choose the right action
did it respect the policy
did it produce the right structured output
did it avoid unsupported claims
did it stay within the expected scope

For platform teams, graders are the contract between evaluation and training. A good grader keeps the loop honest by forcing the team to define success before training starts. It also makes regression testing more useful, because the same rubric can be reused across experiments and checkpoints.

The practical rule is simple: if a human expert can reliably judge the output, and the task can be expressed as a clear score, that is usually a good candidate for a custom grader.

The five-step RFT loop

The public docs describe a five-step loop:

Implement a grader that assigns a numeric reward to each model response.
Upload your prompt dataset and designate a validation split.
Start the fine-tune job.
Monitor and evaluate checkpoints, then revise the data or grader if needed.
Deploy the resulting model through the standard API.

That loop matters because RFT is not a one-shot training run. You are building a measurement system, then using that measurement system to shape the model. If the grader is weak, the model learns the wrong lesson. If the dataset is narrow, the model may overfit to the wrong pattern.

Data prep that actually works

Most RFT projects win or lose on data quality, not on the training command.

Start with tasks that resemble real production usage. Include examples that are easy, borderline, and genuinely hard. Add cases where the model should refuse, ask for more information, or follow a policy instead of guessing.

A practical dataset usually needs:

representative prompts from real users or real workflows
a clean train and validation split
examples that cover edge cases, not just happy paths
stable labeling rules that experts can apply consistently
enough diversity to avoid teaching the model one brittle pattern

If the task uses structured output, be explicit about the schema and evaluate that structure directly. If the task is text based, make sure the grader does not reward stylistic polish at the expense of correctness.

Evaluation discipline for training and rollout

RFT only pays off when teams treat evaluation as part of the product process, not as an afterthought.

Before training, define what success means on the validation set and what failure would look like in production. During training, inspect checkpoints instead of waiting for the final result. After training, run a separate review on representative prompts to check for overfitting, reward hacking, or behavior drift.

The most useful rollout safeguards are:

keep a validation split separate from training data
compare against a prompt-only baseline
track cost and latency alongside quality
test on edge cases and refusal cases, not just happy paths
keep a human review gate for high-impact deployments

This is especially important for product teams because a score increase can still hide a worse user experience. A model that wins the grader but makes the workflow slower, riskier, or less clear is not a real improvement.

A realistic decision checklist

Use this checklist before starting an RFT project:

Can we describe the task clearly enough for a grader to score it?
Do experts agree on what a correct response looks like?
Is the task important enough to justify data prep and training overhead?
Is prompt tuning already close to the limit?
Do we have enough representative data for train and validation splits?
Can we monitor checkpoints and compare against a baseline?
Are we comfortable gating rollout if the model regresses on edge cases?

If most of those answers are yes, RFT is probably worth trying. If the task is still vague, broad, or mostly about wording, prompt tuning is usually the cheaper move.