Skip to content
←

Split View: OpenAI RFT with Custom Graders: A Practical Guide for Product and Platform Teams

|

OpenAI RFT with Custom Graders: A Practical Guide for Product and Platform Teams

What RFT is, in plain language

OpenAI reinforcement fine-tuning, or RFT, adapts a reasoning model using a feedback signal that you define through a programmable grader. The goal is not just to make the model sound better. The goal is to push it toward expert-level performance inside a specific domain.

That makes RFT different from prompt tuning alone. Prompt work can improve instruction following, tone, or formatting. RFT is stronger when the task has a clear success condition and the model can be trained against a repeatable signal.

The current public docs describe RFT as supported on o-series reasoning models, and the guide names o4-mini and o4-mini-2025-04-16. OpenAI's October 6, 2025 AgentKit launch post also says RFT reached general availability on o4-mini, with private beta support for GPT-5 and custom graders in beta.

When to use RFT instead of prompt tuning

Use prompt tuning first when the problem is mostly about clarity, style, or a small behavior shift. Reach for RFT when the model must get better at a task that is hard to express in a prompt but easy to score.

RFT is a strong fit when:

  • the task is unambiguous
  • success criteria are specific and stable
  • domain experts can agree on what a good answer looks like
  • you need better reasoning in a narrow workflow
  • failures are expensive enough that a stronger training signal is worth the setup cost

That is why RFT is a good fit for things like policy review, security analysis, legal passage selection, structured decision support, and other workflows where the answer can be judged with a tight rubric.

Prompt tuning is still valuable for shaping voice and instructions. RFT becomes the better option when prompt changes stop producing durable gains and you need the model to internalize the behavior.

How custom graders help

Custom graders are the core of RFT. They turn your business rules into a training signal.

For product teams, that means you can encode what actually matters in the product:

  • did the model choose the right action
  • did it respect the policy
  • did it produce the right structured output
  • did it avoid unsupported claims
  • did it stay within the expected scope

For platform teams, graders are the contract between evaluation and training. A good grader keeps the loop honest by forcing the team to define success before training starts. It also makes regression testing more useful, because the same rubric can be reused across experiments and checkpoints.

The practical rule is simple: if a human expert can reliably judge the output, and the task can be expressed as a clear score, that is usually a good candidate for a custom grader.

The five-step RFT loop

The public docs describe a five-step loop:

  1. Implement a grader that assigns a numeric reward to each model response.
  2. Upload your prompt dataset and designate a validation split.
  3. Start the fine-tune job.
  4. Monitor and evaluate checkpoints, then revise the data or grader if needed.
  5. Deploy the resulting model through the standard API.

That loop matters because RFT is not a one-shot training run. You are building a measurement system, then using that measurement system to shape the model. If the grader is weak, the model learns the wrong lesson. If the dataset is narrow, the model may overfit to the wrong pattern.

Data prep that actually works

Most RFT projects win or lose on data quality, not on the training command.

Start with tasks that resemble real production usage. Include examples that are easy, borderline, and genuinely hard. Add cases where the model should refuse, ask for more information, or follow a policy instead of guessing.

A practical dataset usually needs:

  • representative prompts from real users or real workflows
  • a clean train and validation split
  • examples that cover edge cases, not just happy paths
  • stable labeling rules that experts can apply consistently
  • enough diversity to avoid teaching the model one brittle pattern

If the task uses structured output, be explicit about the schema and evaluate that structure directly. If the task is text based, make sure the grader does not reward stylistic polish at the expense of correctness.

Evaluation discipline for training and rollout

RFT only pays off when teams treat evaluation as part of the product process, not as an afterthought.

Before training, define what success means on the validation set and what failure would look like in production. During training, inspect checkpoints instead of waiting for the final result. After training, run a separate review on representative prompts to check for overfitting, reward hacking, or behavior drift.

The most useful rollout safeguards are:

  • keep a validation split separate from training data
  • compare against a prompt-only baseline
  • track cost and latency alongside quality
  • test on edge cases and refusal cases, not just happy paths
  • keep a human review gate for high-impact deployments

This is especially important for product teams because a score increase can still hide a worse user experience. A model that wins the grader but makes the workflow slower, riskier, or less clear is not a real improvement.

A realistic decision checklist

Use this checklist before starting an RFT project:

  • Can we describe the task clearly enough for a grader to score it?
  • Do experts agree on what a correct response looks like?
  • Is the task important enough to justify data prep and training overhead?
  • Is prompt tuning already close to the limit?
  • Do we have enough representative data for train and validation splits?
  • Can we monitor checkpoints and compare against a baseline?
  • Are we comfortable gating rollout if the model regresses on edge cases?

If most of those answers are yes, RFT is probably worth trying. If the task is still vague, broad, or mostly about wording, prompt tuning is usually the cheaper move.

Official links

OpenAI RFT with Custom Graders: A Practical Guide for Product and Platform Teams

What RFT is

OpenAI reinforcement fine-tuning, or RFT, adapts a reasoning model using a feedback signal defined through a programmable grader. The point is not only to improve style or compliance. The point is to aim for expert-level performance inside a specific domain.

That makes RFT different from prompt tuning alone. Prompting is great for clearer instructions, better formatting, and quick iteration. RFT is better when the task has a clear success criterion and the model can be trained against a repeatable signal.

Current public docs describe RFT as supported on o-series reasoning models, and the guide names o4-mini and o4-mini-2025-04-16. OpenAI's October 6, 2025 AgentKit launch post says RFT reached general availability on o4-mini, with private beta support for GPT-5 and custom graders in beta.

When to use RFT instead of prompt tuning

Prompt tuning should come first when the problem is mainly about wording, tone, or a small behavior adjustment. RFT becomes attractive when you need the model to get reliably better at a task that is hard to specify in prose but easy to score.

RFT is usually a good fit when:

  • the task is unambiguous
  • success criteria are stable
  • experts can agree on what good looks like
  • the workflow depends on stronger reasoning in a narrow domain
  • failures are expensive enough to justify the extra setup work

That makes RFT a strong option for policy review, security analysis, legal passage selection, structured decision support, and similar workflows where a rubric can be applied consistently.

Why custom graders matter

Custom graders are the center of the RFT workflow. They convert business rules into training signal.

For product teams, the grader can encode what matters in the product itself:

  • did the model choose the right action
  • did it follow policy
  • did it produce the expected structured output
  • did it avoid unsupported claims
  • did it stay in scope

For platform teams, the grader is the bridge between evaluation and training. It forces the team to define success before the run starts, and it makes checkpoint comparisons much more useful because the same rubric can be reused across experiments.

If a human expert can score the output reliably and the task can be reduced to a clear numeric reward, that is often a strong candidate for a custom grader.

The five-step loop

OpenAI's docs describe RFT as a five-step loop:

  1. Implement a grader that assigns a numeric reward to each model response.
  2. Upload your prompt dataset and designate a validation split.
  3. Start the fine-tune job.
  4. Monitor and evaluate checkpoints, then revise the data or grader if needed.
  5. Deploy the resulting model through the standard API.

That loop is important because RFT is not just a training command. You are building a measurement system first, then using it to shape model behavior. If the grader is weak, the model can learn the wrong lesson. If the dataset is too narrow, it can overfit to an unhelpful pattern.

Data preparation that holds up in practice

Most RFT projects succeed or fail on the quality of the data.

Start with real production-like tasks. Include easy examples, borderline examples, and genuinely hard cases. Add situations where the model should refuse, ask for more information, or follow a policy instead of guessing.

A practical dataset usually includes:

  • representative prompts from real workflows
  • separate training and validation splits
  • edge cases, not only happy paths
  • labeling rules experts can apply consistently
  • enough variety to avoid brittle shortcuts

If the task uses structured output, define the schema clearly and evaluate that structure directly. If the task is text based, make sure the grader does not reward polished wording at the expense of correctness.

Evaluation discipline for training and rollout

RFT works best when evaluation is treated as part of the product process.

Before training, define what success means on the validation set and what production failure would look like. During training, inspect checkpoints instead of waiting only for the final model. After training, review representative prompts separately to check for overfitting, reward hacking, or behavior drift.

The safest rollout habits are:

  • keep validation data separate from training data
  • compare against a prompt-only baseline
  • track quality, cost, and latency together
  • test edge cases and refusal cases
  • keep human review for high-impact releases

That discipline matters because a better grader score does not always mean a better product. A model that wins the rubric but slows the workflow down or creates new risk is not a real improvement.

A decision checklist

Before starting an RFT project, ask:

  • Can we describe the task clearly enough for a grader to score it?
  • Do domain experts agree on what a correct answer looks like?
  • Is the task important enough to justify data prep and training overhead?
  • Has prompt tuning already reached diminishing returns?
  • Do we have enough data for clean training and validation splits?
  • Can we compare checkpoints against a baseline?
  • Are we willing to gate rollout if the model regresses on edge cases?

If most of those answers are yes, RFT is likely worth testing. If the task is still broad, vague, or mostly about wording, prompt tuning is usually the simpler and cheaper choice.

Official links