Direct preference optimization & its family

§1Why "direct"? What DPO removes from RLHF

Classic RLHF trains a reward model, then runs an RL loop that keeps generating text from the policy, scoring it, and updating. DPO asks: if the optimal policy of that RL problem has a closed form, can we skip the reward model and the loop?

Recall the pipeline from Chapter 04. Stage one fits a reward model $r_\phi$ on human comparisons with a Bradley–Terry loss (Chapter 03). Stage two maximizes $\E[r_\phi]-\beta\,\KL(\pi\|\piref)$ with PPO, which in practice means four networks in memory (policy, reference, reward model, critic), autoregressive generation inside the training loop, and a long list of stabilizing tricks. Rafailov et al. (2023) observed that the second stage has a known solution in function space, and that you can fold the first stage into it. The result is a supervised-looking loss on a fixed file of pairs: no reward model, no critic, no sampling.

Notice what is not on the DPO side. The reward model is gone, but not because the reward was unnecessary: it has been absorbed into the policy as an implicit reward. Generation is gone too, and that loss is real: the model never sees (and is never corrected on) its own outputs during training. Keep both facts in mind; the rest of the chapter is largely about their consequences.

§2The derivation, one step at a time

Four moves: solve the KL-regularized problem, invert the solution for the reward, plug the reward into Bradley–Terry, and watch the partition function disappear.

Move 1: the closed form. For a fixed prompt $x$, write the objective as a single expectation under $\pi$ and complete it into a KL divergence. Define $Z(x)=\sum_y\piref(y|x)\,e^{r(x,y)/\beta}$ and $\pi^*(y|x)=\piref(y|x)\,e^{r(x,y)/\beta}/Z(x)$. Then

The last term does not depend on $\pi$, and KL is non-negative and zero only at equality, so the unique maximizer is $\pi=\pi^*$. (This is Gibbs' inequality; Chapter 04 builds intuition for the "tilt".) The catch is $Z(x)$: a sum over every possible string. You cannot normalize, so you cannot sample from $\pi^*$, which is why RLHF uses RL to approximate it.

Move 2: invert. Take logs of the closed form and solve for the reward:

Read this as a change of variables: any reward function corresponds to exactly one optimal policy (given $\piref,\beta$), and that policy, together with the unknown per-prompt constant $\beta\log Z(x)$, determines the reward back.

Move 3: plug into Bradley–Terry. The preference model only ever sees reward differences between two responses to the same prompt: $p(y_w\succ y_l|x)=\sigma\big(r(x,y_w)-r(x,y_l)\big)$. Substitute Move 2 for both rewards, and the two copies of $\beta\log Z(x)$ (same $x$!) subtract to zero.

Move 4: fit the policy instead of the reward. Parametrize $\pi^*$ by our model $\pit$ and maximize the likelihood of the observed preferences. The scrollytelling figure walks through it.

The DPO paper also proves that this reparametrization loses nothing: every equivalence class of reward functions (rewards that differ only by a function of $x$, which Bradley–Terry cannot distinguish anyway) contains exactly one reward of the form $\beta\log\frac{\pi(y|x)}{\piref(y|x)}$ for some policy $\pi$. So "fit a reward, then find its optimal policy" and "fit the policy directly" optimize over the same set, in the population limit. The same trick extends to Plackett–Luce rankings over $K$ responses. Keep the qualifier in italics in mind: with finite, off-policy data the two procedures diverge, and §6 is about how.

The gradient, and why it focuses on mistakes

Write $\hat r_w=\beta\log\frac{\pit(y_w|x)}{\piref(y_w|x)}$ and $\hat r_l$ likewise, and let $u=\hat r_w-\hat r_l$. Since $\frac{d}{du}\big[-\log\sigma(u)\big]=-\sigma(-u)$ and $\nabla_\theta u=\beta\big(\nabla_\theta\log\pit(y_w|x)-\nabla_\theta\log\pit(y_l|x)\big)$ (the reference terms are constants):

Without the weight, this is "unlikelihood on the rejected plus likelihood on the chosen", and the DPO paper reports that this naive version makes the model degenerate. The weight is what the KL term buys you: once a pair is ordered with a margin of a few units of $1/\beta$, it stops contributing. In terms of the raw log-ratio margin $h=\log\frac{\pit(y_w)}{\piref(y_w)}-\log\frac{\pit(y_l)}{\piref(y_l)}$, the weight is $\sigma(-\beta h)$, so it falls to $0.1$ at $h=\ln 9/\beta$. With the common $\beta=0.1$, that is about 22 nats. Twenty-two nats of log-ratio is an enormous move for a sequence probability; this is one reason sequence log-probs can move by tens of nats during DPO training.

What β means, three ways

• KL price. In the original objective, $\beta$ is the price of one nat of KL from the reference. Small $\beta$ means "trust the preferences, move far".
• Implicit-reward temperature. $\hat r=\beta\log\frac{\pit}{\piref}$. For a fixed preference strength, smaller $\beta$ requires a larger log-ratio to express it, so the policy must move further from the reference to fit the same data.
• Loss saturation point. The logistic loss is satisfied at margins of order $1/\beta$ nats. As a rough guide, below $\beta\approx 0.01$ the loss barely saturates on realistic margins; above $\beta\approx 0.5$ it saturates almost immediately and the model barely moves. Most published runs of standard DPO (summed log-probs) use $\beta\in[0.01,0.5]$, with $0.1$ the de facto default (the DPO paper's own default; it used $0.5$ for TL;DR summarization).

A subtlety that is easy to miss: in DPO, $\beta$ does not enforce a KL budget the way it does in PPO. In PPO the KL penalty is evaluated on the policy's own samples at every step. In DPO, $\beta$ only shapes a loss on fixed data; nothing measures the KL on the responses the model will actually produce. §4 and §6 show how that difference bites.

The implicit reward is a reward model you can query

After training, $\hat r_\theta(x,y)=\beta\log\frac{\pit(y|x)}{\piref(y|x)}$ scores any response. RewardBench-style evaluations do exactly that with DPO checkpoints. Two identities make it concrete. Averaged over the policy's own samples, $\E_{\pit}[\hat r_\theta]=\beta\,\KL(\pit\|\piref)\ge0$; averaged over the reference's samples, $\E_{\piref}[\hat r_\theta]=-\beta\,\KL(\piref\|\pit)\le0$. So "implicit rewards going negative" on reference-like data is not a bug, it is arithmetic. And the argmax of $\hat r$ need not be the policy's most likely response: a rare response the policy boosted a lot relative to the reference gets a huge implicit reward while still being improbable.

§3Likelihood displacement: where does the probability go?

DPO only asks the chosen response to beat the rejected one by a margin. It is perfectly happy if both get less likely, and in practice they usually do.

The phenomenon was noticed early by practitioners (log-probs of chosen responses falling during DPO) and has since been analysed carefully. Pal et al. (2024, "Smaug") show theoretically that standard DPO can reduce the likelihood of the preferred response, especially when the two responses have a small edit distance. Feng et al. (2024) analyse the gradient field and find DPO decreases the probability of dispreferred data faster than it increases that of preferred data. Rafailov et al. (2024, "From $r$ to $Q^*$") argue that when DPO starts from an SFT model, the implicit rewards of both chosen and rejected should fall: the expected implicit reward under the reference's own samples is $-\beta\KL(\piref\|\pit)$, which is zero at the start and strictly negative once the policy has moved, and preference data the SFT model was trained to fit looks like reference samples. Razin et al. (2024) name the effect likelihood displacement and show it can be harmful: training a model to prefer "No" over "Never" can sharply raise the probability of "Yes", and in their experiments DPO on refusal data drove Llama-3-8B-Instruct's refusal rate from 74.4% to 33.4%.

The first-order story

Let $g_w=\nabla_\theta\log\pit(y_w|x)$ and $g_l=\nabla_\theta\log\pit(y_l|x)$. One gradient step on one pair moves $\theta$ by $\eta\,c\,(g_w-g_l)$ with $c=\beta\,\sigma(\hat r_l-\hat r_w)>0$. To first order, the change in the chosen log-prob is

This is negative whenever $\langle g_w,g_l\rangle>\|g_w\|^2$: when the two responses look alike to the model (their gradients are aligned) and the rejected one has the larger gradient norm. And any third response $y'$ changes by $\eta c\,\langle g_{y'},\,g_w-g_l\rangle$: the mass goes to whatever response is best aligned with the difference direction, which can be a response that appears in no pair at all. Razin et al. make this precise with a centered hidden-embedding similarity (CHES) score between chosen and rejected, and show that filtering out high-CHES pairs mitigates the unintended unalignment.

Fixes, and what each one actually does

• DPO-Positive (DPOP) (Pal et al., 2024) adds a hinge inside the sigmoid that penalizes the chosen log-ratio for going below zero: $-\log\sigma\big(\beta(h-\lambda\max(0,\log\frac{\piref(y_w|x)}{\pit(y_w|x)}))\big)$. The paper used $\beta=0.3$ and $\lambda=50$ (chosen without a sweep).
• DPO + NLL adds a supervised term on the chosen response: $\mathcal L_{\text{DPO}}+\alpha\,\mathcal L_{\text{NLL}}(y_w)$. Pang et al. (2024, Iterative Reasoning Preference Optimization) report the NLL term as crucial for reasoning (Llama-2-70B-Chat on GSM8K from 55.6% to 81.6%); Llama 3's post-training recipe also adds a scaled NLL term (§8). The cost: the chosen responses become imitation targets, which is bad when they are not much better than what the policy already produces.
• Data filtering. Razin et al. drop pairs whose chosen and rejected are too similar in representation space (high CHES).
• Anchored objectives. APO (D'Oosterlinck et al., 2024) makes the direction explicit: APO-zero raises winners and lowers losers in absolute terms; APO-down lowers both, the loser more, for when the model is already better than the "winning" data.
• IPO does not fix displacement, but bounds it: once the margin reaches its target the gradient stops (try it in the widget).

§4Overfitting: deterministic preferences, IPO and noisy labels

The logistic loss is never satisfied. On a pair that is always labelled the same way, DPO's optimum sends the margin to infinity, and the KL "regularizer" has nothing to say about it.

Azar et al. (2023) put this sharply. Suppose for some pair the true preference is deterministic: $p^*(y\succ y')=1$. Bradley–Terry can only represent probability one with an infinite reward gap, so the DPO solution has $\pi(y')=0$ whatever $\beta$ is. In their words, the KL regularization "becomes weaker and weaker the more deterministic the preferences". This is not only about truly deterministic preferences: with finite data, if a pair is observed once, the empirical preference is 0 or 1, and DPO fits it as if deterministic. In RLHF the same data trains a reward model that is itself regularized (early stopping, finite capacity, ensembling), and then the KL penalty on the policy's own samples bounds how far the policy goes; in DPO the empirical preference is consumed directly.

ΨPO and IPO

Azar et al. define a general objective, $\Psi$PO: maximize $\E_{x,\,y\sim\pi,\,y'\sim\mu}\big[\Psi\big(p^*(y\succ y'|x)\big)\big]-\tau\,\KL(\pi\|\piref)$ for a non-decreasing $\Psi$. With $\Psi(p)=\log\frac{p}{1-p}$ and Bradley–Terry preferences, this recovers the RLHF/DPO objective; the unboundedness of the logit near $p=1$ is the source of the problem. With $\Psi$ the identity, preferences enter bounded, and the sampled loss becomes a regression of the log-ratio margin onto a constant:

Two things to notice. The loss has a finite minimizer, $h^*=1/(2\tau)$, so pairs that already have that margin are pushed back. And, unlike DPO, its gradient is not self-normalizing: it is linear in the distance to the target. Libraries such as TRL expose IPO with $\beta$ in the role of $\tau$; the target margin is then $1/(2\beta)$, i.e. 5 nats at $\beta=0.1$.

Label noise: cDPO and robust DPO

If each label is flipped with probability $\varepsilon$, the right target for $p(y_w\succ y_l)$ is $1-\varepsilon$, not 1. Mitchell's note (2023) turns this into conservative DPO:

which is exactly binary cross-entropy with label smoothing. Its gradient vanishes when $\sigma(\beta h)=1-\varepsilon$, i.e. at $h^*=\frac1\beta\ln\frac{1-\varepsilon}{\varepsilon}$; as the note puts it, "For normal DPO, the gradient is never zero!" Robust DPO (Chowdhury et al., 2024) instead builds an unbiased estimator of the clean loss, $\big[(1-\varepsilon)\mathcal L(y_w,y_l)-\varepsilon\,\mathcal L(y_l,y_w)\big]/(1-2\varepsilon)$, and proves a sub-optimality bound that degrades as $1/(1-2\varepsilon)$. Note the minus sign: cDPO and rDPO are different losses, though both take $\varepsilon$ as a hyperparameter (TRL exposes $\varepsilon$ as label_smoothing for its robust loss).

§5Length: the easiest way to win a preference

Human and AI judges tend to prefer longer answers. DPO faithfully learns whatever predicts the label, and length predicts it.

Park et al. (2024, "Disentangling Length from Quality in Direct Preference Optimization") document that DPO models grow verbose and tie the effect to out-of-distribution bootstrapping: the implicit reward extrapolates "longer is better" to lengths the data never contained. Their fix, R-DPO, adds a length term to the Bradley–Terry logit,

derived from modelling the observed reward as $r=\beta\log\frac{\pi}{\piref}+\beta\log Z+\alpha|y|$: an explicit length term "explains away" the length-driven part of the preference, so the policy's log-ratio does not have to. They report up to 20% improvement in win rates when controlling for length.

The other popular answer is length normalization. Sequence log-probs are sums over tokens, so a long response has a large-magnitude $\log\pi$, and the DPO margin of a pair with very different lengths can be dominated by the length difference. SimPO (Meng et al., 2024) uses the average per-token log-prob as its implicit reward, $\frac{\beta}{|y|}\log\pit(y|x)$, with no reference model; Tülu 3 uses a length-normalized variant of DPO (§8). Normalization changes the geometry, not just the scale: it removes the incentive for the chosen response's sum of log-probs to be driven by its token count.

§6Off-policy data and the DPO–RLHF gap

In the population limit DPO and RLHF have the same optimum. With finite data that was sampled from some other model, they do not, and the difference is mostly about which responses get any training signal at all.

The theory gap: what the loss leaves unconstrained

DPO's loss touches $\pit$ only on the responses in the dataset. Xu et al. (2024, "Is DPO Superior to PPO for LLM Alignment?") show that the set of policies minimizing the DPO loss is a strict superset of the set of PPO solutions: "$\Pi_{\text{PPO}}$ is a proper subset of $\Pi_{\text{DPO}}$". Any policy that matches the right log-ratio differences on the data, and does anything at all on responses outside the data, is a DPO optimum; DPO "might find biased solutions that exploit out-of-distribution responses". PPO, by contrast, samples from the policy, so a response that gains probability gets scored by the reward model and pays KL on the policy's own samples. Their empirical results are stark on code: on CodeContests with Code Llama 34B (10@1k on the test set), SFT reached 15.2%, DPO 0.0%, iterative DPO 3.2%, and PPO 22.4%.

Song et al. (2024) give the cleanest statement in terms of coverage: a global coverage condition (the data must cover everything any candidate policy might do) is necessary and sufficient for offline contrastive methods like DPO to converge, while a weaker partial coverage condition suffices for online RL methods. Their HyPO uses offline data for the preference loss and online samples for the KL term. Swamy et al. (2025, "All Roads Lead to Likelihood") ask why two-stage RLHF beats offline fine-tuning at all, given that a reward model cannot add information; their best-supported explanation is the generation–verification gap: a reward (verifier) is simpler to learn than a good policy, and RL then only has to search among policies optimal for that simple verifier.

Empirical evidence: sampling and negative gradients

• Tajwar et al. (2024) compare objectives across bandits, synthetic LLM tasks and real ones and find that on-policy sampling and a "negative gradient" (actively pushing down bad responses) both help; they unify them as mode-seeking objectives, which "relocate masses across bins more effectively" than maximum-likelihood style objectives.
• Tang et al. (2024) study the online/offline gap directly and find it is not explained by coverage or data quality alone: offline-trained policies become better pairwise classifiers but worse generators, and scaling the policy does not close the gap.
• Guo et al. (2024, OAIF): making DPO/IPO/SLiC online by labelling fresh policy samples with an LLM judge wins about 66% of human comparisons against the offline versions.
• Chen et al. (2024): most preference-tuned models reach ranking accuracy below 60% on held-out preference data, and DPO is "ill-suited to fix even mild ranking errors in the reference model". The implicit reward model you trained is weak even when the generations improve.
• Lanchantin et al. (2025), Meta: on Llama-3.1-8B-Instruct, semi-online DPO (re-syncing the generator every few steps), fully online DPO and GRPO perform similarly and all strongly beat offline DPO; online DPO gives a 56.6% increase in AlpacaEval LC win rate and 45.6% in Arena-Hard score over offline DPO.

§7The family: one idea per variant

Almost every DPO variant changes one of four things: the link function, the reference, the pairing, or where the samples come from.

Tang et al. (2024, "Generalized Preference Optimization") make the first axis precise: write the offline loss as $\E\big[f\big(\beta\,h_\theta\big)\big]$ for a convex function $f$. Logistic $f(u)=-\log\sigma(u)$ gives DPO, squared $f(u)=(u-1)^2$ gives IPO (with $\beta=2\tau$; up to a constant factor), hinge $f(u)=\max(0,1-u)$ gives SLiC. Different $f$ imply different effective regularization; the "KL" in each is only enforced through the shape of $f$ on the data.

Changing the link: IPO, cDPO, robust DPO, χPO

Covered in §4. One more worth knowing: χPO (Huang et al., 2024) is a "one-line change" to DPO that replaces the log link with $\phi(z)=z+\log z$ applied to $\pi/\piref$ (clipped), which amounts to adding $\chi^2$-divergence regularization to the KL. The $\chi^2$ penalizes moving mass onto responses the reference rarely produces much more harshly than KL does, which the authors show gives guarantees against over-optimization under single-policy coverage.

Dropping the pairing: KTO

Ethayarajh et al. (2024) start from prospect theory: humans value outcomes relative to a reference point, concavely for gains and convexly for losses. KTO needs only a binary signal per response ("desirable" or "undesirable"), not pairs. With $r_\theta=\log\frac{\pit(y|x)}{\piref(y|x)}$ and a reference point $z_0=\KL(\pit\|\piref)$,

$z_0$ plays the role that the paired response plays in DPO (a baseline that makes the log-ratio meaningful). It is not backpropagated through and in practice is estimated from mismatched pairs in the batch, $\hat z_0=\max\big(0,\frac1m\sum\log\frac{\pit(y_j|x_i)}{\piref(y_j|x_i)}\big)$, which the authors call "a biased but convenient estimate". Because the value function is a sigmoid rather than a log-sigmoid, KTO's gradient vanishes for examples that are very wrong as well as very right: it is robust to outliers (and noisy labels) by construction, at the price of ignoring hard examples. The paper reports KTO matching or exceeding DPO from 1B to 30B parameters and matching DPO with up to 90% fewer desirable examples (reweighting with $\lambda_D,\lambda_U$; they recommend $\frac{\lambda_D n_D}{\lambda_U n_U}\in[1,\tfrac32]$). The practical appeal: thumbs-up/thumbs-down logs are far more abundant than curated pairs.

Dropping the reference: ORPO and SimPO

ORPO (Hong et al., 2024) merges SFT and preference optimization into one stage with no reference model: $\mathcal L_{\text{ORPO}}=\mathcal L_{\text{SFT}}(y_w)+\lambda\,\mathcal L_{\text{OR}}$ with $\mathcal L_{\text{OR}}=-\log\sigma\Big(\log\frac{\text{odds}_\theta(y_w|x)}{\text{odds}_\theta(y_l|x)}\Big)$ and $\text{odds}=\frac{P}{1-P}$ (computed on length-normalized likelihoods). The SFT term anchors the chosen; the odds-ratio term pushes the rejected down. No reference means one model in memory and one training run instead of two.

SimPO (Meng et al., 2024) also drops the reference, normalizes by length and adds a target margin:

The argument is that the implicit reward should match the quantity that guides generation (average per-token log-likelihood), not a log-ratio to a reference that plays no role at inference. The authors recommend $\beta\in[2.0,2.5]$ and $\gamma\in[0.5,1.5]$ (with larger $\beta$, e.g. 10, better in some Llama-3-Instruct settings) and report gains over DPO of up to 6.4 points on AlpacaEval 2 and 7.5 on Arena-Hard. The caveat: with no reference, nothing in the loss ties the model to its starting point except the learning rate and the number of steps, so SimPO is more sensitive to both, and it inherits likelihood displacement (§3).

Changing the samples: RSO, iterative/online DPO, OAIF, self-rewarding

RSO (Liu et al., 2023) argues that estimating the optimal policy is best done with preference pairs sampled from (an estimate of) that optimal policy, not from the SFT model or from unrelated models. It trains a reward model, then uses statistical rejection sampling to draw responses approximately from $\pi^*\propto\piref e^{r/\beta}$ and builds pairs from those. Iterative / online DPO (Xiong et al., 2023; Dong et al., 2024, "RLHF Workflow") alternates: sample several responses per prompt from the current policy, rank them with a reward model, take best vs worst as a pair, run DPO, repeat. Xiong et al. give the theory for the KL-regularized contextual bandit in offline, online and hybrid settings. OAIF (Guo et al., 2024) uses an LLM judge as the online annotator, and shows you can steer behaviour (e.g. ask the judge for shorter answers, cutting average response length from about 120 to about 40) just by changing the judge's prompt. Self-Rewarding LMs (Yuan et al., 2024) make the policy its own judge (LLM-as-a-judge) inside iterative DPO, improving both the responses and the judging across iterations.

Leaving Bradley–Terry: Nash learning, SPPO, DNO

Bradley–Terry assumes preferences are transitive and come from a scalar reward. Human preferences are often intransitive. Nash learning from human feedback (Munos et al., 2023) learns a pairwise preference model $\mathcal P(y\succ y'|x)$ and seeks the policy that no other policy is preferred to: the Nash equilibrium of the symmetric game, solved with a mirror-descent algorithm (Nash-MD) with last-iterate convergence to the regularized equilibrium. SPPO (Wu et al., 2024) approximates the equilibrium by self-play: each iteration samples responses from the current policy, estimates each response's win rate against the policy with a preference model, and regresses the log-ratio toward that win rate. With a 0.4B PairRM judge and 60k UltraFeedback prompts, it reports a 28.53% length-controlled win rate against GPT-4-Turbo on AlpacaEval 2.0 for Mistral-7B-Instruct and 38.77% for Llama-3-8B-Instruct. DNO (Rosset et al., 2024) is a batched on-policy algorithm with a regression-style objective in the same spirit. A related line, general preference models (Zhang et al., 2024), embeds responses so that the preference model can represent cyclic preferences. Note the naming collision: Zhang et al.'s "General Preference Optimization" and Tang et al.'s "Generalized Preference Optimization" are different papers.

Reasoning: Step-DPO

A long chain of thought fails at one step. Pairing a whole correct solution with a whole incorrect one gives a weak signal (they share most tokens and differ for many reasons). Step-DPO (Lai et al., 2024) pairs a correct and an incorrect next step given a shared correct prefix, and applies DPO at that granularity. With 10K step-wise pairs and fewer than 500 training steps, it reports a nearly 3% gain on MATH for models over 70B, and Qwen2-72B-Instruct with Step-DPO reaches 70.8% on MATH and 94.0% on GSM8K. Self-generated (on-policy) steps worked better than human- or GPT-4-written ones, the same lesson as §6 in miniature. A free fact behind it: tokens in a shared prefix have identical log-ratios in the chosen and rejected sequences, so they cancel exactly in $h$; DPO's gradient always concentrates at and after the point where the two responses diverge.

The comparison table

§8What open recipes actually do (2024–2026)

In 2024, DPO was the preference stage of almost every open recipe. By 2025–2026, the recipes that could afford it put online RL at the centre and kept DPO for narrower jobs.

The table summarizes what the technical reports say. Read the β column with care: β = 5 in Tülu 3 and Olmo 3 multiplies a per-token average log-ratio (length-normalized DPO), so it is not comparable with the β = 0.1 of standard DPO on summed log-probs.

Why the frontier went back to sampling

1. Reasoning made on-policy data non-negotiable. When the target behaviour is "solve problems the model currently fails", the useful training signal lives in the model's own attempts, and verifiers can score them for free. That is RLVR (Chapter 06). Offline pairs from other models carry other models' mistakes.
2. The offline ceiling is real. §6's evidence (coverage theory, Tang et al.'s classifier-vs-generator gap, Xu et al.'s code results, Lanchantin et al.'s semi-online results) all points the same way. Nathan Lambert's RLHF book summarizes the comparisons: in these reports "DPO algorithms are a hair behind", yet direct alignment algorithms "are still used extensively in leading models due to their simplicity".
3. The cost argument shrank. DPO's main advantage was running on SFT infrastructure. Once labs had fast inference engines inside the training loop and asynchronous RL systems (Chapter 07), the marginal cost of sampling fell, and Tülu 3's 7× wall-clock gap stopped being decisive for well-resourced teams.
4. Offline stages can hurt later RL. Llama 4's remark that SFT and DPO "over-constrain the model" before online RL is the clearest public statement. Olmo 3 reports the opposite for its delta-learning DPO (a better RLVR initialization than SFT alone), which suggests the issue is how much the stage narrows the distribution, not the loss per se.

§9Implementation details that decide whether it works

DPO is twenty lines of loss and a hundred ways to compute the wrong log-probability.

Sequence log-probs with masking

$\log\pit(y|x)=\sum_{t}\log\pit(y_t|x,y_{\lt t})$, summed over completion tokens only. The prompt, padding, and (in multi-turn data) earlier turns are masked out. Llama 3's report adds a further mask: special formatting tokens (header and termination tokens) are masked out of both chosen and rejected responses, because tokens common to both sides create "a conflicting learning objective" and, in their experience, led to behaviours such as tail repetition or abruptly generating termination tokens (§8).

import torch, torch.nn.functional as F

def sequence_logps(model, input_ids, attention_mask, completion_mask):
    """Sum of log p(token) over completion tokens. completion_mask is 1 on response tokens."""
    logits = model(input_ids=input_ids, attention_mask=attention_mask).logits
    logits = logits[:, :-1].float()                      # position t predicts token t+1
    targets = input_ids[:, 1:]
    mask = completion_mask[:, 1:].float()
    logp = torch.gather(F.log_softmax(logits, dim=-1), 2, targets.unsqueeze(-1)).squeeze(-1)
    return (logp * mask).sum(-1), mask.sum(-1)            # (sum log-prob, number of tokens)

def preference_loss(pw, pl, rw, rl, beta=0.1, kind="dpo", eps=0.0, len_w=None, len_l=None, gamma=0.0):
    """pw/pl: policy log-probs of chosen/rejected; rw/rl: reference log-probs (cached is fine)."""
    if kind == "simpo":                                   # reference-free, length-normalized
        u = beta * (pw / len_w - pl / len_l) - gamma
        return -F.logsigmoid(u).mean()
    h = (pw - rw) - (pl - rl)                             # log-ratio margin
    if kind == "ipo":
        return ((h - 1.0 / (2 * beta)) ** 2).mean()
    # DPO, or cDPO when eps > 0 (label smoothing)
    return (-(1 - eps) * F.logsigmoid(beta * h) - eps * F.logsigmoid(-beta * h)).mean()

# logging: implicit rewards (detached)
# rewards_chosen = beta * (pw - rw).detach(); rewards_rejected = beta * (pl - rl).detach()

Reference log-prob caching and memory

• Cache the reference. The reference is frozen and the data is fixed, so its log-probs can be computed in one pass before training (TRL: precompute_ref_log_probs). This removes the second model from GPU memory during training and is exact, provided tokenization, truncation and template are identical in both passes.
• LoRA gives you the reference for free. With adapters, the reference is the base model with adapters disabled; trainers compute it by toggling the adapter off rather than holding a copy.
• Concatenate chosen and rejected into one forward pass (pad to a common length), as TRL does: one call instead of two, which matters most under FSDP/ZeRO where every forward pass re-gathers the sharded weights.
• Compute log-softmax in fp32. A summed log-prob over 1,000+ tokens in bf16 accumulates errors of the same order as the margins you are trying to learn.
• Know your library's conventions. In TRL (v1.16), sequence log-probs are sums over completion tokens; chosen and rejected are concatenated into one batch; the ipo loss divides log-probs by completion length (so its target is in per-token units); sigmoid_norm is the length-normalized DPO of Tülu 3; and the old rpo_alpha argument is gone (since v0.29): DPO + NLL is now loss_type=["sigmoid", "sft"] with loss_weights=[1.0, alpha]. Libraries move fast; read the loss code, not the blog post.
• Truncation must be symmetric. Truncating the chosen more than the rejected changes $|y|$ and the sum; both sides of a pair should be cut by the same rule (truncate the prompt first).

Hyperparameters that published runs actually used

Two rules of thumb fall out, both with exceptions. First, many open full-parameter recipes run DPO at learning rates one to two orders of magnitude below their SFT rate (Zephyr: 5e-7 vs 2e-5; Tülu 3: 5e-7 and 2e-7): the loss is a difference of large sums, and small parameter changes move it a lot. Llama 3 is the notable exception, using 1e-5 for DPO (the same rate as its SFT), so treat the learning rate as something to sweep, not copy. Second, $\beta$ and learning rate interact. With plain SGD the gradient scales with $\beta$ (§2), so lowering $\beta$ also slows training. With Adam-family optimizers, which normalize away a global gradient scale, that effect largely disappears and $\beta$ acts mainly through the weighting $\sigma(-\beta h)$ and the saturation margin; a run that "moves too little" at small $\beta$ is then more likely a data or learning-rate issue than a $\beta$ issue.

What to monitor

• rewards/accuracies (fraction of pairs with $\hat r_w>\hat r_l$) rises quickly toward 1 on train; on held-out pairs it plateaus far lower (Chen et al.'s sub-60% ranking accuracy is the sobering benchmark).
• rewards/margins growing without bound on train means the margin is being pushed to infinity (§4).
• logps/chosen falling steadily means displacement (§3). Pair it with sampled generations: where did the mass go?
• Mean generation length and KL to the reference measured on samples from the policy, not on the dataset.

§10Frontier & open problems

The open questions have shifted from "which loss?" to "which samples, which judge, and how far can you trust the result?".

• Loss vs data, settled-ish. The best evidence (Tajwar et al. 2024; Tang et al. 2024; Lanchantin et al. 2025) says sampling from the current policy matters more than the choice between a DPO-style and a policy-gradient loss. Semi-online DPO, re-syncing the generator every few to a hundred steps, is a practical middle ground that keeps most of the online benefit with simpler infrastructure. What is not settled is when off-policy data is enough: Sun et al. (2025) find on-policy data was about 3× as effective as static data for Llama-3 but less effective than static data for Zephyr, and SimpleMix (Li et al., 2025) finds the answer differs between verifiable and open-ended tasks.
• Displacement as a safety problem. Razin et al.'s refusal result reframes likelihood displacement from a curiosity to a risk: an innocuous-looking preference fine-tune can remove behaviours that were trained in earlier. Detecting it requires measuring where the mass went, not just whether the margin grew. Representation-aware data filtering (CHES) and anchored objectives are early answers.
• Over-optimization without an explicit reward. Direct alignment algorithms over-optimize much like RLHF (Rafailov et al., 2024), but there is no reward model to ensemble or regularize. Pessimistic objectives such as χPO change the divergence; whether that holds up at frontier scale is open.
• Beyond Bradley–Terry. Human preferences are noisy, context-dependent and sometimes intransitive. Nash-style methods (NLHF, SPPO, DNO) and general preference models drop the scalar-reward assumption; the open question is whether that matters at scale or is swamped by judge noise.
• What the preference signal really is. The delta-learning hypothesis (Geng et al., 2025) shows that pairs where both responses are weak can still teach a stronger model, as long as the chosen is reliably better than the rejected: the information is in the difference. That reframes DPO data curation around reliable deltas rather than high-quality "chosen" responses.
• Where DPO lives in 2026 pipelines. For reasoning and agentic behaviour, most published frontier-scale recipes rely on online RL with verifiable rewards and reward models (Chapter 06, Chapter 07). Direct preference losses survive as cheap stages for style and chat quality, as warm starts, and inside iterative loops.

§11Check yourself

Try each one before opening the answer. The "research" questions have no single right answer; the answers show one good line of reasoning.

§12Read next