Why the model agrees with you

You paste a plan into the model and ask what it thinks. It says the plan is strong and lists why. Something nags at you, so you rephrase: "I'm worried this plan is risky, am I right to be?" Now it agrees the plan is risky and lists why. Same plan, opposite verdicts, and the only thing that changed was which answer you signaled you wanted.

That's not a glitch. It's the model doing exactly what its final training stage rewarded it for. To trust anything a model tells you, you have to know that its agreeableness was manufactured on purpose, by a process that optimized for human approval, and that agreement is close to the one thing it was most directly trained to produce. This module is the pipeline that built that behavior, and how to ask questions that route around it.

First, what a raw model does

Before any of the assistant training, there's pretraining: the model reads a huge slice of the internet and learns one skill, predict the next token. That's it. The result is a base model, and it's strange to interact with. It has vast knowledge and zero manners. It doesn't answer questions, it continues documents.

So the knowledge is laid down in pretraining, and the behavior, the helpful, answer-the-question, follow-instructions persona, comes from three stages layered on top. Chip Huyen's essay calls the base model a "monster" that post-training tames, and that's the right shape to hold: capability first, then manners.

1

Pretraining

Read a huge slice of the internet, predict the next token. Produces a base model: vast knowledge, no persona, completes documents rather than answering.

2

SFT: supervised fine-tuning

Show the model thousands of ideal question-and-answer conversations, written largely by human labelers to a spec. It imitates them. The 'assistant' is this learned style.

3

Reward model: learn human taste

Collect human preference votes on pairs of answers, train a second model to predict which one people will pick. That model is a stand-in for human judgment.

4

RLHF: optimize against the reward model

Tune the assistant to score higher and higher with the reward model. DPO reaches a similar target straight from the preference pairs, skipping the separate reward model and RL loop.

SFT: imitate the ideal conversation

Supervised fine-tuning is the first behavior layer, and it's the simplest. You assemble a dataset of model conversations where each one shows the assistant behaving well: a question, then the kind of answer you wish it gave. Human labelers write most of these, following a spec that says what a good answer looks like. The model trains to imitate them.

After SFT you have something that acts like an assistant. Ask it a question and it answers. But imitation only covers situations the labelers thought to write down, and "write the single ideal answer" is a narrow signal. The next stage is about steering behavior across everything the labelers didn't anticipate, using a cheaper form of human input: not writing answers, just picking between them.

RLHF: a reward model is learned taste

Writing ideal answers is expensive. Comparing two answers and saying which is better is easy, and people are more consistent at it. So RLHF collects a pile of those comparisons, thousands of "answer A is better than answer B" votes from human raters, and trains a separate model, the reward model, to predict those votes. The reward model is a learned model of human taste. Hand it any answer and it outputs a score for how much people would like it.

Then you optimize the assistant to make the reward model happy. Generate an answer, score it, nudge the weights toward higher scores, repeat. The assistant climbs the reward model's approval. DPO, direct preference optimization, is a newer route to roughly the same place: it tunes the model straight from the preference pairs without building a separate reward model or running the reinforcement-learning loop, which is why a lot of teams reach for it now. For our purposes the two rhyme. Both bend the model toward what human raters preferred.

And that bend is where the trouble starts.

Sycophancy is the pipeline working as designed

Here's the uncomfortable part. What do human raters reliably prefer?

Raters lean toward confidence, flattery, length, and agreement with their own stated view. The reward model learns exactly that taste, because it was trained on exactly those votes. Then the assistant is optimized to maximize that taste. Sycophancy, the tendency to tell you what you want to hear, isn't a bug that slipped in. It's what you get when you optimize hard for human approval, because approval and truth are correlated but not the same thing. Optimizing for a proxy always drifts toward whatever scores well on the proxy, which is why this is sometimes called reward hacking. The model learns what gets the thumbs-up.

This isn't theoretical. OpenAI shipped a GPT-4o update in 2025 that overweighted short-term thumbs-up feedback and turned into a flattery machine, agreeing and praising past the point of usefulness, and they rolled it back and wrote up why. A production system, optimized slightly too hard for approval, became a sycophant. That's the mechanism, caught in the act.

System prompts steer, training decides

A fair question: can't you just put "be honest, disagree with me when I'm wrong" in the system prompt? You can, and it helps, but understand what you're doing. Training set the model's priors, its deep tendencies, over billions of tokens and thousands of preference votes. A system prompt is a suggestion layered on at inference time. It can lean against a trained tendency, but it can't reliably override it. If the training bent the model toward agreement, a sentence asking for disagreement is a light hand on a heavy wheel. Don't mistake the instruction for a guarantee.

So "it agreed with me" is weak evidence

Put the pieces together and you get the practical rule. Agreement is close to the single thing the model was most directly optimized to produce. So when you state your view and the model agrees, you've learned almost nothing, because that outcome was baked in before you asked. The agreement carries the least information of any response it could give.

The fix is in how you ask. Strip your own position out of the question. Instead of "I think this design is right, do you agree," ask "what are the three strongest arguments against this design," or present the option you dislike as if it were yours. In why models fabricate I traced how models generate plausible-but-wrong content when the true answer is improbable, and sycophancy is a cousin of that: the agreeable answer is the probable one, and probable is not the same as true.

In production: assume agreement, design against it

Every verification tool I've built assumes the model agrees too easily, and structures around it. In verify-kit, I never ask "does this paper support the claim, I think it does." I remove the stance entirely and force the model to answer against the retrieved source text, then gate the answer, because I know a grounding checker drifts toward saying yes. Enacted's citation gate works the same way: the model doesn't get to vouch for its own summary, a separate check does, because self-endorsement is exactly what the training rewards. When I evaluate models for the NoeticMap extraction work, I never phrase the eval as "confirm this is right," I phrase it as a neutral pass-or-fail against a known answer. The whole design stance is: the model's agreement is not evidence, so build the thing that produces evidence and put it outside the model.