A couple of years ago I shipped an ads ranking model that looked like a clean win. Click-through rate went up. The offline metrics held, the A/B test was solid, and the launch note practically wrote itself.

Then I looked at what those extra clicks were doing for the advertisers paying for them. Too often, the answer was nothing. People clicked, glanced, and left. The merchant paid for the click. Nobody bought.

That gap is one of the easiest traps in ads ranking. Click data is abundant, easy to model, and gives you fast feedback, so teams naturally optimize for it. But a model trained to maximize clicks will happily surface ads that attract attention without creating value. The click is cheap. The purchase is what pays.

I've worked on ads ranking at DoorDash, and previously on YouTube Ads and Assistant NLP at Google. Across those systems, I learned that the most expensive mistakes weren't model bugs. They were objective bugs.

The objective is the bug

Here is the mechanism. A high-CTR ad that never converts looks great on your dashboard and quietly loses money for the advertiser. Do that across a marketplace and you teach your merchants to distrust the channel. CTR is a vanity metric in one specific sense: it is easy to move and easy to mistake for value.

What you actually want to rank on is expected value - P(click) × P(purchase | click) × purchase value. Roughly, the probability someone clicks, times the probability that click turns into a purchase, times what the purchase is worth. Predicting the click is the easy part. The conversion is where the money and the difficulty both live. The ad-ranking field figured this out a while ago; the research moved from chasing click-through toward directly modeling conversions more than a decade back. Production systems are often slower to follow than papers.

What we changed

Over most of 2025 I led a rebuild of the ranking model on one of our core ads surfaces. The central move was switching from a single-objective click model to a multi-task, multi-label one: a single shared representation of the user and context, with separate prediction heads for clicking and for converting. The diagram up top is the whole idea. One model learns the shared part once, then specializes per task, and we rank on the combination rather than on clicks alone.

This is not an exotic architecture. Google published the canonical version, multi-gate mixture-of-experts, back in 2018, and variants are now standard in large recommender stacks. What was new for us was putting a unified model that predicts both clicks and purchases into this surface for the first time. We were not just trying to show ads people would notice. We were trying to show ads people would buy from.

The results held up where it counts. Click-through rate went up about 7%, which was roughly a fifth of the org's annual goal on its own. Conversions went up 2.1%, with real incremental revenue behind that. The number I care about more is the conversion lift, because that is the one the merchant actually feels in their sales.

Why one model instead of two

The obvious alternative is to keep your click model and bolt a separate conversion model next to it. Plenty of teams do. I would argue against it, for reasons I learned the slow way.

A single model with shared layers learns one representation of the user and context, then branches. That shared base tends to make the conversion head better than a standalone conversion model would be, because conversions are sparse and the abundant click signal helps anchor the representation. You also get one feature pipeline, one serving path, one set of things that can break, instead of two models quietly drifting apart on their own schedules. And the development speed matters more than it sounds. When every prediction about user behavior lives in one place, the next engineer does not have to reconcile two systems to ship an improvement.

It is not free. Multi-task models fight themselves.

The parts that are actually hard

Conversions are sparse, and they show up late. Clicks are immediate; purchases are rare and delayed. Someone clicks now and buys forty minutes later, or tomorrow, or never. At training time your conversion labels are a moving target, and some of the "negatives" in your data are just purchases that have not happened yet. If you ignore that, you systematically underpredict conversion and train on a label that is still settling. This is a known and genuinely hard problem in the literature, usually filed under delayed feedback, and handling it well mattered more to model quality than any architecture tweak.

Here is the analogy I keep using on my team. You are a batter facing a pitch, and you want to start your swing off the ball's mid-air trajectory. By the time you have a clean read on the trajectory, the ball is past you. The good hitters start their swing off the pitcher's arm, off the earlier and noisier signal, because that is the one that is actually available in time. Conversion modeling is the same. Recent data is better, but your pipelines cannot always populate everything you need down to the second. Often the win is recognizing the right freshness your model will see in production, rather than squeezing the model on a signal it will never get at serving time.

The two objectives do not pull in the same direction. Click-optimized gradients and conversion-optimized gradients disagree, and the abundant click signal will drown out the sparse conversion one if you let it. A lot of the real work was task weighting and making sure the conversion head got enough signal to learn anything at all, instead of getting bullied by the easy task.

You have to ship the win without breaking the neighbors. This is the part I never see in tutorials. A ranking change does not live alone. It sits inside an auction, a budgeting and pacing system, relevance guardrails, and downstream metrics that other teams own. Improving conversions while staying neutral on the metrics you have promised not to move is its own engineering problem, and it is where a lot of good models die before they ever launch. Being right offline is maybe half the job. The other half is proving you did not quietly tax something else.

A model can only rank what retrieval hands it

Once the ranker was finally optimizing the right objective, we discovered the next bottleneck wasn't ranking at all. It was retrieval. Your model can only choose among the ads that retrieval surfaces, and plain keyword retrieval is worse at that than people assume. Two products can mean the same thing to a shopper and share almost none of the same words. Keyword matching never sees the connection, so the best ad never even enters the auction.

That is the problem behind the semantic retrieval work I have been doing more recently: representing queries and products as high-dimensional embeddings so the system matches on intent instead of exact words, then serving that match under real-time latency at high QPS (queries per second). Facebook's writeup on embedding-based retrieval in search is a good map of the terrain. The hard parts are familiar in shape: keeping the relevance model honest under distribution shift, and making embedding retrieval fast enough to sit in the serving path without blowing the latency budget.

The same idea showed up in a more business-shaped form when I led the work to let ads run on branded search queries we had previously blocked. That was not one model change. It was an end-to-end pass through search, ranking, relevance, targeting, budgeting, and retrieval, across several orgs, to safely widen the set of ads allowed to compete. Different surface, same lesson: the candidate set is part of the model.

What I would tell an engineer building this

Pick the objective that matches the money before you pick anything else. A better model aimed at the wrong target is just a faster way to lose.

Expect multi-task pain and budget for it. The architecture diagram is the easy part. Task weighting, sparse and delayed labels, and the daily fight between objectives are where the time actually goes.

Treat retrieval and ranking as one system. A perfect ranker over a bad candidate set is still a bad system.

And respect the guardrails. Half of shipping is proving you did not break something another team owns. That work is not glamorous and it is not optional.

I spent the early part of my career believing the model was the hard part: the architectures, the optimization, the papers. The model is the part that gets you into the room. What keeps you there is being honest about what you are predicting, and whether it is the thing that pays.