The GPU cost model nobody writes down before buying hardware

The comparison everyone runs

Somewhere in every build-versus-buy conversation there is a spreadsheet with two numbers on it: the per-million-token price of an API, and the hourly rental of a GPU divided by an assumed throughput. The second number is usually three to ten times smaller than the first, and the meeting ends.

The spreadsheet is not wrong about the arithmetic. It is wrong about the denominator. It divides by the throughput the card can deliver, not the throughput it will deliver.

The number that actually decides it

Utilisation. A rented A100 or H100 billed by the hour costs the same whether it serves one request or ten thousand. The API charges you for tokens you actually consume. So the crossover is not a question of raw price, it is a question of how many hours per day your card is genuinely busy.

Take a concrete shape: steady traffic of a few million tokens a day, spread across working hours. That is a workload which saturates a single card for perhaps a quarter of the day and leaves it idle otherwise. Priced per token actually served, the idle hours are pure overhead, and the API wins comfortably. Push the same traffic to a twenty-four-hour sustained load with batching and the ranking flips.

What the spreadsheet leaves out

Two costs are systematically missing. The first is the engineer. Somebody has to own the serving stack, the model upgrades, the CUDA and driver compatibility, the autoscaling policy, and the on-call. That is not a line item you can invoice, but it is a real fraction of a person, and it does not scale down when traffic does.

The second is the retry and failure budget. Managed APIs absorb their own bad hours. Your cluster absorbs yours, in the form of incidents, and incidents during a customer pilot cost more than the compute they interrupt.

How to actually decide

Measure before you buy. Run the workload on rented hardware for two weeks with real traffic, record tokens served and GPU-hours billed, and compute your realised cost per million tokens. Compare that with the API price for the same quality tier. The result is specific to your traffic shape, which is exactly why a generic comparison table cannot answer it for you.

Then decide on the non-cost factors, which are usually the real drivers: data residency, latency to your users, rate limit independence, and whether you need a model nobody serves.