On-premise vs cloud AI: the real cost comparison

Per-token pricing looks small until you multiply it by an organisation. Self-hosting looks expensive until you depreciate it. Here is how to compare the two honestly.

The cost structures are different in kind

API spend is a pure variable cost: it scales with every user, workflow and agent you add, forever. Self-hosting is mostly fixed: hardware up front, then power, hosting and an operations share each month. Variable costs feel safe at low volume and punishing at high volume; fixed costs are the reverse.

Where break-even usually lands

Across our modelling, self-hosting starts to win clearly at sustained volumes of several million tokens per day. A workload of 5M tokens/day at a blended $8 per million tokens is $1.2M a year in API fees; a capable inference server for that workload costs a fraction of one year's spend, plus modest running costs. Below roughly 1M tokens/day, APIs are usually cheaper once you count operations honestly — and pretending otherwise helps nobody.

What the per-token price hides

• Growth coupling: every successful AI rollout raises the bill; success is penalised.
• Pricing power: you carry the risk of vendor price changes and deprecations on a capability you now depend on.
• Compliance overhead: external processing means vendor assessments, data-protection agreements and audit questions that in-perimeter inference never raises.

What self-hosting advocates hide

• Operations are real: GPU drivers, serving stacks, monitoring and model updates need genuine engineering time — budget for it or buy it as a managed service.
• Utilisation risk: hardware sized for peak sits idle off-peak; sizing discipline matters.
• Frontier gaps: for a small set of hardest tasks, top closed models still lead. Pretending open models win everywhere is as dishonest as pretending they win nowhere.

The pattern most enterprises land on

Hybrid: bulk, predictable, sensitive workloads run on owned infrastructure at fixed cost; rare frontier-grade tasks route to an external API under policy control. This captures most of the savings and all of the data-control benefits while keeping an escape hatch for edge cases.

Run your own numbers in our break-even calculator — then pressure-test them against real concurrency and latency targets in a sizing review.