What 8x Cheaper Inference Actually Means for Your Business

Yesterday I posted about TurboQuant, a Google Research paper that compresses the main memory bottleneck in large language models by over 6x. The paper itself was published last year, but Google posted it on X this week and the open-source community has moved fast — implementations already work on Llama and Mistral out of the box.

The questions I keep seeing aren’t about the technique. They’re about what it means for businesses actually running AI workloads. I want to try to work through that properly, because I think the implications are bigger than the technical community is letting on.

Your AI vendor just got dramatically more efficient. What happens next?

Start with the obvious: if inference costs drop by anything close to the magnitude TurboQuant suggests, the companies selling you AI services just got a lot more profitable. Briefly. Because that’s not where this ends.

Competition does what competition always does. Open-source models get the same efficiency gains for free — TurboQuant works on Llama and Mistral out of the box. Rival providers match the cost improvements or lose customers. Within twelve to eighteen months, the savings get competed away and prices come down across the board. We watched exactly this happen with cloud computing over the past decade, except the efficiency gains in AI inference are arriving faster and the open-source alternative is more credible than anything the cloud market had to deal with.

What this means practically: the per-unit cost you’re paying for AI today is a poor guide to what you’ll be paying in eighteen months. That’s not a reason to wait — the productivity gains are real right now. But it is a reason to think carefully about contract structure and vendor flexibility, because the pricing floor hasn’t been found yet.

The build-vs-buy maths changed — but not in the way I first described

An important correction from my post yesterday: I said a model that needed 48GB of GPU memory could now run in 8GB. That was wrong, and a commenter rightly called it out. TurboQuant compresses the KV cache — the memory that grows as a conversation gets longer — not the model weights themselves. A 7B parameter model still needs roughly the same VRAM to load whether or not you’re using TurboQuant.

What TurboQuant actually changes is how much headroom you have once the model is loaded. The KV cache is what balloons as context length increases — it’s why long conversations slow down, why providers charge more for longer contexts, and why running models with 100k+ token windows has been prohibitively expensive on anything less than top-tier data centre hardware. Compressing that cache by 6x means the same GPU can handle dramatically longer contexts, serve more concurrent users, or both. That’s a real and significant cost reduction for inference workloads, even if it’s not the “shrink the whole model” story I initially told.

The build-vs-buy shift is still real, but the mechanism is different from what I described. It’s not that models suddenly run on tiny hardware. It’s that the ongoing cost of running inference — especially for the long-context, high-throughput workloads that enterprises actually care about — drops substantially. Combined with the separate trend of model weight quantisation (which tools like llama.cpp have been doing for a while, compressing models from 16-bit to 4-bit), the total picture is that self-hosted inference is getting more viable on more modest hardware. TurboQuant is one piece of that puzzle, not the whole thing.

The cost comparison still holds directionally. A mid-size company spending $48,000 a year on AI API access has a credible path to self-hosting on hardware that costs a fraction of that — but the economics depend on the specific workload, the model size, and how much the weight quantisation and cache compression compound in their particular use case. It’s not the simple “buy a $1,600 GPU and you’re done” story I told yesterday. The reality is more nuanced, and I should have been more careful with the numbers.

The usual pushback is that frontier models from OpenAI, Anthropic, and Google are still meaningfully better than open-source alternatives. That’s true, and I don’t want to hand-wave it away. But it’s less true than it was a year ago, and for the tasks that make up the bulk of enterprise AI usage — document processing, summarisation, classification, internal search, first-draft generation — open-source models crossed the “good enough” line a while back. Compression doesn’t make them smarter. It makes them cheap enough to run yourself.

Here’s what I think most of the commentary misses

The shift from API to self-hosted AI isn’t primarily a cost story. It’s a risk story, and this is where my background in insurance makes me see something that I think the tech commentary largely overlooks.

When you consume AI through an API, your provider carries a meaningful chunk of the operational risk. They manage uptime, model quality, safety filtering, and they process your data on their infrastructure under their terms. You have a vendor relationship, and with it comes the familiar apparatus of third-party risk management — due diligence, SLAs, contractual protections. Procurement teams know how to handle this.

When you self-host, that entire risk profile inverts. You might save money, but you’re now responsible for everything: model behaviour, data security, output quality, maintenance. If the model hallucinates something problematic in a customer-facing context, there’s no vendor to call. If the outputs drift over time, that’s your problem to detect and fix.

I work in insurance, and I can see both sides of this vividly. An insurer running AI models internally for claims triage or underwriting support gets genuine advantages — sensitive customer data never leaves their infrastructure, which simplifies data protection compliance enormously. But they also absorb model risk directly. If an internally-hosted model produces a biased underwriting recommendation, the insurer owns that outcome completely. There’s no vendor indemnification, no shared liability. The risk hasn’t disappeared — it’s moved from a vendor management problem to an operational risk problem, and most organisations aren’t set up to manage it yet.

This applies well beyond insurance. Any regulated industry — banking, healthcare, legal — faces the same calculus. And frankly, even unregulated companies should be thinking about it, because “we run AI models and we’re accountable for what they produce” is a statement that has legal and reputational weight whether or not there’s a regulator looking over your shoulder.

Sovereignty matters more than cost

I keep coming back to this: the strongest argument for self-hosting isn’t the money. It’s control.

Vendor lock-in in AI is a real and growing problem. Companies build workflows around a specific model’s API, and then discover that switching costs are substantial. Models get deprecated, sometimes without much warning. Terms of service change. Pricing structures shift in ways that don’t favour the customer. Your provider’s strategic priorities don’t necessarily align with yours, and once you’re dependent on their infrastructure, your leverage is limited.

Self-hosting with open-source models doesn’t make you fully independent — you still depend on the model’s development community, on hardware vendors, on the ecosystem. But it distributes that dependency instead of concentrating it in a single provider. You can swap models if something better comes along. You can fine-tune for your specific use case in ways that no general API allows. You can freeze a version that works and not be forced into an upgrade that breaks your workflow.

For industries where continuity and predictability are the baseline expectation — which, honestly, is most industries — that kind of control has tangible value that goes well beyond the cost saving.

Where this leaves me

I don’t think this is a simple story with a clean conclusion. The efficiency gains are real and accelerating. The build-vs-buy calculus has genuinely shifted. But the risk implications of that shift are underappreciated, and I think there’s a meaningful gap between “the economics now favour self-hosting” and “your organisation is actually ready to self-host responsibly.”

The companies that navigate this well will be the ones that treat it as a risk decision, not just a procurement decision. That means thinking about model governance, output accountability, and operational resilience before the cost savings make the move feel inevitable.

There’s a bigger question underneath all of this that I keep turning over: if inference becomes commodity — and I think it will — where does durable value actually accumulate in the AI stack? I have some thoughts forming on that, but it needs more work before I’m confident enough to put a stake in the ground.