The requirement is a document, not a feeling
Start by finding the sentence. "We can't send that to the cloud" is a conclusion someone reached, and the useful question is what they read to reach it. In practice the source is one of four things: a regulation that applies to your sector, a clause in a contract you signed with your own client, an internal policy written before current vendor offerings existed, or a security team's general rule of thumb. Each of those calls for a different response, and only one of them is genuinely immovable on your timeline.
Then read the actual language rather than the summary of it. Requirements almost always constrain specific properties: where data is processed, who may access it, whether it may be used to train a model, how long it is retained, which subprocessors are involved, how quickly a breach must be disclosed. Very few of them say the model weights must sit on hardware you own. That distinction decides the whole question. If the clause constrains retention, residency, training use, and access, a contract can satisfy it. If it constrains physical custody or network egress, meaning no customer data leaves your network under any terms, then no contract will help and you are looking at a real self-hosting project.
Check who can change the requirement, too. Client contracts get renegotiated at renewal. Internal policies get amended by the person who wrote them. If a single clause in one client agreement is driving a six-figure infrastructure decision for your whole business, the cheapest path may be a conversation with that client's security lead about what they were actually trying to prevent.
The middle option most people skip
Between the default API and your own servers sits a category of hosted enterprise and business tiers that exist specifically to answer this question. What they commonly offer, as contract terms rather than blog posts, includes commitments not to train on your content, defined or zero retention windows, processing in a chosen region, tenant isolation, audit logging, single sign-on and user provisioning, a data processing agreement, published compliance reports, and a disclosed subprocessor list. The specifics vary by vendor, by product, and by tier, and they change, so get the current terms in writing for the exact product and tier you would actually buy.
A large fraction of self-hosting mandates end here. If the concern is training on your data, retention, or a counterparty who wants a signed data processing agreement, an enterprise agreement addresses it directly and self-hosting adds cost without adding an answer. One caveat worth stating plainly to whoever has to approve the decision: contract terms are enforceable promises with remedies attached, not laws of physics. If your actual concern is that the provider could be compromised, or compelled by legal process, or that a foreign jurisdiction could reach the data, that is a different claim and self-hosting does change your exposure to it. Decide which claim you are making, because the two get blurred constantly and they lead to different budgets.
One more thing to verify before you sign: the protections you negotiated usually apply to a specific account and product, not to every place your team touches a model. The enterprise terms on your API account say nothing about the free consumer account someone uses on their phone, or the coding assistant installed in an IDE, or the meeting notetaker in your calendar. Coverage gaps like that are more common than architecture gaps, and they are much cheaper to close.
What self-hosting costs after the hardware
The GPU is the visible line item and it is rarely the largest one. Sizing is the first surprise: you provision for peak concurrency and for the longest context you plan to support, not for average load, so a workload that is busy for two hours a day still pays for capacity the rest of the day unless you can genuinely shut it down or share it with something else. Add a non-production environment for testing upgrades, and add redundancy if the service is something people wait on during business hours.
The recurring cost is people and process. Someone owns the inference server and its tuning, the quantization choices, capacity planning, and security patching of the serving stack. Someone is on call when generation stops at nine in the morning. Someone decides what to do each time a new open-weight release lands, and that decision requires an evaluation run, because swapping a model that works for a model that benchmarks better is how quietly broken systems happen. Hosted providers absorb all of this, which is most of what you are paying them for. You need an evaluation set either way, but self-hosting means you own every upgrade decision rather than only the ones you initiate.
Then there is the quality question, which is the one most likely to be argued at you with confidence in both directions. The honest position is that the gap between the strongest open-weight models and the strongest hosted models moves every time either side ships, so treat any blanket claim about it as stale, including this one. Test candidates on your own tasks with your own data. The gap tends to show up most in long-context work, multi-step reasoning, reliable tool use, and languages other than English. A model that performs somewhat below the frontier can be completely fine for classification, extraction, and routing, and not fine for drafting something a client will read. Where the gap costs you accuracy, it shows up on the budget as additional human review time, so put it there rather than treating it as a matter of taste.
Cost it at your volume, not at hypothetical scale
Cost comparisons circulate as if there is a universal answer, and they usually assume a busy GPU running near capacity around the clock. At that utilization, self-hosting often does win on unit cost. Most small and mid-sized business workloads look nothing like that: they are bursty, they follow the working day, and they go quiet on weekends. Your number is the only one that matters, and it is not hard to produce.
Build the hosted side from measurements. Sample real requests and record input and output tokens for each, count requests per business day, find your peak requests per minute, and separate interactive traffic from work that can run as an overnight batch. Multiply that against the published rates on the day you are deciding, and recheck before you sign anything, since those rates move. Build the self-hosted side from capacity sized for peak, amortized over how long you will keep the hardware or the reserved instance, plus power and hosting if it is on premises, plus a realistic fraction of an engineer's month. Then compute effective utilization: seconds of actual inference divided by seconds of capacity you paid for. If that ratio is low, per-token comparisons are not telling you anything true. Finally, re-run both sides at roughly three times today's volume. If a crossover point is anywhere near your growth path, you want to know that now rather than after the purchase order.
Latency deserves the same treatment. Self-hosting can reduce network round trips and remove the variance of sitting behind other tenants in a shared queue. It can also be slower, if your hardware is modest and two users hit it at once. Write the requirement down as a percentile and a number, tied to what a person is actually waiting for, then measure both options against it. A back-office summarization job that runs overnight has no latency requirement at all, and pretending otherwise has bought a lot of unnecessary hardware.
Hybrid: run the sensitive slice locally and nothing else
Very few systems have uniform data sensitivity. The obligation usually attaches to one category: patient records, a specific client's material, source code under a confidentiality clause, financial account details. Once you classify, you can route. The constrained subset goes to a local model, everything else goes to the hosted API. This keeps the self-hosted footprint small enough to run on modest hardware, keeps the quality-sensitive work on the stronger model, and turns an all-or-nothing architecture decision into a sizing exercise.
Several variations are worth knowing. You can keep embeddings and the vector index inside your environment while generation stays hosted, so the index sees the whole corpus but each prompt carries only a retrieved passage. You can redact or tokenize identifiers at the boundary, which sometimes removes the obligation entirely because what leaves is no longer regulated data. You can run a small local model as a first pass and escalate only what it cannot handle. And if exactly one client's contract is driving the whole requirement, you can stand up an isolated deployment for that client alone and price the engagement accordingly, rather than rebuilding your default stack around their terms.
Hybrid is not free, and the honest cost is complexity. You now maintain two model stacks, evaluate both, and debug two sets of failure modes. More importantly, the routing rule becomes a security control: if the classifier sends the wrong request to the hosted model, you have caused the exact incident the architecture was built to prevent. So treat the router like a control. Test it against edge cases before it sees real traffic, log every routing decision so you can show an auditor where any given request went, and make it fail closed by keeping anything uncertain local.
Get the requirement in writing before you choose an architecture. Most mandates that sound like they require self-hosting are satisfied by a hosted enterprise agreement, and most of the remainder are satisfied by running only the sensitive slice of the data locally.