Moving from cloud AI to local models eliminates one well-defined set of risks — and introduces a different one that most teams discover only after their first incident.
Why Organisations Go Local
The case for running models inside your own network has become unusually compelling over the last eighteen months. Data stays on premises, which removes an entire category of regulatory and contractual headaches around third-party processing. There is no dependency on external API providers whose pricing, availability, or model behaviour can change without warning. Sensitive prompts and responses never traverse a public network, which closes off a meaningful exposure for organisations that handle medical records, financial transactions, classified or commercially sensitive material, or personal data subject to strict locality rules. For any regulated industry — healthcare, finance, defence, or any sector bound by strict data-handling and security obligations — local deployment is often not a preference but a precondition for being allowed to use modern AI at all.
The temptation, however, is to read this list of advantages as proof that local deployment is automatically more secure than the cloud alternative. It is not. What local deployment does is move the security responsibility from a vendor’s operations team to your own. Whether the resulting posture is stronger or weaker depends entirely on whether your team is prepared to take that responsibility seriously, with the same tooling, cadence and discipline that the cloud provider applied to the same surface.
The New Risk Surface
When you migrate from a cloud API to a self-hosted inference stack, you trade one set of risks for another. The risks you eliminate are real and worth eliminating: data transmission to third parties, dependency on a vendor’s privacy policy and contractual posture, the perennial problem of API keys being leaked into source repositories, and operational dependence on a provider’s rate limits, regional availability and incident timeline. These are not trivial gains.
But the risks you take on instead are also real, and most of them are silent in the way that classical cybersecurity is silent: nothing visibly breaks until something has been exfiltrated, modified or exploited. The first of these is the question of model weights at rest. A model file sitting unencrypted on a server is, in commercial terms, a compressed copy of whatever institutional knowledge went into producing or fine-tuning it; in security terms, it is a file with the same blast radius as your most sensitive database export. Without encryption at rest and strict access controls on the directory that holds the weights, a compromised server is a compromised model, and a model that has been quietly modified by an attacker is, by definition, a poisoned model — with all the downstream consequences described in the literature on supply-chain attacks.
The second is the absence of the input and output filtering layers that cloud providers apply by default. When you call OpenAI, Anthropic or Google, your request passes through provider-side moderation, prompt-injection heuristics and policy enforcement before it ever reaches the model, and the response passes back through similar filters before it reaches you. Self-hosted stacks have none of this unless you build it. A naive deployment of llama.cpp, Ollama or vLLM exposes the model directly to whatever input the application layer happens to forward, which means that every prompt-injection technique published in the past two years applies to your deployment until you explicitly defend against it.
The third is the audit-trail problem. Cloud providers log API calls by default, retain those logs for at least the contractual period, and make them available for incident investigation. Local deployments have no audit trail unless you build one — and when an incident occurs, the absence of records makes it nearly impossible to determine what happened, which inputs produced which outputs, who was affected, and whether the issue is bounded. The time to invest in input and output logging is before the incident, not afterwards.
The fourth is exposed inference endpoints. The popular local inference servers all expose HTTP APIs, and a network misconfiguration is enough to make those APIs reachable from places you did not intend. Production inference servers are regularly found reachable from the public internet — sometimes because a firewall rule was written too permissively, sometimes because the server was placed on the wrong network segment, sometimes simply because nobody verified that the listening interface was bound only to localhost. The cost of finding this in an assessment is recoverable; the cost of an attacker finding it first is not.
The fifth is the operational tax of staying current. Inference stacks are evolving rapidly: llama.cpp, vLLM, the various CUDA driver layers, the container runtimes that hold them, and the model formats themselves all see meaningful security and stability patches on a near-monthly cadence. Without a patching discipline that treats these components like the production infrastructure they have become, an organisation accumulates known vulnerabilities — exactly the failure mode that classical infrastructure security spent two decades learning to avoid.
What Secure Local Deployment Looks Like
Building a defensible local deployment is not difficult, but it requires deliberate work at three points in the lifecycle. Before deployment, the team needs an honest threat model that maps the attack paths specific to this system: who can reach the API in normal operation, who could reach it under plausible misconfiguration, what data passes through, and what the consequences would be if the model were compromised, exfiltrated or replaced. The same pre-deployment phase is the right time to verify model checksums and to architect network segmentation so that the inference API is not reachable from the same network segment as internet-facing services.
At deployment, the deceptively simple rule is that no endpoint should be reachable without authentication, even inside what is nominally a trusted network. API keys are the minimum bar; mutual TLS or short-lived OIDC tokens are appropriate for high-sensitivity deployments. Every request and response should be logged in a store that the inference application itself cannot modify, because a logging system controlled by the application it is supposed to audit is not really a logging system. Resource limits — token caps per request, rate limits per caller, request timeouts — protect against both abusive behaviour and the genuine operational risk of a runaway inference job consuming an unbounded amount of GPU time.
On an ongoing basis, dependency updates need to be treated with the same seriousness as patches to any other production component. Subscribing to CVE feeds for the specific components in use is one cheap way to make this concrete. Reassessment of the deployment against the current threat landscape should happen whenever the underlying model is updated, whenever new threat intelligence emerges in the public literature, and at minimum annually whether or not anything visible has changed.
The OWASP LLM Top 10 as a Baseline
The OWASP LLM Top 10, in its current 2025 edition, gives a usable baseline checklist for any LLM-based system, and it is particularly useful for local deployments because it forces consideration of the items that cloud providers were silently handling on your behalf. Prompt injection (LLM01) and insecure output handling (LLM02) are the two that most directly map to the loss of provider-side filtering. Sensitive information disclosure (LLM06) addresses the data-exfiltration scenarios that emerge once you fine-tune a model on internal corpora or feed it sensitive context. Vector and embedding weaknesses (LLM08) become directly relevant once you deploy retrieval-augmented generation against your own document store. Misinformation (LLM09) is the category that addresses the consequences of hallucinated or confidently wrong outputs in operational settings. Running a deployment through each of these items, with documented findings and remediation, is a reasonable baseline before going to production.
The Bottom Line
Local deployment is the right choice for many organisations, and for some it is the only choice that allows AI to be used at all. But it is not a security shortcut. It is a transfer of responsibility — the security work that your cloud provider was performing is now yours to perform, with your tools, your people and your operational cadence. The organisations that get this right treat their inference stacks like any other production infrastructure: with threat modelling before deployment, with security reviews against an explicit checklist, with operational hygiene around patching and monitoring, and with the assumption that something will eventually go wrong and the audit trail will need to answer real questions.
Contact us to discuss your local AI deployment
This article is for informational purposes only and does not constitute legal advice.