Self-hosted AI: what it is, what it costs and how to size it
13 min read
Self-hosted AI explained: laptop, server or private cloud, the software layers, sizing numbers and a worked cost example against API prices, with dated sources.
Self-hosted AI means running an AI model on hardware your organisation controls, so prompts and documents are processed there and are not sent to a vendor's service. The same idea is sold as a self-hosted LLM, a private LLM, an on-premise LLM, on-prem AI and private AI for business. The terms differ mainly in where the hardware sits.
The options, with hardware and prices read on 15 November 2026:
Option | Example hardware | Model memory | Cost read today | Who runs it |
|---|---|---|---|---|
Laptop or desktop staff already have | Apple silicon Mac, or a PC with a GPU, with 16 GB or more | Room for a 14 GB model file such as gpt-oss:20b | No new hardware | The user. IT manages the device |
Desktop AI machine | Apple Mac Studio with M5 Ultra, 96 GB, configurable to 512 GB. NVIDIA DGX Spark, 128 GB | Room for a 65 GB model file such as gpt-oss:120b | Mac Studio 96 GB: £5,499 in the UK, $5,499 in the US. NVIDIA's DGX Spark page lists no price | IT, or one team |
On-premises GPU server | One or more 80 GB data-centre GPUs, such as NVIDIA H100 | 80 GB per GPU | No public list price found for an H100 server. NVIDIA AI Enterprise, if you use it, lists at $4,500 per GPU a year | Your infrastructure team |
Private cloud tenancy | A rented GPU instance in your own account, such as 1x H100 PCIe 80 GB at Lambda | 80 GB per GPU | $3.79 per GPU hour, about $2,767 a month if left on | Your cloud team. The provider runs the building |
Hosted API, for contrast | None | None | Per token: Claude Sonnet 5.5 and GPT-6.1 Sol are $2 input and $10 output per million tokens | The provider |
Every figure is sourced at the end. Prices and model sizes change often, so check the review date before you quote them.
What "self-hosted" covers
Three places count.
A laptop or desktop. The model runs on the machine in front of the user. Nothing has to cross a network, and the work continues offline. The limit is memory: a 16 GB machine runs small models.
A server you own. A machine in your office or data centre runs a larger model and other machines send requests to it. This is what "on-premise LLM" and "on-prem AI" mean. The data stays in the building. You carry the purchase, power, cooling and patching.
A private cloud tenancy. You rent a GPU machine in your own cloud account and run the model on it. You choose the region and control the machine. The provider controls the building and the hardware. Many "private LLM" offers mean this.
A hosted model reached through a private endpoint, or under a "no training" contract, does not count. A third party still processes your request. It may meet your policy, but it is a different architecture. If the question is where data may legally be processed, read our guide to AI data residency. If it is national control of models and compute, read what sovereign AI is. If the environment has no external network at all, that is air-gapped AI.
The software layers
A self-hosted setup has four layers. An inference engine such as Ollama is one of them.
Layer | Job | Open-weight and open-source examples | Commercial examples |
|---|---|---|---|
Model | The weights that produce the answer | gpt-oss (OpenAI, Apache 2.0), Gemma 4 (Google, Apache 2.0), Qwen3-Coder (Alibaba, Apache 2.0), DeepSeek-R1 (MIT), Llama 3.3 (Meta, Llama 3.3 Community Licence) | Closed models such as Claude and GPT-6 are sold through an API and cannot be self-hosted |
Inference engine | Loads the model into memory and answers requests | llama.cpp (MIT), Ollama (MIT), vLLM (Apache 2.0) | LM Studio (free at home and at work since 8 July 2025, with an Enterprise plan for SSO and model gating). NVIDIA NIM (the Production Branch needs an NVIDIA AI Enterprise subscription). Aventail™ Link |
Gateway | One endpoint in front of several models: keys, routing, fallback, logs | LiteLLM (MIT, with a separately licensed enterprise folder), Portkey Gateway (MIT), Kong (Apache 2.0) | Aventail™ Link with Aventail™ Control |
Workspace | The chat and document screen staff use | Open WebUI (its own licence), LibreChat (MIT), AnythingLLM (MIT) | Aventail™ Workspace |
Notes from the sources, all read on 15 November 2026:
GitHub stars: Ollama 187,000, Open WebUI 159,000, llama.cpp 135,000, vLLM 97,000, AnythingLLM 68,000, LiteLLM 63,000, LibreChat 47,000.
llama.cpp runs on CPUs and GPUs and can split a model between them. vLLM batches incoming requests continuously, which suits a shared server. Both offer an OpenAI-compatible API.
Hugging Face's Text Generation Inference repository is archived, so older guides that recommend it are out of date.
The Open WebUI Licence forbids removing its branding when a deployment has more than 50 users in a 30-day period, unless you hold written permission or an enterprise licence.
A gateway alone does not run a model. The difference is covered in LLM gateway vs inference engine.
Tool comparisons have their own pages: Ollama alternatives, vLLM vs Ollama and the best local LLMs for coding.
Sizing: how much hardware a model needs
Memory decides what you can run. Four numbers set it.
1. The model file. The file must fit in GPU memory, or in unified memory on a Mac, with room to spare. Sizes in the Ollama library today:
Model | File size | Context window | Memory guidance from the publisher |
|---|---|---|---|
gpt-oss:20b | 14 GB | 128K | Runs on systems with as little as 16 GB |
qwen3-coder:30b | 19 GB | 256K | Not stated |
gemma4:31b | 19 to 20 GB | 256K | Not stated |
llama3.3 (70B) | 43 GB | 128K | Not stated |
gpt-oss:120b | 65 GB | 128K | Fits on a single 80 GB GPU |
2. The context length. A longer context uses more memory on top of the file. Ollama sets its default from GPU memory: 4K tokens below 24 GiB, 32K from 24 to 48 GiB and 256K at 48 GiB or more. It advises at least 64,000 tokens for agents and coding tools. Aventail™'s docs say the same for Claude Code, and add that download size is not the memory requirement and that no single minimum-memory figure fits every model of a given size.
3. The number of people at once. Ollama serves one request per model at a time by default. Raise OLLAMA_NUM_PARALLEL and required memory scales by parallel requests times context length: a 2K context with four parallel requests allocates 8K. Requests beyond that wait in a queue of 512 by default.
4. Speed. Published speed figures for large models on specific machines are scarce. Aventail™'s own benchmark from June 2026 measured Gemma 4 E2B, a 2.9 GB file, at 196 tokens a second on an NVIDIA RTX 4070 with 12 GB and 39 tokens a second on an Apple M2 with 16 GB, one request at a time. Larger models are slower on the same hardware. Measure your model on your machine before you buy.
For reference, an NVIDIA RTX 5090 has 32 GB, and NVIDIA says a 128 GB DGX Spark runs inference on models of up to 200 billion parameters.
Costs against API pricing: a worked example
The example is a 50-person team using an assistant for drafting and document questions.
Inputs
Input | Value | Source |
|---|---|---|
Requests | 50 people x 40 a working day x 21 days = 42,000 a month | Assumption |
Tokens per request | 3,000 input, 500 output | Assumption |
Monthly tokens | 126 million input, 21 million output | 42,000 x 3,000 and 42,000 x 500 |
Claude Sonnet 5.5, GPT-6.1 Sol | $2 input, $10 output per million | Sources 14 and 15 |
Claude Fable 5.2 | $8 input, $40 output per million | Source 14 |
gpt-oss-120b on OpenRouter | $0.15 input, $0.60 output per million | Source 16 |
Mac Studio, M5 Ultra, 96 GB | $5,499, written off over 36 months | Source 17 |
Rented H100 PCIe 80 GB | $3.79 per GPU hour before tax | Source 20 |
Working hours | 10 hours x 21 days = 210 a month | Assumption |
Monthly cost
Route | Arithmetic | Per month |
|---|---|---|
Hosted open-weight model (gpt-oss-120b) | 126 x $0.15 + 21 x $0.60 | $31.50 |
Mid-tier API (Sonnet 5.5 or GPT-6.1 Sol) | 126 x $2 + 21 x $10 | $462.00 |
Frontier API (Fable 5.2) | 126 x $8 + 21 x $40 | $1,848.00 |
Owned Mac Studio | $5,499 / 36 | $152.75 plus power |
Rented H100, working hours only | 210 x $3.79 | $795.90 |
Rented H100, left on | 730 x $3.79 | $2,766.70 |
Power for the owned machine: at its 480 W maximum for 210 hours the Mac Studio uses about 101 kWh. At Ofgem's household cap of 26.32p per kWh for October to December 2026, used here as a reference because business tariffs differ, that is about £27.
What the numbers say
Against a mid-tier API, one request costs $0.011. The owned machine breaks even at about 13,900 requests a month, which is 17 people at this usage. The rented GPU left on breaks even at about 251,500 requests, which is 300 people.
Against the same open-weight model bought through an API, self-hosting loses on price. One request costs $0.00075, so the owned machine needs about 203,700 requests a month to break even.
The owned machine costs the same busy or idle. The rented GPU bills every hour it is on. Utilisation is the assumption that moves the answer most.
The example leaves out staff time, a second machine for resilience, and whether one machine can carry the load. This team needs about 28 output tokens a second on average through the working day, more at peaks.
In this example self-hosting is not the cheapest way to get tokens. Organisations choose it for control over where data is processed, for work that must continue without a connection, and for a cost that does not rise with usage.
The risks
Quality. An open-weight model may perform differently from the hosted model staff already know. A successful chat does not show that tool use or long documents work. Test your own task.
Operations. You own patching, capacity, backups and model updates.
Open ports. Ollama's local API needs no authentication and binds to 127.0.0.1 by default. Changing
OLLAMA_HOSTto share it puts an open endpoint on the network. Aventail™'s docs give the same warning about Link: the tested endpoint used HTTP, and signing in to Control does not protect a serving port.Licences. Open-weight does not mean unrestricted. Apache 2.0 and MIT models allow commercial use. The Llama 3.3 Community Licence adds an acceptable use policy and a separate licence above 700 million monthly users.
Supply chain. Model files and engines are downloaded software. Deploying AI Systems Securely, published on 15 April 2024 by the NSA with CISA, the UK NCSC and others, gives practices for externally developed AI systems.
Data still leaving. A self-hosted gateway can forward requests to an external API. The Ollama library lists
-cloudtags that run on Ollama's servers. Check every destination and every fallback.Prompt injection. A document can still carry instructions the model follows. Hosting the model yourself does not change that.
Where Aventail™ fits
Aventail™ covers the engine, gateway and workspace layers. Aventail™ Link is an inference engine and gateway that runs on each machine: macOS on Apple silicon, Linux x64 and Windows x64. It runs GGUF language models with a llama.cpp engine. Aventail™ Control holds routing rules, and Link evaluates them on the device. Aventail™ Workspace is the chat and document interface. Aventail™ Connect configures Claude Desktop, Claude Code, OpenCode and Codex to send their model requests through Link.
For a self-hosted deployment that gives two lanes:
Work that must stay inside runs on the user's device or on an approved server running Link.
Work that may use an outside model goes to an approved provider through your own account with that provider.
A rule names a destination and up to three fallback hops, and the first matching enabled rule wins. A request with no permitted destination fails with an error. Hosted Control receives management, device and routing events, and does not receive prompts or responses.
The docs state the limits. Each request runs on one destination, so machines are not pooled. A serving port needs its own access controls. Peer and hosted routing are in beta. Link is not open source as of 4 November 2026.
The Free plan covers three connected devices, including business use. Starter is £50 a month excluding VAT for ten. Routing policies need Control for Enterprise. Aventail™ holds SOC 2 Type II and Cyber Essentials.
The rollout steps are in our private AI workspace deployment guide. For sites with no internet access, see our air-gapped AI guide.
Frequently asked questions
What is the difference between self-hosted, private and on-premise AI?
Self-hosted means you run the model. On-premise means the hardware is in your building. Private usually means only your organisation can reach the deployment, on-premise or in your own cloud account.
Is self-hosted AI cheaper than using an API?
Sometimes. In the example above an owned machine beats a mid-tier API from about 17 users, and loses to the same open-weight model bought through an API. A rented GPU left on needs about 300 users to beat a mid-tier API.
What hardware do I need for a self-hosted LLM?
Enough memory for the model file plus the context. A 16 GB machine runs a 14 GB model such as gpt-oss:20b. A 65 GB model such as gpt-oss:120b fits on a single 80 GB GPU, according to its publisher.
Do I need a GPU?
No, but you will want one. llama.cpp runs on a CPU and can split a model between CPU and GPU. Models that spill out of GPU memory answer more slowly.
Can I self-host ChatGPT or Claude?
No. Anthropic and OpenAI sell their flagship models through an API. OpenAI does publish gpt-oss under Apache 2.0, and you can run that yourself. Connecting Claude Desktop or Claude Code to a local model changes the model behind the app. It does not install Claude's weights.
Is a self-hosted LLM secure?
It removes one risk: sending data to a third party. It adds the risks of running a service, such as open ports, unpatched software and unreviewed model files.
Can self-hosted AI run without internet?
Yes, once the model and software are installed. Whether the whole application works offline is a separate test. See air-gapped AI.
Related guides
Sources
All read on 15 November 2026.
Ollama library, gpt-oss: https://ollama.com/library/gpt-oss
Ollama library, qwen3-coder, gemma4 and llama3.3: https://ollama.com/library/qwen3-coder, https://ollama.com/library/gemma4, https://ollama.com/library/llama3.3
Ollama, Context length: https://docs.ollama.com/context-length
Ollama, FAQ: https://docs.ollama.com/faq
Ollama, Authentication: https://docs.ollama.com/api/authentication
Hugging Face model cards: https://huggingface.co/google/gemma-4-31B-it, https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct, https://huggingface.co/deepseek-ai/DeepSeek-R1, https://huggingface.co/openai/gpt-oss-120b
Meta, Llama 3.3 Community License Agreement: https://www.llama.com/llama3_3/license/
GitHub repositories: https://github.com/ggml-org/llama.cpp, https://github.com/ollama/ollama, https://github.com/vllm-project/vllm, https://github.com/BerriAI/litellm, https://github.com/Portkey-AI/gateway, https://github.com/Kong/kong, https://github.com/LibreChat-AI/LibreChat, https://github.com/Mintplex-Labs/anything-llm, https://github.com/huggingface/text-generation-inference
Open WebUI Licence: https://github.com/open-webui/open-webui/blob/main/LICENSE
LM Studio, LM Studio is free for use at work (8 July 2025): https://lmstudio.ai/blog/free-for-work
NVIDIA, NIM offerings: https://docs.nvidia.com/nim/large-language-models/latest/about-nim-llm/nim-offerings.html
NVIDIA Enterprise Licensing Guide, Pricing (last updated 2 September 2026): https://docs.nvidia.com/ai-enterprise/planning-resource/licensing-guide/latest/pricing.html
NVIDIA, DGX Spark and GeForce RTX 5090: https://www.nvidia.com/en-gb/products/workstations/dgx-spark/ and https://www.nvidia.com/en-gb/geforce/graphics-cards/50-series/rtx-5090/
Anthropic, Pricing: https://platform.claude.com/docs/en/about-claude/pricing
OpenAI, API pricing: https://developers.openai.com/api/docs/pricing
OpenRouter, models API (openai/gpt-oss-120b): https://openrouter.ai/api/v1/models
Apple, Mac Studio store pages, UK and US: https://www.apple.com/uk/shop/buy-mac/mac-studio and https://www.apple.com/shop/buy-mac/mac-studio
Apple, Mac Studio technical specifications: https://www.apple.com/uk/mac-studio/specs/
Ofgem, Energy price cap unit rates, 1 October to 31 December 2026: https://www.ofgem.gov.uk/information-consumers/energy-advice-households/get-energy-price-cap-standing-charges-and-unit-rates-region
Lambda, GPU cloud pricing (prices take effect 16 November 2026): https://lambda.ai/pricing
CISA, Joint Guidance on Deploying AI Systems Securely (15 April 2024): https://www.cisa.gov/news-events/alerts/2024/04/15/joint-guidance-deploying-ai-systems-securely
Aventail™, Benchmarks: https://aventail.co.uk/benchmarks
Aventail™ docs, Compatibility and versions: https://docs.aventail.co.uk/docs/introduction/versions-and-supported-models
Aventail™ docs, Manage models and Glossary: https://docs.aventail.co.uk/docs/guides/models and https://docs.aventail.co.uk/docs/introduction/glossary
Aventail™ docs, Routing: https://docs.aventail.co.uk/docs/fleet/routing
Aventail™ docs, Security and access: https://docs.aventail.co.uk/docs/guides/security
Aventail™ docs, Connect overview: https://docs.aventail.co.uk/docs/connect/overview
Aventail™, Trust and Pricing: https://aventail.co.uk/trust and https://aventail.co.uk/#pricing