Skip to content
Aventail™ investigation into UK government AI dependence featured in Computer Weekly.Read the story
Aventail™
All guides

Self-hosted AI: what it is, what it costs and how to size it

13 min read

Self-hosted AI explained: laptop, server or private cloud, the software layers, sizing numbers and a worked cost example against API prices, with dated sources.

Self-hosted AI means running an AI model on hardware your organisation controls, so prompts and documents are processed there and are not sent to a vendor's service. The same idea is sold as a self-hosted LLM, a private LLM, an on-premise LLM, on-prem AI and private AI for business. The terms differ mainly in where the hardware sits.

The options, with hardware and prices read on 15 November 2026:

Option

Example hardware

Model memory

Cost read today

Who runs it

Laptop or desktop staff already have

Apple silicon Mac, or a PC with a GPU, with 16 GB or more

Room for a 14 GB model file such as gpt-oss:20b

No new hardware

The user. IT manages the device

Desktop AI machine

Apple Mac Studio with M5 Ultra, 96 GB, configurable to 512 GB. NVIDIA DGX Spark, 128 GB

Room for a 65 GB model file such as gpt-oss:120b

Mac Studio 96 GB: £5,499 in the UK, $5,499 in the US. NVIDIA's DGX Spark page lists no price

IT, or one team

On-premises GPU server

One or more 80 GB data-centre GPUs, such as NVIDIA H100

80 GB per GPU

No public list price found for an H100 server. NVIDIA AI Enterprise, if you use it, lists at $4,500 per GPU a year

Your infrastructure team

Private cloud tenancy

A rented GPU instance in your own account, such as 1x H100 PCIe 80 GB at Lambda

80 GB per GPU

$3.79 per GPU hour, about $2,767 a month if left on

Your cloud team. The provider runs the building

Hosted API, for contrast

None

None

Per token: Claude Sonnet 5.5 and GPT-6.1 Sol are $2 input and $10 output per million tokens

The provider

Every figure is sourced at the end. Prices and model sizes change often, so check the review date before you quote them.

What "self-hosted" covers

Three places count.

A laptop or desktop. The model runs on the machine in front of the user. Nothing has to cross a network, and the work continues offline. The limit is memory: a 16 GB machine runs small models.

A server you own. A machine in your office or data centre runs a larger model and other machines send requests to it. This is what "on-premise LLM" and "on-prem AI" mean. The data stays in the building. You carry the purchase, power, cooling and patching.

A private cloud tenancy. You rent a GPU machine in your own cloud account and run the model on it. You choose the region and control the machine. The provider controls the building and the hardware. Many "private LLM" offers mean this.

A hosted model reached through a private endpoint, or under a "no training" contract, does not count. A third party still processes your request. It may meet your policy, but it is a different architecture. If the question is where data may legally be processed, read our guide to AI data residency. If it is national control of models and compute, read what sovereign AI is. If the environment has no external network at all, that is air-gapped AI.

The software layers

A self-hosted setup has four layers. An inference engine such as Ollama is one of them.

Layer

Job

Open-weight and open-source examples

Commercial examples

Model

The weights that produce the answer

gpt-oss (OpenAI, Apache 2.0), Gemma 4 (Google, Apache 2.0), Qwen3-Coder (Alibaba, Apache 2.0), DeepSeek-R1 (MIT), Llama 3.3 (Meta, Llama 3.3 Community Licence)

Closed models such as Claude and GPT-6 are sold through an API and cannot be self-hosted

Inference engine

Loads the model into memory and answers requests

llama.cpp (MIT), Ollama (MIT), vLLM (Apache 2.0)

LM Studio (free at home and at work since 8 July 2025, with an Enterprise plan for SSO and model gating). NVIDIA NIM (the Production Branch needs an NVIDIA AI Enterprise subscription). Aventail™ Link

Gateway

One endpoint in front of several models: keys, routing, fallback, logs

LiteLLM (MIT, with a separately licensed enterprise folder), Portkey Gateway (MIT), Kong (Apache 2.0)

Aventail™ Link with Aventail™ Control

Workspace

The chat and document screen staff use

Open WebUI (its own licence), LibreChat (MIT), AnythingLLM (MIT)

Aventail™ Workspace

Notes from the sources, all read on 15 November 2026:

  • GitHub stars: Ollama 187,000, Open WebUI 159,000, llama.cpp 135,000, vLLM 97,000, AnythingLLM 68,000, LiteLLM 63,000, LibreChat 47,000.

  • llama.cpp runs on CPUs and GPUs and can split a model between them. vLLM batches incoming requests continuously, which suits a shared server. Both offer an OpenAI-compatible API.

  • Hugging Face's Text Generation Inference repository is archived, so older guides that recommend it are out of date.

  • The Open WebUI Licence forbids removing its branding when a deployment has more than 50 users in a 30-day period, unless you hold written permission or an enterprise licence.

  • A gateway alone does not run a model. The difference is covered in LLM gateway vs inference engine.

Tool comparisons have their own pages: Ollama alternatives, vLLM vs Ollama and the best local LLMs for coding.

Sizing: how much hardware a model needs

Memory decides what you can run. Four numbers set it.

1. The model file. The file must fit in GPU memory, or in unified memory on a Mac, with room to spare. Sizes in the Ollama library today:

Model

File size

Context window

Memory guidance from the publisher

gpt-oss:20b

14 GB

128K

Runs on systems with as little as 16 GB

qwen3-coder:30b

19 GB

256K

Not stated

gemma4:31b

19 to 20 GB

256K

Not stated

llama3.3 (70B)

43 GB

128K

Not stated

gpt-oss:120b

65 GB

128K

Fits on a single 80 GB GPU

2. The context length. A longer context uses more memory on top of the file. Ollama sets its default from GPU memory: 4K tokens below 24 GiB, 32K from 24 to 48 GiB and 256K at 48 GiB or more. It advises at least 64,000 tokens for agents and coding tools. Aventail™'s docs say the same for Claude Code, and add that download size is not the memory requirement and that no single minimum-memory figure fits every model of a given size.

3. The number of people at once. Ollama serves one request per model at a time by default. Raise OLLAMA_NUM_PARALLEL and required memory scales by parallel requests times context length: a 2K context with four parallel requests allocates 8K. Requests beyond that wait in a queue of 512 by default.

4. Speed. Published speed figures for large models on specific machines are scarce. Aventail™'s own benchmark from June 2026 measured Gemma 4 E2B, a 2.9 GB file, at 196 tokens a second on an NVIDIA RTX 4070 with 12 GB and 39 tokens a second on an Apple M2 with 16 GB, one request at a time. Larger models are slower on the same hardware. Measure your model on your machine before you buy.

For reference, an NVIDIA RTX 5090 has 32 GB, and NVIDIA says a 128 GB DGX Spark runs inference on models of up to 200 billion parameters.

Costs against API pricing: a worked example

The example is a 50-person team using an assistant for drafting and document questions.

Inputs

Input

Value

Source

Requests

50 people x 40 a working day x 21 days = 42,000 a month

Assumption

Tokens per request

3,000 input, 500 output

Assumption

Monthly tokens

126 million input, 21 million output

42,000 x 3,000 and 42,000 x 500

Claude Sonnet 5.5, GPT-6.1 Sol

$2 input, $10 output per million

Sources 14 and 15

Claude Fable 5.2

$8 input, $40 output per million

Source 14

gpt-oss-120b on OpenRouter

$0.15 input, $0.60 output per million

Source 16

Mac Studio, M5 Ultra, 96 GB

$5,499, written off over 36 months

Source 17

Rented H100 PCIe 80 GB

$3.79 per GPU hour before tax

Source 20

Working hours

10 hours x 21 days = 210 a month

Assumption

Monthly cost

Route

Arithmetic

Per month

Hosted open-weight model (gpt-oss-120b)

126 x $0.15 + 21 x $0.60

$31.50

Mid-tier API (Sonnet 5.5 or GPT-6.1 Sol)

126 x $2 + 21 x $10

$462.00

Frontier API (Fable 5.2)

126 x $8 + 21 x $40

$1,848.00

Owned Mac Studio

$5,499 / 36

$152.75 plus power

Rented H100, working hours only

210 x $3.79

$795.90

Rented H100, left on

730 x $3.79

$2,766.70

Power for the owned machine: at its 480 W maximum for 210 hours the Mac Studio uses about 101 kWh. At Ofgem's household cap of 26.32p per kWh for October to December 2026, used here as a reference because business tariffs differ, that is about £27.

What the numbers say

  • Against a mid-tier API, one request costs $0.011. The owned machine breaks even at about 13,900 requests a month, which is 17 people at this usage. The rented GPU left on breaks even at about 251,500 requests, which is 300 people.

  • Against the same open-weight model bought through an API, self-hosting loses on price. One request costs $0.00075, so the owned machine needs about 203,700 requests a month to break even.

  • The owned machine costs the same busy or idle. The rented GPU bills every hour it is on. Utilisation is the assumption that moves the answer most.

  • The example leaves out staff time, a second machine for resilience, and whether one machine can carry the load. This team needs about 28 output tokens a second on average through the working day, more at peaks.

In this example self-hosting is not the cheapest way to get tokens. Organisations choose it for control over where data is processed, for work that must continue without a connection, and for a cost that does not rise with usage.

The risks

  • Quality. An open-weight model may perform differently from the hosted model staff already know. A successful chat does not show that tool use or long documents work. Test your own task.

  • Operations. You own patching, capacity, backups and model updates.

  • Open ports. Ollama's local API needs no authentication and binds to 127.0.0.1 by default. Changing OLLAMA_HOST to share it puts an open endpoint on the network. Aventail™'s docs give the same warning about Link: the tested endpoint used HTTP, and signing in to Control does not protect a serving port.

  • Licences. Open-weight does not mean unrestricted. Apache 2.0 and MIT models allow commercial use. The Llama 3.3 Community Licence adds an acceptable use policy and a separate licence above 700 million monthly users.

  • Supply chain. Model files and engines are downloaded software. Deploying AI Systems Securely, published on 15 April 2024 by the NSA with CISA, the UK NCSC and others, gives practices for externally developed AI systems.

  • Data still leaving. A self-hosted gateway can forward requests to an external API. The Ollama library lists -cloud tags that run on Ollama's servers. Check every destination and every fallback.

  • Prompt injection. A document can still carry instructions the model follows. Hosting the model yourself does not change that.

Where Aventail™ fits

Aventail™ covers the engine, gateway and workspace layers. Aventail™ Link is an inference engine and gateway that runs on each machine: macOS on Apple silicon, Linux x64 and Windows x64. It runs GGUF language models with a llama.cpp engine. Aventail™ Control holds routing rules, and Link evaluates them on the device. Aventail™ Workspace is the chat and document interface. Aventail™ Connect configures Claude Desktop, Claude Code, OpenCode and Codex to send their model requests through Link.

For a self-hosted deployment that gives two lanes:

  • Work that must stay inside runs on the user's device or on an approved server running Link.

  • Work that may use an outside model goes to an approved provider through your own account with that provider.

A rule names a destination and up to three fallback hops, and the first matching enabled rule wins. A request with no permitted destination fails with an error. Hosted Control receives management, device and routing events, and does not receive prompts or responses.

The docs state the limits. Each request runs on one destination, so machines are not pooled. A serving port needs its own access controls. Peer and hosted routing are in beta. Link is not open source as of 4 November 2026.

The Free plan covers three connected devices, including business use. Starter is £50 a month excluding VAT for ten. Routing policies need Control for Enterprise. Aventail™ holds SOC 2 Type II and Cyber Essentials.

The rollout steps are in our private AI workspace deployment guide. For sites with no internet access, see our air-gapped AI guide.

Frequently asked questions

What is the difference between self-hosted, private and on-premise AI?
Self-hosted means you run the model. On-premise means the hardware is in your building. Private usually means only your organisation can reach the deployment, on-premise or in your own cloud account.

Is self-hosted AI cheaper than using an API?
Sometimes. In the example above an owned machine beats a mid-tier API from about 17 users, and loses to the same open-weight model bought through an API. A rented GPU left on needs about 300 users to beat a mid-tier API.

What hardware do I need for a self-hosted LLM?
Enough memory for the model file plus the context. A 16 GB machine runs a 14 GB model such as gpt-oss:20b. A 65 GB model such as gpt-oss:120b fits on a single 80 GB GPU, according to its publisher.

Do I need a GPU?
No, but you will want one. llama.cpp runs on a CPU and can split a model between CPU and GPU. Models that spill out of GPU memory answer more slowly.

Can I self-host ChatGPT or Claude?
No. Anthropic and OpenAI sell their flagship models through an API. OpenAI does publish gpt-oss under Apache 2.0, and you can run that yourself. Connecting Claude Desktop or Claude Code to a local model changes the model behind the app. It does not install Claude's weights.

Is a self-hosted LLM secure?
It removes one risk: sending data to a third party. It adds the risks of running a service, such as open ports, unpatched software and unreviewed model files.

Can self-hosted AI run without internet?
Yes, once the model and software are installed. Whether the whole application works offline is a separate test. See air-gapped AI.

Sources

All read on 15 November 2026.

  1. Ollama library, gpt-oss: https://ollama.com/library/gpt-oss

  2. Ollama library, qwen3-coder, gemma4 and llama3.3: https://ollama.com/library/qwen3-coder, https://ollama.com/library/gemma4, https://ollama.com/library/llama3.3

  3. Ollama, Context length: https://docs.ollama.com/context-length

  4. Ollama, FAQ: https://docs.ollama.com/faq

  5. Ollama, Authentication: https://docs.ollama.com/api/authentication

  6. Hugging Face model cards: https://huggingface.co/google/gemma-4-31B-it, https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct, https://huggingface.co/deepseek-ai/DeepSeek-R1, https://huggingface.co/openai/gpt-oss-120b

  7. Meta, Llama 3.3 Community License Agreement: https://www.llama.com/llama3_3/license/

  8. GitHub repositories: https://github.com/ggml-org/llama.cpp, https://github.com/ollama/ollama, https://github.com/vllm-project/vllm, https://github.com/BerriAI/litellm, https://github.com/Portkey-AI/gateway, https://github.com/Kong/kong, https://github.com/LibreChat-AI/LibreChat, https://github.com/Mintplex-Labs/anything-llm, https://github.com/huggingface/text-generation-inference

  9. Open WebUI Licence: https://github.com/open-webui/open-webui/blob/main/LICENSE

  10. LM Studio, LM Studio is free for use at work (8 July 2025): https://lmstudio.ai/blog/free-for-work

  11. NVIDIA, NIM offerings: https://docs.nvidia.com/nim/large-language-models/latest/about-nim-llm/nim-offerings.html

  12. NVIDIA Enterprise Licensing Guide, Pricing (last updated 2 September 2026): https://docs.nvidia.com/ai-enterprise/planning-resource/licensing-guide/latest/pricing.html

  13. NVIDIA, DGX Spark and GeForce RTX 5090: https://www.nvidia.com/en-gb/products/workstations/dgx-spark/ and https://www.nvidia.com/en-gb/geforce/graphics-cards/50-series/rtx-5090/

  14. Anthropic, Pricing: https://platform.claude.com/docs/en/about-claude/pricing

  15. OpenAI, API pricing: https://developers.openai.com/api/docs/pricing

  16. OpenRouter, models API (openai/gpt-oss-120b): https://openrouter.ai/api/v1/models

  17. Apple, Mac Studio store pages, UK and US: https://www.apple.com/uk/shop/buy-mac/mac-studio and https://www.apple.com/shop/buy-mac/mac-studio

  18. Apple, Mac Studio technical specifications: https://www.apple.com/uk/mac-studio/specs/

  19. Ofgem, Energy price cap unit rates, 1 October to 31 December 2026: https://www.ofgem.gov.uk/information-consumers/energy-advice-households/get-energy-price-cap-standing-charges-and-unit-rates-region

  20. Lambda, GPU cloud pricing (prices take effect 16 November 2026): https://lambda.ai/pricing

  21. CISA, Joint Guidance on Deploying AI Systems Securely (15 April 2024): https://www.cisa.gov/news-events/alerts/2024/04/15/joint-guidance-deploying-ai-systems-securely

  22. Aventail™, Benchmarks: https://aventail.co.uk/benchmarks

  23. Aventail™ docs, Compatibility and versions: https://docs.aventail.co.uk/docs/introduction/versions-and-supported-models

  24. Aventail™ docs, Manage models and Glossary: https://docs.aventail.co.uk/docs/guides/models and https://docs.aventail.co.uk/docs/introduction/glossary

  25. Aventail™ docs, Routing: https://docs.aventail.co.uk/docs/fleet/routing

  26. Aventail™ docs, Security and access: https://docs.aventail.co.uk/docs/guides/security

  27. Aventail™ docs, Connect overview: https://docs.aventail.co.uk/docs/connect/overview

  28. Aventail™, Trust and Pricing: https://aventail.co.uk/trust and https://aventail.co.uk/#pricing

We use optional cookies and tracking to identify visiting companies, load our contact form, measure Google Ads download clicks and understand page use with PostHog on our homepage, Connect and How it works pages, Ollama and LM Studio comparison guides and campaign pages. Cookie policy