Local AI for a development team: three setups, sizing and rollout
12 min read
How to give a whole development team local AI for coding: a model on every laptop, one shared server, or a routed mix. Costs, failure modes and sizing numbers.
There are three ways to give a whole development team local AI for coding: put a model on every laptop, run one shared inference server, or route between the two with a rule that decides where each request runs. For more than a few developers, put the large model on a shared server, keep a small one on each laptop, and write the rule that chooses between them.
The table compares the three. The numbers under it are published measurements, read on 15 November 2026.
A model on every laptop | One shared inference server | A routed mix | |
|---|---|---|---|
Hardware cost | A machine with 32 GB or more for each developer, for the 27B to 30B coding models. Cost grows with headcount | One GPU server, and laptops of any size. As a reference, CloudRift rents an RTX 5090 for $0.65 an hour and an RTX PRO 6000 for $1.20 | The laptops you have, one server, and provider usage on your own account if a rule allows cloud |
Who maintains it | Every developer, unless a fleet tool pushes the model and settings | Whoever owns the server: runtime, model, access control and uptime | An admin who owns the routing policy, and the server owner |
Where code goes | Nowhere. Prompts stay on the laptop | To the server, over your network. It stays inside if the server is yours and its port is not exposed | Where the rule says: the laptop, the server, or an approved provider |
Failure behaviour | One laptop fails and one developer stops. Works offline once the model is installed | The server fails and everyone stops | The rule's fallback order applies. With no permitted destination the request fails with an error |
Largest model | Set by the weakest laptop | Set by the server's GPU memory | Small on laptops, large on the server |
Speed for one developer | 23.7 tokens a second for a 30B model on an Apple M4 Max | 74.9 tokens a second for the same model on an RTX 5090, shared with whoever else is asking | Depends on the destination chosen |
1. A model on every laptop
Each developer runs a model with a local runtime such as Ollama or LM Studio and points the coding tool at localhost. Setup for one machine is in the guides for Claude Code with Ollama and OpenCode with Ollama.
It suits a small team where everyone has a recent machine with 32 GB or more. It is the only setup that works on a train. Three things go wrong as the team grows:
Uneven hardware. The developer with 16 GB runs a 9B model while a colleague runs a 27B model, and they get different results on the same task. The best local LLMs for coding lists what fits each memory size.
No shared configuration. Model, context length and endpoint live in each person's shell profile. Ollama starts with a 4K context below 24 GiB of VRAM, and coding agents need 64K, so one missed setting breaks tool calls.
No record. Nobody can show which machine sent which request where.
2. One shared inference server
One machine with a large GPU serves the model, and every coding tool points at it. Laptops need no extra memory. The team gets one model and one place to update it.
Three points decide whether it works:
The runtime's concurrency. Ollama processes one request at a time for each model by default. Raising
OLLAMA_NUM_PARALLELmultiplies the context memory: four parallel requests at 64K need the memory of a 256K context. When the queue of 512 requests is full, Ollama returns a 503. The VMware test in the sizing section used vLLM, which batches concurrent requests; the trade-offs are in vLLM vs Ollama.Access to the port. Ollama binds to 127.0.0.1 by default, and you change that with
OLLAMA_HOST. Its local API accepts any key: the documentation marks the token as "required but ignored". Once the port is on the network, anyone who can reach it can use it, so put it behind your own access control and never on the public internet.The single point of failure. A driver update or a stuck process stops the whole team. Decide who is on call for it.
3. A routed mix
Each laptop runs a small model for quick edits and for work offline. The server runs the large model. A rule on each laptop decides which one a request uses, and what happens when the first choice is unavailable. If policy allows it, the last fallback is a cloud provider on the company's own account.
This is the setup that needs a routing layer in addition to a runtime. Without one, each developer switches base URLs by hand and the "rule" is a wiki page. The questions to settle are the same whichever tool you use:
Which work may never leave the laptop or the server.
Which destination is first, and which fallbacks are allowed, in order.
What the developer sees when nothing is allowed. It should be an error, with no quiet switch to another provider.
How you check where a request ran.
Sizing: how many developers one server can carry
No published figure tells you the number for your team. The measurements below give the range.
What was measured | Hardware | Model and runtime | Result | Source |
|---|---|---|---|---|
One stream | RTX 5090 | Muse Glimmer-30B, 4-bit, llama.cpp | 74.9 tokens a second; 233.4 with speculative decoding | Meta model card, August 2026 |
One stream | Apple M5 Max and M4 Max | Muse Glimmer-30B, 4-bit, ExecuTorch | 26.6 and 23.7 tokens a second; 50.2 and 37.8 with speculative decoding | Meta model card, August 2026 |
One stream, small model | RTX 4070, 12 GB | Gemma 4 E2B Q4_K_M on Aventail™ Link | 196 tokens a second (Ollama: 166) | Aventail™ benchmarks, June 2026 |
One stream, small model | Apple M2, 16 GB | Gemma 4 E2B Q4_K_M on Aventail™ Link | 39 tokens a second (Ollama: 41) | Aventail™ benchmarks, June 2026 |
Total throughput, many requests | One RTX 4090, RTX 5090, RTX PRO 6000 | Qwen3-Coder-30B AWQ on vLLM, 8K context | 2,259, 4,570 and 8,425 tokens a second | CloudRift GPU benchmarks |
Total throughput, many requests | Four RTX 5090 | Qwen3-Coder-30B AWQ on vLLM, 8K context | 12,744 tokens a second | CloudRift GPU benchmarks |
Concurrent users inside a latency target | Four H100 | gpt-oss-120b on vLLM, internal coding assistant | 300 users | VMware Cloud Foundation blog, 30 April 2026 |
Concurrent users inside a latency target | Two H200 | gpt-oss-120b on vLLM, internal coding assistant | 85 users | VMware Cloud Foundation blog, 30 April 2026 |
How to use these numbers:
Count concurrent sessions instead of headcount. VMware's figures are users active at the same moment, with pauses between turns. That works out at about 75 users per H100 and about 42 per H200 for a 120B model in their test.
Expect less than the batch benchmarks. CloudRift measured an 8K context. VMware's developers sent far more: 70% of users grew from 5,000 to 50,000 tokens, 20% from 15,000 to 120,000, and 10% started near 75,000. Long prompts were what limited the server, through attention compute and memory bandwidth, and GPU memory was not the limit.
Do not scale by GPU count. Four H100 carried about 3.5 times the users of two H200 in the same test.
Start from the single-stream speed. In Meta's test, one stream on an RTX 5090 ran about three times as fast as on an M4 Max laptop with the same 30B model. Batching on a server raises the total, and each person's share falls as sessions overlap.
Budget memory for parallel contexts. Weights are loaded once. Context is held for each parallel request.
Measure with your own prompts. VMware's authors say it directly: "We cannot claim to know the optimal number of users for your deployment".
A workable first step is one server with a GPU that holds your chosen model, a short pilot with a few developers, and a count of how often requests queue.
Rollout steps with Aventail™
Aventail™ covers the routed mix. Aventail™ Link is the inference engine and gateway that runs on each laptop and on the server. Aventail™ Control holds the fleet configuration and the routing rules; Link evaluates the rules on the device, and hosted Control receives device and routing events, not prompts or responses. Aventail™ Connect points Claude Code, OpenCode and Codex at Link. Fleet configuration and routing policies are part of Control for Enterprise.
These steps follow Aventail™'s deployment guide and documentation. Include a Linux and a Windows machine in your pilot, so every platform your team uses is covered.
Pick one workflow and its owner. Write down the data it handles and the destinations it may use.
Create a fleet key. In Control, open Fleet operations, choose Add devices, then Create a key. The key is shown once. You need the Fleet Admin or Super Admin role.
Enrol one test laptop. On macOS or Linux:
curl -fsSL https://get.aventail.co.uk/install.sh | AVENTAIL_FLEET_KEY='YOUR_FLEET_KEY' shOn Windows PowerShell:
$env:AVENTAIL_FLEET_KEY = 'YOUR_FLEET_KEY'try { & ([scriptblock]::Create((irm https://get.aventail.co.uk/install.ps1))) }finally { Remove-Item Env:AVENTAIL_FLEET_KEY }Confirm the device shows online in Devices. Packages exist for macOS on Apple silicon, Linux x64 and Windows x64.
Deploy and serve a coding model. Link runs GGUF language models. Start serving, then send a harmless prompt. A downloaded file is not a running endpoint.
Connect the coding tool. In Link, open Connected apps and choose Connect beside Claude, OpenCode or Codex. Claude Code needs at least 64K context, and Link warns below 128K.
aventail connect --listshows the state from a terminal.Add the shared server. Enrol it the same way, serve the larger model, and turn on Serve other devices. Review its listener, authentication, TLS and firewall rules before other machines reach it: the docs note that the tested endpoint used HTTP and that Control sign-in does not protect the serving port.
Write the routing policy. Tag the test devices and create a policy for the tag. A rule has a condition, a primary destination and up to three fallback hops, and the first matching enabled rule wins. Destinations are On device, a network peer, or a provider connection on your own account. A peer must offer the exact model named in the rule, and a peer can be a primary destination only, not a fallback hop. Peer and hosted routing are in beta.
Check delivery, then a real request. Wait for the device's Delivery state to show applied. Send a harmless request and read the destination in Recent usage. Then stop local serving on a test device and confirm the result is the fallback you configured, or an error.
Publish the configuration and roll out in rings. A fleet configuration revision lists the models and whether devices serve them. Apply it ring by ring, and check the applied revision and a response before you advance. Stopping a rollout does not reverse changes already applied, so record the previous configuration and who can restore it.
Each request runs on one destination. Link does not pool machines or split a request across them, so size the server on its own. Aventail™'s Free plan covers three connected devices. Starter is £50 a month excluding VAT with ten devices, and £5 a month for each extra device. The staged plan for laptops and servers together is in deploying a private AI workspace.
Which setup to choose
Your situation | Start with |
|---|---|
A small team, all on 32 GB or more | A model on every laptop |
Mixed laptops, code must stay on your network | One shared server |
Travel, offline work or a server that must not stop the team | A routed mix with a small model on each laptop |
Some work needs a frontier model | A routed mix with an approved provider as the last fallback |
Frequently asked questions
How can I give my whole team local AI for coding?
Run one shared inference server with a 27B to 30B coding model, point each developer's coding tool at it, and keep a small model on each laptop as the fallback. Use a routing rule so the fallback is automatic and recorded.
How many developers can one GPU serve?
It depends on the model, the context length and how many sessions overlap. VMware reported 300 concurrent users on four H100 and 85 on two H200 for a 120B model in April 2026. Pilot with your own prompts before you buy.
Do we need a GPU server, or are laptops enough?
Laptops are enough if every developer has 32 GB or more and you accept that each person maintains their own setup. A server is the better choice when laptops vary or when you want one model for everyone.
Can a team share one Ollama server?
Yes. Set OLLAMA_HOST so it listens on the network and raise OLLAMA_NUM_PARALLEL above its default of 1. Ollama's local API does not check keys, so restrict who can reach the port.
Does our code leave the building with a shared server?
Not if the server is on your network and the tools point only at it. Check the coding tool's own telemetry and web tools separately, and check any fallback to a cloud provider.
What happens when the shared server is down?
Without a fallback, everyone stops. With a routing rule, requests go to the next permitted destination in order, and fail with an error if none is permitted.
Which model should a team standardise on?
One that supports tool calling and fits the server with room for parallel contexts. The current options by memory size are in best local LLMs for coding.
Related guides
Sources
Meta, Muse Glimmer-30B model card (speed table): https://huggingface.co/meta-models/Muse-Glimmer-30B
CloudRift, GPU benchmarks for LLM inference: https://www.cloudrift.ai/gpu-benchmarks
VMware Cloud Foundation blog, How many users can your LLM server really handle? (30 April 2026): https://blogs.vmware.com/cloud-foundation/2026/04/30/how-many-users-can-your-llm-server-really-handle/
Aventail™, Benchmarks: Aventail™ Link vs Ollama (June 2026): https://aventail.co.uk/benchmarks
Ollama, FAQ (concurrent requests, queue, network binding): https://docs.ollama.com/faq
Ollama, Context length: https://docs.ollama.com/context-length
Ollama, Anthropic compatibility: https://docs.ollama.com/api/anthropic-compatibility
Ollama, Claude Code and OpenCode integrations: https://docs.ollama.com/integrations/claude-code and https://docs.ollama.com/integrations/opencode
Aventail™, Deploying a private AI workspace across laptops and servers: https://aventail.co.uk/guides/private-ai-workspace-deployment
Aventail™, Add devices: https://docs.aventail.co.uk/docs/fleet/overview
Aventail™, Fleet configuration: https://docs.aventail.co.uk/docs/fleet/config-and-rollouts
Aventail™, Routing: https://docs.aventail.co.uk/docs/fleet/routing
Aventail™, Connect overview: https://docs.aventail.co.uk/docs/connect/overview
Aventail™, Security and access: https://docs.aventail.co.uk/docs/guides/security
Aventail™, Plan a deployment: https://docs.aventail.co.uk/docs/guides/deployment
Aventail™, Compatibility and versions: https://docs.aventail.co.uk/docs/introduction/versions-and-supported-models
Aventail™, Pricing: https://aventail.co.uk/#pricing
All sources were read on 15 November 2026.