Local LLM Deployment on One Shared GPU Server for Small Teams
One GPU server can serve a whole small team, with the same model, the same settings, and documents that never leave your own hardware — as long as you plan for the moment everyone asks at once. Here is how to size it, share it fairly, and update it without breaking your team's prompts.
Why one shared server beats five laptops
Most small teams do not need five people each running a model on five laptops. One decent machine with a consumer GPU can serve the whole team through a private API. Everyone keeps the same model, the same settings, and the same results, and sensitive documents never leave your own hardware.
Sharing also makes costs honest. A single graphics card, its electricity, and one machine to maintain are usually cheaper than a cloud subscription for every seat. But sharing only works if you plan for the moment everyone asks for an answer at once. A setup that feels instant for one person can crawl when six people hit it together.
Do not share a server if your work is mostly one long, urgent job at a time, such as live transcription during a meeting. In that case, a dedicated machine, or a cloud API for the urgent job, is the calmer choice. Shared servers suit steady, mixed work: drafting, summarising, coding help, and document questions spread across the day.
Choose a sharing model before you buy anything
There are three practical ways to share one GPU. Pick one up front, because switching later means rebuilding how your team connects.
| Model | How it works | Best for |
|---|---|---|
| One model at a time | The server loads one model and unloads it when idle. Simple and predictable. | Teams that mostly use one model for one job. |
| Several small models side by side | Two or three small models stay loaded at once, each on part of the GPU memory. | Teams that need a fast small model and a stronger one for harder tasks. |
| Router with a queue | Requests wait in line and are sent to the right model in turn, with timeouts and priorities. | Teams of six or more, or mixed urgent and background work. |
Start with the simplest model that fits. Most teams of two to six people are happy with one model at a time, or two small models side by side. Add a queue only when people actually start waiting. If you later need a queue, put it in front of the same API instead of rebuilding the server: your team's tools keep calling one address, and only the plumbing behind it changes.
Serve the model through an OpenAI-compatible API, so your team's existing tools keep working without a custom integration for each one.
Size memory for concurrency, not for one user
The mistake almost everyone makes is sizing the GPU for a single chat. Model weights are only the first part of the bill. Each active conversation also needs working memory for its context, and that grows with longer documents and longer chats.
As a rough starting point, an 8-billion-parameter model in 4-bit quantisation needs around 5 GB for its weights, plus working memory on top. A 70-billion-parameter model in the same format needs roughly 40 GB for weights alone, which already rules out most consumer cards. Do not treat these as promises: measure with your own models, context lengths, and documents.
A simple sizing rule
- Pick the largest model you actually need, not the largest you have heard of.
- Add up the weights, then reserve at least a third of the GPU memory again for working memory and headroom.
- Test with three simultaneous long requests. If the server slows sharply, you are oversubscribed.
When memory runs out, the graceful answer is a clear queue message, not a crash. Set a maximum context length and a maximum number of concurrent requests, and return a short "busy, try again in a moment" response instead of letting the server thrash. For help choosing the model and quantisation for your hardware, read Local LLM Model Comparison for Budget Hardware.
Set up the server in an afternoon
Keep the setup boring. One machine, one serving tool, one API address, and a short runbook your teammates can actually follow. Ollama is the easiest start for most small teams; vLLM earns its extra complexity when you need higher throughput on a stronger card. Both can sit behind the same kind of API, so you can start simple and move later without retraining your team.
Put the server on your private network or behind a VPN, never directly on the public internet without authentication. Give each person their own API key, even if you all trust each other. Keys are not about trust; they are about knowing who made the request that filled the queue, and being able to turn off one key without changing everyone else's setup. Log which key, which model, and how long each request took, but do not log the full text of sensitive documents unless your policy explicitly allows it.
Defaults worth setting on day one
- Idle unload: unload a model after a few idle minutes, so the next job starts from a clean slate. In Ollama this is the
keep_alivesetting. - Request timeouts: cap how long one request can hold the GPU, so a runaway job cannot freeze the team.
- Context cap: set a default context length that fits your card, and raise it only for jobs that prove they need it.
Write down the exact model name and version you deployed, and the command that starts the server. Six months from now, "it worked in November" is not a runbook.
Fair-use rules that keep the peace
Shared hardware creates shared manners. Agree the rules before the first slow afternoon, while everyone is still calm. Three rules cover most teams.
First, separate interactive work from batch work. Drafting, coding help, and questions while someone waits should go first. Long jobs — summarising a hundred documents, or re-running a whole folder — should run outside working hours. If your serving tool cannot prioritise, use a shared rule instead: batch jobs run after 6 p.m.
Second, cap the big jobs. A single request that asks for a very long output, or pastes a very long document, can hold the GPU for minutes. Set a maximum output length and a maximum input size for interactive use, and give batch jobs their own limits.
Third, review usage once a week for the first month. Look at peak times, the slowest requests, and who is waiting. Most "we need a bigger GPU" moments turn out to be one batch job running at the wrong hour. For a broader cost view, see Local LLM Deployment: Cost Control for Small Teams.
Update models without breaking everyone's prompts
Models change, and a new version that is better on paper can still break the prompts your team relies on. Never let the server silently switch to a new model. Pin the exact model name and version, and treat an update as a small release, not a background download.
- Keep the old model installed until the new one has proved itself. Disk is cheap; a broken Monday is not.
- Run a fixed test set: ten to twenty real prompts from your team's daily work, with the answers you already accept. Compare the new model side by side.
- Let one or two people try it first for a few days, on real work, before the whole team moves.
- Announce the switch with the date, the model name, and how to go back. If anything regresses, roll back to the pinned version in minutes.
Store your test prompts and the accepted answers with the runbook, not in someone's chat history. When the next update arrives, you will rerun the same set instead of arguing from memory about whether the new model is "worse." If a prompt needs rewriting for the new model, rewrite the shared prompt template once, so everyone benefits.
Related reading
Limits and notes
One shared server is a team tool, not a product. It will not match a cloud provider's uptime, it needs someone to restart it and apply updates, and it becomes a single point of failure if the whole team depends on it. Keep a fallback for the day the server is down: a cloud API account, or a smaller model on a laptop.
Start small, measure real usage, and grow only when the queue proves you must. The goal is not the biggest model you can fit on a card. It is a server your team can rely on every working day.