Local LLM

Local LLM API Server Setup for Multi-Model Development Workflows

A practical walkthrough of running multiple local LLM API servers on a single Mac — combining llama.cpp, Ollama, and custom endpoints to build a multi-model development environment that costs zero API fees.

Free Last tested: 2026-07-04 Audience: Developers, AI Prototypers

Why run local LLM API servers?

Cloud-based LLM APIs are convenient, but they add up fast. A team of three developers testing prompt variations against GPT-4 or Claude can burn through $200–$600 per month before writing a line of production code. For prototyping, evaluation, and iteration cycles, running local models via a standardized API interface gives you unlimited calls for the one-time cost of hardware.

The key insight: modern open-weight models (Llama 3, Qwen 2.5, DeepSeek, Mistral) are good enough for structured tasks like code generation, data extraction, JSON output parsing, and classification. And they all support OpenAI-compatible API endpoints — meaning you swap one line in your code to switch between a local model and a cloud API.

Architecture: one Mac, three API servers

We set up three local LLM backends on a Mac Studio (M2 Ultra, 64 GB unified memory). Each backend exposes an OpenAI-compatible endpoint on a different port:

Server A — llama.cpp server (port 8080): Best for speed. Runs 4-bit quantized models (Q4_K_M) natively optimized for Apple Silicon via Metal. Single-model server, low overhead, best latency.

Server B — Ollama (port 11434): Best for model switching. Supports 60+ models via a single binary. Manage memory by switching models on the fly. Slower per-request than llama.cpp, but more flexible for testing different families.

Server C — Custom Python server (port 8000): Best for structured output. Wraps a smaller model with custom pre/post-processing logic, JSON schema validation, and retry logic. Useful for production-like testing without cloud costs.

Step 1: Install and configure llama.cpp

llama.cpp is the gold standard for local LLM inference on Apple Silicon. It uses Metal (GPU acceleration) and is constantly updated. The API server mode is built in — no need for third-party wrappers.

# Install via Homebrew brew install llama.cpp # Or build from source for latest Metal optimizations: git clone https://github.com/ggerganov/llama.cpp cd llama.cpp && make clean && LLAMA_METAL=1 make -j8 # Download a model in GGUF format # — Llama-3.2-3B-Instruct (Q4_K_M): ~2 GB # — Qwen2.5-7B-Instruct (Q4_K_M): ~4.5 GB # — DeepSeek-Coder-V2-Lite (Q4_K_M): ~3.5 GB # Start API server (Metal + 6 CPU threads + 4096 context) ./llama-server \ --model /path/to/model.gguf \ --port 8080 \ --host 0.0.0.0 \ --n-gpu-layers -1 \ --ctx-size 4096 \ --threads 6

The --n-gpu-layers -1 flag offloads all layers to the GPU. On M2 Ultra, a 7B model runs at ~40–50 tokens/second. Use --mlock to prevent swapping if you have enough RAM for the model.

Step 2: Set up Ollama for rapid model switching

Ollama manages model downloads, quantization, and serving behind a single CLI. Its REST API is compatible with the OpenAI chat completions format — you can swap your client from https://api.openai.com/v1 to http://localhost:11434/v1 and existing code just works.

# Install brew install ollama # Start the server ollama serve & # (runs on port 11434 by default) # Pull models (automatically quantized) ollama pull llama3.2:3b # ~2 GB ollama pull qwen2.5:7b # ~4.5 GB ollama pull mistral:7b # ~4 GB ollama pull deepseek-coder:6.7b # ~3.8 GB # Test your endpoint curl http://localhost:11434/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "llama3.2:3b", "messages": [{"role": "user", "content": "Say hello in JSON format."}], "temperature": 0 }'

Ollama's killer feature: you can install 10 models and switch between them in your app by just changing the model field in the request. Perfect for eval runs where you test the same prompt across 5 models.

Step 3: Build a unified router for multi-model workflows

The real power comes from routing requests to different models based on task type. A simple Python router can dispatch:

import requests, json MODELS = { "code": "http://localhost:8080/v1/chat/completions", "reasoning": "http://localhost:11434/v1/chat/completions", "extract": "http://localhost:8000/v1/chat/completions", } def query(task: str, prompt: str, model_key: str = "code"): endpoint = MODELS.get(model_key, MODELS["code"]) payload = { "model": "local", # Most local servers ignore model name "messages": [{"role": "user", "content": prompt}], "temperature": 0.1 if model_key == "extract" else 0.3, } resp = requests.post(endpoint, json=payload) return resp.json()["choices"][0]["message"]["content"] # Usage: dispatch by task type code_result = query("code", "Write a Python function to sort a dict by value") reasoning_result = query("reasoning", "Explain the trade-offs of microservices") structured_result = query("extract", "Extract dates and amounts from: ...")

This pattern means you use the cheapest/fastest model for simple tasks (3B param models), a more capable model for reasoning (7B), and a custom pipeline for structured output — all without paying per-token fees.

Cost comparison: local vs cloud over a month

We tracked usage over 30 days with two developers running daily prompt iterations:

Cloud-only approach (GPT-4.1 + Claude Sonnet 4):
~45,000 API calls, average 2,000 tokens per call = 90M tokens
GPT-4.1: $2.0/M input + $8/M output ≈ $480/month
Claude Sonnet 4: $3.0/M input + $15/M output ≈ $720/month
Total: $1,200/month (for two devs)

Local-only approach (M2 Ultra, Qwen2.5-7B + Llama3.2-3B):
One-time: $0 (already owned hardware)
Electric cost: ~$30/month (machine runs 8 hours/day)
Total: $30/month

That's a 40× cost reduction. The trade-off: local models score ~15–20% lower on some reasoning benchmarks, but for prototyping, data extraction, code generation, and classification, the quality gap is often negligible — and the speed of unlimited iteration compensates for the quality difference.

Limits and notes

Memory is your cap. A 7B Q4_K_M model uses ~4.5 GB VRAM + ~1 GB overhead. On a 64 GB system you can run two 7B models simultaneously (one on each server) and still keep a 3B model in reserve. Beyond that, models start swapping to system RAM and speed drops 5–10×. The fix: use Ollama's ollama stop <model> between sessions to free memory.

Mac Studio's unified memory handles this better than discrete GPU setups because there's no PCIe transfer bottleneck — but the hard limit is still the 64/96/128 GB pool. For teams pushing 32K+ context windows, consider a dedicated Linux box with 2× RTX 4090s (~$5,000 one-time).

OpenAI-Compatible API Endpoints for Local LLMs → Running RAG on Consumer Hardware → Local LLM Model Comparison on Budget Hardware → Browse all articles → Run Local LLMs as OpenAI-Compatible API Endpoints →