The fastest way to deploy an LLM locally: install Ollama, then run one command — ollama run qwen3. It downloads a quantized model, drops you into a chat, and quietly starts an OpenAI-compatible API at http://localhost:11434/v1 that your code can call. Total time: about five minutes on a decent connection.

That is the whole answer for most people. The rest of this piece is about picking the right model for your hardware, the alternatives worth knowing (llama.cpp, LM Studio, vLLM), how to expose a local model as an API, and when you should not bother running locally at all.

The answer in one screen#

# 1. Install Ollama (macOS/Linux one-liner; Windows has an installer at ollama.com)
curl -fsSL https://ollama.com/install.sh | sh

# 2. Download and run a model — this drops you into an interactive chat
ollama run qwen3

# 3. That's it. Ollama is now also serving an API at http://localhost:11434
#    Test it from another terminal:
curl http://localhost:11434/api/generate -d '{
  "model": "qwen3",
  "prompt": "Say hi in five words."
}'

If qwen3 is too big for your machine, swap in a smaller tag like ollama run qwen3:4b or ollama run gemma3:4b. Everything below is detail on top of these three steps.

1. The fastest path: Ollama#

Ollama has become the default way to run local models for a reason: it hides everything annoying. It manages downloads, picks sensible quantization, uses your GPU if you have one (NVIDIA, AMD, or Apple Silicon) and falls back to CPU if you don't, and serves a local HTTP API without any configuration. As of mid-2026 its model library lists well over a hundred models.

The commands you actually need:

ollama run llama3.3          # download (if needed) and chat
ollama pull gemma3:27b       # download without starting a chat
ollama list                  # see what you've downloaded
ollama ps                    # see what's currently loaded in memory
ollama rm gemma3:27b         # delete a model to reclaim disk

Inside a chat, /bye exits and /? shows session commands. Models are pulled as quantized GGUF files, so a "7B" model is a few gigabytes on disk, not the ~14 GB the full-precision weights would be.

One 2026 note: Ollama Cloud now lets you run larger models on datacenter hardware using the exact same commands and API, which is a clean escape hatch when a model is too big for your laptop but you don't want to change your code. Local stays local; you opt into cloud per model.

2. Choosing a model for your hardware#

The single question that decides what you can run is: how much VRAM (or unified memory) do you have? Everything else is secondary.

The rule of thumb for the common Q4_K_M quantization is roughly 0.6 GB per billion parameters, plus some headroom for context. That gives you a quick mental table:

Your VRAMComfortable model size (Q4_K_M)Good picks in mid-2026
8 GB~7-8Bqwen3:8b, llama3.1:8b, gemma3:4b
12 GB~8-14Bqwen3:14b, gemma3:12b
16 GB~14-20Bgpt-oss:20b, qwen3:14b
24 GB (e.g. RTX 4090)~27-32Bgemma3:27b, qwen3:32b
48 GB~70Bllama3.3:70b, qwen3:32b (roomy)
80 GB (datacenter)~120B MoEgpt-oss:120b

A few things worth knowing about the models themselves:

On quantization: don't overthink it. Q4_K_M is the sweet spot — roughly a quarter of the memory of full precision for a small, usually unnoticeable quality drop. Go to Q5 or Q6 if you have spare VRAM and want a little more fidelity; drop to Q3 only if it's the difference between fitting and not. Ollama picks a reasonable default for you, so this mostly matters when you hand-pick GGUF files.

If you're choosing hardware rather than a model, our cheapest 16 GB VRAM cards for local AI breakdown is the companion read, and on Apple Silicon specifically, MLX vs llama.cpp covers which engine to run on a Mac.

3. The alternatives worth knowing#

Ollama is the easy default, but three other tools each win a specific job. All four, importantly, speak the OpenAI API.

llama.cpp — maximum control. This is the C++ inference engine that Ollama and LM Studio are both built on top of. Going straight to it gives you custom compile flags, the newest features first, and the leanest CPU/edge performance. Its llama-server binary launches an OpenAI-compatible API (default port 8080):

# Build with GPU support, or grab a prebuilt release from the repo
llama-server -hf ggml-org/Qwen3-8B-GGUF --port 8080
# Now http://localhost:8080/v1 is an OpenAI-compatible endpoint

LM Studio — the GUI. If you'd rather not touch a terminal, LM Studio is a desktop app with a model browser, one-click Hugging Face downloads, a chat window for testing, and a local server that exposes an OpenAI-compatible API on port 1234. It runs on macOS, Windows, and Linux. Same underlying GGUF models as Ollama — just a friendlier front door.

vLLM — serving and throughput. When you're past "run it on my laptop" and into "serve this to real users," vLLM is the answer. It's built for high concurrency and throughput (via PagedAttention and continuous batching) rather than single-user desktop use, and it's GPU-first:

pip install vllm
vllm serve Qwen/Qwen3-8B
# OpenAI-compatible server at http://localhost:8000/v1

The short version of the trade-off, laid out well in Red Hat's llama.cpp vs vLLM comparison: llama.cpp for portability and one user, vLLM for a server handling many.

4. Expose it as an OpenAI-compatible API#

This is the part that makes local deployment genuinely useful: you almost never have to change your application code. Every runtime above serves the same /v1/chat/completions shape the OpenAI SDK expects. Point the SDK at your local URL and pass any non-empty string as the key.

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1",  # Ollama; 8080 llama.cpp, 1234 LM Studio, 8000 vLLM
    api_key="ollama",                        # required by the SDK, but ignored locally
)

resp = client.chat.completions.create(
    model="qwen3",
    messages=[{"role": "user", "content": "Explain quantization in one sentence."}],
)
print(resp.choices[0].message.content)

The same thing in plain curl:

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3",
    "messages": [{"role": "user", "content": "One sentence on why local LLMs matter."}]
  }'

Because the interface is standard, switching from a cloud API to your local model — or between local runtimes — is a base-URL change, not a rewrite. That portability is worth designing for even if you use cloud today.

5. When NOT to run locally#

Local isn't always the right call. Be honest with yourself:

Local deployment wins clearly on three things: privacy (data never leaves your machine), offline availability, and marginal cost (near-zero per token once the hardware exists). If any of those is the point, run locally. If raw capability or elastic scale is the point, don't force it.

The bottom line#

Deploying an LLM locally in 2026 is a five-minute task, not a project. Install Ollama, run ollama run qwen3, and match the model tag to your VRAM. Reach for llama.cpp when you need control, LM Studio when you want a GUI, and vLLM when you need to serve. And because every one of them speaks the OpenAI API, the model on your desk plugs into the code you've already written.