Run LLM Locally: the Complete Setup Guide for Laptops, Macs and Desktops

Key Takeaway

• 🧮 Memory is the only rule that matters: a model must fit in your VRAM (or your Mac’s unified memory); at the standard Q4_K_M quantization each billion parameters costs about 0.6 GB — an 8 GB card runs 7-8B class, a 16 GB card runs 20-26B mixture-of-experts models, a 24-32 GB card runs the 27-35B dense class, and a 48-64 GB machine runs a 70B.
• ⚙️ To run LLM locally, the runner decides the experience: Ollama for the one-command default, LM Studio for a window instead of a terminal, llama.cpp for the exact fit when a model is too big, and vLLM when you serve more than one person at once.
• 🚦 The speed cliff for any run-LLM-locally attempt is real and checkable: 100% GPU load is the target — the moment a model spills to system RAM you lose roughly tenfold; run ollama ps and fix by choosing a smaller quantization, a shorter context, or a smaller model — never by buying a faster CPU.
• 💵 The honest cost math: a GPU you already own runs words for free; a dedicated ~$500-2,500 card pays back against cloud bills only at sustained volume — the same math that prices token bills decides owning versus renting.
• 🔐 Local is not automatically safe: 175,000 Ollama servers sit exposed on the open internet because their operators skipped the one configuration step that matters — bind the server to localhost; our self-hosted LLM security guide carries the checklist.

Table of Contents

Run LLM locally and you change three things about how you use AI at once — the reasons to run LLM locally stack: the words you type never leave your machine, the bill stops moving, and the model stops changing under you. Those are the reasons the practice spread from a tinkering hobby to a mainstream working setup. This guide walks the whole path in the order that matters — memory first, runner second, model third, tuning fourth — with the numbers each step turns on and the failure modes the forum threads keep rediscovering.

The mechanism is simpler than the marketing: generating one token means streaming essentially every model weight out of memory once. Capacity decides whether a model runs at all; memory bandwidth decides how fast. That is why the two questions a newcomer asks — “which GPU?” and “which model?” — both reduce to one first question: how many gigabytes fit.

What run LLM locally means — and why it changed

To run LLM locally is to run inference on hardware you own: the model’s weights sit on your drive, each token is produced by your own CPU or GPU, and no prompt travels to a vendor. You are almost never training — inference needs orders of magnitude less memory than training, which is why a single consumer GPU or a well-equipped Mac runs systems that rival last year’s paid cloud assistants.

What changed in 2026 is the catalog. Quantized open-weight models ship at fixed size classes — roughly 1B, 3B, 8B, 14B, 27B, 35B-A3B-style mixture-of-experts, 70B — and the current generation at a given class beats the previous generation one class up: a recent 9B outmatches a 2024-era 70B on most everyday tasks. At the same time the runners matured: Ollama’s engine picks Apple’s MLX runtime automatically on Apple Silicon, the ollama launch command (January 2026) boots coding agents like Claude Code, OpenCode and Codex against local models with one command, and LM Studio ships a headless daemon for servers. The gap that used to need an engineer now closes inside a lunch break.

The privacy claim deserves its exact wording, because this is where the hype misleads: a locally run model sends nothing to any inference provider — your prompts and the generated text stay on device. What it does not guarantee is security from everything else: an unpatched runner, a misconfigured bind address, or a model downloaded from a hostile mirror can burn the same device. Local is a data-flow guarantee, not a security guarantee, and the two get confused constantly — our 9.9 CVSS command-execution story is the same class of confusion at enterprise scale.

The one number that decides: memory

Find your VRAM first: on Windows, Task Manager → Performance → GPU reads “Dedicated GPU memory”; on Linux, nvidia-smi; on an Apple Silicon Mac, About This Mac shows unified memory. Memory bandwidth is the second number: roughly, tokens per second ≈ bandwidth ÷ bytes read per token — the same model on a card with double the bandwidth finishes about twice as fast.

The quantization math — what every run LLM locally decision turns on — is the third piece, and it is what makes consumer hardware viable. Models ship as GGUF files at graded compression levels; the standard working point is Q4_K_M at roughly 0.56-0.6 GB per billion parameters:

On top of the weights sits the KV cache — the memory a context window consumes — which adds roughly 1 GB at an 8K context on an 8B model and scales near-linearly to tens of GB for long contexts on big models. The working rule: pick the largest model that fits at Q4_K_M with your context budgeted, because a larger model at Q4 beats a smaller model at Q8 within the same memory.

The memory bands map to what you already own: a 16-32 GB RAM CPU-only laptop runs the 1-8B class slowly but readably; an 8 GB card (RTX 4060 class, ~$299 new) holds 7-8B models with short context; 12 GB (RTX 3060 12 GB, the secondhand budget king) reaches 14B; a 16 GB card (RTX 5060 Ti 16 GB, 4070 Ti Super, 5070 Ti, 5080) fits 14B dense or the 20-26B mixture-of-experts tier — MoE architectures store many parameters but activate only a few billion per token, which is why a 35B-A3B model reads only ~3B’s worth of weights per token and generates three to five times faster than a dense 27B; 24-32 GB (RTX 3090/4090/5090; the 5090’s 32 GB is 2026’s consumer ceiling at ~$1,999-2,500) holds the 27-35B dense class; and a 48-64 GB unified-memory machine (Mac Studio-class, two-CPU-buffered workstations, secondhand dual-GPU rigs) runs a 70B at Q4.

Apple Silicon is its own column, and a real one. Unified memory means no PCIe bottleneck and no separate VRAM — a Mac with 64-128 GB of unified memory runs models no single consumer NVIDIA card can hold, trading raw bandwidth (an M4 Max streams ~546 GB/s against ~1.79 TB/s on an RTX 5090) for capacity. The practical result: fewer tokens per second on big models, but models three to four times the size — and on Gemma-class models with the MLX engine, measured output lands in the 69+ tokens/second range with speculative decoding, competitive with a mid-range RTX card for the 7-12B class.

One hardware warning buys a month of debugging: AMD support in Ollama requires ROCm v7 installed from AMD’s own driver page (the kernel-bundled version misses key features), and multi-GPU setups — automatically split in Ollama but 15-30% slower in tokens/second than one bigger card — are the classic budget trap. One larger card beats two smaller ones; the inter-GPU communication tax is real.

Pick the runner: Ollama, LM Studio, llama.cpp, vLLM

Five runners cover nearly every way to run LLM locally; they wrap the same two engines (llama.cpp everywhere, MLX additionally on Apple), and the difference is how many decisions each makes for you.

Ollama is the default pick, with two caveats worth knowing before they waste a night. First, its default context window is conservative — historically 4,096 tokens, with newer releases scaling the default by available VRAM — and every model you serve for coding or documents needs more; set OLLAMA_CONTEXT_LENGTH=8192 (or the app’s context slider) deliberately, because a bigger context costs real VRAM. Second, its library tags default to Q4_K_M unless you name the quantization — which is the standard anyway, but verify it matches what you benchmarked. The January 2026 ollama launch command is the quiet killer feature: it wires Claude Code, OpenCode or Codex to your local model in one line with no environment plumbing, which is exactly the bridge the agent-tools ledger tracks for local backends.

When the fit is tight, drop to llama.cpp — directly, from winget, brew or a source build: it exposes the two flags Ollama hides (how many layers go on the GPU: -ngl; and the KV-cache type: -ctk q8_0 -ctv q8_0, which halves cache memory for ~no quality loss) and can pull a quantized model straight from Hugging Face: llama-server -hf MODEL:Q4_K_M -c 8192 -ngl 99 serves an OpenAI-compatible endpoint with a built-in web UI. LM Studio hides the same dials behind checkboxes and marks which files fit your card — the friendlier path if a terminal is not your habitat; its 0.4 release added parallel requests, a stateful REST API with MCP tool support, and the headless daemon for boxes without displays. vLLM is the when-one-box-serves-a-team answer: continuous batching, concurrent streams, production-grade throughput on server-class cards — and our Ollama-on-Hostinger guide walks the managed-server variant of the same pattern for people who would rather rent the box.

The first model: five minutes, checked

Step 1 — install: download Ollama’s installer from ollama.com (Windows/macOS), or on Linux run the one-line script from the project’s homepage; the desktop app ships a chat window, model picker and context slider so the terminal is optional after day one.

Step 2 — pick and pull a model that fits your memory: with 8 GB VRAM, the safe first pull is an 8B-class instruct model (~4.9 GB at Q4_K_M); with 16-24 GB, start at the 14-27B class; on Apple with 32+ GB unified memory, the 27B class runs comfortably. The 2026 shortlist by class — Gemma 4 (E4B and 12B), Qwen3.5 (4B, 9B), Qwen3.6-27B for dense coding, Qwen3.8-27B, Llama 3.3 70B, DeepSeek-R1 8B, Phi-3 Mini, gpt-oss 20B (native MXFP4), Mistral Small 22B, Qwen2.5-Coder 7B/14B/32B — gets the full task-and-hardware treatment in our local LLM models field guide; the first pull only needs one honest entry from that table.

Step 3 — run and verify the GPU: when you run LLM locally at full residency, ollama pull MODEL, then ollama run MODEL, ask something, and while it is loaded run ollama ps — the PROCESSOR column must read “100% GPU”. A CPU/GPU split means the model or its context spilled: drop the context, drop the model size, or drop the quantization. This verification is the single habit that prevents the classic “local AI feels slow” report — the runner never warns you, it just quietly offloads to system RAM and your tokens per second collapse.

Step 4 — a working context budget: chat works at 4-8K; document summaries want 16K+; coding agents chew context aggressively and deserve 32K (the practical minimum for multi-file loops, per the same memory-band math as above). Set it once per model rather than per prompt, and keep Flash Attention plus the q8_0 KV cache on (both are single-line environment variables in Ollama: OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q8_0) — up to 40% of context VRAM reclaimed for under 0.1% quality cost.

Tuning: context, quantization, offload

The offload hierarchy decides any attempt to run LLM locally, in plain numbers: 100% GPU is the target — the experience feels like the cloud but slower-slope. ~90+/10 split is usable interactive. A 50/50 split is batch-job territory. A 100% CPU load, below 3B models, is patience-dependent. A 7B model at ~45 tokens/second on a 24 GB card drops to single digits the moment even 10% of its layers hit system RAM — the fix in any run LLM locally slowdown is always a smaller model or a shorter context; a faster CPU never rescues it.

Quantization choices, honestly ranked: Q4_K_M is the default and the floor that preserves most of the benchmark spread in blind use; Q5_K_M if the model fits with headroom; Q8_0 only when memory is abundant (it doubles Q4’s footprint for a small perceptual gain); sub-4-bit (Q3, IQ3, Q2) trades quality faster, especially on coding and math, and only makes sense to rescue a run from offload. The one-line rule a 2026 user needs: bigger model at Q4 beats smaller model at Q8 in the same memory.

Serving apps once you run LLM locally: the OpenAI-compatible endpoints

Every mainstream runner exposes an OpenAI-compatible endpoint — the hinge of any plan to run LLM locally into real software of the whole local workflow: most AI software — the editor extension, the document pipeline, your own scripts — switches to your local model by changing the base URL and model name, leaving everything else in place. Open WebUI gives the browser chat with history and document upload over Ollama; coding assistants point at the local server for private completion; the local-model variant of our agent-tools ledger tracks the local backend options; and on the agentic side, OpenClaw, the open-source self-hosted agent, runs its loop against the same endpoint so the agent and the model both live on your box.

What run LLM locally costs — and when cloud wins

When you run LLM locally, hardware is the only bill that does not scale with use. The tiers: $0 if you already own a 16 GB+ RAM laptop (1-8B class, slow but real); ~$300-500 for an 8 GB entry card (the RTX 4060 tier — productive chat, coding assistance, simple RAG); ~$1,000-2,000 for the 24-32 GB enthusiast tier (the 27-35B class, the “professional gold standard” for local work); ~$2,000-2,500 for a 32 GB RTX 5090. Then electricity, which on the PH grid’s typical residential rates runs in single-digit centavos per hour of use for a mid-range card at load — the honest entry in the ledger is your time for updates and troubleshooting, the hidden cost every honest guide names.

The break-even math decides it, and it is one division. A sustained daily cloud bill for heavy document work — say 2 million input and 200,000 output tokens a day on a $2/$10-per-million frontier model — is roughly $6/day; a $2,500 local rig then pays back in about 417 days, twice the volume in 208, half the volume in 833. Light users who touch the cloud a few times a week never hit the crossover: the pay-as-you-go model is cheaper for them on any horizon that matters. The complete decision table, with the rental alternative priced at the hourly rates our GPU rental guide tracks, is in the local vs cloud piece — the one-paragraph version: local for privacy at any volume and for steady heavy volume; cloud for peak quality and bursty use.

Security: local does not mean open

The exposed-Ollama incident is the lesson, not a warning about Ollama: 175,000 servers were reachable on the open internet with no password not because they were hacked, but because their operators installed a server-mode runner and left the default bind address on 0.0.0.0. The fix is one line — bind to 127.0.0.1 and never expose the API to the internet without an authenticated proxy — and the full seven-step hardening (auth, model provenance, updates, logging) lives in our self-hosted LLM security guide. Two other practices close the loop for work use: download models only from verified sources or official catalogs (model provenance matters as much as software provenance), and treat the local endpoint like any other internal service — visible to your apps, invisible to the internet. If the machine holds regulated data and must never phone home, the air-gapped deployment guide is the compliance-grade version of the same setup.

WorldNgayon Insight: own the trade-off, don’t dodge it

The local-AI debate keeps producing two dishonest shapes: the hobbyist piece that pretends everyone should self-host everything, and the vendor piece that pretends the cloud is the only serious option. Both dodge the structure of the actual decision, which has three independent axes — data (must it leave?), volume (is the bill moving?), and quality (is frontier-grade needed?). A locally run answer prices each axis: data-sovereignty wins that the cloud cannot match at any volume, cost-crossovers that arrive on their own schedule, and quality gaps that close class by class with each model generation — the useful truth is that the gap between “the best local model” and “the best hosted model” narrows roughly one size-class per generation, which is precisely why the 2026 catalog already covers most working traffic for most people. The right posture is portable: know your memory, know your volume, and switch per task — which is why this guide ends with a checklist instead of a verdict.

Run LLM locally — the checklist in one glance

• 🧮 Memory first, before you run LLM locally: find your VRAM (or unified memory) and size the model at ~0.6 GB per billion parameters at Q4_K_M, with your context budgeted on top.
• ⚙️ Runner second: Ollama to start; LM Studio for the GUI; llama.cpp when the fit is tight; vLLM when you serve a team.
• 🚦 Verify 100% GPU to run LLM locally at speed: ollama ps after every load — a split is the sign to shrink something (context, model, or quantization), not the signal to buy faster compute.
• 🎚️ Set context deliberately: 4-8K chat, 16K+ documents, 32K for agent loops — with Flash Attention and the q8_0 KV cache on.
• 🔐 Bind to localhost: the exposed-server incident only touched people who skipped this; the seven-step hardening takes an afternoon once.
• 💵 Check the crossover: hardware cost ÷ daily cloud bill = payback days — local is a privacy decision at any volume and a cost win at steady heavy volume, not automatically a money win for light use.

What this means for work, money and the Philippines

Budgets: the entry price of productive local AI collapsed from workstation money to secondhand-GPU money: a used 24 GB RTX 3090 (~$600-800 in the 2026 secondary market) reaches the 27-35B class — the reason the local route now competes with renting for steady users, per the same rental arithmetic our GPU guide prices. Careers: the skill “size a model, verify GPU residency, tune context” is the new baseline a freelancer or in-house hire shows up with; it is interview-checkable in five minutes and invisible on a résumé — build the demo, not the claim. For Filipino professionals and OFWs: the practical stack is the laptop-first path — an 8-16 GB machine runs the 7-9B class on CPU or an entry GPU, which is enough for private document work (contracts, payslips, remittance records, client data) that should not enter any vendor’s cloud, with the hard tasks escalated to the cloud model of your choice — keep both, use each for what it is good at, and document the boundary: which workloads are local-only is now a real, enforceable line.

Verdict: which setup is yours

If you already own an 8 GB+ gaming GPU: run LLM locally tonight — install Ollama,, pull the largest 2026 model that fits, verify 100% GPU residency, done. If you own a Mac with 32 GB+: same path, MLX engine, the 27B class. If your machine is a CPU-only laptop with 16 GB of RAM: the 1-8B class runs tonight, and the productive tier costs a one-card upgrade — buy the 16 GB band before anything faster, because capacity beats speed in this market. If your volume is light and your data is public: none of the above matters, use the cloud free tier and skip the hardware — the checklist says so, and the checklist is the verdict.

Financial Disclaimer

This article is editorial analysis of publicly reported specifications, prices and usage patterns, not investment or financial advice, and no outcome is promised. Hardware prices and cloud pricing change; verify with the named sources before any purchase. Technology purchases carry substantial risk.