September 25, 202611 min read

How much memory do you need to run local LLMs in 2026? (And why Mac minis keep selling out)

Weights, KV cache and OS overhead, worked out from bits per weight. A size table for 3B to 70B models and why bandwidth sets tokens per second.

ByLokesh Kumar· Founder & Builder, Vyra
Share on XShare on LinkedIn

The short answer

Budget memory as weights plus KV cache plus headroom. Weights are parameters times bits per weight divided by 8: roughly 0.6 GB per billion parameters at a 4-bit quant, about 1.06 GB at 8-bit, 2 GB at FP16. An 8B model at 4-bit fits comfortably in 16 GB. A 70B model at 4-bit needs around 48 GB before long contexts.

Why is everyone suddenly asking this?

Because the hardware ran out. On 30 August, MacRumors reported, citing The Information, that Apple's unusually early Mac mini and Mac Studio refresh was "driven by unexpectedly strong enterprise appetite for AI hardware," and that a global memory shortage plus AI demand has left many configurations out of stock for months. MacRumors attributes this to The Information, not to an Apple statement, so treat it as reported.

Kevin Lewis's writeup of his local model setup on an M4 Pro Mac mini and Federico Viticci's M5 Ultra Mac Studio review followed on the Hacker News front page.

The first question for any local model is whether it fits; the second is how fast the machine can stream it. Both are memory questions. If you are new to running models offline, our guide to running AI locally covers the tooling; this post is the arithmetic.

How do you calculate the memory a model's weights need?

Weights dominate. The formula is:

weight bytes ≈ parameters × bits per weight ÷ 8

The trap is that "4-bit" is not 4.0 bits. Quantized formats store scales and other metadata alongside the weights, so the effective figure is higher. The llama.cpp quantize README publishes measured bits per weight for Llama-3.1-8B in each format:

For a Q4_K_M "4-bit" file, 4.89 is a more honest multiplier than 4. Hugging Face's Llama 3.1 launch post gives the idealized version of the same arithmetic: 8B needs 16 GB at FP16, 8 GB at FP8 and 4 GB at INT4 "just to load the model checkpoint," and 70B needs 140, 70 and 35 GB.

Applying the llama.cpp multipliers to common size classes (decimal GB, nominal parameter counts, weights only):

Model sizeQ4_K_M (4.89 bpw)Q8_0 (8.50 bpw)FP16 (16 bpw)
3B3 × 4.89 ÷ 8 ≈ 1.8 GB≈ 3.2 GB6 GB
8B8 × 4.89 ÷ 8 ≈ 4.9 GB≈ 8.5 GB16 GB
14B≈ 8.6 GB≈ 14.9 GB28 GB
32B≈ 19.6 GB≈ 34 GB64 GB
70B≈ 42.8 GB≈ 74.4 GB140 GB

Sanity check: 4.9 GB for 8B at Q4_K_M matches llama.cpp's measured 4.58 GiB (4.58 GiB is about 4.92 GB). Real models drift a little because "8B" is 8.03B and some layers, such as embeddings, are often kept at higher precision.

Mixture-of-experts models change one thing. Memory is set by total parameters, since every expert must be resident. Lewis's Qwen3.6-35B-A3B at 4-bit uses "about 20GB of RAM" on his 48 GB Mac mini, which is what the table predicts for a ~35B model, even though only about 3B parameters are active per token.

How much does context length add?

This is the part people forget. The KV cache stores keys and values for every token in the context so they are not recomputed each step. Per token, it costs:

2 (K and V) × layers × KV heads × head dimension × bytes per element

For Llama 3.1 8B, the Llama 3 paper (Table 3) lists 32 layers, 8 key/value heads and a model dimension of 4,096 across 32 attention heads, so a head dimension of 128. At FP16 (2 bytes): 2 × 32 × 8 × 128 × 2 = 131,072 bytes, or 128 KiB per token. The 70B has 80 layers with the same 8 KV heads, so 320 KiB per token.

Multiply out and you get the figures in the Hugging Face post:

Model1k tokens16k tokens128k tokens
Llama 3.1 8B0.125 GB1.95 GB15.62 GB
Llama 3.1 70B0.313 GB4.88 GB39.06 GB

At full 128k context, the 8B model's KV cache is three times larger than its 4-bit weights. That is why a model that "fits in 8 GB" can fall over the moment you paste in a codebase.

Two mitigations exist. Most runtimes default to a modest window; the Ollama FAQ says it uses 4,096 tokens by default unless you set OLLAMA_CONTEXT_LENGTH. And the cache itself can be quantized: Ollama documents that q8_0 "uses approximately 1/2 the memory of f16" and q4_0 about a quarter, when Flash Attention is enabled. Other architectures have different per-token costs, so check the model's config.

What about the OS and everything else?

Whatever the OS, your editor, your browser and the inference runtime's own buffers use is memory the model cannot. There is no universal number for this, so measure it on your own machine with nothing loaded. If you run locally for privacy reasons, a machine that swaps will push you back to the cloud.

On Macs there is an extra, specific limit. By default macOS caps how much unified memory the GPU may wire. In a long-running llama.cpp discussion, users report that "only 2/3 or 3/4 of the unified memory can be used for Metal inference" by default, and that sudo sysctl iogpu.wired_limit_mb=<mb> raises it (the setting resets on reboot). llama.cpp prints the effective ceiling as recommendedMaxWorkingSetSize at load time. The same thread warns that the OS "needs some reasonable amount of memory," so raising it to the full amount is asking for swap.

A workable rule: add weights and KV cache for the context you actually use, then keep a quarter to a third of total RAM free. By that rule a 16 GB machine is comfortable for 8B at 4-bit, 32 GB for 14B at 8-bit or 32B at 4-bit with short contexts, and 64 GB is the floor for 70B at 4-bit.

Why does unified memory matter compared with a GPU's VRAM?

On a PC with a discrete GPU, the model runs fast only if it fits in VRAM. NVIDIA's flagship consumer card, the RTX 5090, has 32 GB of GDDR7. From the table, that holds a 32B model at 4-bit with a modest context, and nothing in the 70B class without spilling layers to system RAM over PCIe. Once layers spill, generation speed drops toward what the CPU side can manage. Ollama makes this visible: ollama ps shows a Processor column such as "100% GPU" or a split like "48%/52% CPU/GPU."

Apple Silicon has one pool shared by CPU and GPU. Current Mac mini specs go up to 64 GB with the M5 Pro. The Mac Studio goes to 128 GB with M5 Max and to 256 GB or 512 GB with M5 Ultra. MacStories notes the 512 GB configuration is "coming October." A 70B model at 8-bit (about 74 GB) does not fit in any single consumer GPU but fits in a 128 GB Mac Studio with room for context.

Capacity per box, not raw compute, is the plausible reason these machines sell out. The tradeoff shows up in prompt processing speed, which depends on GPU compute, not in whether the model loads.

How does memory bandwidth turn into tokens per second?

Generating one token requires reading every active weight from memory once. So a hard ceiling is:

max tokens/sec ≈ memory bandwidth ÷ bytes of active weights

It is an upper bound; real runtimes lose efficiency to KV cache reads and overhead. But it explains the data. The community benchmark thread Performance of llama.cpp on Apple Silicon measures LLaMA 7B text generation (TG128) across chips:

ChipBandwidthQ8_0 tok/sQ4_0 tok/s
M168 GB/s7.9214.19
M4 Max546 GB/s54.0583.06
M2 Ultra800 GB/s66.6494.27

Two patterns fall out. Generation scales with bandwidth across chips. And on the same chip, fewer bytes per weight means more tokens: the M4 Max goes from 31.64 tok/s at F16 to 83.06 at Q4_0. Quantization is a speed optimization as much as a capacity one.

Apply the ceiling to the table above. A 70B model at Q4_K_M is about 42.8 GB. On an M5 Pro's 307 GB/s, that is at most about 7 tokens per second in theory; on an M5 Ultra's 1.2 TB/s, about 28. Those are arithmetic bounds, not measurements.

MoE models break the link between size and speed. Because only active parameters are read per token, Lewis reports his 35B-total, 3B-active Qwen averaging "34 tok/s in token generation" and "325 tok/s in processing prompts" on an M4 Pro with 273 GB/s. At the top end, Viticci measured the M5 Ultra at 108 tok/s generation against 70 tok/s on an M3 Ultra (819 GB/s) for Qwen3.8-Flash-Next prose at 16K context, and time to first token at 256K context of 104 s versus 246 s. Its prompt processing figures (2,887 vs 1,143 tok/s) track GPU compute more than bandwidth, which is why a big context still means waiting before the first token.

Do 1.58-bit and ternary models change the math?

Eventually, perhaps. Ternary models store each weight as -1, 0 or +1, which is about 1.585 bits of information. A September paper, Breaking the 1.58-bit Barrier for Ternary LLMs, found that zeros make up "up to 51.5% of all weights" across 29 ternary models and exploits that to reach 1.485 bits per weight on the sparsest one, with end-to-end throughput gains of up to 1.18× on CPUs and 1.27× on GPUs. Plug 1.5 bpw into the formula and a 70B model's weights drop to roughly 13 GB. The catch is that this applies to models trained as ternary, not to the 4- and 8-bit checkpoints most people run today. For whether a small model is enough in the first place, see small language models vs LLMs.

Common questions about memory for local LLMs

Is 16 GB enough to run a local LLM?

Yes, for models up to about 8B at 4-bit with contexts in the low tens of thousands of tokens. Weights take about 5 GB and the KV cache for 16k tokens adds about 2 GB for Llama 3.1 8B. A 14B model at 4-bit (about 8.6 GB) also loads, but leaves little room for context and other apps.

How much RAM do I need for a 70B model?

At Q4_K_M the weights alone are about 43 GB, so 64 GB is the practical minimum, and only with short contexts. Full 128k context adds about 39 GB of FP16 KV cache for Llama 3.1 70B, which pushes you toward 128 GB or a quantized cache.

Is 8-bit worth it over 4-bit?

It costs about 1.7 times the memory and, per the llama.cpp benchmarks, generates noticeably slower. Lewis cites 4-bit as losing "about 1-2 points on most benchmarks compared to BF16" for the model he uses. Try 4-bit first and move up only if you see errors that matter for your task.

Why is my local model so slow even though it fits?

Either it does not fully fit on the GPU (check ollama ps for a CPU/GPU split, or the macOS wired memory limit), or you are bandwidth-bound. Divide your machine's bandwidth by the model file size to get the ceiling you are working against.

Where Vyra by Vyraagi stands

Vyra by Vyraagi is a desktop AI agent in closed alpha. Its 5-tier model router falls through to a local Ollama tier when offline, so the sizing above decides which local model that tier can realistically run on a given machine. Its episodic memory and structured world model are held on-device, which is a separate budget from the model's weights and KV cache. OS support is confirmed at beta; more on the offline side is on our offline AI assistant page and in local vs cloud models with offline fallback.

Sources

Want an agent whose offline tier runs on the hardware you already own? Vyra by Vyraagi is in closed alpha; join the waitlist.

Vyra is in closed alpha now, with a Founders Beta ahead of public launch.

Join the WaitlistSee Plans

Related reading

How to Run AI Locally in 2026: A No-Nonsense Guide to Private, Offline AI

What hardware you actually need, which model size to pick, and how to get a private AI running on your own laptop in about ten minutes, plus the honest limits of local models.

Small Language Models vs LLMs: Why the Next AI Wave Is Getting Smaller

Small language models are 10–30× cheaper to run than LLMs and could handle 40–70% of an agent's work. When small beats big, when it doesn't, and why.

AI Assistant Privacy: What Local-First Actually Changes

"Local-first" gets used as a privacy claim far more often than it earns one. Here's what running an assistant on your own machine genuinely protects, what it doesn't, and the questions worth asking.