Honey, I Shrunk the Weights (And Blew Up the KV Cache)

If you follow local machine learning on Apple Silicon (or spend an unhealthy amount of time lurking on r/LocalLLaMA, like I know I have), the marketing narrative is seductive: Apple laptops have unified memory, unified memory gives the GPU direct access to system RAM, and therefore your MacBook Pro is an autonomous AI workstation in a backpack. Right? Right?

In practice, anyone who has tried running serious local models on a 16 GB, 24 GB, or even 36 GB MacBook Pro (assuming you survived Apple’s extortionate RAM upgrade tax) runs into a brick wall.

The open-weight ecosystem has settled into two uncomfortable extremes:

1. The 8B Lightweight Tier (Llama 3 8B, Mistral 7B): Quantized to 4-bit, an 8B model fits comfortably in roughly 5 GB. It runs fast, but for real-world agentic work, it falls apart. The moment you ask an 8B model to run a 10-step autonomous loop (parsing dirty tool schemas, tracking terminal outputs, and reasoning across multi-file codebases), it begins looping and hallucinating. It is an impressive demo, but an unreliable daily driver.
2. The 70B Heavyweight Tier (Llama 3 70B, Qwen 72B): Phenomenal reasoning depth and strict instruction fidelity. But even quantized aggressively to 4-bit, a 70B model demands over 40 GB of dedicated VRAM. That immediately locks out 95% of software engineers, requiring a multi-thousand-dollar top-tier Mac Studio or a rack of server GPUs.

To get genuine agentic competence on a laptop, you need models in the 27B to 32B tier. Models in this weight class possess the parameter density required to handle complex tool calling without collapsing into loops.

So where is my Jarvis running on my laptop, all local and everything? Where is it?

Time to address the elephant in the room that benchmarks and devrels politely ignore: The KV Cache.

The Geometry of Memory: How Context Balloons Past Your RAM

When developers evaluate whether a model fits on their machine, they usually calculate static weight footprint in half-precision (FP16):

Static Footprint = 27B parameters * 2 bytes = 54 GB

Most people look at that number and assume: “Fine, I will just quantize it to 4-bit.”

At 4-bit (INT4), a 27B model shrinks to roughly 14.5 GB.

If you own a base 16 GB MacBook Pro, you are dead on arrival before token zero. You have 14.5 GB of static model weights fighting against 4 to 5 GB of macOS display and system buffers. The unified memory pool is already underwater on model initialization.

And if you paid extra for a 24 GB machine thinking you had breathing room, that illusion vanishes the moment you attempt actual autonomous workflows.

In a trivial chat session generating fifty tokens, memory is mostly static. But in real-world agentic tasks (where system prompts inject complete repository ASTs, documentation, terminal logs, and test traces), context quickly swells to 16,000 or 32,000 tokens.

For a 27B model with 64 attention layers:

• The uncompressed FP16 Key-Value (KV) cache consumes roughly 1.25 GB per 4,000 tokens.
• At a realistic 32k agentic context window, the KV cache alone demands 10 GB of unified memory.
• Activation buffers and scratch tensors consume another 2 GB.
• macOS itself requires 4 to 5 GB just to keep the desktop environment alive.

When you sum the arithmetic:

Static Weights (14.5 GB) + KV Cache (10 GB) + Activations (2 GB) + macOS (5 GB) = 31.5 GB

Even on a 24 GB machine, the operating system panics into compressed SSD swap. Token generation drops from an interactive pace down to an unusable crawl of 0.3 tok/s.

The physical constraint is unforgiving: If you cannot compress static model weights to under 7 GB, long-context agentic AI on an everyday laptop is physically impossible.

The Ternary Promise (And The False Summit)

In late 2024, extreme 1.58-bit ternary quantization (represented by models like Prism ML’s Ternary-Bonsai-2-27B and the broader BitNet lineage) showed a way out of this trap.

Instead of traditional 4-bit integers, ternary models constrain weights to just three discrete states: 1.

By encoding weights in base-3, a 27-billion parameter model compresses into a static footprint of just 6.5 GB.

At 6.5 GB, the geometry of local memory changes completely. A 27B model can sit comfortably on a 24 GB MacBook Pro with more than 14 GB of unified RAM left over for deep context windows, IDEs, and local developer tooling.

It sounded like the local AI breakthrough we had been waiting for. But when I actually sat down to run ternary models on Apple Silicon, I ran into a major false summit.

Bonsai published eye-catching headline numbers: 143 tok/s. But look at the fine print: that benchmark was measured on a monster workstation running an NVIDIA RTX 5090 with 1,792 GB/s of dedicated memory bandwidth.

When you moved over to Apple Silicon, the story changed dramatically. To run it on a Mac, you had to pull down a custom MLX fork with brittle build dependencies. And even inside that fork, performance on everyday laptops stalled: 28 tok/s on high-tier M-series chips, dropping sharply on consumer hardware.

Why? Because compressing the weights is only half the battle. If your runtime software does not match the physical memory architecture of the GPU, you have only traded a capacity bottleneck for an execution bottleneck.

The Execution Bottleneck: Why Stock Kernels Choke on Base-3

Apple’s standard mlx.core.quantized_matmul is a brilliant implementation, but it was designed for binary power-of-two representations (2-bit, 4-bit, 8-bit).

When you force a 1.58-bit Base-3 ternary model through a standard 2-bit binary execution path on Apple Silicon, you hit three massive architectural friction points:

1. The 25% Empty Bit Tax

A 2-bit binary number represents 4 discrete states. A ternary weight only needs 3 states. Forcing ternary values into standard 2-bit registers throws away 25% of theoretical data density on unused bit patterns. You are streaming wasted bytes across the memory bus on every single token decode.

2. Disjoint Memory Strides

In standard affine quantization, weights are stored in one array, FP16 scale factors in a second array, and biases in a third array. During autoregressive decode, the GPU memory controller has to fetch data from three completely separate addresses in unified memory. This pointer scatter destroys L2 cache efficiency. The GPU compute units spend over 65% of their cycles stalled, waiting on memory bus arbitration.

3. Unpacking Modulo Arithmetic in the Shader

Because Apple GPUs lack a native 1.58-bit assembly instruction, unpacking base-3 weights at runtime requires arithmetic inside the shader. Naive kernels unpack packed trits using integer division and modulo math:

// The execution bottleneck in naive shaders
int trit = (packed_byte / power_of_3) % 3;

Integer division on a GPU is notoriously slow. Running modulo operations billions of times per second saturates integer pipelines and starves the actual floating-point tensor arithmetic.

The Architectural Contrast: Stock MLX vs. Horizon

The Core Question: Can We Hack Our Way to Full Hardware Saturation?

An off-the-shelf M4 Pro MacBook Pro provides 273 GB/s of unified memory bandwidth. That is an astonishing amount of memory throughput packed into a thin laptop chassis.

Yet when running ternary models through off-the-shelf software stacks, the system was leaving over 70% of that physical bandwidth completely untouched. The hardware was capable of blistering local inference. The software runtime was the bottleneck holding it hostage.

That realization turned into a personal engineering obsession:

Is local AI in 2026 actually ready for autonomous agentic workflows on an everyday M4 Pro? Can an engineer bypass framework bloat, build a custom Metal JIT kernel from scratch, and actually saturate Apple Silicon’s unified memory bandwidth?

Over the next three logs, I am documenting the entire engineering trail:

1. Escape from ABI Hell: Ditching C++ builds and nanobind to build a pure-Python Metal JIT compiler.
2. The Horizon Architecture: Packing 5 trits into 8 bits with a 256-entry register LUT and 32-byte cacheline interleaving.
3. Dual-Dispatch & 4-Bit KV Cache: Splitting decode and prefill to run 32,000-token agentic context windows locally under 11 GB total memory.

The hardware is ready. It is time the software caught up.

(Follow the project or run benchmarks locally on GitHub).