>_faiz.palwala

MLX Ternary

[IN PROGRESS]Systems series active on Noggin. Custom Metal bit-packing kernels and MLX C++ bindings in development.

macOS Memory Pressure under 27B Local Model

The Problem: The Working Memory Wall

Running capable, autonomous AI models locally on consumer Apple Silicon hits a physical memory ceiling in 2026. The open-weight ecosystem is polarized between two extremes:

  1. 8B Lightweight Tier (Llama 3 8B, Mistral 7B): Fits in ~5GB at 4-bit, but lacks the parameter density for complex, multi-step agentic execution (parsing dirty schemas, analyzing multi-file codebases, and recovering from errors without looping).
  2. 70B Heavyweight Tier (Llama 3 70B, Qwen 72B): Exceptional reasoning, but demands 40GB+ of dedicated VRAM even at 4-bit, locking out anyone without a multi-thousand-pound Mac Studio or server cluster.

The sweet spot for local agents is the 27B to 32B tier. However, deploying a standard 27B model on a 24GB MacBook Pro fails the moment context scales:

  • Static Weights (INT4): ~14.5 GB
  • KV Cache (32k tokens, 64 layers): ~10.0 GB (uncompressed FP16 consumes ~1.25 GB per 4,000 tokens)
  • Activations & Scratch Tensors: ~2.0 GB
  • macOS System & Display Buffers: ~4.5 GB
  • Total Working Memory Demand: 31.0 GB

On a 24GB unified memory bus, macOS immediately panics into compressed SSD swap, dropping autoregressive decode throughput from an interactive 25+ tok/s down to an unusable crawl of 0.3 tok/s.

To make long-context 27B agentic workflows physically viable on standard hardware, static model weights must compress to under 7 GB.

The 1.58-Bit Opportunity & The Apple Silicon False Summit

Extreme 1.58-bit ternary quantization (constraining weights to three discrete states: ${-1, 0, +1}$) compresses a 27B parameter model down to 6.5 GB. At 6.5 GB, a 27B model fits comfortably on a 24GB Mac with more than 14GB of unified RAM remaining for deep context, IDEs, and local services.

However, stock runtimes on Apple Silicon cannot exploit this theoretical advantage. Apple’s stock mlx.core.quantized_matmul was designed for binary power-of-two representations (2-bit, 4-bit, 8-bit). Running base-3 ternary models through binary execution paths hits three core architectural bottlenecks:

  1. The 25% Empty Bit Tax: Storing 3 states in standard 2-bit registers ($2^2 = 4$) wastes 25% of theoretical information density on unassigned bit patterns, streaming useless bytes across the memory bus during every token decode.
  2. Disjoint Memory Strides: Storing weights, FP16 scale factors, and biases in separated arrays causes severe pointer scatter across unified memory, thrashing L2 cache and leaving GPU compute units stalled over 65% of the time.
  3. Shader Modulo Overhead: Because Apple GPUs lack native 1.58-bit assembly instructions, naive kernels unpack trits using integer division and modulo math in the shader, bottlenecking ALU pipelines.

The Systems Architecture

MLX Ternary is an open research engine and custom Metal kernel extension designed to saturate Apple Silicon memory bandwidth for ternary architectures:

1. Optimal Base-3 Bit Packing (5 Trits per Byte)

By packing 5 trits into a single 8-bit byte ($3^5 = 243 < 256$), the encoding achieves 95% theoretical information density, recovering the 25% bandwidth wasted by naive 2-bit register storage and reducing the static 27B footprint to its true physical floor.

2. Cache-Line Aligned Interleaved Memory Strides

Instead of separate pointer arrays for weights, scales, and offsets, the runtime interleaves quantized weights and FP16 scale factors into contiguous, 128-byte cache-line-aligned memory blocks. This eliminates pointer scatter and maximizes L2 cache hit rates during memory bus streaming.

3. Branchless Lookup Table (LUT) Shader Decoding

Eliminates runtime modulo and division instructions inside Metal threadgroups. Packed bytes are unpacked using fast compile-time lookup tables and SIMD shuffle operations directly within on-chip register space.

4. Direct MLX C++ Extension Binding

Integrates custom Metal compute pipelines directly into Apple MLX via its C++ API, enabling zero-copy tensor interoperability and seamless execution within existing MLX model architectures.

Published Engineering Logs

The development and benchmarking of MLX Ternary is documented publicly on Noggin:

  • Part 1: Honey, I Shrunk the Weights (and Blew Up the KV Cache) — The geometry of memory, 27B context math, and the hardware false summit.
  • Part 2 (In Progress): Escape from ABI Hell — Interfacing C++ Metal compute pipelines with Apple MLX.
  • Part 3 (In Progress): The Horizon Architecture — 5-trit bit-packing kernels, cache alignment, and decode benchmarks.