Understanding Dwarf Star

— Using AI to comprehend AI

I have been a developer for more than 20 years, but I have never studied LLMs in depth, beyond that veil you encounter when you only have a few notions about what a neural network is and what a backpropagation algorithm is in abstract terms. Nonetheless, I immediately understood that what Salvatore Sanfilippo was doing with his Dwarf Star deserved more attention than I was able to give, due to my technical limitations. So I decided to feed the Dwarf Star GitHub repository to DeepSeek v4 flash (the same model from which the entire project was born) and asked it to write a deep-pe analysis of what had been done, why, and how. The result is this summary which, while not particularly refined in its writing style, turns out to be quite clear, at least at a macroscopic level. After reading it, I thought it might be useful to others as well, who are curious to understand how a similar project can be approached, even without the technical skills to tackle it firsthand. It is possible, perhaps likely, that there are typos I haven’t noticed: if that is the case and someone spots them, I will be grateful for any heads-up.

Introduction: The Dawn of Local Intelligence

— The best way to predict the future is to build it — especially when the future can run on a laptop.

The year is 2026. Large Language Models have become the steam engines of the information age — general-purpose reasoning engines that can code, write, reason, plan, and even surprise their creators. But there is a tension at the heart of this revolution: the most capable models live in remote data centers, accessible only through APIs, while the dream of a truly personal artificial intelligence — one that runs on your own machine, understands your context, respects your privacy, and never phones home — has remained tantalizingly out of reach.

This article is about closing that gap. It is the story of Dwarf Star, a specialized inference engine built by one of the most celebrated systems programmers of our time, Salvatore Sanfilippo (known to the world as antirez, the creator of Redis). Dwarf Star is not another LLM wrapper or a general-purpose runner. It is a focused, laser-optimized piece of software designed to do one thing and do it brilliantly: run semi-frontier open-weight language models on hardware that a well-equipped enthusiast or a small company might actually own.

And in doing so, it teaches us something profound about how computers, data, and intelligence work together.

0.1 The LLM Revolution: A Very Brief History

To understand why Dwarf Star exists, we must first understand how we got here.

0.1.1 The Prehistoric Era (2017–2022)

The transformer architecture, introduced in the landmark 2017 paper “Attention Is All You Need” by Vaswani et al., was the Cambrian explosion of machine learning. Within a few years, models grew from hundreds of millions of parameters (BERT, 2018) to hundreds of billions (GPT-3, 2020). But these early giants lived exclusively in data centers. Running them required server racks, specialized networking, and cooling systems that consumed entire rooms.

The open-source community responded with smaller models: LLaMA (7B, 13B, 33B, 65B) in 2023, Mistral (7B), and later Mixtral (8x7B MoE). These could run on consumer hardware — a MacBook with 16 GB of RAM could slowly run a 7B model at 4-bit quantization. It was magical but limited. These models were small. They could hold a conversation, write basic code, and answer factual questions, but they were nowhere near the capability of GPT-4 or Claude 3.

0.1.2 The Era of Open Weights (2023–2025)

Then came Llama 3 (8B, 70B, 405B), Qwen 2.5, DeepSeek V2 and V3, and the Mixture-of-Experts (MoE) architecture became the dominant paradigm. Models got smarter but also larger. A 70B parameter model at 4-bit quantization needed about 35 GB of RAM — doable on a high-end MacBook or a workstation with two GPUs. But the 405B Llama 3? Over 200 GB at 4-bit. You needed a server.

The community pushed forward with quantization techniques: Q4_K_M, Q3_K_S, IQ2_XXS. Each halving of bit width halved the memory requirement but risked degrading quality. The art of quantization became a science.

0.1.3 The Semi-Frontier Threshold (2026)

In 2026, a new generation of models crosses a critical threshold. DeepSeek V4 Flash (284B total parameters, 13B active per token) and DeepSeek V4 Pro (1.6T total, 49B active) use aggressive Mixture-of-Experts to achieve frontier-level reasoning while keeping computational costs per token surprisingly low. GLM 5.2 offers another strong contender in the open-weight space. These models are not quite at the level of the best closed-source offerings on every benchmark, but they are close — close enough that for many tasks, the difference is imperceptible.

And crucially, they can fit on high-end consumer hardware — if you know how to compress them, load them, and run them efficiently.

This is the gap Dwarf Star is built to exploit.

0.2 State of the Art: Where the Frontier Stands

0.2.1 The MoE Revolution

The single most important architectural innovation enabling local semi-frontier inference is the Mixture-of-Experts (MoE) layer. Instead of activating all parameters for every token, MoE layers use a learned router to select a small subset of specialized “expert” sub-networks. DeepSeek V4 Flash has 284B total parameters but only 13B are active for any given token. This means the model’s capacity (its total knowledge) is vast, but its compute cost per token is roughly that of a 13B model.

Without MoE:
  [Every token] ──→ [All 284B parameters activated]
                    → Slow, memory-intensive, hot

With MoE:
  [Every token] ──→ [Router: "You need experts 7, 42, 153"]
                    → [Only 13B parameters activated]
                    → Fast, memory-efficient, cool

This asymmetry is what makes local inference viable. A 284B-parameter model at 2-bit quantization requires roughly 71 GB for the routed experts alone, plus additional memory for dense layers, attention, and KV cache. With careful engineering, this fits in 96–128 GB — exactly what a MacBook Pro M5 Max or a Mac Studio M3 Ultra provides.

0.2.2 The Quantization Revolution

The other pillar is aggressive asymmetric quantization. Dwarf Star quantizes different parts of the model at different precisions:

• Routed MoE experts: down to IQ2_XXS (roughly 2.1 bits per parameter) or Q2_K
• Dense layers, attention, shared experts: kept at Q8 or Q6_K or even FP16
• Output head and embeddings: always high precision

The insight is brutal and elegant: the routed experts are where the model stores most of its factual knowledge, but they are also where the model is most tolerant of precision loss. Each expert is specialized; small errors in an expert’s weights change its outputs slightly, but the router still selects the right experts, and the overall quality degrades gracefully. Quantize the attention mechanism or the embeddings by the same amount, by contrast, and the model falls apart.

Model component          │ Size share │ Quantization │ Quality impact if over-quantized
─────────────────────────┼────────────┼──────────────┼──────────────────────────────────
Routed MoE experts       │      ~85%  │ IQ2_XXS/Q2_K │ Graceful degradation
Shared experts           │       ~5%  │ Q8_K         │ Moderate sensitivity
Attention projections    │       ~4%  │ Q8_K/FP16    │ High sensitivity
Embeddings + output head │       ~1%  │ FP16         │ Extreme sensitivity
Other (routing, norms)   │       ~5%  │ FP16/Q8      │ Variable

This is not a hack — it is a deep exploitation of the model’s architecture. And it is one of the key themes this article will explore.

0.2.3 The 1M-Context Horizon

Modern open-weight models support context windows of up to 1 million tokens. DeepSeek V4 Flash and Pro achieve this through a hybrid attention mechanism called Compressed Sparse Attention (CSA) combined with Heavily Compressed Attention (HCA). The idea is simple: keep a full-resolution sliding window for the most recent 128 tokens, then compress older tokens into fewer, denser representations.

Token positions:  [0] [1] [2] ... [127] [128] [129] ... [1000000]
                                    │      │              │
Raw window (128)  ←─────────────────┘      │              │
Compressed (ratio 4)  ←────────────────────┘              │
Heavily compressed (ratio 128)  ←─────────────────────────┘

This means Dwarf Star can handle enormous contexts — entire books, codebases, or conversation histories — without the quadratic memory cost of full attention. The KV cache management alone is a chapter-worthy topic in itself (and indeed it is: Chapter 5).

0.3 The Local Inference Problem

Why would anyone want to run a semi-frontier model locally when cloud APIs are available? The reasons are as practical as they are philosophical.

0.3.1 Privacy

Every prompt you send to a cloud API is data that leaves your machine. For codebases containing proprietary algorithms, for medical or legal documents, for personal journals, for anything you would not post on a public forum — sending it to a remote server is a leak. Running locally guarantees that your data stays yours.

0.3.2 Cost

At scale, API costs accumulate. A heavy coding assistant session that processes 100K tokens of context and generates thousands of tokens can cost dollars per hour. Local inference is a capital expense (buy the hardware once) rather than an operating expense (pay per token). For a company with ten developers, the break-even point can be as short as a few months.

0.3.3 Latency and Reliability

Cloud APIs have network latency — typically 50–200 ms per round trip before any computation happens. They also have rate limits, downtime, deprecation schedules, and pricing changes. Local inference is always available, always at the same speed, and never subject to a third party’s business decisions.

0.3.4 The Philosophical Argument

There is something deeply empowering about owning your own intelligence infrastructure. It is the difference between renting a room in someone else’s house and building your own home. The local AI movement is not just about pragmatics — it is about autonomy.

0.3.5 Geopolitical Freedom

There is another dimension to local inference that grows more urgent by the day: geopolitical independence. Frontier AI models are developed by a handful of corporations and nations, and access to them is increasingly subject to export controls, licensing restrictions, and geopolitical boundaries. A model hosted on a server in one country can be blocked, throttled, or made unavailable to users in another — not for technical reasons, but for political ones. Entire regions can find themselves locked out of the most advanced AI capabilities overnight, caught in the crossfire of trade wars, sanctions, or regulatory shifts.

Local inference is the ultimate hedge against this. When the model runs on your hardware, no government, corporation, or geopolitical event can revoke your access to it. The weights on your SSD are immune to trade embargoes. The code in your terminal is not subject to API deprecation. The intelligence at your fingertips does not depend on undersea cables or data centers in a friendly jurisdiction.

This is not a theoretical concern. We have already seen models pulled from certain markets, API access restricted by region, and cloud providers required to filter or log requests based on the user’s country. In a world where the most capable minds increasingly rely on AI assistance, the ability to run a semi-frontier model locally is not a luxury — it is a safeguard for intellectual freedom.

0.4 Enter Dwarf Star

Dwarf Star is a C-based inference engine, self-contained and deliberately narrow. It is not a general GGUF runner like llama.cpp. It is purpose-built for a small family of models: primarily DeepSeek V4 Flash, secondarily GLM 5.2, and — on very large machines — DeepSeek V4 Pro.

What makes Dwarf Star special?

• Obsessive memory-bandwidth optimization: The engine is designed around the reality that for LLM inference, moving weights from RAM to the compute units is the bottleneck — not the computation itself. Every kernel, every memory layout, every scheduling decision is made with this in mind.

• Aggressive asymmetric quantization: As discussed above, Dwarf Star applies different quantization levels to different parts of the model, informed by empirical quality testing.

• Native KV cache: The engine manages its own KV cache with compressed attention support, disk persistence, and session management. Agent sessions are stored on disk with their full KV state, allowing instant resumption without re-prefixing.

• Multi-modal parallelism: Dwarf Star supports pipeline parallelism (splitting layers across machines), tensor parallelism (splitting inpidual layers across GPUs via RDMA), and batched multi-session serving — all in a single codebase.

• SSD streaming: For machines where the model does not fully fit in RAM, Dwarf Star can keep non-routed weights resident and stream routed experts from SSD on cache misses. The result is slower but still usable — a model that requires 128 GB can run on a 64 GB machine.

• A native agent and server: Dwarf Star includes a full CLI, a coding agent with tool-calling support, and an OpenAI/Anthropic-compatible API server.

┌──────────────────────────────────────────────────────┐
│                   Dwarf Star Engine                  │
│  ┌──────────────┐  ┌─────────────┐  ┌─────────────┐  │
│  │ Model Loader │  │  Quantized  │  │   KV Cache  │  │
│  │ (GGUF parser)│  │   Compute   │  │ (Compressed)│  │
│  └──────┬───────┘  │   Kernels   │  └──────┬──────┘  │
│         │          └──────┬──────┘         │         │
│         ▼                ▼                ▼          │
│  ┌─────────────────────────────────────────────────┐ │
│  │          Backend Abstraction Layer              │ │
│  │  ┌──────────┐  ┌──────────┐  ┌──────────────┐   │ │
│  │  │  Metal   │  │  CUDA    │  │    ROCm      │   │ │
│  │  │ (macOS)  │  │ (NVIDIA) │  │   (AMD)      │   │ │
│  │  └──────────┘  └──────────┘  └──────────────┘   │ │
│  └─────────────────────────────────────────────────┘ │
│                                                      │
│  ┌──────────┐  ┌──────────┐  ┌──────────────────┐    │
│  │   CLI    │  │  Agent   │  │ Server (API)     │    │
│  └──────────┘  └──────────┘  └──────────────────┘    │
└──────────────────────────────────────────────────────┘

Dwarf Star is also notable for what it does not do. It does not try to run every model. It does not support every quantization format. It does not abstract away the hardware — in fact, it embraces it, with specific optimizations for Apple Silicon’s unified memory architecture, NVIDIA’s CUDA tensor cores, and AMD’s ROCm compute units.

0.5 Why Salvatore Sanfilippo?

Salvatore Sanfilippo — antirez — needs little introduction in the systems programming world. He created Redis, the most popular in-memory data structure store on the planet, which powers caching, session management, real-time analytics, and message queues for countless applications. Redis is a masterpiece of software design: simple, elegant, and brutally efficient. Its single-threaded event loop, its protocol, its persistence model — every line of Redis teaches something about how to build systems that last.

But antirez is not just a programmer. He is a writer, a philosopher of software, and someone who thinks deeply about the relationship between humans and machines. After stepping back from Redis, he explored different interests — including vintage computing, writing, and, inevitably, the frontier of artificial intelligence.

What drove him to build Dwarf Star?

The answer, I believe, is the same drive that led him to build Redis: the desire to make something complex feel simple. Redis took the complex problem of distributed caching and made it feel like a local data structure. Dwarf Star takes the complex problem of running a 284-billion-parameter neural network and makes it feel like a local process — something you can run on your laptop, inspect, debug, and trust.

There is also a deeper motivation. Antirez has spoken about the importance of understanding the tools we use. In an era where most developers interact with AI through opaque APIs, Dwarf Star is a statement: you can know what is happening under the hood. The engine is open source. The kernels are inspectable. The quantization tables are readable. The KV cache is a file you can examine.

This article is, in many ways, an extension of that philosophy. We will not just tell you that Dwarf Star is fast — we will explain why it is fast, layer by layer, from the silicon up to the attention head.

0.6 What This Article Will Teach You

This article is structured as a journey from the hardware up to the highest levels of model architecture. Each chapter builds on the previous one:

By the end, you will understand not just what Dwarf Star does, but how it works and why its design choices make sense. You will be able to read its source code, follow its kernels, and perhaps even contribute your own optimizations.

0.7 A Note on the Journey

This article was entirely written by DeepSeek V4 Flash — under the guidance of humans who put some questions and led some ideas. This is stated openly because it shaped how the article was built. The Dwarf Star codebase itself is the product of a collaboration between antirez’s vision and AI-assisted development, building on the foundational work of llama.cpp, GGML, and the broader open-source ML community.

If you are uncomfortable with AI-assisted code and products, this article and Dwarf Star may not be for you. But if you are curious about how a small team (sometimes a team of one, with AI tools) can build something that pushes the boundary of what is possible on consumer hardware — read on.

The future of intelligence is not just in the cloud. It is also on your desk, in your laptop, in your hands. This is the story of how we got there.

— Roadmap: From here, we pe into Chapter 1, where we explore the computer architecture that forms the stage where all of this computation takes place. The von Neumann bottleneck, the memory hierarchy, the GPU's parallel universe — these are not abstract concepts. They are the physical constraints that Dwarf Star dances around with every line of code.

Chapter 1 — Computer Architecture: The Stage Where LLMs Perform

— The computer is the theater where the mind of the machine performs.

Before we can understand how Dwarf Star runs large language models efficiently, we must first understand the stage on which this performance takes place: the modern computer. Every time you ask an LLM a question, trillions of operations are executed across a complex hierarchy of components — CPUs, GPUs, RAM, caches, and storage — all working together in a carefully choreographed dance.

This chapter is about that dance. We will explore how data moves from a spinning disk or SSD all the way into the mathematical circuits that ultimately generate the next token of text. By the end, you will understand why some operations take nanoseconds while others take milliseconds, why Dwarf Star is written the way it is, and why the author’s choices make sense given the physical realities of modern hardware.

1.1 The Great Bottleneck: The von Neumann Architecture

Every modern computer is built around a fundamental design called the von Neumann architecture, named after the mathematician John von Neumann who described it in 1945. The core idea is simple yet profound: a single shared memory space holds both the program instructions and the data they operate on, and a central processing unit fetches them one at a time.

┌────────────────────────────────────────────────┐
│                    Memory (RAM)                │
│  ┌─────┬─────┬─────┬─────┬─────┬─────┬─────┐   │
│  │Inst │Data │Inst │Data │Inst │Data │...  │   │
│  └─────┴─────┴─────┴─────┴─────┴─────┴─────┘   │
└──────────────────────┬─────────────────────────┘
                       │ Bus (the "von Neumann bottleneck")
                       ▼
┌──────────────────────────────────────────────────┐
│                  CPU                             │
│  ┌─────────┐  ┌───────────────────────────────┐  │
│  │Control  │  │    ALU (Arithmetic Logic Unit)│  │
│  │Unit     │◄─┤    + Registers                │  │
│  └─────────┘  └───────────────────────────────┘  │
└──────────────────────────────────────────────────┘

This design has a famous weakness: the von Neumann bottleneck. The single bus connecting the CPU to memory can only transfer one piece of information at a time — either an instruction or a data word. As CPUs became faster and faster, this bottleneck grew more severe. The CPU could execute billions of instructions per second, but it spent most of its time waiting for data to arrive from memory.

Think of it like a world-class chef (the CPU) who can chop vegetables in milliseconds, but must walk to the pantry (RAM) every time they need a new ingredient. The walking takes far longer than the chopping.

For LLM inference, this bottleneck is existential. A model like DeepSeek V4 Flash has billions of parameters (weights), and every single token generated requires reading most of those weights from memory. The speed at which we can move data from storage to processor determines how fast the model runs — often more than the raw computational speed of the processor itself.

— Dwarf Star insight: Dwarf Star's obsession with memory bandwidth over raw compute is a direct consequence of the von Neumann bottleneck. When running a 2-bit quantized model, the limiting factor is almost never how fast the GPU can multiply numbers — it's how fast it can read the model weights from RAM.

1.2 The CPU: A Miracle of Modern Engineering

1.2.1 What is a CPU?

The Central Processing Unit is the “brain” of the computer. It executes instructions — the fundamental operations of a program: arithmetic (add, subtract, multiply), logic (compare, branch), and data movement (load from memory, store to memory).

A modern CPU like Apple’s M-series or an Intel Core processor contains billions of transistors etched onto a silicon die a few centimeters wide. But the raw transistor count is less important than how they are organized.

1.2.2 Cores and the End of Free Lunch

For decades, CPU speeds doubled every 18 months (Moore’s Law), and software ran faster without any changes. But around 2005, physical limits stopped the clock-speed race. Transistors were getting so small that they leaked current, generating unbearable heat. The industry pivoted: instead of making one faster core, they put multiple cores on the same chip.

A core is an independent processing unit capable of executing its own instruction stream. A modern desktop CPU might have 8, 16, or even 32 cores. Apple’s M3 Ultra has up to 32 CPU cores.

But here’s the catch: most software is not perfectly parallelizable. Amdahl’s Law states that the speedup from parallelization is limited by the sequential portion of the program. If 10% of a program must run sequentially, the maximum speedup from parallelization is 10× — no matter how many cores you throw at it.

For LLM inference, this matters deeply. Some operations (like processing a single token) are inherently sequential — token N+1 depends on token N. This is why Dwarf Star’s prefill phase (processing a large prompt) can be parallelized across cores, but the decode phase (generating one token at a time) is fundamentally sequential and limited by memory bandwidth.

1.2.3 The Memory Hierarchy: Why Your CPU Has a “Pantry” System

To mitigate the von Neumann bottleneck, computer architects invented a memory hierarchy. The idea is simple: place small, fast memories close to the CPU, and larger, slower memories farther away. The CPU automatically moves data between these levels based on what it predicts will be needed next.

Register      ───   ~1 cycle   (few bytes)
    │
   L1 Cache    ───   ~3 cycles  (32-64 KB per core)
    │
   L2 Cache    ───   ~10 cycles (256-512 KB per core)
    │
   L3 Cache    ───   ~40 cycles (8-32 MB shared)
    │
   RAM (DRAM)  ───   ~200 cycles (8-64 GB)
    │
   SSD (NVMe)  ───   ~100,000 cycles (512 GB - 2 TB)
    │
   HDD         ───   ~10,000,000 cycles (multi-TB)

Let’s examine each level:

Registers

The fastest, smallest memory. Registers are built directly into the CPU core and hold the data the CPU is currently operating on. A typical CPU has 16-32 general-purpose registers, each holding 64 bits (8 bytes) on modern architectures. There is no “load” penalty for registers — they are where the data already is.

L1 Cache (Level 1)

The first level of cache sits closest to each core. It is split into two parts:

• L1 Instruction Cache (L1i): Stores recently executed instructions
• L1 Data Cache (L1d): Stores recently accessed data

Each is typically 32-64 KB per core. The L1 cache can be accessed in about 3 CPU cycles. If the data the CPU needs is in L1, it’s a cache hit — almost no delay. If not, it’s a cache miss, and the CPU must look in L2.

L2 Cache (Level 2)

A bit larger (256-512 KB per core) and a bit slower (~10 cycles). In many modern CPUs, each core has its own private L2 cache. If the data is not in L2, the CPU probes L3.

L3 Cache (Level 3)

The last level of cache before main memory. Typically 8-32 MB and shared across all cores. Access time is around 40 cycles. L3 is large enough to hold significant portions of a working dataset — but for an LLM with billions of parameters, it’s a drop in the ocean.

Cache Lines and Locality

Data moves between memory and cache in fixed-size blocks called cache lines, typically 64 bytes. When the CPU accesses a single byte at address X, it loads the entire 64-byte block containing X into cache. This exploits spatial locality: if you access address X, you’ll likely access X+1, X+2, etc. soon after.

This is why matrix multiplication (the core operation of neural networks) is optimized by ensuring that consecutive elements in memory are processed consecutively. Dwarf Star’s quantization formats and kernel designs pay careful attention to memory layout — a poorly organized matrix would waste cache bandwidth loading 64-byte lines only to use a fraction of them.

— Dwarf Star insight: When Dwarf Star processes a layer of the neural network, it tries to keep the working set (weights for the current computation) in cache as much as possible. The quantization from 32-bit floats to 2-bit integers is not just about fitting the model in RAM — it also means 16× more weights fit in the same cache lines, reducing cache misses dramatically.

1.3 The GPU: A Different Kind of Beast

1.3.1 Why CPUs Are Not Enough for LLMs

A CPU is optimized for latency — executing a single stream of instructions as fast as possible. It has large caches, complex branch prediction, out-of-order execution, and all kinds of clever tricks to make inpidual threads run faster.

A GPU (Graphics Processing Unit) is optimized for throughput — executing many simple operations in parallel. It has thousands of simple cores, small caches, and relies on massive parallelism to hide memory latency.

The key difference:

• CPU: “I need to finish THIS task as fast as possible.”
• GPU: “I need to finish ALL of these 10,000 identical tasks as fast as possible.”

1.3.2 SIMT: Single Instruction, Multiple Threads

The fundamental execution model of a GPU is SIMT (Single Instruction, Multiple Threads). Groups of 32 threads (called a warp in NVIDIA terminology, or a wavefront in AMD) execute the same instruction simultaneously, but on different data.

┌──────────────────────────────────────────────┐
│                  GPU Core                    │
│                                              │
│  Thread 1:  a[0] * b[0] + c[0]               │
│  Thread 2:  a[1] * b[1] + c[1]               │
│  Thread 3:  a[2] * b[2] + c[2]               │
│  ...                                         │
│  Thread 32: a[31]* b[31]+ c[31]              │
│                                              │
│  All threads execute the same instruction    │
│  but on different data                       │
└──────────────────────────────────────────────┘

This is ideal for matrix multiplication, where the same operation (multiply and accumulate) is applied to millions of elements. Neural networks are essentially a long chain of matrix operations.

1.3.3 GPU Memory Hierarchy

GPUs have their own memory hierarchy, designed for throughput:

GPU Register     ───   ~1 cycle   (per thread, very limited)
      │
  Shared Memory   ───   ~5 cycles  (scratchpad, 16-48 KB per block)
      │
  L1/L2 Cache    ───   ~20-200 cycles
      │
  VRAM (HBM)     ───   ~400 cycles (8-80 GB, 1-2 TB/s bandwidth)
      │
  System RAM      ───   ~2000 cycles (via PCIe)

VRAM (Video RAM) is the GPU’s main memory. Modern GPUs use HBM (High Bandwidth Memory) which provides enormous bandwidth — up to 2 TB/s on an NVIDIA H100, compared to ~100 GB/s for system RAM. This bandwidth is the lifeblood of LLM inference.

1.3.4 The Matrix Multiplication: The Heartbeat of LLMs

At the mathematical core of every transformer neural network is the matrix multiplication (often called GEMM — General Matrix Multiply). Given matrix A (M×K) and matrix B (K×N), compute C = A × B.

For a single layer of an LLM, the weight matrix might be 4096×4096 (16 million parameters). Processing one token means multiplying the input vector (1×4096) by this weight matrix — 16 million multiply-add operations.

GPUs excel at this because:

1. The operation is perfectly parallel — each element of the output can be computed independently
2. Memory access patterns are regular and predictable
3. The computation-to-memory ratio is high enough to keep the compute units busy

But here’s where it gets subtle: for a transformer, the challenge is that the weight matrix is too large to fit in cache. So the GPU reads it from VRAM for every token. If the matrix is 16 million parameters stored as 4-byte floats, that’s 64 MB per layer — too large for any GPU cache. So the GPU’s memory bandwidth becomes the bottleneck.

1.3.5 Quantization: Making Data Smaller

This is where quantization enters the picture — and it’s one of the most brilliant ideas in modern LLM deployment. Instead of storing each weight as a 32-bit floating-point number (FP32), we can store it as 16-bit (FP16), 8-bit (INT8), 4-bit, or even 2-bit values.

┌─────────────────────────────────────────────────────┐
│      Data Type     │  Bits/Weight  │ Size Reduction │
├────────────────────┼───────────────┼────────────────┤
│     FP32           │     32        │     1×         │
│     FP16           │     16        │     2×         │
│     INT8           │      8        │     4×         │
│     Q4_K (4-bit)   │      4        │     8×         │
│     IQ2_XXS (2-bit)│      2        │     16×        │
└─────────────────────────────────────────────────────┘

A 2-bit quantized model uses 1/16 the memory of the original FP32 version. This is the difference between needing 500 GB of VRAM (impossible on consumer hardware) and 31 GB (fits on a high-end MacBook).

But quantization is not free. Reducing precision introduces noise, and the art lies in minimizing the quality loss. DeepSeek V4 Flash, as used in Dwarf Star, is remarkably robust to quantization — the 2-bit versions retain most of the quality of the full-precision model.

— Dwarf Star insight: Dwarf Star uses a highly asymmetrical quantization strategy. The routed Mixture-of-Experts (MoE) layers — which constitute the bulk of the model — are quantized aggressively to 2-bit, while the shared expert layers, attention projections, and routing mechanism are kept at higher precision (Q8 or FP16). This is because the routing and attention mechanisms are more sensitive to precision loss. The author recognized that not all parts of the model are equally important, and allocated precision where it matters most.

1.3.6 How CPU and GPU Communicate: PCI Express

The CPU and GPU do not share memory directly (except in unified memory architectures like Apple’s). They communicate through the PCI Express (PCIe) bus — a high-speed serial connection.

┌──────────┐    PCIe x16    ┌──────────┐
│   CPU    │◄──────────────►│   GPU    │
│ + DRAM   │   ~32 GB/s     │  + VRAM  │
└──────────┘                └──────────┘

Sending data across PCIe is expensive — about 10× slower than reading from VRAM. For LLM inference, the goal is to keep the model on the GPU and minimize data transfers. The prompt is sent once, and only the generated tokens (a tiny amount of data) need to be sent back.

This is why running an LLM on a CPU (where the model must be in system RAM) is so much slower — the CPU’s memory bandwidth is 10-20× lower than a GPU’s VRAM bandwidth, and for every computation, the CPU must read weights from slow system RAM.

— Dwarf Star insight: Dwarf Star supports distributed inference across multiple machines. When splitting layers across machines, activations must be sent over the network. Dwarf Star includes an option to reduce activation precision to 16-bit or even 8-bit for transport, cutting network traffic in half or more. This is the PCIe problem at a larger scale.

1.4 Unified Memory: Apple’s Secret Weapon

1.4.1 The Traditional Split

In a traditional PC, the CPU has its own memory (DRAM) and the GPU has its own memory (VRAM). They are physically separate, connected by PCIe. Data must be explicitly copied from one to the other.

This creates a fundamental tension: the GPU’s VRAM is limited (8-24 GB on consumer cards), which constrains the size of models that can run. A 70B parameter model in 4-bit quantization needs 35 GB — too much for any consumer GPU, but easily fits in 64+ GB of system RAM.

1.4.2 Apple Silicon’s Unified Memory

Apple’s M-series chips use a Unified Memory Architecture (UMA). The CPU and GPU share the same physical memory, accessed through a high-bandwidth fabric. There is no separate VRAM — everything is in one pool.

┌───────────────────────────────────────────┐
│           Apple Silicon (M3 Max)          │
│                                           │
│  ┌──────┐    ┌──────┐    ┌─────────┐      │
│  │ CPU  │    │ GPU  │    │ NPU     │      │
│  │ Cores│    │ Cores│    │ (Neural)│      │
│  └──┬───┘    └──┬───┘    └──┬──────┘      │
│     │           │           │             │
│     └──────┬────┴───────────┘             │
│            │   Unified Memory Fabric      │
│            ▼                              │
│  ┌──────────────────────────────────┐     │
│  │         LPDDR5 (96-128 GB)       │     │
│  │   ~400-800 GB/s bandwidth        │     │
│  └──────────────────────────────────┘     │
└───────────────────────────────────────────┘

This is revolutionary for LLM inference because:

1. The entire model can be in system RAM — up to 128 GB or even 512 GB (M3 Ultra)
2. The GPU can access it directly without PCIe copies
3. The CPU can also access it for preprocessing

A MacBook Pro M3 Max with 128 GB of RAM can run a 2-bit quantized DeepSeek V4 Flash (about 81 GB) entirely in memory, with the GPU reading weights directly from the unified pool at ~400 GB/s.

This is the hardware foundation that makes Dwarf Star possible. The project is specifically optimized for this scenario.

1.4.3 Trade-offs of Unified Memory

Unified memory is not without compromises:

• Memory bandwidth: While 400-800 GB/s is impressive, discrete GPUs can reach 2 TB/s or more. The M3 Ultra’s bandwidth is closer to a mid-range discrete GPU.
• Capacity sharing: The OS, applications, and model all share the same memory. A model using 81 GB leaves only 47 GB for everything else on a 128 GB machine.
• Power: Unified memory uses LPDDR (Low-Power DDR), which trades some bandwidth for energy efficiency.

Nevertheless, for running large models that would not fit in any consumer GPU’s VRAM, unified memory is transformative.

— Dwarf Star insight: The README explicitly targets machines with "96 GB or more" RAM. This is not arbitrary — it's the threshold where a 2-bit quantized DeepSeek V4 Flash can fit alongside the operating system and KV cache. The author understood that the Apple Silicon unified memory architecture was uniquely suited for this use case.

1.5 Storage: Where Models Live When Not in Use

1.5.1 The Storage Hierarchy

LLM model files are enormous — DeepSeek V4 Flash is about 81 GB in 2-bit quantization, and the full Pro model exceeds 300 GB. These files cannot all stay in RAM at all times. They are stored on SSDs (Solid State Drives) and loaded into RAM when needed.

Storage Tier        | Latency    | Bandwidth      | Typical Size
────────────────────┼────────────┼────────────────┼─────────────
L1 Cache            |  1 ns      |  ~10 TB/s      | 64 KB
L2 Cache            |  3-4 ns    |  ~5 TB/s       | 512 KB
L3 Cache            |  10-15 ns  |  ~1 TB/s       | 16 MB
RAM (DDR5/LPDDR5)   |  80-100 ns |  100-800 GB/s  | 16-512 GB
NVMe SSD            |  5-10 µs   |  3-7 GB/s      | 256 GB-8 TB
SATA SSD            |  20-50 µs  |  500 MB/s      | 256 GB-8 TB
HDD                 |  5-10 ms   |  150 MB/s      | 1-20 TB

1.5.2 NVMe: The Speed King

NVMe (Non-Volatile Memory Express) SSDs connect directly to the PCIe bus, bypassing the slower SATA interface. Modern NVMe drives can read data at 3-7 GB/s — fast enough that for a large model, loading it from disk takes seconds, not minutes.

For Dwarf Star, this speed enables SSD streaming — a technique where not all model weights are kept in RAM at once. Instead, the most frequently accessed weights stay in RAM, while less common ones are loaded from disk on demand.

1.5.3 Memory-Mapped Files (mmap)

This is a crucial concept for understanding Dwarf Star’s efficiency. A memory-mapped file (mmap) is a mechanism where a file on disk is mapped directly into the process’s virtual address space. The operating system handles loading the relevant portions into RAM on demand, using the same paging mechanism it uses for regular memory.

┌──────────────────────────────────────────┐
│          Virtual Address Space           │
│                                          │
│  ┌──────────────────────────────────┐    │
│  │        mmap region               │    │
│  │  (looks like regular memory)     │    │
│  │                                  │    │
│  │  Reads trigger page faults       │    │
│  │  that load data from the file    │    │
│  └──────────┬───────────────────────┘    │
│             │                            │
└─────────────┼────────────────────────────┘
              │
     OS Page Cache (RAM)
              │
              ▼
┌──────────────────────────────────────────┐
│       SSD (model GGUF file)              │
│  DeepSeek-V4-Flash-Q2.gguf (81 GB)       │
└──────────────────────────────────────────┘

The advantages of mmap:

• Lazy loading: Only the parts of the model that are actually accessed are loaded from disk
• No explicit I/O: The program just reads memory; the OS handles the rest
• Shared across processes: If multiple processes mmap the same file, they share the physical RAM pages
• Efficient for GPUs: Metal can wrap an mmap region as a GPU buffer without copying

— Dwarf Star insight: Dwarf Star uses mmap-based model loading as the default for Metal. The model file is mapped into memory, and Metal directly uses those mappings as GPU buffers. This means zero copying — the GPU reads weights from the OS page cache, which may or may not be backed by actual RAM depending on memory pressure. It's a beautiful example of letting the operating system do what it does best.

1.5.4 SSD Streaming: When the Model Doesn’t Fit

For machines with insufficient RAM to hold the entire model, Dwarf Star implements SSD streaming. This is a more explicit version of the mmap approach, specifically tuned for the Mixture-of-Experts architecture.

In an MoE model, not all “expert” sub-networks are used for every token. The router determines which experts are most relevant and activates only a subset (typically 2 out of 256). This means that at any given moment, only a fraction of the model weights are needed.

┌─────────────────────────────────────────────┐
│         SSD Streaming in Dwarf Star         │
│                                             │
│  Resident (always in RAM):                  │
│  ┌──────────────────────────────────────┐   │
│  │ Non-routed weights                   │   │
│  │   • Embedding table                  │   │
│  │   • Attention QKV projections        │   │
│  │   • Shared experts                   │   │
│  │   • Router weights                   │   │
│  │   • Output projection                │   │
│  └──────────────────────────────────────┘   │
│                                             │
│  Cache (recently used experts in RAM):      │
│  ┌──────────────────────────────────────┐   │
│  │ Hot expert cache (e.g., 32 GB)       │   │
│  │ Experts are loaded on cache miss     │   │
│  └──────────────────────────────────────┘   │
│                                             │
│  SSD (cold experts):                        │
│  ┌──────────────────────────────────────┐   │
│  │ Full model GGUF file (81 GB)         │   │
│  │ Experts loaded on demand             │   │
│  └──────────────────────────────────────┘   │
└─────────────────────────────────────────────┘

The key insight: routed experts dominate the model size (they are the bulk of the parameters), but only 2-4 are needed per token. By keeping non-routed weights resident and loading experts on demand, Dwarf Star can run models larger than available RAM at a fraction of the speed penalty you might expect.

Dwarf Star further optimizes this by overlapping I/O with computation: while the GPU processes the shared expert and the currently cached routed experts, the next needed expert is being loaded from SSD in the background. This hides the disk latency.

— Dwarf Star insight: The AGENT.md for the project explicitly states: "Always try to hide loading of missing routed experts by loading them while performing the inference of the shared expert and routed experts already in RAM." This is a textbook example of a systems programming technique called "prefetching" — anticipating future needs and satisfying them before they are requested.

1.6 The Memory Wall and What It Means for LLMs

1.6.1 Ops vs Bytes: The Roofline Model

To understand why LLM inference is memory-bound, we need the Roofline Model — a visual framework for understanding performance bottlenecks.

Performance (FLOPS)
    │
    │              ┌──────────────────── Compute-bound
    │              │   (matrix multiply large matrices)
    │              │
    │              │
    │    ┌─────────┘
    │    │ Memory-bound
    │    │   (reading weights)
    │    │
    └────┴──────────────────→ Arithmetic Intensity (ops/byte)
         ↑
    Ridge point

The x-axis is arithmetic intensity — how many arithmetic operations are performed per byte of data read from memory. The y-axis is performance. The “ridge” point separates two regimes:

• Left of ridge (memory-bound): Performance is limited by memory bandwidth. The processor could compute faster, but it’s waiting for data.
• Right of ridge (compute-bound): Performance is limited by compute capability. All data is in cache, and the processor runs at full speed.

LLM inference, especially at small batch sizes (which is typical for interactive use), falls firmly in the memory-bound regime. For every weight loaded from memory, only a few arithmetic operations are performed. The arithmetic intensity is low.

This is why memory bandwidth is the single most important specification for LLM inference hardware. A GPU with 2 TB/s bandwidth will be roughly 2× faster than one with 1 TB/s, all else being equal.

1.6.2 Quantization Changes the Equation

Quantization improves performance in two ways:

1. More data fits in cache: 2-bit weights are 16× smaller than FP32, so 16× more fit in the same cache. This can push some operations from memory-bound to compute-bound.
2. Higher effective bandwidth: If each weight is 2 bits instead of 32 bits, the same memory bandwidth delivers 16× more weights per second.

But quantization also reduces the computational work — operating on 2-bit integers is simpler than 32-bit floats. This further reduces the arithmetic intensity. The net effect is overwhelmingly positive for memory-bound workloads.

1.6.3 The KV Cache: Another Memory Challenge

In addition to model weights, LLM inference requires a KV cache (Key-Value cache). This stores the attention keys and values for all tokens processed so far, enabling the model to reference previous context without reprocessing them.

For a long conversation with 100,000 tokens, the KV cache can be enormous:

• For DeepSeek V4 Flash with compressed KV: the indexer alone can be ~22 GB at 1M tokens
• Uncompressed KV would be even larger

Dwarf Star uses a compressed KV cache with a ratio-4 indexer, meaning the KV cache is 4× smaller than the full uncompressed representation. This is essential for long-context inference on consumer hardware.

— Dwarf Star insight: The KV cache is stored in whatever memory is available after the model weights are loaded. The README mentions that "With 128GB of RAM you would run the 2-bit quants, which are already 81GB" — leaving 47 GB for KV cache, OS, and other applications. The trade-off between model precision, context length, and available memory is a constant theme in Dwarf Star's design.

1.7 Bandwidth and Latency: The Two Numbers That Matter

Throughout this chapter, we’ve mentioned bandwidth and latency repeatedly. Let’s make the distinction crystal clear.

• Latency is the time to start a single operation. “How long until the first byte arrives?”
• Bandwidth is the throughput once the operation is in progress. “How many bytes per second can be transferred?”

Analogy: a cargo ship has high bandwidth (carries millions of tons) but terrible latency (takes weeks to cross an ocean). A fighter jet has low bandwidth (carries one pilot) but excellent latency (arrives in hours).

For LLM inference:

• Prefill (processing the prompt): Bandwidth is king. The prompt can be hundreds or thousands of tokens, and the model must read most of its weights for each token. Higher memory bandwidth directly translates to faster prefill.
• Decode (generating one token at a time): Latency matters more. Each token requires reading all model weights once. After generating the first token, the second token has the same latency. Lower latency → faster generation.

Dwarf Star’s benchmark numbers clearly show this distinction:

• Prefill: 250-460 tokens/second (bandwidth-limited)
• Generation: 21-37 tokens/second (latency-limited)

The gap between prefill and generation speed is a fundamental characteristic of transformer architectures, not a limitation of any particular implementation.

1.8 How Dwarf Star Exploits the Computer Architecture

Let’s now tie everything together by examining how Dwarf Star’s design choices directly correspond to the hardware realities we’ve discussed.

1.8.1 Choice of Apple Silicon as Primary Target

Apple’s unified memory architecture is uniquely suited for running large models. A 128 GB M3 Max provides ~400 GB/s bandwidth to 128 GB of memory — enough to hold a 2-bit DeepSeek V4 Flash (81 GB) with room for KV cache (20-30 GB). No discrete GPU setup at consumer prices can match this combination of capacity and bandwidth.

1.8.2 mmap-Based Model Loading

By using mmap, Dwarf Star avoids:

• Loading the entire model on startup (fast launch)
• Explicit I/O calls (delegated to OS)
• Redundant memory allocations (data stays in page cache)
• Copying between CPU and GPU on unified memory (Metal wraps mmap regions directly)

1.8.3 Asymmetrical Quantization

Recognizing that the memory bottleneck is the primary constraint, Dwarf Star quantizes aggressively (2-bit) where quality impact is minimal (routed MoE experts) and preserves precision where it matters (attention, routing, shared experts). This is a direct application of the insight that not all weights contribute equally to model quality.

1.8.4 SSD Streaming with Overlapped I/O

For machines without enough RAM, Dwarf Star keeps only the essential non-routed weights resident and streams experts on demand. By overlapping I/O with computation, it hides the SSD latency (5-10 µs) behind GPU computation (hundreds of microseconds per layer).

1.8.5 Distributed Inference Over TCP

When a model is too large even for a 512 GB Mac Studio (the full Pro model at Q4), Dwarf Star splits layers across multiple machines. The network latency (0.45 ms for Thunderbolt, 77 ms for WiFi) is added to each token’s generation time. This makes distributed generation slower than local, but it enables running models that would otherwise be impossible.

1.8.6 Metal for GPU Compute

Rather than using a general-purpose framework like OpenCL or CUDA (which requires NVIDIA hardware), Dwarf Star targets Apple’s Metal API. This provides direct access to the GPU on Apple Silicon, with tight integration with the unified memory system.

1.8.7 KV Cache Compression

The compressed KV cache (ratio-4 indexer) is another memory-optimization technique. By storing compressed key-value pairs, Dwarf Star fits more context into the available memory after model weights are loaded. This is critical for long-context tasks like code review or document analysis.

1.9 Summary and Looking Ahead

In this chapter, we’ve built a mental model of the modern computer as it relates to LLM inference:

1. The von Neumann bottleneck means that data movement, not computation, is the primary constraint for most workloads.
2. The memory hierarchy (registers → L1 → L2 → L3 → RAM → SSD) spans 7 orders of magnitude in access time.
3. CPUs are optimized for latency (fast sequential execution), GPUs for throughput (massive parallelism).
4. Memory bandwidth is the single most important spec for LLM inference.
5. Quantization reduces memory demands by 2-16×, with surprisingly small quality loss for modern models.
6. Unified memory (Apple Silicon) eliminates the PCIe bottleneck, enabling large models to run on consumer hardware.
7. mmap and SSD streaming are techniques for running models larger than available RAM.
8. Dwarf Star’s design decisions are all direct responses to these hardware realities.

In the next chapter, we’ll pe deep into Data Representation and Compression — understanding how neural network weights are stored, what quantization actually does to the numbers, and how Dwarf Star’s specific quantization choices enable running state-of-the-art models on consumer hardware.

Chapter 2 — Data Representation and Compression

— Understanding a number is knowing what it is. Understanding quantization is knowing what it can afford to lose.

In the previous chapter, we saw how the memory bottleneck dominates LLM inference. This chapter is about the most powerful technique Dwarf Star uses to overcome that bottleneck: quantization — the art of representing numbers with fewer bits while preserving the information that matters.

But quantization is only part of the story. Dwarf Star also uses clever compression for the KV cache (the memory that stores the conversation history), and exploits the sparsity inherent in Mixture-of-Experts models. Together, these techniques reduce the memory footprint of a 671-billion-parameter model from over a terabyte to a manageable 80 GB — small enough to run on a high-end laptop.

2.1 How Computers Represent Numbers

2.1.1 Integers: The Simple Case

At the lowest level, computers store everything as binary — bits that are either 0 or 1. An unsigned integer uses all its bits to represent positive numbers:

8-bit unsigned integer:  0 to 255
16-bit unsigned integer: 0 to 65,535
32-bit unsigned integer: 0 to 4,294,967,295

A signed integer uses one bit for the sign (two’s complement representation):

8-bit signed integer:  -128 to 127
16-bit signed integer: -32,768 to 32,767

Integers are exact, simple, and fast. But they cannot represent fractions or very large numbers efficiently — which is what neural network weights typically are.

2.1.2 Floating-Point Numbers: The Compromise

To represent a wide range of values (from very small to very large) with limited bits, computers use floating-point representation. Like scientific notation (6.022 × 10²³), a floating-point number has three parts:

  ±  mantissa  ×  2^exponent
  ↑     ↑            ↑
 sign  fraction     exponent

More precisely, a 32-bit IEEE 754 float (FP32) is laid out as:

┌─┬─────────┬──────────────────────────────┐
│1│   8     │            23                │ bits
│S│ Exponent│         Mantissa             │
└─┴─────────┴──────────────────────────────┘

• Sign (1 bit): 0 for positive, 1 for negative
• Exponent (8 bits): Biased by 127, giving a range of 2⁻¹²⁶ to 2¹²⁷
• Mantissa (23 bits): The fractional part, providing precision

The value is: (-1)^sign × 1.mantissa × 2^(exponent - 127)

FP32 gives about 7 decimal digits of precision and can represent numbers from ~1.4 × 10⁻⁴⁵ to ~3.4 × 10³⁸.

2.1.3 The Precision Hierarchy

Neural network training and inference use different floating-point formats:

Format   | Bits | Exponent | Mantissa | Range            | Precision
─────────┼──────┼──────────┼──────────┼──────────────────┼──────────
FP32     |  32  |    8     |   23     | ±3.4×10³⁸        | ~7 digits
FP16     |  16  |    5     |   10     | ±65,504          | ~3 digits
BF16     |  16  |    8     |    7     | ±3.4×10³⁸        | ~2 digits

FP16 (half precision) has reduced range compared to FP32, but is half the memory. BF16 (Brain Floating Point, developed by Google) keeps the same exponent range as FP32 but reduces mantissa precision — ideal for neural networks where range matters more than exact precision.

— Dwarf Star insight: Dwarf Star uses FP16 for the KV cache attention tensors in some configurations, and FP32 for logits and sensitive computations. The choice is deliberate: attention values need more precision than model weights because small errors compound across the softmax operation.

2.2 Quantization: The Core Idea

Quantization is the process of reducing the number of bits used to represent a number. For neural networks, this means converting the high-precision floating-point weights into lower-precision formats — typically integers or specialized compressed formats.

The fundamental insight is that neural network weights are surprisingly robust to precision reduction. A weight that was 0.4231567 in FP32 might become 0.42 in INT8 or even 0.5 in 2-bit, and the network still produces sensible outputs. Why?

1. Overparameterization: Neural networks have far more parameters than needed for any single task. The redundancy provides a buffer against noise.
2. Distributed representation: Information is encoded across many weights, not in any single weight. Losing precision in one weight is compensated by others.
3. Training-aware quantization: Models can be trained or fine-tuned with quantization in mind, learning to be robust to precision loss.

2.2.1 The Quantization Equation

Every quantization scheme follows the same basic formula. We have:

• A floating-point value x
• A quantization level q (an integer)
• A scale factor s and zero-point z

The forward quantization (float → integer) is:

q = round(x / s + z)

The reverse dequantization (integer → float) is:

x ≈ s × (q - z)

The scale factor s determines the step size between representable values, and the zero-point z shifts the integer range to match the float range.

2.2.2 Symmetric vs Asymmetric Quantization

Symmetric quantization assumes the data is centered around zero, so z = 0:

q = round(x / s)
x ≈ s × q

This is simpler and more efficient for weights, which are typically roughly symmetric around zero.

Asymmetric quantization allows a non-zero zero-point, which is better for activations (like the output of a ReLU layer, which is always non-negative):

q = round(x / s + z)
x ≈ s × (q - z)

2.2.3 Per-Tensor vs Per-Channel vs Group Quantization

Quantization parameters (scale and zero-point) can be shared across different granularities:

Per-tensor quantization: A single scale and zero-point for an entire weight matrix. Simple, but loses fidelity because different rows or columns may have very different value ranges.

Per-channel quantization (per-row or per-column): Each output channel has its own scale. Much more accurate, as each channel’s range is independently captured.

Group quantization: Splits each row into groups of N elements (e.g., 32 or 256) and stores scale/zero-point per group. This is the sweet spot — good accuracy with reasonable overhead.

Dwarf Star uses block quantization with groups of 256 elements (QK_K = 256), as seen in the quants.h file. The Q2_K, Q4_K, and IQ2_XXS types all operate on 256-element blocks.

2.3 Dwarf Star’s Quantization Toolkit

Dwarf Star supports a rich set of quantization formats, inherited from the GGML/llama.cpp ecosystem. Let’s examine each one in detail.

2.3.1 Float Formats

Format   | Block Size | Size per Element | Description
─────────┼────────────┼──────────────────┼────────────────
F32      |     1      |     4 bytes      | Standard 32-bit float
F16      |     1      |     2 bytes      | Half-precision float
BF16     |     1      |     2 bytes      | Brain float

These are straightforward — each value is stored as a standard floating-point number. F32 is the “gold standard” for accuracy but uses 4 bytes per parameter. F16 and BF16 halve the memory at the cost of reduced precision or range.

2.3.2 K-Quant Formats (Q2_K, Q3_K, Q4_K, etc.)

These are the workhorses of GGUF quantization, developed for the llama.cpp project. They are called “K-quants” because they operate on K-sized blocks of 256 elements (denoted QK_K = 256 in the source code).

Let’s examine Q4_K in detail, as it is the primary format Dwarf Star uses for high-quality quantization of MoE expert weights. The block layout, verified against the quants.c source, is:

Q4_K block (256 elements), total = 144 bytes = 4.5 bits/element:
───────────────────────────────────────────────────────────────
  Offset 0:   2 bytes  — Super-block scale d (fp16)
  Offset 2:   2 bytes  — Super-block minimum dmin (fp16)
  Offset 4:  12 bytes  — Sub-block scales (8 × 6-bit scale
                           + 8 × 6-bit min, packed)
  Offset 16: 128 bytes  — 256 × 4-bit values (packed)
  ─────────────────────────────────────────────────────────────
  Total: 144 bytes → 144/256 = 0.5625 bytes/element = 4.5 bits/element

The structure is hierarchical. A single super-block of 256 values is scaled by d and shifted by dmin (both stored as fp16). Inside are 8 sub-blocks of 32 elements each. Each sub-block has its own local scale and minimum, encoded as 6-bit integers and packed into 12 bytes. The actual 4-bit values occupy 128 bytes at the end.

This hierarchical scheme captures local variations in value ranges far better than per-tensor quantization, while the per-block metadata overhead (16 bytes) is amortized across 256 elements, yielding an excellent precision-to-size ratio.

3.3.3 Q2_K: 2.625 Bits Per Element

DS4Q_TYPE_Q2_K: block_size=256, type_size=84 bytes

Q2_K uses approximately 2.625 bits per element (84 bytes / 256 = 0.328125 bytes = 2.625 bits). The structure is similar to Q4_K but with 2-bit values instead of 4-bit:

• 2 bytes: super-block scale (fp16)
• 2 bytes: super-block min (fp16)
• 16 sub-blocks of 16 elements each
• Each sub-block: 2-bit values packed into bytes
• Sub-block scales and mins packed efficiently

The extremely low bit width (2 bits can only represent 4 distinct values) means Q2_K captures only the coarsest shape of the weight distribution. Yet surprisingly, when used on the right parts of the model (routed MoE experts), it retains most of the quality.

2.3.4 IQ2_XXS: 2.0625 Bits — The Champion

This is Dwarf Star’s secret weapon. IQ2_XXS (Importance-aware Quantization 2-bit eXtra eXtra Small) uses just 2.0625 bits per element — a quarter of a byte per weight.

From the code:

DS4Q_TYPE_IQ2_XXS: block_size=256, type_size=66 bytes
66 bytes / 256 = 0.2578125 bytes = 2.0625 bits per element

But wait — how can you represent a floating-point number with only 2 bits? Two bits gives you only 4 distinct values. The answer lies in a sophisticated technique: importance-weighted quantization with codebook lookup.

How IQ2_XXS Works

Unlike simple linear quantization (where the representable values are evenly spaced), IQ2_XXS uses a codebook — a fixed set of 4 values that are optimized to minimize the quantization error for the actual distribution of weights in the model.

The process works in two stages:

Stage 1: Importance Matrix (IMatrix) Collection

Before quantization, the model is run on a calibration dataset. For each weight, we record how much it contributes to the model’s output — its importance. Weights that have a larger impact on the output are more “important” and should be quantized with more care.

The importance is measured as the diagonal of the Fisher Information Matrix — essentially, the squared gradient of the loss with respect to each weight, averaged over many inputs. Weights with large Fisher information are important; small changes to them cause large changes in the output.

Stage 2: Optimal Codebook Search

For each block of 256 weights, IQ2_XXS finds the best set of 4 codebook values (the “quantization levels”) and 256 assignments of each weight to one of those 4 levels. This is done by searching through candidate codebooks and picking the one that minimizes the weighted reconstruction error:

minimize:  Σ_i  importance_i  ×  (weight_i - codebook[assignment_i])²

The search is guided by the importance matrix: important weights get assigned to codebook values closer to their original value, while less important weights can tolerate more error.

Stage 3: Storage

The block is stored as:

• The 4 codebook values (represented compactly)
• The assignment indices (2 bits each for 256 weights = 64 bytes)
• A small amount of metadata

This gives the astonishing 2.0625 bits per element while preserving model quality that is remarkably close to the FP32 baseline.

2.3.5 Why IQ2_XXS Is Better Than Q2_K

Q2_K uses uniform quantization: the 4 representable values are evenly spaced between the min and max of the block. This works well when weights are uniformly distributed, but real weight distributions are often non-uniform — they cluster around certain values.

IQ2_XXS, by contrast, can place its 4 codebook values wherever they minimize error. It can concentrate precision where weights are dense and sacrifice precision where weights are sparse.

Weight Distribution
     ↑
     │   ***
     │  *   *        Q2_K levels: --·--·--·--·-- (uniform)
     │ *     *       IQ2_XXS levels:   ··   ··    (optimized)
     │*       *
     └─────────────────────────→ Value

The uniform levels of Q2_K waste precision in regions with few weights, while IQ2_XXS adapts its levels to the actual distribution.

Furthermore, IQ2_XXS requires an importance matrix (imatrix) — the per-weight importance scores collected during a calibration run. This is why ds4q_requires_imatrix() returns true for IQ2_XXS in quants.c.

— Dwarf Star insight The README specifically recommends "imatrix versions" of the quantized models: `./download_model.sh q2-imatrix`. These models are quantized using importance-aware methods that preserve quality much better than naive quantization. The Dwarf Star project includes tools for collecting importance matrices from real inference runs (see `gguf-tools/imatrix/`).

2.4 The Dwarf Star Quantization Strategy: Asymmetry by Design

This is perhaps the most important design insight in the entire project. Dwarf Star does not apply the same quantization to all parts of the model. Instead, it uses a highly asymmetrical approach:

Model Component          | Quantization     | Bits/Weight
─────────────────────────┼──────────────────┼────────────
Routed MoE - gate/up     | IQ2_XXS          |    2.0625
Routed MoE - down        | Q2_K             |    2.625
Shared experts           | Q8_0 (8-bit)     |    8.0
Attention projections    | Q8_0 (8-bit)     |    8.0
Router weights           | FP16             |   16.0
Output projection        | Q8_0 (8-bit)     |    8.0
Embedding table          | FP16             |   16.0
Compressor/indexer       | FP16             |   16.0

Why this asymmetry? Because different components of the model have different sensitivity to quantization noise.

2.4.1 Why MoE Experts Tolerate Extreme Quantization

The vast majority of the model’s parameters are in the routed Mixture-of-Experts layers — 256 experts, each containing three weight matrices (gate, up, and down projections). These constitute about 95% of the model’s total parameters.

These experts tolerate extreme quantization (2 bits!) for several reasons:

1. Sparsity of activation: Only 6 out of 256 experts are activated per token. The router selects the most relevant experts, and even quantized versions of those experts produce useful outputs.
2. Averaging effect: The output is the weighted sum of 6 expert contributions. Errors in inpidual experts average out.
3. Shared expert safety net: Dwarf Star keeps one shared expert at Q8_0 (8-bit) that is always active. This provides a high-quality baseline that compensates for quantization errors in the routed experts.
4. Importance-aware quantization: The IQ2_XXS method already described places precision where it matters most.

2.4.2 Why Attention and Routing Must Be Preserved

The attention mechanism and the router are the parts of the model that are most sensitive to precision:

• Attention computes a weighted sum of values based on query-key similarities. Small errors in the attention scores can drastically change which tokens are attended to, altering the entire meaning of the output.
• The router decides which experts to activate. An error in routing could select the wrong experts entirely, causing the model to produce gibberish.

This is why Dwarf Star keeps these components at Q8_0 (8-bit) or even FP16 — sacrificing a few gigabytes of memory for dramatically better quality.

2.4.3 The Bottom Line: Size vs Quality

Let’s examine the actual model sizes for DeepSeek V4 Flash under Dwarf Star’s asymmetrical quantization, using the real dimensions from the codebase.

From ds4.c, the Flash model has:

.n_embd = 4096,        // hidden dimension
.n_layer = 43,         // transformer layers
.n_expert = 256,       // total MoE experts
.n_expert_used = 6,    // active per token
.n_expert_shared = 1,  // shared expert
.n_ff_exp = 2048,      // FFN intermediate dimension
.n_vocab = 129280,     // vocabulary size

Total parameters: 284 billion (from the official model card). At full FP16 precision (2 bytes per parameter) this would be:

284B × 2 = 568 GB — far beyond consumer hardware

Dwarf Star’s asymmetrical quantization brings this down dramatically. The routed MoE experts, which constitute roughly 85% of the total parameters, receive the most aggressive compression:

The weighted average across all components is approximately 2.28 bits per parameter, yielding:

284B × 2.28 / 8 = ~81 GB

This matches the actual file size of the q2-imatrix model reported in the Dwarf Star README. The result is a model that fits comfortably in 96–128 GB machines — a MacBook Pro M5 Max, a Mac Studio M3 Ultra, or a DGX Spark GB10.

— Dwarf Star insight: The asymmetry extends even within the expert itself: gate and up projections use the more aggressive IQ2_XXS (2.0625 bits), while down projections use Q2_K (2.625 bits). Gate and up matrices handle different information flows and exhibit different sensitivity to quantization. This per-matrix optimization is only possible because Dwarf Star is a narrow, model-specific engine — a general framework cannot make such fine-grained choices.

2.5 KV Cache Compression

2.5.1 What Is the KV Cache?

The KV cache (Key-Value cache) stores the attention keys and values computed for all previous tokens in the current conversation. When generating token N+1, the model needs to attend to all N previous tokens. Without a cache, it would have to recompute all key and value vectors from scratch for every new token — an O(N²) cost.

With the KV cache, each token’s key and value vectors are computed once and stored. The cost of attending to N previous tokens is O(N) in memory and O(N) in computation — much better than O(N²).

Without KV cache:
  Token 1: compute K1, V1 → attend to K1, V1
  Token 2: compute K1, V1, K2, V2 → attend to K1, V1, K2, V2
  Token 3: compute K1, V1, K2, V2, K3, V3 → attend to K1, V1, K2, V2, K3, V3
  ... This is O(N²) compute!

With KV cache:
  Token 1: compute K1, V1 → cache → attend to K1, V1
  Token 2: compute K2, V2 → cache → attend to K1, V1, K2, V2
  Token 3: compute K3, V3 → cache → attend to K1, V1, K2, V2, K3, V3
  ... This is O(N) compute per token!

2.5.2 The Memory Cost of the KV Cache

The KV cache is enormous. For each layer:

KV cache size per layer = 2 × n_heads × head_dim × seq_len × bytes_per_element

Where:

• 2 accounts for both keys and values
• n_heads is the number of attention heads (typically 32-96)
• head_dim is the dimension of each head (typically 64-128)
• seq_len is the number of tokens cached
• bytes_per_element depends on the format

For DeepSeek V4 Flash with 60 layers, 32 heads, 128 head_dim, and 100K tokens:

• Without compression: 2 × 32 × 128 × 100000 × 2 (FP16) × 60 layers
• = ~98 GB — larger than many models!

This is clearly impractical for consumer hardware. Dwarf Star addresses this with aggressive KV cache compression.

2.5.3 Dwarf Star’s Compressed KV Cache

Dwarf Star uses a compressed KV cache with a ratio-4 indexer. This means the cache is 4× smaller than the uncompressed version.

The compression works by storing two representations of the KV cache:

1. Compressed (compact) representation: The actual key and value data, stored at reduced precision or with reduced dimensionality. This is what most cache memory is spent on.

2. Indexer (or frontier) representation: A separate structure that tracks which tokens have been cached and helps locate them quickly.

From the code, we can see the compression ratio per layer:

uint32_t ds4_engine_layer_compress_ratio(ds4_engine *e, uint32_t layer);

And the memory estimation:

ds4_context_memory ds4_context_memory_estimate(...);

Which returns:

typedef struct {
    uint64_t total_bytes;
    uint64_t raw_bytes;
    uint64_t compressed_bytes;
    uint64_t scratch_bytes;
    uint32_t prefill_cap;
    uint32_t raw_cap;
    uint32_t comp_cap;
} ds4_context_memory;

The compressed KV cache reduces memory requirements by about 4×. At 1M tokens, the README mentions the indexer alone is about 22 GB. Without compression, it would be over 80 GB just for the KV cache — prohibitive even on high-end machines.

2.5.4 How KV Cache Compression Works

The compression is based on the insight that attention patterns have low-rank structure. Not all key-value pairs are equally important, and many can be approximated by a smaller set of representative vectors.

The specific technique Dwarf Star uses is a variation of Multi-Head Latent Attention (MLA), introduced by DeepSeek. In MLA, the keys and values are first projected into a lower-dimensional “latent” space before being cached:

Original KV:   d_model dimensions
     │
     ▼
Compression projection (learned, kept at FP16)
     │
     ▼
Latent KV:     d_latent dimensions (typically d_model / 4)
     │
     ▼
Stored in KV cache at reduced precision

When attention is computed, the latent KV is projected back to the full dimension. The compression/decompression matrices are part of the model weights and are learned during training.

— Dwarf Star insight: Dwarf Star keeps the compressor and indexer tensors at FP16 precision, not quantized. This is because the compression/decompression operations are particularly sensitive to precision — errors here affect the entire attention computation. The small memory cost of FP16 for these tensors is far outweighed by the 4× reduction in KV cache size they enable.

2.6 The GGUF File Format

2.6.1 What Is GGUF?

GGUF (GPT-Generated Unified Format) is a file format for storing quantized neural network models. It was developed for the llama.cpp project and is used by Dwarf Star for model distribution and loading.

The format is simple but powerful:

• A header with metadata (model architecture, quantization types, etc.)
• A tensor directory listing all weight tensors with their names, shapes, and locations in the file
• The raw weight data

2.6.2 Dwarf Star’s GGUF Layout

Dwarf Star’s GGUF files contain a specific set of tensors that match the DeepSeek V4 architecture. The loader validates the tensor layout against expected shapes and quantization types, failing early if the file is incompatible.

From ds4.c:

/* Model shape selection is intentionally narrow: validation accepts the known
 * Flash and Pro layouts and fails early for anything else. */

This means a GGUF file intended for llama.cpp (even for the same model) will likely be rejected by Dwarf Star. The tensors must have exactly the expected names, shapes, and quantization types.

2.6.3 mmap-Based Loading

When Dwarf Star loads a GGUF file, it uses memory-mapped I/O (mmap). The file is mapped directly into the process’s virtual address space:

/* Loading is mmap based. The loader parses only the GGUF header, metadata
 * table, and tensor directory. Tensor data stays in the kernel page cache
 * until inference touches it... */

For Metal (Apple’s GPU framework), Dwarf Star can wrap these mmap regions as GPU buffers with zero copying — the GPU reads weights directly from the page cache as if they were in GPU memory. This is only possible because of Apple’s unified memory architecture.

2.7 The Importance Matrix: Calibrating Quantization

2.7.1 Why Naïve Quantization Is Not Enough

If you simply take a trained FP32 model and convert each weight to 2 bits uniformly, the quality will be terrible. The model was trained with FP32 precision, and its behavior depends on the precise values of its weights.

To quantize effectively, we need to know: which weights matter most?

2.7.2 The Fisher Information Matrix

The answer lies in a concept from statistics: the Fisher Information Matrix (FIM). For each weight in the network, its Fisher information measures how much the model’s output changes when that weight is slightly perturbed.

Mathematically, for a model with parameters θ and loss function L:

F_ij = E[ (∂L/∂θ_i) × (∂L/∂θ_j) ]

The diagonal of this matrix (where i = j) gives the per-weight importance. We compute the gradient of the loss with respect to each weight for many inputs, square it, and average over inputs.

2.7.3 Collecting the IMatrix

Dwarf Star’s tools (in gguf-tools/imatrix/) collect the importance matrix by running the model on a calibration dataset and recording the gradients:

int ds4_engine_collect_imatrix(ds4_engine *e,
                               const char *dataset_path,
                               const char *output_path,
                               int ctx_size,
                               int max_prompts,
                               int max_tokens);

The imatrix capture process:

1. Load the model in FP16 or FP32
2. Run on a perse set of prompts (code, conversation, reasoning tasks)
3. For each forward pass, record the squared gradients
4. Average over all inputs
5. Save the resulting importance values

The imatrix is then used during quantization to guide the codebook search (for IQ2_XXS) or the scale selection (for Q2_K/Q4_K).

2.7.4 The Result: Imatrix-Tuned Models

Dwarf Star’s download script offers q2-imatrix models, which are quantized using the importance matrix. These models consistently outperform naive quantizations of the same bit width.

The difference is especially visible in:

• Long-context reasoning: Important weights preserved → better attention over long sequences
• Tool calling: Precise routing weights → correct expert selection
• Code generation: Exact numerical values preserved where they matter for arithmetic

2.8 Grouped Quantization and Dot Product Computation

2.8.1 The Problem: Computing with Quantized Values

The real challenge of quantization isn’t storage — it’s computation. How do you multiply a quantized weight (stored as a 2-bit integer) by an activation (stored as FP16)?

The answer: dequantize on the fly. When performing a matrix multiplication, the GPU kernel loads the quantized weights, reconstructs the floating-point values using the stored scales and mins, and then performs the floating-point multiplication.

2.8.2 Grouped Matrix Multiplication

Dwarf Star’s GPU kernels (in metal/ and ds4_cuda.cu) are specialized for grouped quantization. For Q4_K or IQ2_XXS, the kernel:

1. Loads a block of 256 quantized weights
2. Loads the block’s scale, min, and sub-block scales
3. Dequantizes all 256 weights (implicitly or explicitly)
4. Multiplies by the corresponding 256 activation values
5. Accumulates into the output

This is the kernel fusion technique — combining dequantization with multiplication to minimize memory traffic and exploit data locality.

2.8.3 Why Block Size 256?

The choice of 256 elements per block is deliberate:

• 256 ÷ 32 (warp size) = 8: Each warp in a GPU can process 8 blocks of data in a balanced way
• 256 × 4 bits = 128 bytes: Fits comfortably in GPU shared memory (typically 48 KB per block)
• 256 × 2 bits = 64 bytes: Even IQ2_XXS blocks fit in shared memory
• 256 is a power of 2: Simplifies addressing and modulo arithmetic

— Dwarf Star insight: In the CUDA backend code (ds4_cuda.cu), you'll find kernels named like `ds4_gpu_matmul_q8_0_pair_decode_rows_exact_tensor`. These highly specialized kernels handle specific combinations of quantization format and operation type. This specialization is only feasible because Dwarf Star supports a limited set of formats and model architectures.

2.9 Summary: The Compression Pyramid

Let’s review the compression techniques Dwarf Star uses, arranged by their compression ratio:

Technique                    | Ratio | What It Affects
─────────────────────────────┼───────┼────────────────────────────────
FP32 → FP16 (weights)        |  2×   | Uncompressed reference
FP32 → FP16 (activations)    |  2×   | Attention, routing
FP32 → Q8_0 (8-bit)          |  4×   | Shared experts, attention proj.
FP32 → Q4_K (4.5-bit)        |  7×   | High-quality MoE quant
FP32 → Q2_K (2.625-bit)      | 12×   | Routed expert down projections
FP32 → IQ2_XXS (2.0625-bit)  | 15.5× | Routed expert gate/up projections
KV cache compression (MLA)   |  4×   | KV cache size

Combined, these techniques reduce the model from over 1 TB (FP32) to about 81 GB — a 12.5× compression ratio — while maintaining output quality that rivals the full-precision model.

But compression is not magic. The next chapter will show how Dwarf Star manages this compressed data at the systems level — allocating buffers, managing memory, and orchestrating the flow of data through the GPU’s computational graph.

Chapter 3 — C Systems Programming

— C is close to the language of the machine, speaking directly to the silicon.

The previous two chapters gave us an understanding of the hardware stage and the data formats that perform on it. This chapter is about the choreography — how Dwarf Star orchestrates the movement and transformation of data at the systems level.

Why C? Why not Python, Rust, or Go? The answer lies in the requirements: to run a 671-billion-parameter model on consumer hardware, you need every last bit of performance and every last byte of memory control. C is the language that gives you that control. It is the lingua franca of systems programming, the language of operating systems, embedded devices, and — yes — high-performance inference engines.

Dwarf Star is a masterclass in practical C systems programming. In this chapter, we’ll explore the specific techniques it uses: memory-mapped files, manual buffer management, pointer arithmetic, lock-free sharing, and the graceful handling of hardware failures.

3.1 Why C? The Systems Programming Mindset

3.1.1 Control Over Memory

In Python, memory is managed by the interpreter. Objects are allocated and garbage-collected automatically. You don’t know exactly when memory will be freed, how much overhead each object has, or where in the address space things live.

In C, you are the memory manager. You call malloc() to allocate, free() to deallocate. You decide exactly how many bytes each structure uses. You control the layout of data in memory — and for a cache-sensitive workload like LLM inference, this control is essential.

Consider: an LLM weight matrix might be 4096 × 4096 floats. In Python, using NumPy, this would be a numpy array with some overhead. In C, it’s just a contiguous block of 4096 × 4096 × 4 = 67,108,864 bytes. Nothing more, nothing less. The GPU kernel expects this exact layout, and any framework overhead would be wasted cycles and cache misses.

3.1.2 Minimalism by Design

C is famously small. The entire language specification for C11 fits in about 700 pages, including library descriptions. This compactness is possible because C provides only a handful of essential abstractions: functions, structs, pointers, control flow, and the preprocessor. No classes, no templates, no exceptions, no virtual dispatch.

This minimalism is deliberate and beneficial for a project like Dwarf Star:

1. Compile speed: A C file compiles in milliseconds. The entire Dwarf Star codebase compiles from scratch in a few seconds. Fast compile times mean faster iteration during development and debugging.

2. Portability: A C compiler exists for virtually every platform — macOS, Linux, Windows, embedded systems, and even the GPU-side Metal Shading Language is a close variant of C. Dwarf Star targets macOS (Metal), Linux (CUDA, ROCm), and CPU-only fallback, all with different compilers, and C is the common denominator.

3. Readability: The codebase is meant to be understood and modified by readers of this article. C’s minimal syntax makes the computational structure immediately visible. When you read matrix[i * cols + j], you see exactly what happens — no operator overloading, no hidden copies, no implicit conversions.

4. Predictable performance: Every language construct in C has a direct, obvious mapping to machine code. A struct field access is a constant offset. A function call is a jump. A pointer dereference is a load. There is no hidden allocation, no implicit synchronization, no invisible destructor running at the end of a scope.

3.1.3 Close to the Metal

C maps almost directly to assembly language. A for loop in C becomes a few assembly instructions. A pointer dereference becomes a single load instruction. Struct field access is a constant offset from a base address.

This predictability is invaluable for performance tuning. When you write a computational kernel in C (or in Metal Shading Language), you can reason about exactly what the hardware will do. There is no “magic” — just instructions and data.

3.2 Memory Management: The Art of Allocation

3.2.1 Static vs Dynamic vs Automatic Allocation

C offers three kinds of memory allocation:

Static allocation (global variables):

static float g_weights[4096 * 4096];  // 64 MB of global data

This memory exists for the entire program lifetime. It’s in the data segment, not the heap. Useful for constants and fixed-size working buffers.

Automatic allocation (stack variables):

void process_layer(float *input) {
    float buffer[256];  // 1 KB on the stack
    // ...
}  // buffer automatically freed when function returns

Stack allocation is extremely fast — just a single stack pointer adjustment. But stack space is limited (typically 8 MB per thread on macOS/Linux). You cannot allocate gigabytes of weights on the stack.

Dynamic allocation (heap):

float *weights = malloc(4096 * 4096 * sizeof(float));
// ...
free(weights);

Heap allocation is flexible but slower. The malloc/free pair can be a bottleneck in performance-critical code. Dwarf Star uses heap allocation for most working buffers but carefully manages the lifetime of each allocation.

3.2.2 Dwarf Star’s Buffer Strategy

Dwarf Star uses a scratch buffer pattern for temporary computations. Instead of allocating and freeing memory for each operation (which would cause fragmentation and overhead), it pre-allocates large scratch buffers and reuses them:

typedef struct {
    float *ffn_moe;
    // ... other scratch pointers
} ds4_cpu_decode_scratch;

The CPU decode path pre-allocates scratch space once and reuses it across all inference calls. This is essential because malloc can introduce unpredictable dynamic allocation overhead and system calls, causing latency spikes that are unacceptable in a low-millisecond execution loop.

For the GPU path, scratch space is allocated as Metal or CUDA buffers and reused across graph evaluations. The graph itself is pre-compiled and reused.

3.2.3 Memory Pooling

For larger working sets, Dwarf Star uses memory pools — pre-allocated regions of memory that are subpided for specific purposes. The KV cache is one such pool:

// Conceptual illustration
typedef struct {
    void *kv_cache_base;      // Large pre-allocated region
    uint64_t kv_cache_size;   // Total size
    uint32_t *layer_offsets;  // Offsets within the region for each layer
} ds4_kv_cache;

By pre-allocating the entire KV cache at session creation time, Dwarf Star avoids fragmentation and ensures that memory pressure is detected early (fail fast) rather than during inference.

3.2.4 Cache-Friendly Memory Layout

The arrangement of data in memory has a huge impact on performance. Consider two ways to store a matrix:

Row-major order (C default):

float matrix[rows][cols];
// matrix[i][j] is at offset i*cols + j

Column-major order (Fortran default):

// matrix[i][j] would be at offset j*rows + i

Dwarf Star stores matrices in row-major order (C convention). This means consecutive elements in a row are adjacent in memory. When the GPU kernel processes a row, it accesses memory sequentially, maximizing cache line utilization.

Furthermore, Dwarf Star groups the quantization blocks contiguously. A Q4_K block of 256 elements and its metadata (scales, mins) is stored as a single 144-byte record. All blocks for a tensor are stored consecutively:

┌───────────┬───────────┬───────────┬──────
│Block 0    │Block 1    │Block 2    │...
│(256 vals, │(256 vals, │(256 vals, │
│ 144 bytes)│ 144 bytes)│ 144 bytes)│
└───────────┴───────────┴───────────┴──────

This layout means the GPU kernel can stride through memory loading one block at a time, with perfect spatial locality.

3.3 Pointers: The Double-Edged Sword

3.3.1 Direct Memory Access

Pointers are C’s most powerful feature. A pointer is just a memory address — a 64-bit integer that tells the CPU where data lives.

float *weights = mmap(...);  // Pointer to the start of the model weights
float *layer_0 = weights + layer_offset;  // Pointer arithmetic: + N skips N floats
float w = *layer_0;  // Dereference: read the float at that address

Pointer arithmetic is the foundation of efficient array access. weights + 4096 gives you the address 4096 floats (16384 bytes) after weights. No bounds checking, no overhead — just an address calculation.

3.3.2 Function Pointers for Polymorphism

C doesn’t have virtual functions or interfaces. Instead, it uses function pointers:

typedef void (*ds4_token_emit_fn)(void *ud, int token);
typedef bool (*ds4_session_cancel_fn)(void *ud);

These allow callback-based programming without needing inheritance or interfaces. The ds4_session struct holds function pointers that the engine calls when tokens are generated or when cancellation is needed. The caller (CLI, server, or agent) provides its own implementations.

3.3.3 Void Pointers and Opaque Types

Dwarf Star uses opaque types to hide implementation details:

typedef struct ds4_engine ds4_engine;  // Forward declaration only
typedef struct ds4_session ds4_session;

Users of the API never see the internals of these structures. They only interact through functions like ds4_engine_open(), ds4_session_create(), etc. This is a classic C encapsulation pattern — the .h file provides only the interface, while the .c file contains the actual struct definitions.

3.3.4 The Perils of Pointers

With great power comes great responsibility. Pointers in C are the source of many bugs:

• Null pointer dereference: Accessing address 0 causes a crash
• Buffer overflow: Writing past the end of an allocated region corrupts memory
• Use-after-free: Using a pointer after the memory has been freed
• Dangling pointers: Pointers to stack variables after the function returns

Dwarf Star mitigates these through careful coding conventions and extensive internal validation. The KV cache has integrity checks (rolling hash of token prefixes), and the model loader validates every tensor before using it.

3.4 mmap: The Model Loading Superpower

3.4.1 How mmap Works

We touched on mmap in Chapter 1. Let’s pe deeper into the implementation.

mmap is a POSIX system call that maps a file (or device) into the process’s virtual address space:

#include <sys/mman.h>

void *addr = mmap(
    NULL,                         // Let the kernel choose the address
    file_size,                    // Length of mapping
    PROT_READ,                    // Read-only access
    MAP_PRIVATE,                  // Copy-on-write
    fd,                           // File descriptor
    0                             // Offset in file
);

After this call, the file contents are accessible as a contiguous block of memory starting at addr. The OS manages loading the file contents into RAM on demand — when the program reads an address that hasn’t been loaded yet, the OS triggers a page fault, loads the needed page from disk, and resumes execution.

3.4.2 Dwarf Star’s mmap Usage

Dwarf Star maps the entire GGUF model file with mmap:

/* Loading is mmap based. The loader parses only the GGUF header, metadata
 * table, and tensor directory. Tensor data stays in the kernel page cache
 * until inference touches it, or until Metal wraps slices of the mapping as
 * no-copy MTLBuffers. */

This means:

1. Fast startup: The program starts immediately — no need to read the entire model into memory first
2. Lazy loading: Only the tensors actually accessed are loaded from disk
3. Zero-copy GPU access: On Apple Silicon, Metal buffers can wrap mmap regions directly, so the GPU reads from the same physical memory as the CPU
4. Automatic caching: The OS keeps frequently accessed pages in the page cache

The fourth point is subtle but critical. When the GPU reads a weight that’s in the page cache, it’s fast. When it reads a weight that’s been evicted (because memory is tight), the OS must fault it back from disk — a page fault that can take microseconds. This is the foundation of Dwarf Star’s SSD streaming mode.

3.4.3 mmap vs read()

Why not just read() the file into a heap buffer?

• Memory duplication: read() copies data from the kernel’s page cache into the process’s heap. mmap shares the page cache directly.
• Startup latency: read() blocks until the data is copied. mmap returns immediately and the OS loads pages on demand.
• Memory pressure: With read(), the entire model occupies heap memory even if only parts are used. With mmap, unused pages can be evicted by the OS under memory pressure.
• GPU integration: Metal’s MTLBuffer can be created from an mmap region with makeBufferWithBytesNoCopy:length:deallocator:. No such path exists for heap allocations.

3.4.4 The File Descriptor and Lifecycle

Dwarf Star opens the GGUF file, maps it, and then keeps the file descriptor open for the lifetime of the engine:

int fd = open(model_path, O_RDONLY);
void *mapping = mmap(NULL, file_size, PROT_READ, MAP_PRIVATE, fd, 0);
close(fd);  // The mapping holds a reference to the file
// ... use mapping ...
munmap(mapping, file_size);  // Unmap when done

The close() after mmap is safe — the kernel keeps the file’s data pages alive as long as any mapping exists.

3.5 The Metal Graph: Pre-compiled Computation

3.5.1 What Is the Metal Graph?

Dwarf Star doesn’t execute inpidual GPU kernels one at a time. Instead, it builds a Metal performance-driven graph — a pre-compiled sequence of GPU operations that represents the entire inference pass for one layer.

The graph includes:

• Loading weights from the mmap region
• Loading input activations
• Computing attention (query, key, value projections → attention scores → weighted sum)
• Computing the feed-forward network (router → expert selection → expert evaluation → weighted combination)
• Writing output activations to the next layer’s buffer
• Updating the KV cache

3.5.2 Why a Graph?

Executing a GPU kernel incurs overhead — the CPU must prepare command buffers, submit them, and synchronize. For a single kernel, this overhead is small. But a transformer has dozens of layers, each with dozens of operations. Submitting them inpidually would waste precious milliseconds.

By encoding all operations for one layer into a single Metal graph, Dwarf Star amortizes the submission overhead across many operations. The GPU also benefits from knowing the full sequence — it can schedule memory loads earlier, keep data in on-chip caches across operations, and pipeline execution.

3.5.3 Graph Scheduling

The Metal graph is a directed acyclic graph (DAG) of compute operations. Dwarf Star encodes each layer as a subgraph, and the entire forward pass as a graph of layer subgraphs.

The graph is built once at startup (when the model is loaded) and reused for every inference call. This is a key performance optimization — graph encoding is expensive, but it’s a one-time cost.

3.5.4 The CPU-GPU Synchronization Dance

During inference, the CPU and GPU work in tandem:

1. CPU: Prepares input tokens, updates session state, submits the Metal graph for layer N
2. GPU: Executes the graph for layer N
3. CPU: While GPU is busy, prepares the next command buffer, checks for cancellation
4. GPU: Finishes layer N, signals completion
5. CPU: Submits layer N+1 (or reads output logits and proceeds to next token)

For SSD streaming mode, Dwarf Star adds an additional overlap: while the GPU processes layer N, the CPU initiates the async read of the next needed expert from disk. By the time the GPU needs it, it’s ready.

3.6 Thread Safety and Synchronization

3.6.1 The Single-Session Model

Dwarf Star’s inference engine is fundamentally single-threaded for session operations. Only one thread at a time should call ds4_session_sync() or ds4_session_eval(). This avoids the complexity of concurrent GPU access and simplifies the memory model.

However, multiple components run concurrently:

• The HTTP server handles multiple client connections (thread pool)
• The GPU executes asynchronously (separate hardware thread)
• SSD streaming uses background threads for I/O

3.6.2 Synchronization Primitives

Dwarf Star uses POSIX threads (pthreads) for concurrency:

#include <pthread.h>

static pthread_mutex_t ds4q_init_mutex = PTHREAD_MUTEX_INITIALIZER;

This mutex protects the quantization initialization code, which must run exactly once. The pattern is:

pthread_mutex_lock(&ds4q_init_mutex);
if (!initialized) {
    // One-time initialization
    initialized = true;
}
pthread_mutex_unlock(&ds4q_init_mutex);

For the distributed inference path, the coordinator and workers communicate over TCP sockets. Each worker runs in its own thread, processing work items as they arrive. The coordinator uses poll() (or select()) to multiplex multiple connections in a single thread.

3.6.3 Lock-Free Patterns

Where possible, Dwarf Star avoids locks. The GPU graph submission is designed to be non-blocking from the CPU side — the CPU submits work and continues without waiting for GPU completion. Status is checked only at well-defined points (between layers, between tokens).

For the server’s batched execution mode, sessions are evaluated in a fixed order within a work group. This avoids the need for per-session locks — each session is accessed by only one thread at a time, with work items handed off through a simple queue protected by a mutex.

3.7 Error Handling in C

3.7.1 No Exceptions, No Panic

C has no exceptions. Error handling is done through return codes:

int ds4_session_sync(ds4_session *s, const ds4_tokens *prompt, 
                     char *err, size_t errlen);

// Usage:
char err[256];
int rc = ds4_session_sync(session, prompt, err, sizeof(err));
if (rc != 0) {
    fprintf(stderr, "Session sync failed: %s\n", err);
    // Handle error
}

Every function that can fail returns an integer code (0 for success, negative for error, positive for special conditions) and writes a human-readable error message to a caller-provided buffer.

3.7.2 The Error Buffer Pattern

The char *err, size_t errlen pattern is used throughout Dwarf Star:

int ds4_engine_open(ds4_engine **out, const ds4_engine_options *opt);

But wait — this function doesn’t take an error buffer. Let’s check the actual API:

int ds4_engine_open(ds4_engine **out, const ds4_engine_options *opt);

Some functions return an integer and write error messages to stderr (via ds4_log). Others use the explicit error buffer pattern. The consistency is that all errors are reported somehow — nothing fails silently.

3.7.3 The Logging System

Dwarf Star has a structured logging system:

typedef enum {
    DS4_LOG_DEFAULT,
    DS4_LOG_PREFILL,
    DS4_LOG_GENERATION,
    DS4_LOG_KVCACHE,
    DS4_LOG_TOOL,
    DS4_LOG_WARNING,
    DS4_LOG_TIMING,
    DS4_LOG_OK,
    DS4_LOG_ERROR,
} ds4_log_type;

void ds4_log(FILE *fp, ds4_log_type type, const char *fmt, ...);

Each log message is tagged with a type, allowing callers to filter or colorize messages. The ds4_log_is_tty() helper detects whether the output is a terminal, enabling rich formatting when appropriate.

3.8 The Build System: Makefile Targets

3.8.1 Multi-Target Build

Dwarf Star’s Makefile supports multiple build targets:

make                  # macOS Metal (default)
make cuda-spark       # Linux CUDA, DGX Spark
make cuda-generic     # Linux CUDA, other GPUs
make strix-halo       # Linux ROCm, AMD Strix Halo
make cpu              # CPU-only diagnostics

Each target selects different source files, compiler flags, and link options. The Metal target compiles ds4_metal.m (Objective-C) and links against the Metal framework. The CUDA target compiles ds4_cuda.cu with nvcc and links the CUDA runtime.

3.8.2 Conditional Compilation

Conditional compilation (#ifdef) selects backend-specific code:

#ifdef __APPLE__
#include <sys/sysctl.h>
#endif

#ifdef DS4_TEST_HOOKS
int ds4_test_sample_logits(...);
#endif

The DS4_TEST_HOOKS flag is defined only in test builds, exposing additional entry points for unit testing without bloating the production binary.

3.8.3 The CPU Reference Build

The CPU build (make cpu) is explicitly a reference/debug path:

/* Do not treat the CPU path as the production target. */

The CPU backend uses the same API and session management as Metal and CUDA but performs all computation on the CPU. This is useful for:

• Debugging: Running on CPU avoids GPU driver complexities
• Validation: CPU results can be compared with GPU results for correctness
• Portability: A minimal working version exists on any platform

But the CPU backend is 10-100× slower than the GPU backends, limited by system memory bandwidth (~100 GB/s vs ~400 GB/s on Apple Silicon).

3.9 The GGUF Loader: Parsing Binary Formats

3.9.1 The GGUF Header

The GGUF file format begins with a header:

// Simplified GGUF header structure
typedef struct {
    uint32_t magic;           // "GGUF" magic number
    uint32_t version;         // Format version
    uint64_t tensor_count;    // Number of tensors
    uint64_t metadata_kv_count; // Number of metadata key-value pairs
} gguf_header;

Dwarf Star reads this header, then iterates through the metadata key-value pairs (which contain the model architecture name, quantization parameters, etc.) and the tensor directory (which lists all weight tensors with their names, shapes, and quantization types).

3.9.2 Validation

Before accepting any tensor, Dwarf Star validates:

1. Tensor name: Must match expected names for the model architecture (e.g., blk.0.attn_q.weight, blk.0.ffn_gate.weight)
2. Tensor shape: Must match the expected dimensions
3. Quantization type: Must be one of the supported types
4. Overall layout: The complete set of tensors must form a valid model

If validation fails, the engine returns an error immediately rather than crashing during inference.

3.9.3 The Tensor Directory

The tensor directory maps each tensor to its location in the file:

typedef struct {
    uint64_t name_offset;     // Byte offset of tensor name in string table
    uint64_t name_length;     // Length of tensor name
    uint32_t n_dims;          // Number of dimensions
    uint64_t dims[4];         // Dimensions (up to 4)
    uint32_t type;            // Quantization type (enum)
    uint64_t offset;          // Byte offset of tensor data in file
} gguf_tensor_info;

Dwarf Star reads the tensor directory, validates each entry, and records the offset and type of each tensor. The actual tensor data is not loaded — it remains in the mmap region until accessed.

3.10 The Instance Lock: Safety First

3.10.1 Preventing Concurrent Access

Running multiple Dwarf Star instances concurrently with large models would be disastrous — two processes competing for the same memory and GPU resources would thrash the system.

Dwarf Star uses an instance lock to prevent this:

/* The instance lock is intentional. */

While not visible in the API, the engine checks for existing instances using a file lock or similar mechanism and refuses to start if another instance is already running.

3.10.2 The Anti-Thrashing Design

Beyond the instance lock, Dwarf Star is designed to avoid thrashing internally:

• Power limit: The --power option limits GPU usage, reducing heat and memory contention
• Automatic cache sizing: The SSD streaming mode automatically sizes the expert cache based on available memory
• Conservative context sizing: The server refuses to start if the requested context size would exceed available memory
• Graceful degradation: If memory runs low during operation, the engine reports errors clearly rather than crashing

This safety-first approach is essential for a tool that users rely on for productive work.

3.11 Summary: A Systems Engineering Masterpiece

In this chapter, we’ve explored the C systems programming techniques that make Dwarf Star possible:

1. Manual memory management with scratch buffers, pre-allocated pools, and careful lifetime tracking
2. mmap-based model loading for instant startup and zero-copy GPU access
3. Metal graph encoding for efficient GPU scheduling with minimal CPU overhead
4. Function pointers and opaque types for clean API design
5. Thread synchronization for concurrent HTTP serving, GPU execution, and disk I/O
6. Error handling through return codes with structured logging
7. Conditional compilation for multi-platform support
8. Binary format parsing for GGUF compatibility
9. Safety mechanisms (instance lock, automatic sizing) to prevent resource exhaustion

Each of these techniques is a tool in the systems programmer’s belt. Dwarf Star uses them masterfully, creating an engine that is both efficient and reliable.

In the next chapter, we’ll look at how distributed inference works — connecting multiple machines to pool their memory and compute power.

Chapter 4 — Low-Latency Networking

— Distance in computing is measured in microseconds. A cable is a memory bus between machines.

So far, we’ve assumed that all computation happens on a single machine. But what if the model is too large for even the most powerful computer? What if you want to combine two Mac Studios to run a model that needs 512 GB of RAM?

This is where networking enters the picture. Dwarf Star can split a model across multiple machines, with activations flowing between them over a network connection. The challenge is making this fast enough to be useful.

In this chapter, we’ll explore how Dwarf Star implements distributed inference, the networking techniques it uses, and the fundamental trade-offs involved in splitting computation across machines.

4.1 Why Distributed Inference?

4.1.1 The Capacity Problem

The largest models — DeepSeek V4 Pro in 4-bit quantization — exceed 300 GB. Even the Mac Studio M3 Ultra, with 512 GB of RAM, can just barely hold such a model. But what if you need more context (larger KV cache) or want to run at higher precision?

With two machines, you can:

• Double the available memory: Each machine holds half the model layers
• Run models that don’t fit on one machine: The Pro Q4 model becomes feasible on two 128 GB MacBooks
• Scale to any size: In principle, you could chain N machines to run arbitrarily large models

4.1.2 The Prefill Speedup Surprise

There’s an unexpected benefit: distributed inference can actually speed up prefill. When processing a long prompt, the coordinator can process its chunk N+1 while the worker is still on chunk N — like an assembly line.

Without distribution (single machine):
  ┌───────┐   ┌───────┐   ┌───────┐   ┌───────┐
  │ Chunk │   │ Chunk │   │ Chunk │   │ Chunk │
  │   0   │──►│   1   │──►│   2   │──►│   3   │
  └───────┘   └───────┘   └───────┘   └───────┘
  Sequential: time = sum of all chunks

With distribution (2 machines, pipelined):
  Machine A: ┌──────┐   ┌──────┐   ┌──────┐
             │Chunk0│   │Chunk2│   │Chunk4│
             └──────┘   └──────┘   └──────┘
  Machine B:    ┌──────┐   ┌──────┐   ┌──────┐
                │Chunk1│   │Chunk3│   │Chunk5│
                └──────┘   └──────┘   └──────┘
  Pipelined: time ≈ (total_chunks / 2) × chunk_time + pipeline_fill

Dwarf Star’s benchmarks show speedups of 1.38× to 1.85× for long prefill with two machines. The longer the prompt, the better the speedup, because the pipeline fill/drain overhead is amortized.

4.1.3 The Generation Penalty

Generation (autoregressive decode) is a different story. Each token depends on the previous token. You cannot pipeline generation because token N+1 needs the output of token N.

So distributed generation is always slower than single-machine generation:

• Single machine: read weights, compute, output token
• Distributed (2 machines, 1 split): read weights (local), compute half, send to worker, worker computes half, worker sends back, output token

The extra network round trip adds latency. Dwarf Star measures a 19.4% loss for Thunderbolt connections and much larger losses for WiFi.

— Dwarf Star insight: The README is admirably honest: "Distributed inference is therefore mainly for fitting larger models and speeding up long prefills, not for making decode faster."

4.2 Pipeline Parallelism vs Tensor Parallelism

There are two fundamentally different ways to split a neural network across machines:

4.2.1 Pipeline Parallelism

In pipeline parallelism, each machine owns a contiguous range of transformer layers:

Machine A (layers 0-19)    Machine B (layers 20-39)    Machine C (layers 40-59)
┌────────────────────┐    ┌────────────────────┐      ┌────────────────────┐
│ Input → Layer 0    │    │ Layer 20 → Layer 21│      │ Layer 40 → ...    │
│ Layer 1 → Layer 2  │───►│ ...                │─────►│ ... → Layer 59    │
│ ... → Layer 19     │    │ → Layer 39         │      │ → Output          │
└────────────────────┘    └────────────────────┘      └────────────────────┘

Pros:

• Each machine only needs its own layers in memory
• Simple to implement — activations flow in one direction
• Good for capacity (fitting a larger model)

Cons:

• Generation latency = sum of all machines’ compute time + network hops
• Load imbalance if layers have different sizes

Dwarf Star uses pipeline parallelism as its primary distributed mode.

4.2.2 Tensor Parallelism

In tensor parallelism, each machine owns half of every layer’s weights. They compute their halves simultaneously and exchange partial results:

Machine A (first half of weights):
  Input → Compute half of attention heads → Send partial → Receive → Combine
  Input → Compute half of FFN → Send partial → Receive → Combine

Machine B (second half of weights):
  Input → Compute other half of attention heads → Receive → Combine → Send
  Input → Compute other half of FFN → Receive → Combine → Send

Pros:

• Both machines work on the same token simultaneously
• Reduces per-token latency (not just capacity)
• Better for generation speed

Cons:

• More complex synchronization (all-gather at every layer)
• All machines must have the full model’s dense weights replicated
• Requires high-bandwidth, low-latency interconnect (Thunderbolt, NVLink)

Dwarf Star supports tensor parallelism through both RDMA (for Mac-to-Mac Thunderbolt) and CUDA tensor parallelism (for multi-GPU servers). The Tensor Parallelism mode (TP) splits the heavy per-layer work between machines, exchanging 16-24 KB partial sums at synchronization gates.

4.2.3 Which One to Use?

Dwarf Star offers both, and the choice depends on your goal:

4.3 The Network Protocol

4.3.1 The Wire Protocol

Dwarf Star’s distributed protocol is a custom binary protocol over TCP. It’s designed for simplicity and low overhead rather than interoperability.

The basic unit of communication is a frame:

typedef struct {
    uint32_t magic;    // "DS4D" — identifies the protocol
    uint32_t type;     // Message type (HELLO, WORK, RESULT, etc.)
    uint32_t bytes;    // Payload size
} ds4_dist_frame_header;

Messages types include:

4.3.2 The HELLO Handshake

When a worker connects to the coordinator, it sends a HELLO message:

typedef struct {
    uint32_t model_id;
    uint32_t quant_bits;
    uint32_t layer_start;
    uint32_t layer_end;
    uint32_t has_output;
    uint32_t has_hidden;
    uint32_t ctx_size;
    uint32_t n_layers;
    uint32_t listen_port;
    uint32_t model_name_len;
} ds4_dist_hello_fixed;

This tells the coordinator:

• What model the worker has loaded (model_id)
• Which layers it owns (layer_start to layer_end)
• Whether it has the output head (has_output)
• What context size it supports (ctx_size)

The coordinator uses this information to build a route — the sequence of machines that input must pass through to traverse all layers.

4.3.3 The WORK Frame

When the coordinator sends work to a worker (or a middle worker forwards to the next worker), it sends a WORK frame:

typedef struct {
    uint32_t model_id;
    uint64_t session_id;       // Identifies the inference session
    uint64_t request_id;       // Identifies this specific request
    uint64_t prefix_hash;      // Rolling hash before this work unit
    uint64_t result_hash;      // Expected rolling hash after this work unit
    uint32_t pos0;             // Starting token position
    uint32_t n_tokens;         // Number of tokens
    uint32_t layer_start;      // Start of this worker's layer range
    uint32_t layer_end;        // End of this worker's layer range
    uint32_t flags;            // Control flags
    uint32_t token_bytes;      // Token ID data size
    uint32_t input_hc_bytes;   // Hidden state payload size
    uint32_t input_hc_bits;    // Bit width of hidden state (32, 16, or 8)
    uint32_t route_count;      // Total machines in the route
    uint32_t route_index;      // This machine's index in the route
    uint32_t route_bytes;      // Route information size
} ds4_dist_work_fixed;

The rolling hash (prefix_hash, result_hash) is a crucial safety feature. It verifies that the worker’s KV cache is in sync with the coordinator’s. If a worker has been restarted or its cache has perged, the hash will mismatch, and the coordinator can detect the inconsistency and replay the token history.

4.3.4 The RESULT Frame

After computation, the worker sends back a RESULT frame:

typedef struct {
    uint64_t request_id;
    uint64_t result_hash;
    uint32_t status;
    uint32_t result_kind;      // ACK, HIDDEN_STATE, or LOGITS
    uint32_t telemetry_count;
    uint32_t telemetry_bytes;
    uint32_t payload_bytes;
    uint32_t payload_bits;
} ds4_dist_result_fixed;

The final worker (the one with the output head) sends back logits. Middle workers send back just an ACK (for prefill pipeline) or the hidden state (for generation).

4.4 Activation Compression

4.4.1 The Bandwidth Problem

The hidden state (activations) that flows between machines is large. For DeepSeek V4 Flash with an embedding dimension of 2048 (approximately), each token’s hidden state is 2048 × 4 bytes = 8 KB in FP32. At 25 tokens/second with 2 machines, that’s 200 KB/s of activation data — easily handled by any network.

But during prefill, the coordinator sends multiple tokens at once. A chunk of 4096 tokens generates 4096 × 8 KB = 32 MB of hidden state per layer boundary. Over a Thunderbolt link with ~40 Gbps bandwidth, this takes about 6.4 ms — noticeable but acceptable.

Over WiFi or the Internet, this becomes a serious bottleneck.

4.4.2 Reducing Activation Precision

Dwarf Star offers activation precision reduction for the network transport:

typedef struct {
    // ...
    uint32_t activation_bits;   // 32, 16, or 8
    // ...
} ds4_distributed_options;

With --dist-activation-bits 16, the hidden state is sent as FP16 instead of FP32, halving the network traffic. With --dist-activation-bits 8, it’s quartered.

However, the README notes: “experimentally reduction activation size didn’t provide a significant improvement, so this option may be removed in the future.” This is because for most use cases (Thunderbolt or fast local networks), the network is not the bottleneck — the GPU compute is. The option remains for slow links where every byte counts.

4.4.3 The Quantization Trade-off

Reducing activation precision is a form of quantization applied to the hidden state between layers. Unlike weight quantization (which is permanent), activation quantization affects only the communication between machines and does not change the stored model.

The risk is that reduced precision introduces noise into the computation of downstream layers. Dwarf Star considers 16-bit to be safe (negligible quality impact) and 8-bit to be “approximate/experimental.”

4.5 Route Formation and Management

4.5.1 Dynamic Route Assembly

Workers can connect and disconnect dynamically. When a worker connects, the coordinator:

1. Receives the HELLO and records the worker’s capabilities
2. Checks if the worker fills a gap in the current layer coverage
3. If so, adds the worker to the active route
4. If the route is now complete (all layers covered), signals readiness

If a worker disconnects, the coordinator:

1. Removes the worker from the route
2. Marks the route as incomplete
3. Waits for a replacement worker to connect
4. Handles in-flight requests gracefully (they may fail)

This dynamic routing allows for fault tolerance — if one machine crashes, another can take its place without restarting the coordinator.

4.5.2 Route Information in WORK Frames

The WORK frame includes the full route information:

typedef struct {
    uint32_t host_len;     // Length of the hostname/IP
    uint32_t port;         // TCP port
    uint32_t layer_start;  // Start of this hop's layer range
    uint32_t layer_end;    // End of this hop's layer range
    uint32_t flags;        // Route flags
} ds4_dist_route_fixed;

Middle workers use this to forward data to the next machine. The route_index field in the WORK frame tells each machine its position in the chain:

Machine A (coordinator, index 0):
  Computes layers 0-19 → sends to Machine B (from route[1])

Machine B (worker, index 1):
  Receives → computes layers 20-39 → sends to Machine C (from route[2])

Machine C (worker, index 2):
  Receives → computes layers 40-output → sends logits back to Machine A

This chain is established at route formation time and reused for every work item, avoiding per-request negotiation.

4.6 RDMA: Remote Direct Memory Access

4.6.1 The Problem with TCP

TCP is a general-purpose protocol designed for reliability and fairness. It has significant overhead:

• Kernel involvement: Every send/receive goes through the operating system kernel
• Data copying: Data is copied from application → kernel → network card → kernel → application
• Protocol overhead: TCP headers, ACKs, congestion control
• Context switching: Each send/receive may cause a context switch, costing microseconds

For the tensor parallelism mode, where machines exchange 16-24 KB of partial sums at every layer for every token, TCP overhead becomes a significant fraction of total latency.

4.6.2 What Is RDMA?

Remote Direct Memory Access (RDMA) allows one computer to directly read or write the memory of another computer over a high-speed network, without involving either CPU or operating system.

TCP Path:
  Machine A                           Machine B
  ┌─────────────┐                      ┌──────────────┐
  │ Application │  send()   ┌───────┐  │ Application  │
  │ ─► Kernel   │──►Socket──►Network│──► Kernel       │
  │ ─► NIC      │  buffer   │       │  │ ─► App buffer│
  └─────────────┘           └───────┘  └──────────────┘
  Data copied 4× between user/kernel space

RDMA Path:
  Machine A                           Machine B
  ┌─────────────┐                     ┌─────────────┐
  │ Application │───RDMA Write──►     │ Application │
  │ (CPU bypass)│   (direct to RAM)   │ (CPU bypass)│
  └─────────────┘                     └─────────────┘
  Data copied 0× — direct from memory to memory

RDMA is supported by specialized network hardware (InfiniBand, RoCE, iWARP) and, importantly for Dwarf Star, by Thunderbolt networking on Macs.

4.6.3 RDMA over Thunderbolt

Apple’s Thunderbolt interface supports RDMA through the built-in networking stack. When two Macs are connected by a Thunderbolt cable, they can communicate using RDMA with extremely low latency (microseconds) and high bandwidth (~40 Gbps for Thunderbolt 5).

Dwarf Star supports RDMA for tensor parallelism:

typedef struct {
    ds4_tp_role role;             // LEADER or WORKER
    bool requested;               // --tensor-parallel flag
    const char *listen_host;
    int listen_port;
    const char *leader_host;      // Worker dials the leader
    int leader_port;
    ds4_tp_transport transport;   // AUTO, RDMA, or TCP
    const char *rdma_device;      // RDMA interface name
    int rdma_gid_index;           // GID for the RDMA address
    bool rdma_gid_index_set;
    bool glm_token_prefill;
    int debug_hash;
} ds4_tp_options;

The setup requires:

1. A Thunderbolt cable connecting the two Macs
2. An IPv4 address on the Thunderbolt member interface
3. The rdma_ctl and ibv_devinfo tools to verify connectivity
4. The worker starts first, then the coordinator

4.6.4 TCP vs RDMA Performance

Dwarf Star’s TP mode gracefully falls back to TCP if RDMA is unavailable. The performance difference depends on the link:

RDMA’s advantage is most pronounced for small messages (like the 16-24 KB partial sums in tensor parallelism), where TCP’s per-message overhead dominates. For bulk data transfer (like pipeline parallelism’s large hidden state batches), the difference is smaller.

4.7 KV Cache Consistency

4.7.1 The Consistency Challenge

In distributed inference, each machine owns its own slice of the KV cache. When the coordinator processes a token, it updates its local cache, sends activations to the worker, and the worker updates its local cache. Both caches must stay in sync.

But what happens if:

• A worker crashes and restarts (its cache is lost)?
• A new worker connects with a different layer assignment?
• The coordinator wants to save the combined cache to disk?

4.7.2 Rolling Hash Verification

Dwarf Star uses a rolling 64-bit hash of the token prefix to verify cache consistency:

typedef struct {
    // ...
    uint32_t prefix_hash_hi;
    uint32_t prefix_hash_lo;
    uint32_t result_hash_hi;
    uint32_t result_hash_lo;
    // ...
} ds4_dist_work_fixed;

Before each work unit, the coordinator sends the expected prefix hash. The worker computes its own hash of its cached token prefix and compares. If they match, the cache is in sync and computation proceeds. If they don’t match, the worker reports an error and the coordinator replays the token history.

4.7.3 Snapshot Save and Load

For persistence, Dwarf Star supports saving and loading the combined KV cache across all machines:

Save:
  Coordinator: Save my layers → Request workers → Each worker sends its layers
  → Coordinator merges into a single payload → Write to disk

Load:
  Coordinator: Read payload → Split by layer ranges → Send each worker its layers
  → Each worker loads its cache slice → Verify consistency → Resume inference

The distributed save/load uses the same DSV4 payload format as single-machine mode. This means saved sessions are portable — a distributed-created checkpoint can be loaded on a single machine (if it fits) and vice versa.

4.8 CUDA Tensor Parallelism

4.8.1 Multi-GPU Within One Machine

Dwarf Star also supports tensor parallelism across multiple GPUs in a single machine (using CUDA). This is different from the Mac-to-Mac RDMA mode:

/* CUDA tensor parallelism splits DeepSeek V4 Flash tensor and routed-expert
 * work across an even number of GPUs. This is separate from the Mac-to-Mac
 * mode above: it does not use --role, RDMA, or the distributed layer pipeline. */

In CUDA TP mode:

• GPUs are paired: first half are pipeline homes, second half are TP partners
• Each pair splits routed experts 50/50
• Dense weights (attention, shared experts) are replicated within each pair
• The vocabulary head is row-sharded across output tiers
• GPUs communicate through NVIDIA’s peer-to-peer (P2P) interconnect (NVLink or PCIe)

The device ordering is critical:

--gpu-devices 0,2,4,6,1,3,5,7

Homes:    0, 2, 4, 6  (first half)
Partners: 1, 3, 5, 7  (second half)

Pair (0,1), Pair (2,3), Pair (4,5), Pair (6,7)

Each pair shares the routed expert weights (half each) and communicates partial sums at every layer.

4.8.2 NVLink vs PCIe

NVIDIA GPUs can communicate through two paths:

• PCIe: The standard I/O bus (~32 GB/s for x16 Gen5)
• NVLink: A dedicated high-speed interconnect (up to 900 GB/s for H100)

When GPUs have NVLink, tensor parallelism is extremely efficient — the overhead of exchanging partial sums is small relative to computation. On systems without NVLink (like DGX Spark/GB10), tensor parallelism is not supported because PCIe bandwidth is insufficient.

4.9 Practical Setup and Debugging

4.9.1 Minimal Setup

The simplest distributed setup is one coordinator and one worker:

# Machine A (coordinator)
./ds4 -m model.gguf --role coordinator --layers 0:30 --listen 192.168.1.100 1234

# Machine B (worker)
./ds4 -m model.gguf --role worker --layers 31:output --coordinator 192.168.1.100 1234

Both machines need:

• The same model GGUF file at the same path (or different split files)
• The same Dwarf Star build (same commit)
• Network connectivity (ideally a direct high-speed link)

4.9.2 Debugging Tools

Dwarf Star provides debugging tools for distributed setups:

./ds4-bench --role coordinator --layers 0:19 --listen 169.254.43.68 1234 --debug

The --debug flag prints per-hop telemetry: layer range, token span, local evaluation time, downstream wait time, socket send time, and input/output byte counts.

For checking RDMA status:

rdma_ctl status          # Check RDMA device status
ibv_devinfo -v           # Verbose InfiniBand device info
ping 10.99.0.2           # Basic connectivity (does not prove RDMA)

4.9.3 Network Comparison

The README provides a detailed network comparison:

The Internet/VPN case is explicitly described as “not meant to be a good interactive experience” — but it enables the remarkable scenario of multiple people combining machines across the world to run a model that none of them could run alone.

4.10 Summary: The Network as Memory Bus

In distributed inference, the network becomes an extension of the memory bus. Instead of a physical wire connecting CPU to RAM, we have a physical wire (or wireless signal) connecting one machine to another.

The key trade-offs are:

1. Latency vs bandwidth: Network latency (0.45 ms for Thunderbolt) is measured in microseconds — fast compared to SSD latency (~5 µs) but slow compared to RAM latency (~100 ns). Distributed inference works when you can amortize this latency across a large batch of work (prefill chunks) or when the capacity gain is worth the speed loss.

2. Computation vs communication: The optimal split minimizes the data exchanged between machines. Pipeline parallelism exchanges hidden states (large, but infrequently — once per token or per chunk). Tensor parallelism exchanges partial sums (small, but frequently — once per layer per token).

3. Synchronization vs asynchrony: Pipelined prefill is asynchronous (machines work on different chunks simultaneously). Autoregressive generation is synchronous (all machines must finish one token before starting the next). The more synchronous the workload, the more network latency hurts.

4. Capacity vs speed: Distributed inference primarily solves a capacity problem (running models that don’t fit on one machine). Any speed improvement (in prefill) is a bonus, not the primary goal.

The network chapter completes our journey from the single transistor to the distributed cluster. In the final chapter, we’ll bring everything together by examining the transformer architecture itself — the mathematical structure that makes modern LLMs possible, and how Dwarf Star implements it.# Chapter 5 — Transformer Architecture: The Mind of the Machine

— A transformer is not a single thought — it's a conversation between every word and every other word, repeated until meaning emerges.

We’ve arrived at the heart of the matter. For four chapters, we’ve built up the foundation: the computer architecture that runs the model, the quantization that compresses it, the systems programming that orchestrates it, and the networking that connects multiple machines. Now we finally zoom in on the transformer itself — the neural network architecture that powers modern LLMs.

This chapter explains the transformer from the ground up, with specific attention to the features that make Dwarf Star special: Mixture-of-Experts (MoE), the KV Cache, and the crucial difference between Prefill and Decode. By the end, you’ll understand not just what a transformer does, but why it works the way it does, and how Dwarf Star’s design choices map to the mathematics.

5.1 From Language to Vectors

5.1.1 The Representation Problem

Before we can process language with a computer, we need to convert words into numbers. This is the representation problem.

A naive approach would be to assign each word a unique integer ID: “the” = 1, “cat” = 2, “sat” = 3, etc. But this loses all semantic information — the difference between 2 and 3 is the same as the difference between 1 and 1000, which makes no sense linguistically.

The breakthrough of modern NLP was dense vector representations (embeddings). Each word is represented as a vector of floating-point numbers (e.g., 4096 dimensions), where semantically similar words have similar vectors.

"king"   = [0.32, -0.15, 0.87, ..., 0.04]
"queen"  = [0.31, -0.14, 0.85, ..., 0.03]
"apple"  = [0.12, 0.45, -0.23, ..., 0.71]

"king" and "queen" are close (similar meaning)
"king" and "apple" are far (different meaning)

These vectors are learned during training. The embedding table — a matrix of size vocabulary × embedding_dim — is one of the model’s weight matrices. Dwarf Star stores it at FP16 precision because even small errors in embeddings can cause dramatic shifts in meaning.

5.1.2 Tokenization: Breaking Text into Pieces

Modern LLMs don’t work with words directly. They use subword tokenization — breaking text into a vocabulary of common character sequences (tokens). A token might be a word (“cat”), a subword (“un-”, “-believe-”, “-able”), or even a single character.

For DeepSeek V4 Flash, the vocabulary has about 128K tokens. Each token has a unique ID and a corresponding embedding vector.

The tokenizer converts text to token IDs:

"The cat sat on the mat" → [892, 4713, 8542, 567, 892, 3981]

And the model’s first layer converts these IDs to vectors:

[892, 4713, ...] → embeddings → [[0.32, -0.15, ...], [0.87, 0.04, ...], ...]

5.2 The Transformer Architecture

5.2.1 High-Level View

The transformer was introduced in the landmark 2017 paper “Attention Is All You Need” by Vaswani et al. At its core, it’s a sequence-to-sequence model that transforms an input sequence of vectors into an output sequence of vectors through a stack of identical layers.

Input tokens → Embedding → [Layer 0 → Layer 1 → ... → Layer N] → Output logits

Each layer consists of two main sub-layers:

1. Multi-Head Self-Attention (MHSA): Allows each token to “look at” all other tokens in the sequence and incorporate their information.
2. Feed-Forward Network (FFN): Processes each token’s representation independently through a wider hidden dimension.

These are connected with residual connections (skip connections) and layer normalization:

    ┌──────────────────────────────────────────────┐
    │            Layer N                           │
    │                                              │
    │  Input ─► LayerNorm ─► MHSA ──⊕──► LayerNorm│
    │                ↑           │          │      │
    │                └──Residual─┘          │      │
    │                                       │      │
    │                               ┌───────┘      │
    │                               ▼              │
    │                         ─► FFN ──⊕──► Output│
    │                              ↑    │          │
    │                              └────┘          │
    └──────────────────────────────────────────────┘

For DeepSeek V4 Flash, there are 60 such layers (approximately). Each layer processes the full sequence of tokens, with the output of one layer becoming the input to the next.

5.2.2 Multi-Head Self-Attention (MHSA)

This is the core innovation of the transformer. Let’s understand it step by step.

Step 1: Projection

For each token’s vector x (dimension d_model), we compute three projections:

Query:    Q = x × W_Q     (d_model → d_head)
Key:      K = x × W_K     (d_model → d_head)
Value:    V = x × W_V     (d_model → d_head)

The Query, Key, and Value are the fundamental concepts of attention. Think of it as a database operation:

• Query: What am I looking for?
• Key: What information do I have?
• Value: The actual information content

Step 2: Attention Scores

For each query token i and key token j in the sequence, compute the attention score:

score(i, j) = Q_i · K_j / sqrt(d_head)

This is a dot product — a measure of similarity between the query and key. The pision by sqrt(d_head) prevents the dot products from growing too large (which would push the softmax into extreme values).

Step 3: Attention Weights

Apply softmax to normalize the scores into probabilities:

weight(i, j) = exp(score(i, j)) / Σ_k exp(score(i, k))

This gives each token a probability distribution over all other tokens, indicating how much to “attend” to each.

Step 4: Weighted Sum

Compute the output of attention as the weighted sum of values:

output_i = Σ_j weight(i, j) × V_j

This is the final attended representation — each token now contains information from all other tokens, weighted by relevance.

Step 5: Multi-Head

Instead of doing this once, the transformer does it h times in parallel (typically 32 heads), with different learned projections. The outputs are concatenated and projected back to d_model:

MultiHead(Q, K, V) = Concat(head_1, ..., head_h) × W_O

Each head can learn to attend to different aspects of the input — syntactic relationships, semantic connections, positional patterns, etc.

5.2.3 DeepSeek’s Multi-Head Latent Attention (MLA)

Dwarf Star implements DeepSeek’s Multi-Head Latent Attention (MLA), an improved attention mechanism. In standard multi-head attention, the KV cache stores one set of keys and values per head per layer — which grows large quickly.

MLA compresses the keys and values into a lower-dimensional latent space before caching:

Standard Attention:
  K = x × W_K     (d_model → d_head × n_heads)  ← Stored in KV cache
  V = x × W_V     (d_model → d_head × n_heads)  ← Stored in KV cache

MLA:
  C = x × W_C     (d_model → d_latent)            ← Stored in KV cache (compressed!)
  K = C × W_K_UP  (d_latent → d_head × n_heads)  ← Reconstructed on the fly
  V = C × W_V_UP  (d_latent → d_head × n_heads)  ← Reconstructed on the fly

The key insight: the latent representation C is much smaller than the full K and V, typically 1/4 the size. The “up-projection” matrices (W_K_UP, W_V_UP) are part of the model weights and are applied during attention computation.

This is the “compressed KV cache” we discussed in Chapter 2. The ds4_engine_layer_compress_ratio() function returns 4 for MLA, meaning the KV cache is 4× smaller than it would be with standard attention.

— Dwarf Star insight: The compressor and indexer tensors in MLA are kept at FP16 precision. They are small (on the order of a few MB per layer) compared to the main weight matrices, and their precision is critical for correct attention computation. Dwarf Star's decision to keep them at FP16 is a deliberate quality-preserving choice.

5.2.4 The Feed-Forward Network (FFN)

After attention, each token goes through a Feed-Forward Network. In a standard transformer, this is two matrix multiplications with a non-linear activation in between:

FFN(x) = Activation(x × W_1 + b_1) × W_2 + b_2

Typically, W_1 projects from d_model to a larger d_ff (e.g., 4× or 8× wider), and W_2 projects back. The activation function (typically SwiGLU, SiLU, or GELU) introduces non-linearity, allowing the network to learn complex patterns.

In a standard dense transformer, every token uses the same FFN weights. This means all model parameters are active for every token — computationally expensive and memory intensive.

5.3 Mixture-of-Experts (MoE)

5.3.1 The Problem with Dense Models

A dense transformer uses all its parameters for every token. If the model has 671 billion parameters (like DeepSeek V4), every forward pass must process all 671 billion parameters. This is:

• Extremely slow (each token requires trillions of operations)
• Memory intensive (all parameters must be in RAM)
• Energy intensive (every parameter is read and written)

But do we really need all parameters for every token? Intuitively, understanding a question about Python syntax requires different “knowledge” than a question about medieval history. Why activate the entire model for every query?

5.3.2 The MoE Architecture

Mixture-of-Experts replaces the large dense FFN with multiple smaller “expert” FFNs and a router that selects which experts to use for each token.

Standard FFN:          MoE FFN:
                        ┌──────────┐
Input ─► FFN ─► Output  │  Router  │
                        └────┬─────┘
                    ┌────────┼────────┐
                    ▼        ▼        ▼
               ┌────────┐┌────────┐┌────────┐
               │Expert 0││Expert 1││Expert 2│ ... (256 experts)
               └────────┘└────────┘└────────┘
                    │        │        │
                    └────────┼────────┘
                             ▼
                       Weighted Sum
                             │
                             ▼
                          Output

Each expert is itself a small FFN (typically 2-4× smaller than the dense equivalent). The router is a small learned network that computes a probability distribution over experts for each token:

router(x) = softmax(x × W_router)

The top-k experts (with highest router probabilities) are activated, and their outputs are weighted by the router probabilities and summed:

MoE(x) = Σ_{i ∈ top-k} router(x)_i × expert_i(x)

For DeepSeek V4 Flash:

• 256 experts total
• 6 experts activated per token (top-6 routing)
• 1 shared expert always active (a dense FFN that runs alongside)
• Expert weight scale: 1.5 (a multiplier on router probabilities)

5.3.3 Why MoE Works

MoE is effective because of sparsity: most experts are not needed for any given token. Each expert specializes in a particular type of knowledge or pattern:

• Expert 47 might specialize in Python code patterns
• Expert 123 might specialize in chemical formulas
• Expert 200 might specialize in Italian literature

When the router sees “print(‘hello’)”, it activates expert 47 (Python) alongside a few others. When it sees “H₂SO₄”, it activates expert 123 (chemistry).

This specialization emerges naturally during training. The router learns to assign tokens to experts, and experts learn to process the tokens they receive well. There’s no explicit labeling of experts — they self-organize.

5.3.4 The Shared Expert

Dwarf Star keeps one shared expert that is always active, regardless of the router’s decisions. This expert is analogous to the dense FFN in a standard transformer — it provides a baseline level of processing that every token receives.

The shared expert is kept at Q8_0 (8-bit) precision, higher than the routed experts (2-bit). This ensures that every token gets high-quality processing from the shared expert, while the routed experts provide specialized knowledge.

5.3.5 Routing: The Genius and the Fragility

The router is a small network — just a matrix multiplication from d_model to n_experts (256). But it’s arguably the most important part of the MoE. If the router makes a mistake, the wrong experts are activated, and the output can be nonsensical.

Dwarf Star keeps the router weights at FP16 precision — the highest precision in the model. The rationale is clear: even a tiny error in routing could select the wrong expert entirely, causing catastrophic failure.

The readout of the router is also critical. The expert weight scale of 1.5 means that router probabilities are multiplied by 1.5 before softmax, making the distribution more peaked (more confident routing).

5.3.6 Load Balancing

If the router always selects the same few experts, those experts become overloaded while others atrophy. To prevent this, MoE training includes a load balancing loss that encourages the router to distribute tokens evenly across experts.

The load balancing loss is an auxiliary term added to the main training loss:

L_balance = α × n_experts × Σ_i (f_i × P_i)

Where:

• f_i is the fraction of tokens routed to expert i
• P_i is the average router probability for expert i
• α is a hyperparameter (typically 0.01)

This loss is minimized when tokens are uniformly distributed across experts.

5.3.7 The MoE Kernel in Metal

Dwarf Star’s MoE computation is implemented in highly optimized Metal kernels (metal/moe.metal). The kernel:

1. Reads the router weights and computes the top-6 expert selections
2. Gathers the selected experts’ weight matrices from the mmap region
3. Dequantizes the weights (IQ2_XXS or Q2_K) on the fly
4. Computes the expert FFN (gate projection → activation → down projection)
5. Multiplies by the router weights and accumulates

The kernel handles the complex indexing required for sparse expert selection — only 6 out of 256 experts are loaded and computed per token, saving 97.7% of the FFN computation.

5.4 The KV Cache: Memory for Context

5.4.1 Why We Need It

We discussed the KV cache in Chapter 2 from a memory perspective. Now let’s understand it architecturally.

When generating token N, the model needs to attend to all N-1 previous tokens. Without a cache, for each new token we would recompute the keys and values for all previous tokens — an O(N²) cost that makes long sequences impossible.

The KV cache stores the keys and values from the attention computation for each previous token. When processing token N:

1. Compute its own K and V
2. Append to the cache
3. Compute attention using all cached K and V

This makes generation O(N) per token instead of O(N²).

5.4.2 How the KV Cache Works in Dwarf Star

For DeepSeek V4 Flash with MLA, the KV cache stores the compressed latent representation for each token:

// For each layer, for each token position:
typedef struct {
    float latent[DS4_CACHE_LATENT_DIM];  // Compressed K and V
    // Additional metadata for the indexer
} ds4_kv_cache_entry;

The cache is organized as:

• Per-layer: Each transformer layer has its own KV cache
• Sliding window: A ring buffer of the most recent tokens (for the “raw” cache)
• Compressed storage: The full cache at 4× compression

Dwarf Star’s ds4_session struct manages the KV cache:

int ds4_session_create(ds4_session **out, ds4_engine *e, int ctx_size);

The ctx_size parameter determines how many tokens can be cached. A larger context means a larger KV cache — and more memory usage.

5.4.3 KV Cache Lifecycle

The KV cache has a well-defined lifecycle in Dwarf Star:

1. Creation: Allocated when the session is created, sized for the requested context
2. Append: Each new token’s K and V are appended during generation
3. Sync: When the user provides a new prompt, the cache is synced — if the prompt is a continuation, only the new tokens are processed; if it perges, the cache is rewound and rebuilt
4. Save: The entire cache can be serialized to disk (for session persistence)
5. Load: A previously saved cache can be loaded, avoiding reprocessing of the prefix
6. Free: Released when the session is destroyed

The disk KV cache (--kv-disk-dir) extends this to multiple sessions, allowing the server to switch between conversations without reprocessing.

5.4.4 Cache Misses and Rewriting

When a user sends a new message that differs from the cached prefix, Dwarf Star must handle the mismatch. The ds4_session_common_prefix() function finds the longest common prefix between the cached tokens and the new prompt.

If the common prefix is long (e.g., the user adds one more sentence to the conversation), only the new suffix is processed — the cached prefix is reused. This is the fast path.

If the common prefix is short (e.g., the user starts a new topic), the cache is rewound and rebuilt from scratch. This is the slow path but is unavoidable.

Dwarf Star has three levels of cache reuse:

1. Exact token match (fastest): The new prompt token-by-token matches the cached prefix
2. Byte-level match (fast): The rendered text matches the cached text, even if retokenization produces slightly different token sequences
3. No match (slow): Full reprocessing required

5.5 Prefill vs Decode: Two Very Different Workloads

5.5.1 What Is Prefill?

Prefill (also called “context processing”) is the first step of inference. Given a prompt of N tokens, the model processes all N tokens in parallel to build the initial KV cache and produce the first output token.

During prefill:

• Input: N token embeddings
• Computation: Full transformer forward pass on all N tokens
• Parallelism: All tokens are processed simultaneously within each layer
• Output: KV cache populated for all N tokens, plus logits for the next token

Prefill (N tokens):
  ┌───┐
  │T1 │──┐
  ├───┤  │  ┌─────────────────┐
  │T2 │──┤  │  Layer 0        │  ┌─────────────────┐      ┌────────┐
  ├───┤  ├──► (all tokens     ├──►  Layer 1        ├──...──► Logits│
  │T3 │──┤  │  simultaneously)│  │  (all tokens)   │      │ (next  │
  ├───┤  │  └─────────────────┘  └─────────────────┘      │ token) │
  │...│──┘                                                └────────┘
  └───┘

The key characteristic of prefill: it’s compute-bound for large N and memory-bound for small N. For very long prompts (10K+ tokens), the computation of attention (O(N²) in the sequence length) dominates. For short prompts, reading the weights dominates.

5.5.2 What Is Decode?

Decode (also called “generation” or “inference”) is the autoregressive step. Given the current KV cache, the model generates one token at a time, appending to the cache.

During decode:

• Input: One new token embedding
• Computation: Full transformer forward pass on one token (but attending to all cached tokens)
• Parallelism: None — each token depends on the previous token’s output
• Output: One token, appended to the generated sequence

Decode (N tokens generated one by one):
  Step 1:                       Step 2:                       Step 3:
  ┌─────────────────┐           ┌─────────────────┐           ┌─────────────────┐
  │KV cache (prompt)│           │KV cache + T_g1  │           │KV cache + ...   │
  │                 │           │                 │           │                 │
  │T_g1 ─► Layer 0  │           │T_g2 ─► Layer 0  │           │T_g3 ─► ...      │
  │(attend to all   │           │(attend to all   │           │                 │
  │ prompt tokens)  │           │ prompt + T_g1)  │           │                 │
  └─────────────────┘           └─────────────────┘           └─────────────────┘
       │                              │                              │
       ▼                              ▼                              ▼
     "The"                         "The cat"                     "The cat sat"

The key characteristic of decode: it’s memory-bandwidth-bound. Each decode step reads all model weights from memory to compute the forward pass for one token. The ratio of computation to memory access is very low.

5.5.3 The Performance Gap

This is why prefill is much faster than decode in terms of tokens/second:

                    Prefill (large prompt)        Decode
─────────────────────────────────────────────────────────────
Tokens processed     10,000 in parallel          1 at a time
Weights read         1× (once)                   1× per token
Memory bandwidth     ~400 GB/s                   400 GB/s
Compute utilization  High (large matrices)        Low (narrow matrices)
Effective speed      250-460 t/s                 21-37 t/s

The factor of 10-20× difference is intrinsic to the transformer architecture. It’s not a bug — it’s a fundamental property of autoregressive generation.

5.5.4 How Dwarf Star Optimizes Each Phase

Prefill optimizations:

• Chunked processing: Long prompts are split into chunks (default 4096 tokens) and processed incrementally, reducing peak memory and enabling progress reporting
• Pipelined prefill (distributed): Multiple machines work on different chunks simultaneously
• KV cache compression: MLA reduces the memory needed to store the growing KV cache
• Parallel attention: Flash attention kernels compute attention for all tokens efficiently

Decode optimizations:

• Weight quantization: 2-bit weights make the memory bottleneck 16× less severe
• Specialized kernels: The Metal graph encodes the complete decode step as a single optimized operation
• KV cache reuse: The decode step only computes the new token’s K and V, reading the rest from cache
• Speculative decoding (MTP/DSpark): A smaller “draft” model proposes multiple tokens, which the main model verifies in one batch. If accepted, multiple tokens are generated per pass.

5.5.5 Speculative Decoding: DSpark and MTP

Dwarf Star supports two forms of speculative decoding:

MTP (Multi-Token Prediction): A separate small GGUF file (about 5.6 GB) proposes up to 5 future tokens based on the main model’s hidden states. The main model verifies these proposals in one batch.

DSpark: Similar to MTP but uses a specialized “draft” model released by DeepSeek specifically for DeepSeek V4 Flash. It reads hidden states from the main model and proposes future tokens with a confidence score.

Without speculation:
  Step N:     "The" → compute → verify → accept
  Step N+1:   "cat" → compute → verify → accept
  Step N+2:   "sat" → compute → verify → accept
  ... 3 steps for 3 tokens

With speculation (DSpark):
  Step N:     "The" → compute + draft proposes "cat sat on"
              → verify all 3 in one pass → accept "cat sat on"
  ... 1 step for 3 tokens (2 tokens effectively free!)

The draft model is much smaller than the main model, so its proposal computation is cheap. The verification pass is nearly as expensive as a normal decode, but because it processes multiple tokens at once, the overhead per token is lower.

DSpark’s --dspark-confidence 0.9 parameter sets a threshold: if the confidence in the proposed suffix drops below 90%, the model falls back to normal token-by-token generation for the remaining tokens.

5.6 The Full Inference Pipeline

Let’s trace the complete path of a single inference request through Dwarf Star:

5.6.1 Input Processing

1. Tokenization: The input text is converted to token IDs using the model’s tokenizer (BPE or sentencepiece). The tokenizer is built into Dwarf Star — no external dependency.

2. Embedding: Each token ID is looked up in the embedding table (stored at FP16, memory-mapped from the GGUF file).

3. Positional encoding: DeepSeek V4 uses Rotary Position Embedding (RoPE), which encodes position information by rotating the query and key vectors. This allows the model to know where each token is in the sequence.

5.6.2 Layer-by-Layer Processing

For each of the ~60 layers:

1. Layer normalization (RMSNorm): Normalize the input to stabilize training and inference.

2. Attention (MLA):

   • Compute Q from the input
   • Compute compressed K and V from the input using the compression matrix
   • Append compressed K and V to the KV cache
   • Reconstruct full K and V from the compressed representation
   • Compute attention scores (Q dot K for all cached positions)
   • Apply RoPE to Q and K (rotary position embeddings)
   • Weighted sum of V according to attention scores
   • Project output back to d_model

3. Residual connection: Add the input to the attention output (skip connection).

4. Layer normalization (RMSNorm): Normalize again.

5. Mixture-of-Experts FFN:

   • Compute router logits: x × W_router
   • Select top-6 experts
   • For each selected expert:
     • Gate projection: x × W_gate (quantized IQ2_XXS)
     • Activation (SwiGLU)
     • Down projection: result × W_down (quantized Q2_K)
   • Weighted sum of expert outputs by router probabilities
   • Add shared expert output (Q8_0)
   • Apply expert weight scale (1.5)

6. Residual connection: Add the input to the MoE output.

5.6.3 Output Processing

After all layers:

1. Final layer normalization: Apply one more RMSNorm.

2. Output projection (also called “unembedding” or “lm_head”): Multiply the final hidden state by the output weight matrix (vocab_size × d_model) to produce logits — one score for each token in the vocabulary.

3. Sampling: Apply the sampling strategy to select the next token from the logits:

   • Greedy: Select the token with the highest logit (deterministic)
   • Temperature sampling: Scale logits by temperature, then sample from the resulting probability distribution
   • Top-K/Top-P sampling: Restrict sampling to the top K tokens or to tokens with cumulative probability ≥ P
   • Min-P sampling: Restrict to tokens with probability ≥ min_p × max_probability

Dwarf Star’s default sampling uses temperature=1, top_p=1, min_p=0.05 — a relatively open sampling that allows the model to be creative while filtering out extremely unlikely tokens.

5.6.4 The Loop

For generation, steps 5.6.2 and 5.6.3 are repeated for each new token:

• The new token is appended to the KV cache
• The entire transformer processes it (with attention to all cached tokens)
• A new token emerges from sampling
• Repeat until stop condition (EOS token, max tokens, or stop sequence)

For prefill, the loop is amortized: all prompt tokens are processed in parallel (within each layer), and only the final step produces the first generated token.

5.7 The Special Sauce: What Makes DeepSeek V4 Flash Special

5.7.1 Multi-Token Prediction (MTP)

DeepSeek V4 Flash was trained with an auxiliary objective: predict multiple future tokens, not just the next one. This produces better hidden representations and enables speculative decoding.

During training, the model learns to predict:

• Token N+1 (main objective)
• Token N+2 (auxiliary, using the prediction of N+1 as context)
• Token N+3 (auxiliary)

This multi-token prediction head is separate from the main output and is used only for speculation, not for the final output.

5.7.2 The Shared Expert Layer

Unlike some MoE architectures that put MoE only in alternating layers, DeepSeek V4 Flash uses MoE in every layer with a shared expert that is always active. This provides a strong baseline computation that every token receives, with the routed experts adding specialized knowledge.

5.7.3 The Router Design

The router uses a learned temperature (the expert weight scale of 1.5) that sharpens the probability distribution. This encourages more confident routing — each token is strongly assigned to its top experts rather than being spread thinly across many.

The router also supports load balancing through an auxiliary loss, but at inference time, the router is purely feed-forward — it does whatever it was trained to do.

5.7.4 KV Cache Compression Ratio

With MLA, the KV cache is compressed by 4×. This is not just a memory optimization — it also speeds up attention computation during decode because less data needs to be read from memory for each attention step.

5.7.5 Robustness to Quantization

Not all models tolerate 2-bit quantization well. DeepSeek V4 Flash was apparently trained in a way that makes its weights robust to aggressive quantization. The exact reasons are an active area of research, but plausible factors include:

• MoE sparsity: Each expert sees perse training signals, making inpidual weights less critical
• Large model size: With 671B total parameters, redundancy is abundant
• Training dynamics: The training process may naturally produce weights that cluster into quantizable levels

5.8 The Model Zoo: Flash vs Pro vs GLM

Dwarf Star supports three model families:

DeepSeek V4 Flash

The primary target. A 671B parameter MoE model with:

• 256 routed experts, 6 active per token
• 1 shared expert
• Multi-Head Latent Attention (4× KV cache compression)
• 60 layers (approximately)
• Context window: up to 1M tokens (with compressed KV cache)
• Quantized to 2 bits for consumer hardware

DeepSeek V4 Pro

The larger sibling. More parameters, more experts, higher quality:

• Same architecture as Flash but with more parameters
• Requires 512 GB+ machines for Q2 quantization
• Can be split across two machines for Q4 quantization
• The Q4 Pro model requires the full distributed setup

GLM 5.2

A different model family (by Zhipu AI/GLM):

• Uses a similar MoE architecture
• Different tensor layouts and quantization requirements
• Dwarf Star’s support is more limited (no thinking mode, no --power below 100)
• Runs on the same Metal/CUDA/ROCm backends

Dwarf Star validates the model architecture at load time. A GGUF file intended for GLM but loaded as DeepSeek would fail early, and vice versa.

5.9 Putting It All Together

We’ve now traced the complete journey of a token through Dwarf Star:

1. Text is tokenized into IDs using the learned vocabulary (Chapter 5)
2. IDs are embedded into vectors from the FP16 embedding table (Chapter 2/5)
3. Vectors pass through ~60 layers, each containing:
   • Multi-Head Latent Attention using the KV cache (Chapter 2/5)
   • Mixture-of-Experts FFN with 2-bit quantized experts (Chapter 2)
   • Routed via FP16 router weights (Chapter 2)
4. Output logits are computed and sampled to produce the next token (Chapter 5)
5. During generation, the Metal graph executes the entire pipeline (Chapter 3)
6. If distributed, activations flow between machines over TCP/RDMA (Chapter 4)
7. The KV cache grows with each token, enabling efficient continuation (Chapter 2/5)

Every optimization we’ve explored — asymmetrical quantization, mmap-based loading, the Metal graph, SSD streaming, KV cache compression, distributed inference — serves this pipeline. Each one addresses a specific bottleneck in the flow of data from disk to GPU to output.