I Ran a 284B-Parameter LLM From 3.2GB of RAM — in Plain C

Introduction

The headline reads like a benchmark trick, but the engineering behind it is straightforward constraint management. A 284-billion parameter transformer requires roughly 568 GB of RAM in FP16. Even aggressive 4-bit quantization leaves you at 142 GB. Fitting that workload into 3.2 GB of physical memory is impossible if you treat the model as a static data structure. It becomes entirely feasible when you treat it as a streaming data pipeline.

We built an out-of-core inference engine in plain C that never loads more than a single transformer block into RAM at any given moment. The system memory-maps the weight file, schedules I/O prefetching ahead of the compute thread, dequantizes blocks on demand, and feeds them directly into fused SIMD kernels. The KV cache and activation buffers are tightly packed and manually aligned to avoid allocator fragmentation. We bypassed the memory wall by making disk bandwidth the primary constraint instead of RAM capacity.

This approach strips away framework overhead, removes runtime dependencies, and gives explicit control over cache lines, page faults, and instruction pipelines. What follows is the architectural breakdown of how we structured the pipeline, the exact C patterns we used, and the production trade-offs you will face when pushing inference to the edge of commodity hardware.

Why This Matters

Inference costs are scaling faster than memory density. Deploying foundation models in production requires either expensive GPU clusters or highly optimized CPU fallback paths. Relying on high-level frameworks often hides memory allocation patterns, GC pauses, and unnecessary data copies that become critical at scale.

Running a 284B model on 3.2 GB of RAM solves three concrete production problems:

1. Cost reduction: Commodity x86/ARM servers with fast NVMe storage can handle large models without dedicated accelerator cards.
2. Privacy and compliance: Data never leaves the host machine, and the small memory footprint simplifies container isolation and sandboxing.
3. Deterministic performance: Plain C with explicit memory management eliminates non-deterministic pauses, making throughput predictable under load.

Engineers care about this because it demonstrates how constraint-driven design forces better memory hierarchy utilization, tighter I/O scheduling, and more efficient compute kernels. The same patterns apply to video transcoding, large-scale log processing, and any workload where data exceeds available memory.

How It Works

The inference pipeline operates as a continuous stream rather than a batch load. We divide the model into fixed-size blocks (typically 128M parameters per slice). The scheduler reads ahead from the disk, places blocks into a ring buffer, and signals the compute thread when data is ready. The compute thread dequantizes, performs matrix multiplication, updates the KV cache, and discards the block immediately. Memory usage stays flat because we never retain more than one active block plus the KV cache and activation buffers.

flowchart TD
  A[SSD Model Weights / 284B Params] --> B[mmap File Descriptor]
  B --> C[Block Scheduler & Prefetch Queue]
  C --> D[On-Demand Dequantizer]
  D --> E[AVX-512 Fused MatMul Kernel]
  E --> F[Activation Router]
  F --> G[Compressed KV Cache Ring Buffer]
  G --> H[Token Sampler & Output]
  H --> I[Next Prompt / Context Window]
  I --> C

The scheduler runs on a dedicated thread and uses madvise to hint sequential access patterns to the kernel. The dequantizer converts stored integer weights back to FP16/FP32 in cache-aligned buffers. The fused kernel computes the forward pass without intermediate allocations. The KV cache ring buffer evicts oldest positions once the context window exceeds the allocated limit. The sampler generates the next token, which loops back to trigger the next block load. Throughput is bounded by sequential read bandwidth and CPU decode width, not by RAM capacity.

Core Concepts

Out-of-Core Computation We treat the model as a stream of contiguous memory blocks. The engine maintains a cursor pointing to the current layer slice. When the compute thread finishes a block, the cursor advances, and the scheduler prefetches the next slice. No global model state exists in RAM.

Block-Wise Quantization Weights are stored as 4-bit integers with per-block scale and zero-point metadata. This reduces storage by 4x while preserving activation distribution. We use affine quantization: FP16_val = scale * (INT4_val - zero_point). The dequantizer applies this formula in parallel using SIMD broadcast instructions.

Memory-Mapped I/O mmap replaces traditional fread loops. The kernel handles page caching, and we leverage PROT_READ | MAP_PRIVATE to avoid copy-on-write overhead. We align mappings to 2MB huge pages where available to reduce TLB pressure.

KV Cache Compression The key-value cache grows linearly with context length. We compress it using rank-2 approximation and store it as INT8. A circular buffer manages eviction. When the context exceeds the limit, oldest tokens are purged, and the buffer wraps without reallocation.

Fused Compute Kernels Matmul, dequantization, and residual addition are fused into a single kernel. This eliminates temporary buffer allocations and keeps data in L1/L2 cache. We use loop tiling to match cache line sizes and prevent cache thrashing.

Examples & Code Walkthrough

The following snippets show the core structures and dispatch loop. They are production-ready, include defensive error handling, and avoid dynamic allocation in the hot path.

#include <sys/mman.h>
#include <stdint.h>
#include <stdbool.h>
#include <string.h>
#include <errno.h>

#define BLOCK_SIZE        (128 * 1024 * 1024) // 128MB per slice
#define CACHE_LINE_SIZE   64
#define MAX_KV_SLOTS      4096

// Quantization metadata per block
typedef struct {
    float scale;
    int32_t zero_point;
    uint32_t row_count;
    uint32_t col_count;
} quant_block_meta_t;

// Memory-mapped model descriptor
typedef struct {
    void *map_addr;
    size_t file_size;
    off_t cursor;
    int fd