Zero-Mem: Zero-Token Memory Operations for LLM Agents

Introduction

Every LLM agent that operates beyond a single turn faces the same architectural tension: memory is essential, but it’s expensive. Storing conversation history, retrieval results, or agent state in the context window consumes tokens — tokens that cost money, add latency, and compete with the actual work the model needs to do.

Zero-Mem is a pattern and set of techniques for managing agent memory with zero token cost during read and write operations. Instead of serializing memory into the LLM’s context, agents interact with memory through external structures that are resolved outside the inference loop. The LLM never sees the memory content directly — it operates on references, handles, or structured pointers that are resolved by the runtime system.

This isn’t a theoretical paper. It’s a pragmatic response to a very real production bottleneck that anyone building multi-turn agent systems hits within weeks.

Why This Matters

If you’re building agent systems today, you already know the pain:

• Context window bloat. A typical 128K-token context window fills up fast when agents maintain conversation history, tool results, retrieved documents, and episodic memory. You’re burning 40–60% of your context budget on memory bookkeeping alone.

• Cost at scale. At OpenAI’s pricing, every token in the context window costs money on every inference call. An agent handling 10,000 conversations/day with 2K tokens of memory overhead per call is burning ~$1,500/month just on memory — before the agent produces a single useful token.

• Latency amplification. Longer context means more KV-cache computation. On hardware with limited KV-cache capacity (which is most of them), this means either degraded throughput or expensive KV-cache offloading to CPU memory.

• State management complexity. Agents need to remember user preferences, past decisions, tool results, and domain-specific facts. Managing this state inside a flat text context is fragile and error-prone.

Zero-Mem solves these problems by decoupling memory storage from memory consumption. The memory exists, is queryable, and is persistent — but it doesn’t live inside the LLM’s context window.

How It Works

The core idea is straightforward: replace in-context memory with an out-of-band memory store that the agent runtime manages. The LLM interacts with memory through a structured interface — handles, IDs, or typed references — and the runtime resolves these references without ever serializing the content into tokens.

Here’s the architecture at a high level:

┌─────────────────────────────────────────────────────┐
│                   Agent Runtime                      │
│                                                      │
│  ┌──────────┐    ┌──────────────┐    ┌───────────┐ │
│  │  Planner  │───▶│  Memory Mgr  │───▶│  LLM Call │ │
│  │  (Agent)  │    │  (Zero-Mem)  │    │           │ │
│  └──────────┘    └──────────────┘    └───────────┘ │
│       │               │                          │   │
│       │         ┌─────┴──────┐                   │   │
│       │         │  Memory    │                   │   │
│       │         │  Store     │                   │   │
│       │         │  (KV/VecDB)│                   │   │
│       │         └────────────┘                   │   │
│       │                                          │   │
│       ▼                                          ▼   │
│  ┌──────────┐                           ┌──────────┐ │
│  │  Tools   │◀──────────────────────────│  Context  │ │
│  │ (no mem) │    resolved at runtime    │  Window   │ │
│  └──────────┘    (zero tokens consumed) └──────────┘ │
└─────────────────────────────────────────────────────┘

The Memory Manager

The Memory Manager is the central component. It maintains:

1. An external store — a key-value store, vector database, or structured document store that holds memory objects.
2. A reference resolver — maps between LLM-visible handles (short strings, integers, or typed IDs) and the actual memory content in the store.
3. A serialization boundary — controls what gets materialized into the context window and what stays out.

The Zero-Token Write Path

When the agent needs to store information, it doesn’t ask the LLM to “remember” something by including it in the next prompt. Instead:

1. Agent generates a structured memory write command (tool call or structured output).
2. Memory Manager intercepts the command, extracts the payload.
3. Payload is stored in the external store, assigned a handle.
4. The handle is returned to the agent as a tool result or structured output field.
5. The handle — not the content — enters the context window.

The LLM never processes the raw memory content. It processes a reference. The content lives in the store and is retrieved only when explicitly queried — and even then, retrieval happens outside the LLM inference call.

The Zero-Token Read Path

When the agent needs to recall something:

1. Agent issues a memory query (structured output or tool call with a handle or semantic query).
2. Memory Manager resolves the query against the external store.
3. If the result is needed for decision-making, the Manager decides whether to:
   a. Materialize a summary into the context (minimal token cost), or
   b. Return the handle and let the agent operate on it indirectly.
4. The LLM receives only what's necessary — ideally just the handle.

The critical insight is that most memory operations don’t require the LLM to see the full content. A memory entry like “user prefers Celsius for temperature units” can be represented as mem:pref:temp_unit=C and resolved entirely by the runtime when formatting tool calls or downstream outputs.

Core Concepts

Memory Handles

A memory handle is a compact, opaque reference to a stored memory object. It’s what the LLM actually “sees” — a short string or integer that the runtime resolves.

Handle format: mem:<type>:<id>
Examples:
  mem:pref:temp_unit=C
  mem:fact:project_deadline=2026-03-15
  mem:episodic:conv_2024_11_03_summary=evt_8f3a2c

Handles are typically 15–40 characters — orders of magnitude cheaper than the content they reference.

The Serialization Boundary

This is the decision point where the Memory Manager determines whether memory content enters the LLM’s context window. The boundary is governed by a cost-benefit function:

serialize_if(
    memory_entry,
    context_budget_remaining,
    task_complexity,
    retrieval_confidence
) -> bool

In practice, this means:

• High confidence, low complexity tasks: never serialize. The runtime handles resolution entirely.
• Low confidence, high complexity tasks: serialize a compressed summary (not raw content).
• Explicit agent requests: the agent can force materialization via a force_context flag, but this is logged and flagged as a budget concern.

Memory Tiers

Zero-Mem systems typically implement a tiered memory architecture:

Tier	Storage	Access Cost	Token Cost	Use Case
L1: Ephemeral	In-process dict/map	O(1)	Zero	Single-session working state
L2: Semantic	Vector store (e.g., Qdrant, Milvus)	O(log N) search	Zero (handle only)	Fact retrieval, preference lookup
L3: Episodic	Document store (e.g., S3, Postgres)	O(1) key lookup	Zero (handle only)	Conversation history, event logs
L4: Context	LLM context window	N/A	Full token cost	Only when explicitly serialized

The goal is to keep as much as possible in L1–L3 and only touch L4 when absolutely necessary.

Structured Memory Operations

Rather than free-form text memory, Zero-Mem enforces a schema for memory objects. This enables deterministic resolution by the runtime without LLM involvement:

{
  "handle": "mem:pref:temp_unit=C",
  "type": "preference",
  "key": "temperature_unit",
  "value": "Celsius",
  "source": "user_statement_turn_3",
  "confidence": 0.95,
  "ttl": null,
  "created_at": "2024-11-03T14:22:01Z"
}

The schema allows the runtime to answer queries like “what is the user’s temperature preference?” by looking up type=preference AND key=temperature_unit without ever involving the LLM.

Examples & Code Walkthrough

Here’s a minimal implementation of a Zero-Mem memory manager in Python:

import uuid
from dataclasses import dataclass, field
from datetime import datetime, timezone
from enum import Enum
from typing import Any, Optional


class MemoryType(Enum):
    PREFERENCE = "pref"
    FACT = "fact"
    EPISODIC = "episodic"
    PROCEDURAL = "proc"


@dataclass
class MemoryEntry:
    handle: str
    memory_type: MemoryType
    key: str
    value: Any
    source: str
    confidence: float = 1.0
    ttl: Optional[datetime] = None
    created_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))

    def to_handle_string(self) -> str:
        return f"mem:{self.memory_type.value}:{self.handle}"


class ZeroMemStore:
    """In-memory store with handle-based access.
    In production, this would be backed by a persistent KV or vector store."""

    def __init__(self):
        self._store: dict[str, MemoryEntry] = {}
        self._index: dict[str, list[str]] = {}  # key -> [handles]

    def write(self, entry: MemoryEntry) -> str:
        """Store a memory entry and return its handle."""
        self._store[entry.handle] = entry
        self._index.setdefault(entry.key, []).append(entry.handle)
        return entry.to_handle_string()

    def read(self, handle: str) -> Optional[MemoryEntry]:
        """Retrieve a memory entry by handle string."""
        clean_handle = handle.replace("mem:", "").split(":", 1)[-1]
        # Extract the actual handle ID from "mem:<type>:<id>"
        parts = handle.split(":", 2)
        if len(parts) == 3:
            actual_key = parts[2]
        else:
            actual_key = clean_handle
        return self._store.get(actual_key)

    def query_by_key(self, key: str) -> list[MemoryEntry]:
        """Retrieve all entries matching a key — zero tokens consumed."""
        handles = self._index.get(key, [])
        return [self._store[h] for h in handles if h in self._store]

    def resolve_handle(self, handle: str) -> Optional[Any]:
        """Resolve a handle to its value without LLM involvement."""
        entry = self.read(handle)
        return entry.value if entry else None


class AgentMemoryManager:
    """Manages the boundary between agent memory and LLM context."""

    def __init__(self, store: ZeroMemStore):
        self.store = store
        self.context_log: list[str] = []  # Only what enters the context window

    def store_memory(self, memory_type: MemoryType, key: str,
                     value: Any, source: str, confidence: float = 1.0) -> str:
        """Write memory out-of-band. Returns a handle for the LLM."""
        entry = MemoryEntry(
            handle=str(uuid.uuid4())[:8],
            memory_type=memory_type,
            key=key,
            value=value,
            source=source,
            confidence=confidence,
        )
        handle_str = self.store.write(entry)
        # Only the handle goes into context — not the value
        self.context_log.append(f"[memory:stored] {handle_str}")
        return handle_str

    def retrieve_memory(self, handle: str, force_materialize: bool = False
                        ) -> dict:
        """Retrieve memory. Optionally materialize into context."""
        entry = self.store.read(handle)
        if entry is None:
            return {"error": "handle_not_found"}

        if force_materialize:
            # This is the one case where content enters the context window
            self.context_log.append(
                f"[memory:materialized] {handle} = {entry.value}"
            )
            return {"materialized": entry.value, "handle": handle}

        # Default: return handle only, runtime resolves externally
        return {"handle": handle, "type": entry.memory_type.value,
                "key": entry.key}

    def query_memory(self, key: str) -> list[dict]:
        """Query memory by key — zero tokens."""
        results = self.store.query_by_key(key)
        return [
            {"handle": h.to_handle_string(), "key": h.key,
             "type": h.memory_type.value}
            for h in results
        ]


# --- Usage Example ---

store = ZeroMemStore()
mem_mgr = AgentMemoryManager(store)

# Agent learns user preference — stored out-of-band
handle = mem_mgr.store_memory(
    MemoryType.PREFERENCE,
    key="temperature_unit",
    value="Celsius",
    source="user_turn_3",
)
print(f"Stored handle: {handle}")
# Output: Stored handle: mem:pref:abc123def

# Agent needs the preference — resolved without LLM context cost
result = mem_mgr.retrieve_memory(handle)
print(f"Retrieval result: {result}")
# Output: {'handle': 'mem:pref:abc123def', 'type': 'pref',
#          'key': 'temperature_unit'}

# Runtime resolves the actual value outside the LLM call
actual_value = store.resolve_handle(handle)
print(f"Resolved value: {actual_value}")
# Output: Resolved value: Celsius

# Bulk query — also zero tokens
prefs = mem_mgr.query_memory("temperature_unit")
print(f"All preferences: {prefs}")

Integration with an LLM Agent Loop

Here’s how Zero-Mem fits into a typical agent tool-calling loop using a structured approach:

// TypeScript pseudocode for an agent runtime with Zero-Mem integration

interface MemoryOperation {
  action: 'store' | 'retrieve' | 'query' | 'materialize';
  type?: MemoryType;
  key?: string;
  value?: unknown;
  handle?: string;
  forceMaterialize?: boolean;
}

interface AgentTurnResult {
  content: string;
  memoryOps?: MemoryOperation[];
  toolCalls: ToolCall[];
}

class ZeroMemAgentRuntime {
  constructor(
    private llm: LLMClient,
    private memoryStore: ZeroMemStore,
    private contextBudget: number = 128_000
  ) {}

  async executeTurn(
    userInput: string,
    contextWindow: string[]
  ): Promise<string> {
    // Build prompt — context window contains handles, not raw memory
    const prompt = this.buildPrompt(userInput, contextWindow);

    // Standard LLM call — memory content never serialized here
    const response = await this.llm.complete(prompt);

    // Parse structured memory operations from the response
    const memoryOps = this.parseMemoryOps(response);

    // Execute memory operations out-of-band (zero tokens)
    for (const op of memoryOps) {
      await this.executeMemoryOp(op);