Dennis Started Keeping Notes Or: what happens when your software refuses to forget what it saw.

Introduction

When we call a system “stateful,” we usually mean it holds enough context to process the next request correctly. But when that context grows unbounded, the system stops functioning as a processor and starts functioning as a hoarder. We refer to this architectural pattern internally as “Dennis”: a service that ingests events, records every decision, and refuses to drop historical context.

Databases are engineered to remember. Write-Ahead Logs, MVCC snapshots, append-only transaction logs, and LSM-trees all exist to guarantee durability. The problem emerges when application-level memory requirements outpace storage engine design. A table that started as a simple audit log becomes a multi-terabyte monolith. Query latency degrades as B-tree depth increases. Compaction jobs stall replication. Backup windows bleed into business hours.

This article examines how to architect persistent memory layers that scale. We will cover append-only ingestion, deterministic compaction, partitioned historical storage, and intentional eviction strategies. The goal is not to build a system that remembers everything forever, but to build one that remembers what matters, discards what does not, and queries the past without choking the present.

Why This Matters

Modern applications demand historical context. AI agents require conversation history. Financial systems require immutable audit trails. Event-sourced microservices require full state reconstruction. Telemetry pipelines require time-bounded aggregation. Every one of these use cases pushes data into a database with the implicit expectation that the past remains queryable.

Naive implementations fail at scale. Inserting raw JSON payloads into a single unpartitioned table causes index bloat. Running DELETE on millions of rows triggers aggressive VACUUM cycles, fragments free space maps, and stalls concurrent reads. Querying unbounded history without time constraints results in full sequential scans that exhaust IOPS and CPU. Compliance requirements clash with cost constraints when retention policies are enforced at the application layer instead of the storage layer.

Engineers need deterministic patterns for managing persistent state. You need to know when to materialize views, how to partition without sharding complexity, how to compact without write amplification storms, and how to enforce retention without locking production tables. This is not theoretical. It is the difference between a service that scales linearly and one that requires emergency migrations at 2 AM.

How It Works

The architecture follows a three-stage pipeline: ingestion, compaction, and query routing. Raw events land in an append-only buffer. A background process materializes aggregated state and archives cold data. Queries are routed based on recency and scope.

flowchart TD
    Ingest[Client Services] -->|Batch Events| Buffer[(Append-Only Buffer)]
    Buffer -->|Sequential Write| RawStore[(Raw Event Store)]
    RawStore -->|Checkpoint Trigger| Compactor[Compaction Engine]
    Compactor -->|Aggregate & Snapshot| ContextStore[(Materialized Context)]
    Compactor -->|Archive & Evict| ColdStorage[(Cold Tier / Object Store)]
    ContextStore -->|Read Path| Router{Query Router}
    Router -->|Time-Bounded| HistoricalAPI[Historical API]
    Router -->|Latest State| RealtimeAPI[Realtime API]
    RetentionPolicy[Retention Configuration] -.->|TTL & Archival Rules| Compactor
    Metrics[Observability Pipeline] -.->|Monitors IOPS & Bloat| RawStore
    Metrics -.->|Monitors Materialization Lag| Compactor

The pipeline operates on deterministic boundaries. Ingestion writes sequentially to a partitioned table, avoiding random I/O. The compaction engine runs on a schedule or event trigger, reading raw events, aggregating them into state snapshots, and marking rows for archival. The query router inspects request parameters and directs traffic to the appropriate storage tier. Observability tracks materialization lag, table bloat, and IOPS to prevent runaway growth.

Core Concepts

Append-Only Ingestion
Writing sequentially is faster than random updates. Databases optimize for sequential I/O. By treating the raw event store as a log, we leverage WAL efficiency and avoid row-level locking contention. Every event receives a monotonically increasing sequence ID and a partition key (usually a timestamp interval).

Deterministic Compaction
Compaction transforms a high-churn event stream into stable state snapshots. Instead of updating rows in place, we materialize aggregated state into a separate table. This isolates write-heavy ingestion from read-heavy querying. Compaction runs idempotently, using watermark tracking to avoid reprocessing.

Partitioned Historical Storage
Partitioning by time intervals (daily, weekly, monthly) enables efficient pruning. When a partition ages out, we detach it, compress it, and move it to object storage. The database catalog shrinks, index depth resets, and query planners can prune partitions automatically.

Intentional Eviction
Forgetting is a feature. Retention policies define what stays hot, what moves cold, and what gets deleted. TTL-based policies work for telemetry. Compliance-driven policies require cryptographic hashing and append-only guarantees. Hybrid systems use both, routing data to the appropriate lifecycle track.

Query Routing & Materialization
Historical queries differ from real-time queries. Real-time reads hit the latest state table. Historical reads hit partitioned archives or materialized views. Routing logic inspects time ranges and aggregation levels, preventing full-table scans on raw event data.

Examples & Code Walkthrough

The following implementation demonstrates a production-grade ingestion and compaction pipeline using PostgreSQL. It includes defensive error handling, batch processing, and deterministic watermark tracking.

import asyncio
import logging
import psycopg2
from psycopg2.extras import execute_values
from datetime import datetime, timedelta
from typing import List, Dict, Any

logger = logging.getLogger("dennis_memory")

# Domain model: raw telemetry/audit event
class MemoryEvent:
    def __init__(self, event_id: str, tenant_id: str, payload: Dict[str, Any], occurred_at: datetime):
        self.event_id = event_id
        self.tenant_id = tenant_id
        self.payload = payload
        self.occurred_at = occurred_at

class MemoryStore:
    def __init__(self, dsn: str, pool_size: int = 10):
        self.dsn = dsn
        self.pool_size = pool_size
        self._connection = None

    def _get_connection(self):
        if not self._connection or self._connection.closed:
            self._connection = psycopg2.connect(self.dsn)
            self._connection.set_session(autocommit=True)
        return self._connection

    def ingest_batch(self, events: List[MemoryEvent]) -> bool:
        """
        Batch inserts raw events into the append-only store.
        Uses execute_values for optimal sequential write performance.
        """
        conn = self._get_connection()
        cur = conn.cursor()
        
        values = [
            (e.event_id, e.tenant_id, psycopg2.extras.Json(e.payload), e.occurred_at)
            for e in events
        ]
        
        query = """
            INSERT INTO public.raw_events (event_id, tenant_id, payload, occurred_at)
            VALUES %s
            ON CONFLICT (event_id) DO NOTHING
        """
        
        try:
            execute_values(cur, query, values, page_size=500)
            conn.commit()
            logger.info(f"Ingested {len(events)} events successfully.")
            return True
        except Exception as exc:
            conn.rollback()
            logger.error(f"Ingestion failed: {exc}", exc_info=True)
            return False
        finally:
            cur.close()

    def compact_window(self, tenant_id: str, window_start: datetime, window_end: datetime) -> int:
        """
        Materializes aggregated state from raw events into a context table.
        Runs idempotently using a watermark table to track completed windows.
        """
        conn = self._get_connection()
        cur = conn.cursor()
        
        # Check if window was already compacted
        cur.execute(
            "SELECT EXISTS(SELECT 1