Position: LLMs Can't Jump

Introduction

Large language models are powerful, but they are bounded. Every LLM has a fixed context window, a static knowledge cutoff, and an inference-time architecture that constrains what it can do in a single forward pass. The industry is slowly internalizing this: you can’t prompt your way past fundamental architectural limits, and no amount of clever instruction-tuning turns a 7B parameter model into an agent that reliably plans across hours of autonomous execution.

This isn’t a dismissal of LLMs. It’s a call for engineers to design systems that respect what these models can’t do—so we can reliably build what they can.

Why This Matters

If you’re building production systems that depend on LLMs—RAG pipelines, autonomous agents, code generation tools, or conversational interfaces—you’ve probably hit one of these walls:

• Context window exhaustion: Your system needs to reason over 200K tokens, but the model’s effective attention degrades well before that.
• Hallucination under distribution shift: The model confidently fabricates facts when asked about domains or time periods outside its training data.
• Compositional reasoning failures: The model handles individual steps correctly but fails when chaining multiple steps into a coherent plan.
• Statelessness: Each inference call is an island. There’s no persistent memory, no implicit state carryover, no “learning” between requests.

These aren’t bugs waiting to be patched. They’re architectural constraints baked into the transformer paradigm. Ignoring them leads to brittle production systems that fail silently and unpredictably.

How It Works

An LLM processes input as a sequence of token embeddings, passes them through a stack of self-attention layers, and produces a probability distribution over the vocabulary for each output position. The “jump” metaphor breaks down when you consider what happens at each stage:

┌─────────────────────────────────────────────────────┐
│                   Input Tokens                      │
│  [tok_1] [tok_2] ... [tok_n]                       │
│  n ≤ context_window (e.g., 4K, 32K, 128K)         │
└──────────────────────┬──────────────────────────────┘
                       │
                       ▼
┌─────────────────────────────────────────────────────┐
│            Token Embedding Layer                    │
│  Each token mapped to dense vector in ℝ^d          │
│  d = model dimension (e.g., 4096 for Llama-70B)    │
└──────────────────────┬──────────────────────────────┘
                       │
                       ▼
┌─────────────────────────────────────────────────────┐
│         Transformer Layers (L layers)               │
│  Self-attention + FFN per layer                   │
│  Attention: softmax(QK^T / √d_k) · V              │
│  - Fixed positional encoding (or rotary)           │
│  - No recurrence, no external memory access        │
└──────────────────────┬──────────────────────────────┘
                       │
                       ▼
┌─────────────────────────────────────────────────────┐
│         Output Probability Distribution             │
│  P(token_{t+1} | token_1...token_t)                │
│  Greedy / sampled / beam search                    │
└─────────────────────────────────────────────────────┘

The key constraint is visible here: the model operates entirely within its fixed parameter weights and the input sequence provided in a single forward pass. It cannot:

1. Retrieve information not present in the input or its weights — no live database lookups, no real-time knowledge.
2. Maintain state across calls — each request is independent unless you explicitly build that mechanism.
3. Perform exact computation — it approximates patterns; it doesn’t execute algorithms.
4. Scale reasoning depth with input size — longer inputs don’t give the model more “thinking time.”

Core Concepts

Context Window: The maximum number of tokens the model can process in a single forward pass. This is a hard architectural limit determined by the attention matrix dimensions (O(n²) in standard transformers, though FlashAttention and sliding window variants reduce the constant factor).

Knowledge Cutoff: The temporal boundary of the model’s training data. The model has no access to events, codebases, or data that didn’t exist during pre-training or fine-tuning.

Attention Mechanism: The core computation where each token attends to every other token in the sequence. The attention weights determine how much “information” flows between positions. This is where the context window constraint lives.

Emergent Capabilities vs. Scaling Laws: Some abilities (chain-of-thought reasoning, few-shot learning) appear to emerge at certain model scales. But they don’t appear suddenly—they scale predictably with compute, data, and parameters. There’s no phase transition where a model suddenly gains an entirely new cognitive faculty.

In-Context Learning: The ability to perform tasks based solely on examples provided in the prompt, without weight updates. Powerful, but bounded by the context window and the model’s pre-trained inductive biases.

Tool Use / Retrieval-Augmented Generation (RAG): The primary engineering pattern for overcoming the knowledge cutoff and context limits. The model delegates information retrieval to external systems and reasoning to itself.

Examples & Code Walkthrough

Here’s a practical pattern for building a RAG system that respects LLM constraints—specifically, the context window limit and the inability to retrieve information outside the input:

# rag_pipeline.py
import asyncio
from dataclasses import dataclass
from typing import List

from sentence_transformers import SentenceTransformer
from rank_bm25 import BM25Okapi

@dataclass
class DocumentChunk:
    content: str
    metadata: dict

@dataclass
class RetrievalResult:
    chunk: DocumentChunk
    score: float

class RAGPipeline:
    """
    Retrieval-Augmented Generation pipeline.
    Designed around the constraint that LLMs cannot
    access information outside their context window or
    training data.
    """

    def __init__(
        self,
        embedding_model_name: str = "all-MiniLM-L6-v2",
        top_k: int = 5,
        max_context_tokens: int = 8192,
    ):
        self.embedder = SentenceTransformer(embedding_model_name)
        self.top_k = top_k
        self.max_context_tokens = max_context_tokens
        self.chunks: List[DocumentChunk] = []
        self.bm25 = None
        self._index_built = False

    def index_documents(self, documents: List[str], chunk_size: int = 512):
        """Split documents into overlapping chunks and build indices."""
        self.chunks = []
        for doc_id, doc in enumerate(documents):
            tokens = doc.split()
            for i in range(0, len(tokens), chunk_size // 2):
                chunk_tokens = tokens[i : i + chunk_size]
                if not chunk_tokens:
                    continue
                self.chunks.append(
                    DocumentChunk(
                        content=" ".join(chunk_tokens),
                        metadata={"doc_id": doc_id, "chunk_idx": i},
                    )
                )

        # Build BM25 for lexical retrieval
        tokenized = [c.content.split() for c in self.chunks]
        self.bm25 = BM25Okapi(tokenized)
        self._index_built = True

    async def retrieve(self, query: str) -> List[RetrievalResult]:
        """Hybrid retrieval: BM25 lexical + semantic embedding."""
        if not self._index_built:
            raise RuntimeError("Index not built. Call index_documents first.")

        # BM25 scores
        tokenized_query = query.split()
        bm25_scores = self.bm25.get_scores(tokenized_query)

        # Embedding similarity
        query_embedding = self.embedder.encode([query])
        chunk_embeddings = self.embedder.encode([c.content for c in self.chunks])
        import numpy as np
        similarities = np.dot(chunk_embeddings, query_embedding.T).flatten()

        # Combine scores (simple weighted average)
        combined = {}
        for idx in range(len(self.chunks)):
            combined[idx] = 0.4 * bm25_scores[idx] + 0.6 * similarities[idx]

        # Rank and return top-k
        ranked = sorted(combined.items(), key=lambda x: x[1], reverse=True)
        results = []
        for idx, score in ranked[: self.top_k]:
            results.append(RetrievalResult(chunk=self.chunks[idx], score=score))
        return results

    def build_prompt(
        self, query: str, retrieval_results: List[RetrievalResult]
    ) -> str:
        """
        Assemble the prompt, respecting the context window.
        This is where we enforce the 'LLMs can't jump' constraint
        by only including retrievable context.
        """
        context_parts = []
        total_tokens = 0
        for result in retrieval_results:
            chunk_tokens = len(result.chunk.content.split())
            if total_tokens + chunk_tokens > self.max_context_tokens:
                break
            context_parts.append(
                f"[Source {result.chunk.metadata['doc_id']}]: "
                f"{result.chunk.content}"
            )
            total_tokens += chunk_tokens

        context = "\n\n".join(context_parts)
        return (
            f"Use only the provided context to answer the question.\n"
            f"If the context doesn't contain the answer, say so.\n\n"
            f"Context:\n{context}\n\n"
            f"Question: {query}"
        )

The critical design decision in this pipeline is the build_prompt method. It explicitly enforces the context window boundary. The LLM can only reason about what’s inside that prompt—nothing more. The retrieval step bridges the gap between the model’s static knowledge and the dynamic world, but the model itself never “jumps” to information it wasn’t given.

Best Practices

1. Treat the context window as a hard budget, not a soft suggestion. Monitor token usage at every stage of your pipeline—input tokens, retrieved context, system prompt, output tokens. Set hard limits and fail gracefully when you approach them. Don’t let your RAG system silently truncate critical context.

2. Separate retrieval from reasoning. The LLM is a reasoning engine, not a database. Build explicit retrieval layers (vector stores, keyword indices, API calls) and feed their output into the model. Don’t expect the model to “remember” facts it wasn’t given in the current context.

3. Validate outputs against the provided context. Implement post-hoc verification: check that the model’s answer is grounded in the retrieved documents. If the model answers a question that wasn’t present in the retrieved context, flag it as a potential hallucination.

4. Design for statelessness. Assume each inference call is independent. If you need state, manage it explicitly in your application layer—not in the model. Use external stores, conversation summaries, or structured memory systems.

5. Benchmark distribution shift explicitly. Test your system on data that falls outside the model’s training distribution. If accuracy drops sharply, your system needs better retrieval, better prompt engineering, or a different model—not more prompt tricks.

6. Prefer deterministic retrieval over probabilistic generation for factual lookups. If you need to check whether a specific fact exists, use a retrieval system with exact matching, not an LLM’s probabilistic approximation of that fact.

Common Mistakes & Anti-Patterns

1. Prompting as a substitute for architecture. Writing a clever system prompt and hoping the model will “just reason correctly” across complex multi-step tasks. This fails at scale. Prompt engineering has diminishing returns; architectural patterns (agent frameworks, tool use, retrieval loops) have compounding returns.

2. Treating the context window as infinite. Engineers stuff 100K tokens into a model’s context and assume it will attend to all of them equally. In practice, attention degrades for tokens far from the query position. Use retrieval to pre-filter, don’t dump everything in.

3. Ignoring the knowledge cutoff in production. Building a customer-facing Q&A system on an LLM without RAG, then being surprised when it can’t answer questions about events after its training cutoff. If your application domain changes over time, you need a retrieval layer that updates independently of the model.

4. Trusting the model as a deterministic system. Treating LLM outputs as reliable function results. They’re probabilistic. Two identical prompts can produce different outputs. In production, you need retry logic, fallback strategies, and confidence scoring—not just happy-path testing.

Performance Considerations

Latency:

• Prefill latency (processing the input prompt) is proportional to sequence length and model size. For a 70B model, expect 50-200ms per token on high-end GPUs.
• Decode latency dominates for long generations. A 1000-token response on a single L40S GPU takes roughly 2-5 seconds.
• Batching requests improves throughput but increases tail latency. Design your serving layer with this tradeoff in mind.

Memory:

• Model weights for a 7B model in FP16 consume ~14GB. A 70B model needs ~140GB. Quantization (INT8, INT4) reduces this but introduces quality degradation.
• KV cache grows linearly with sequence length and batch size. For long-context workloads, KV cache can exceed GPU memory before the model itself does.

Compute Complexity:

• Self-attention: O(n² · d) where n is sequence length and d is model dimension. This is why context window scaling is expensive.
• Retrieval (vector similarity): O(N · d) for exact search, reducible to O(log N · d) with approximate nearest neighbor (ANN) indices like HNSW.

Scalability:

• Horizontal scaling of inference servers is straightforward (stateless requests).
• The bottleneck shifts to retrieval infrastructure and embedding computation at high throughput.
• Consider caching retrieval results and LLM outputs (with appropriate invalidation strategies) to reduce redundant computation.

Real-World Usage

Anthropic’s Claude: Uses a retrieval-augmented approach for its “Projects” feature, where enterprise users can feed large document collections. The system explicitly manages context window boundaries, retrieving relevant chunks before each generation call.

GitHub Copilot: Relies on retrieval from the user’s codebase and indexing of repository structure. It doesn’t “know” the codebase—it retrieves relevant snippets and context, then generates completions grounded in that retrieved context.

Meta’s Llama Index / LangChain ecosystems: These open-source frameworks exist precisely because engineers recognized that LLMs need external scaffolding to overcome their inherent limitations. The retrieval, orchestration, and memory management layers are all explicit engineering constructs, not magic.

OpenAI’s Assistants API: Provides built-in tools (code interpreter, file search, function calling) that externalize computation and retrieval away from the model itself. The model “jumps” by delegating to tools—it doesn’t jump on its own.

Vercel’s AI SDK and vercel/knowledge: Production systems that use streaming retrieval-augmented generation, where the retrieval step runs asynchronously and results are fed incrementally into the LLM context, managing the context budget in real time.

Frequently Asked Questions (FAQ)

Q: Can I make an LLM “jump” by fine-tuning it on my domain data? A: Fine-tuning improves performance within the model’s capabilities, but it doesn’t grant new abilities the model architecture can’t support. It shifts the distribution—it doesn’t expand the model’s fundamental reasoning or memory capacity. Fine-tuning is best for style adaptation, domain vocabulary, and task-specific patterns, not for teaching a model entirely new capabilities.

Q: What’s the practical limit on context window size? A: The theoretical limit is set by the architecture (e.g., 128K tokens for Gemini 1.5 Pro). The practical limit is where attention quality degrades—typically 50-70% of the nominal maximum for standard transformers. Beyond that, you need architectural innovations (Ring Attention, sliding windows, memory layers) or you need to switch to retrieval