# LLM Memory Compaction: Keeping an Agent's Context Window Healthy

## Blog Details

- **Author**: Navneet
- **Date**: August 11, 2026
- **Tags**: LLM, AI Agents, Context Window, Memory Management, Prompt Engineering
- **Read Time**: 9 mins

Every capable LLM agent eventually runs into the same wall: the context window. A model can only attend to a fixed number of tokens at once, but a long-running agent keeps accumulating history, tool outputs, and intermediate reasoning. Left unmanaged, that history grows until it overflows the window, drowns the signal in noise, or quietly runs up a cost and latency bill on every single turn.

Memory compaction is how you keep that from happening. This post walks through what compaction is, why it becomes unavoidable as agents get more capable, and the practical strategies teams use to do it well.

## What Is Memory Compaction

Memory compaction is the deliberate process of reducing the size of an agent's working context while preserving the information that future turns actually need. Instead of carrying the entire raw conversation forward on every request, the agent maintains a compacted representation: shorter, denser, and focused on what matters.

The key word is *deliberate*. Without compaction, an agent's context is simply an ever-growing append-only log: every user message, every model response, every tool call and its full output, stacked end to end. Compaction replaces parts of that log with smaller equivalents. A verbose tool result becomes a one-line fact. Ten turns of back-and-forth become a three-sentence summary. A resolved sub-task collapses into its conclusion. The agent keeps behaving as if it remembers everything, but the tokens it carries stay bounded.

![Raw growing context versus a compacted context](https://d5osvdbc8um23.cloudfront.net/static-asset/blog_images/llm-memory-compaction/01-what-is-compaction.png)

A useful way to think about it: raw context is a transcript, compacted context is a briefing. A transcript records everything that was said. A briefing keeps only what the next decision depends on. Good compaction turns the former into the latter without losing the thread.

## Why It Is Required

Compaction is not a nice-to-have optimization you bolt on later. For any agent that runs for more than a handful of turns, it is a correctness and cost requirement. Four forces make it unavoidable.

**The context window is finite.** This is the hard limit. Once the accumulated history plus the next prompt exceeds the model's window, requests start failing or silently truncating. An agent that cannot compact is an agent with a fixed lifespan measured in turns.

**More context is not more intelligence.** Even well below the hard limit, stuffing the window degrades quality. Models attend less reliably to information buried in the middle of a very long context, and irrelevant history actively distracts from the current task. A focused 8k-token context often produces better decisions than a bloated 80k-token one. Compaction improves reasoning, not just capacity.

**Cost and latency scale with tokens.** Most LLM pricing is per-token, and every turn re-sends the entire context. An un-compacted agent pays for its whole history on every single step, so a long session's cost grows roughly quadratically. Latency rises the same way, since larger prompts take longer to process. Compaction keeps both flat instead of ballooning.

**Signal gets buried in noise.** Raw tool outputs, verbose logs, and repeated boilerplate crowd out the few facts that actually drive the next action. Compaction is as much about *relevance* as size: it raises the density of useful information so the model spends its attention where it counts.

![The four pressures that force compaction](https://d5osvdbc8um23.cloudfront.net/static-asset/blog_images/llm-memory-compaction/02-why-required.png)

## Ways to Perform Memory Compaction

There is no single compaction technique. In practice teams combine several, each suited to a different kind of bloat. The strategies below range from the simplest mechanical trims to policy-driven schemes that fire on events or time. Most production agents layer two or three of them together.

![The family of compaction strategies](https://d5osvdbc8um23.cloudfront.net/static-asset/blog_images/llm-memory-compaction/03-strategies-overview.png)

### Context Trimming

Trimming is the most direct strategy: drop messages from the context according to a simple rule, usually keep-the-most-recent. When the history exceeds a threshold, the oldest turns fall off the front, much like a sliding window over the conversation.

Its strength is that it is cheap, predictable, and requires no extra model calls. Its weakness is that it is *lossy in a blunt way*: it has no idea whether the turn it just dropped contained something important. A user's original goal stated on turn one can vanish by turn twenty. That is why trimming is usually paired with a pinned or protected region: the system prompt, the original task, and any explicitly "sticky" facts are exempt from trimming, and only the free-flowing middle is windowed.

![Context trimming as a sliding window with a pinned region](https://d5osvdbc8um23.cloudfront.net/static-asset/blog_images/llm-memory-compaction/04-context-trimming.png)

Use trimming when recency genuinely correlates with relevance and turns are roughly independent. Avoid relying on it alone when early context carries commitments the agent must honor later.

### Summarization

Summarization compresses rather than discards. Instead of dropping old turns, the agent replaces a block of them with a model-generated summary that preserves their meaning in a fraction of the tokens. The raw exchange goes away; a dense recap takes its place.

This is the workhorse of long-running agents because it is *lossy in an intelligent way*: a good summary keeps decisions, constraints, and open questions while shedding the verbatim phrasing that consumed the tokens. The cost is real, though: summarization requires an extra LLM call, it adds latency at the moment it fires, and a careless summary can drop a detail that turns out to matter later. Quality of the summarization prompt is everything.

![Summarization replacing a block of turns with a dense recap](https://d5osvdbc8um23.cloudfront.net/static-asset/blog_images/llm-memory-compaction/05-summarization.png)

Common refinements include *hierarchical* summarization (summaries of summaries as a session gets very long) and *structured* summaries that force the model to fill fixed fields (goal, decisions so far, current state, open items) so nothing critical is left to the model's discretion.

### Tool Definition and Response Compaction

Agentic contexts are often dominated not by conversation but by tools: the JSON schemas that define what tools exist, and the frequently large payloads those tools return. Both are prime compaction targets, and both are easy to overlook because they do not look like "memory."

On the **definition** side, an agent with many tools spends a surprising fraction of its window just describing them on every call. Compaction here means only presenting the tools relevant to the current phase, trimming verbose parameter descriptions, or collapsing rarely-used tools behind a single dispatcher. On the **response** side, raw tool output (a full API response, a page of search results, a database dump) is rarely needed verbatim. Compaction extracts the few fields the agent actually uses and drops the rest, so a multi-thousand-token result becomes a compact, structured fact before it ever enters the running history.

![Tool schemas and raw responses compacted before entering context](https://d5osvdbc8um23.cloudfront.net/static-asset/blog_images/llm-memory-compaction/06-tool-compaction.png)

This strategy is high-leverage precisely because tool payloads are often the single largest and most redundant part of an agent's context, yet the least information-dense once the task moves on.

### Event-Based Compaction

The strategies above compact based on *size*: you trim or summarize when the context crosses a threshold. Event-based compaction instead compacts when something *meaningful happens* in the agent's lifecycle, regardless of current size.

The clearest example is a sub-agent call. When a parent agent delegates a self-contained task to a sub-agent, that sub-agent may burn through many turns of its own reasoning and tool calls. None of that internal churn needs to survive in the parent's memory; only the sub-agent's final result does. So the natural compaction event is "sub-agent returned": at that boundary, the parent discards the sub-agent's entire working history and keeps just the outcome.

![Event-based compaction at a sub-agent boundary](https://d5osvdbc8um23.cloudfront.net/static-asset/blog_images/llm-memory-compaction/07-event-based.png)

Other natural events include completing a phase of a workflow, resolving a tool-use loop, or reaching a checkpoint. The insight is that agent workflows have *natural seams*: moments where a chunk of context has served its purpose and can be collapsed to its conclusion. Compacting on those seams is often cleaner than any size-based heuristic, because the boundary is semantically meaningful rather than arbitrary.

### Time-Based Compaction

Time-based compaction ages context out. Information is tagged with when it entered, and older material is compacted or dropped as it crosses age thresholds: recent turns stay in full detail, middle-aged turns get summarized, and the oldest are reduced to a thin trace or removed.

This is a natural fit for agents that run over long wall-clock durations, such as assistants that persist across a working day, monitoring agents, or anything where relevance genuinely decays with time. It is essentially trimming or summarization with an age-based policy rather than a purely positional one, and it often maps well to how humans treat their own memory: this morning's details are sharp, last week's are a summary, last month's is a headline.

The caveat mirrors trimming: age is a proxy for relevance, not relevance itself. A constraint set an hour ago may still bind, so time-based schemes also need a pinned region for durable facts.

### Rolling-Over Compaction

Rolling-over compaction is a continuous, incremental scheme rather than an occasional cleanup. Instead of letting the context grow until a big compaction event fires, the agent maintains a *rolling compacted state* that it updates every turn: as each new turn arrives, the oldest content is folded into a running summary, so the window slides forward while a compact carry-over travels along with it.

![Rolling-over compaction folding each turn into a running state](https://d5osvdbc8um23.cloudfront.net/static-asset/blog_images/llm-memory-compaction/08-rolling-over.png)

The benefit is smoothness. There is no periodic latency spike from a large summarization pass and no sudden context reset; the agent always operates on a bounded, continuously-maintained state. The tradeoff is that folding happens incrementally, so information passes through the summarizer repeatedly and small losses can compound over a very long run. It works best when paired with a durable pinned region for the facts that must never degrade, letting the rolling state absorb the churn while the anchors stay exact.

## Bringing It Together

No single strategy is sufficient on its own, and they are not mutually exclusive. A robust agent typically layers them: pin the system prompt and the original goal so they are never touched; compact tool definitions and responses at the point of entry so bloat never accumulates; summarize or roll over the conversational middle to keep it bounded; and compact hard at natural seams like sub-agent returns and phase completions.

The through-line across every technique is the same principle we started with: a transcript records everything, a briefing keeps what the next decision needs. Memory compaction is the discipline of continuously turning the former into the latter, so the agent stays fast, focused, affordable, and, above all, able to keep going.