> ## Content Index > Fetch the complete content index at: https://blog.diengdoh.com/llms.txt > Use this file to discover other available public pages before exploring further. # My AI Agent Only Saw 71% of Its Own Memory - URL: https://blog.diengdoh.com/ai-agent-memory-file-truncation/ - Published: 2026-09-03T10:00:00.000Z - Updated: 2026-10-11T10:16:27.000Z - Description: An AI agent's long-term memory file grew to 70 KB and the bootstrap cut it at 71%. Two automatic writers caused it. How I split curated vs. archive memory. - Author: Lukas - Tags: business-ai, AI The self-hosted AI assistant I run for a small business has a long-term memory: a plain Markdown file, `MEMORY.md`, in its workspace. At the start of every session the framework loads that file into the context, together with a few other bootstrap files. That's how the agent "remembers" who the important people are, how the boss likes his mails written, which projects exist. One day I looked at what actually landed in the context. The memory file had grown to about **70 KB**. The bootstrap was injecting roughly **71%** of it and cutting the rest. The agent had been working with a partial memory, with no warning, for who knows how long. **TL;DR:** Two automatic jobs kept appending to the memory file until it outgrew the bootstrap limit and got silently truncated. Fix: split it into a curated `MEMORY.md` (~~39 KB, always loaded) and an archive file (~~31 KB, not loaded, read on demand), point both automatic writers at the archive, raise the bootstrap limits so the curated part fits with room to spare. Back up before every step. Then measure what's really injected, don't assume. ## Measure first Before changing anything, find out how big the bootstrap files are and how much of them the agent actually gets: ```bash cd /path/to/agent/workspace wc -c *.md | sort -n ``` That gives you sizes on disk. The second half, what gets injected, depends on your framework; compare what it loads against those sizes. That comparison is where my 71% came from. If your framework doesn't expose it, ask the agent to quote the last entry of its memory file. If it can't, and the entry is right there on disk, you have your answer. ## Gotcha: truncation is silent Nothing errored. The agent didn't complain. The bootstrap loaded what fit and cut the rest. From the outside the agent looked fine. The symptom of something like this is an agent that "forgets" things that are definitely in its memory, and those moments are easy to put down to the model having a bad day. That's the dangerous part. A crash gets fixed the same day. A silently partial memory degrades quality slowly, and you blame the model instead of the plumbing. ## How the file got that big Nobody wrote 70 KB of memory on purpose. Two automatic writers did: 1. **A mail watcher.** It processes incoming mails where the assistant is in CC, and for each one it appended "facts from CC mails" to the memory file. One entry per mail, dated, with the subject in the heading. 2. **A daily drive scan job.** It scans the company's cloud storage for new documents and, at the end of every run, appended a block of "learnings" to the same file. Each entry on its own was reasonable. Together they were a log, not a memory. Every day the file got longer, nothing ever got removed, and the hand-curated parts at the top, the stuff that actually matters, were sharing a fixed budget with an ever-growing stream of auto-notes. The same assistant has burned me before with background jobs that nobody watches, [once on duplicate uploads](/idempotency-cron-automation-duplicate-uploads/) and [once on missing container tools](/ai-assistant-scanned-pdf-vision-tools/). Unattended writers are where the surprises live. ## The fix: curated vs. archive The core idea: separate what a human curates from what machines append. - **`MEMORY.md`**: curated. People, preferences, projects, partners, the things the agent needs in every single conversation. About 39 KB after the split. Stays in the bootstrap. - **`MEMORY-ARCHIV.md`**: the automatic entries. ("Archiv" is German for archive.) About 31 KB. **Not** loaded at bootstrap. The agent reads it with its normal file tools when a question needs it. I tested the on-demand part before calling it done: the agent does reach into the archive when it needs something from there. That was the condition for the whole approach. An archive the agent never opens is just a slower way of deleting things. ### Step 1: backups Every file I touched got a dated copy first: ```bash cp MEMORY.md MEMORY.md.bak-before-split-$(date +%Y%m%d) cp config.json config.json.bak-before-bootstrap-limits-$(date +%Y%m%d) ``` Plus an export of the scheduled job definition before editing its prompt. Memory files are hard to reconstruct, and a split is exactly the kind of edit where you cut one section too many. ### Step 2: split by entry type Because both writers used recognisable headings, the split was mechanical: entries starting with the mail watcher's dated heading pattern and the drive job's "learnings" heading went to the archive, everything else stayed. A quick sanity check afterwards: ```bash wc -c MEMORY.md MEMORY-ARCHIV.md MEMORY.md.bak-before-split-* ``` The two new files together should be roughly the size of the backup. If they're much smaller, you lost something. ### Step 3: redirect the writers This is the step that stops it from happening again. Splitting the file once without touching the writers just buys you a few weeks. - The mail watcher is a script. Its target file is a single constant; I pointed it at the archive. - The drive scan job is an agent job with a prompt. I changed its instruction to write learnings to `MEMORY-ARCHIV.md`, and spelled out explicitly *not* `MEMORY.md`, because agents are good at "helpfully" going back to the file they know. ```python # mail watcher, in essence: before MEMORY_FILE = WORKSPACE / "MEMORY.md" # after MEMORY_FILE = WORKSPACE / "MEMORY-ARCHIV.md" ``` ### Step 4: raise the bootstrap limits The curated file alone was still big, so I raised the limits in the framework config. Per-file and total: ```json { "agents": { "defaults": { "bootstrapMaxChars": 60000, "bootstrapTotalMaxChars": 90000 } } } ``` Then I checked again: about **52.8K characters injected**, nothing truncated. The curated memory fits with headroom, and the headroom is there on purpose. The next time the file grows, it grows into free space instead of off a cliff. ## Gotcha: raising limits alone is not a fix The tempting shortcut is to skip the split and just crank the limits up. Two problems: - **It only moves the cliff.** With two writers appending daily, any limit gets hit eventually. You'd be back here in a couple of months with a bigger file. - **Context isn't free.** Every bootstrap character is sent with every session. A memory file full of months-old CC notes costs tokens on every conversation and dilutes the stuff that matters. Background spend is how [this same agent once drained a prepaid API budget](/ai-agent-api-costs-background-loops/). Raising the limit was right, but only after the file was cut down to what deserves to be loaded every time. ## What I'd tell anyone running an agent with memory **Whatever an agent writes on its own grows without bound.** Logs, learnings, notes, "facts". Nobody prunes them, so they only ever grow. Plan for that on day one: give automatic writes their own file, or a rotation, or a size check. **Separate curated from automatic.** Curated memory is small, high-signal and changes rarely. Automatic memory is big, noisy and grows daily. They need different treatment, and mixing them means the noise eats the budget of the signal. **Measure what actually lands in the context.** Not what's on disk, not what the config says. What's injected. Bytes on disk and characters in the context aren't the same thing, and the only number that matters is the one the model actually gets. Check it after every change and occasionally in between. **Back up before every edit.** Especially memory. Nobody else has a copy. ## FAQ **Why not let the agent summarise its own memory?** You can, and it's tempting. But an agent pruning its own memory unsupervised can drop the one fact that mattered, and you won't notice until it's gone. I'd rather curate by hand and let the agent read the archive. **Should I put the archive in a vector database?** For this size, no. A 31 KB Markdown file the agent can read with a file tool is simpler and easier to inspect. If the archive grows into megabytes, search starts to make sense. **How often should I check?** After every change to the agent's jobs, and whenever it "forgets" something it should know. That forgetting is the symptom; truncation is one of the first causes to rule out. --- ### Related posts - [“I Barely Use It” — Where $100 of AI API Credit Actually Went](https://blog.diengdoh.com/ai-agent-api-costs-background-loops/) - [My AI Receipt Sorter Shifted Amounts by 2,000 Euros, So I Took the Numbers Away From It](https://blog.diengdoh.com/ai-receipt-matching-deterministic-parsing/) - [My AI Agent Invented a CLI Flag, and I Called the Error "Cosmetic"](https://blog.diengdoh.com/ai-agent-hallucinated-cli-flags/)