My AI Assistant "Stopped Answering" — It Was Missing Three apt Packages
A self-hosted AI assistant went silent on a document request. Not the model, not permissions: poppler, ImageMagick and Ghostscript were missing.
I run a self-hosted AI assistant for a small business. It lives in a Docker container, talks to people through a chat app, and one of its main jobs is digging documents out of the company's cloud storage. "Send me the signed loan contract for X" is exactly the kind of request it's there for.
Then the user told me: "The bot doesn't answer any more."
It did answer, technically. It answered with an error message. From the user's side that's the same thing as silence, and honestly that's the more interesting bug of the two. But first the boring one, because the boring one was the cause.
TL;DR: The agent finds the right scanned PDF by rendering candidate pages into a single contact sheet image and handing that to a vision tool. The container had no pdftoppm and no montage, so the contact sheet was never created and the vision step failed with "Local media file not found". Fix: install poppler-utils, imagemagick and ghostscript in the container. Do it properly with a derived image, not by hand. And do not loosen ImageMagick's PDF policy while you're at it, because you don't need to.
The quick check
If your agent shells out to tools, check that those tools exist in the place the agent actually runs. Not on the host. In the container:
docker exec agent-gateway sh -c 'for b in pdftoppm montage gs; do command -v $b || echo "MISSING: $b"; done'
In my case that's three MISSING lines.
The immediate fix, as root inside the running container:
docker exec -u root agent-gateway sh -c \
'apt-get update && apt-get install -y --no-install-recommends poppler-utils imagemagick ghostscript'
That gets you back up in a minute. It's also not a real fix, which I'll get to.
What the agent was trying to do
Signed contracts are the worst kind of document for an AI. Somebody printed the thing, signed it with a pen, and scanned it back in. The result is a PDF that's really just a stack of images. There's no text layer. pdftotext gives you nothing, full-text search in the cloud storage only finds it by filename, and the model can't "read" it without looking at it.
So the agent's flow for "find me contract X" was:
- Search the cloud storage, download the PDF candidates.
- Render the first pages into JPGs and tile them into one image, a
contact_sheet.jpg. - Pass that contact sheet to the vision tool and let the model pick the right document.
- Send the winner to the user.
Step 2 is plain Unix tooling. Roughly this shape (my illustration of the idea, not the agent's exact command line):
# PDF -> JPG, first page only, low resolution is enough to recognise a document
for f in candidates/*.pdf; do
pdftoppm -jpeg -r 50 -f 1 -l 1 "$f" "thumbs/$(basename "$f" .pdf)"
done
# tile all thumbnails into one image the vision model can look at in one go
montage thumbs/*.jpg -tile 3x -geometry +4+4 contact_sheet.jpg
pdftoppm comes from poppler-utils. montage comes from imagemagick. Neither was in the container.
How it failed
pdftoppm didn't exist, so no thumbnails. montage didn't exist, so no contact sheet. The agent didn't stop there. It carried on to the vision step and pointed the image tool at a file that was never written:
[tools] image failed: Local media file not found ...contact_sheet.jpg
And then it did the worst possible thing with that: it sent the error to the user and stopped. No document, no "I found three candidates but can't look at them", no "here's the file I think it is". Just a failure. One or two of those and the user's conclusion is "it's broken".
Gotcha: the obvious suspect was innocent
My first guess was permissions. The contract sat in a shared drive, shared drives are a classic source of "the API can't see it", and the request failed exactly at the find-a-document stage.
Wrong. Running the same search the agent runs found the file immediately. Access was never the problem. If I'd gone with my gut I'd have spent an hour on sharing settings and service account scopes while the real cause sat one command -v away.
Lesson I keep relearning: reproduce the agent's steps one by one, in the agent's environment, before you theorise.
The proper fix: a derived image
Installing packages with docker exec survives a docker restart. It does not survive a recreate. The next docker compose up -d after a config change, or the next image pull, and you're back to a container without the tools and an assistant that "stops answering" again, probably on a day you've forgotten all about this.
My first stopgap was to add the three packages to an init script I already run inside the container after a recreate. That works as long as somebody remembers to run it. The container's entrypoint starts the gateway as a non-root user, so the script can't just run itself at boot and apt-get its way through.
The fix that doesn't depend on me remembering anything is a small derived image. The upstream image stays as it is; I layer the tools on top:
# Dockerfile.gateway
FROM your-agent-image:latest
USER root
RUN apt-get update \
&& apt-get install -y --no-install-recommends poppler-utils imagemagick ghostscript \
&& rm -rf /var/lib/apt/lists/*
USER node
And in the compose file, build that instead of pulling the upstream image directly:
services:
gateway:
build:
context: .
dockerfile: Dockerfile.gateway
image: agent-gateway-local:latest
# ... rest unchanged
cp docker-compose.yml docker-compose.yml.bak-$(date +%Y%m%d)
docker compose build gateway
docker compose up -d gateway
docker exec agent-gateway sh -c 'command -v pdftoppm montage gs'
Two things to notice. First, USER root for the install and then back to USER node: the agent should not run as root just because it once needed apt. Second, when the upstream image updates, docker compose build --pull rebuilds on top of the new base and your tools come along for free.
To be honest about where this stood when I wrote it: the packages were installed and in the init script, the user had the contract (plus an apology), and the derived image was the next step, written but not yet rolled out. Until it is, any recreate has to be followed by the init script. That's a known gap, not a solved one.
Gotcha: don't touch ImageMagick's PDF policy
When you search for "imagemagick pdf", half the answers tell you to edit policy.xml and allow the PDF coder. Many distro packages ship ImageMagick with PDF/PostScript reading disabled on purpose, because it hands the file to Ghostscript, and feeding untrusted documents into Ghostscript has a long history of security bugs.
You don't need it here. The pipeline never asks ImageMagick to read a PDF:
pdftoppm(poppler) rasterises PDF to JPG.montage(ImageMagick) only ever sees JPGs and tiles them.
So the policy stays as shipped. Documents that arrive by mail and get pulled out of shared drives are exactly the kind of untrusted input that policy exists for. Ghostscript gets installed because the toolchain around these PDFs expects it, not as a reason to open the door.
The bug behind the bug: silent failure
Missing packages are a five-minute fix. What made this one painful was how it looked from outside. The user didn't see "a tool is missing". They saw an assistant that stopped working.
Two things I changed in how I think about agent containers:
Tools are dependencies. If the agent's instructions say "render a contact sheet with montage", then montage is a dependency, exactly like a Python package or a database. It belongs in the image, it gets checked after every rebuild, and "it worked last week" means nothing after a recreate. The command -v line above is a ten-second smoke test; run it after every rebuild or recreate, not after the next user complaint.
Failures must reach the user in a useful form. "image failed: Local media file not found" is a log line, not a reply. A useful reply would have been "I found the contract but couldn't check the scans, here's the most likely file", or at minimum "something on my side is broken, I've flagged it". The agent should degrade, not go quiet. And an error like this should also land somewhere I see it, not only in the chat of a user who won't report it for a day. I wrote about piping server errors into Discord, and the same idea applies here.
FAQ
Why not use a Python library instead of system tools? You can, and plenty of PDF libraries are pip-installable. Many of them still need poppler or similar system libraries underneath. Either way, the rule is the same: whatever the agent calls has to be baked into the image.
Can't the agent install what it needs itself? In my setup it runs as a non-root user, on purpose. I'd rather have a fixed, reviewed image than an agent that apt-get installs whatever seems handy at runtime.
If you're running an assistant like this for a business, its background jobs deserve the same scrutiny. The mail watcher on this same setup once taught me a hard lesson about idempotency in cron jobs. And if Docker itself is new territory, start with installing Docker on Ubuntu and the Docker vs Podman comparison.
