One of my automations quietly dumped over 1,500 duplicate files into cloud storage from a single incoming email. Same ten attachments, uploaded 153 times, spread across folders, over about five hours. Nothing crashed. No error was raised that anyone acted on. It just kept politely doing the wrong thing every two minutes until I noticed.

The root cause is one of those bugs that's obvious in hindsight and invisible in review, and it comes down to a single question you should ask of every cron job you own: if this runs twice, what happens?

TL;DR: The job did an irreversible side effect (upload) before it marked the work as done, and it only marked the work as done if a fragile later step succeeded. The later step kept failing, so "done" was never written, so the next run repeated the upload. Fix: commit the done-marker right after the side effect, not after the fragile step.

The setup

Simplified, the job looked like this. A cron fires every two minutes, finds unprocessed items, does something with side effects, then triggers a heavier downstream step:

def run():
    processed = load_state()            # ids we've already handled

    for item in find_unprocessed():
        if item.id in processed:
            continue

        upload_attachments(item)                 # <-- side effect: writes to cloud
        ok = trigger_downstream(item, timeout=180) # <-- flaky: 180s call, often times out

        if ok:                          # <-- only here do we record "done"
            mark_read(item)
            processed.add(item.id)
            save_state(processed)

Read that if ok: again. The entire dedup mechanism — mark_read, processed.add, save_state — is gated on trigger_downstream returning success.

What actually happened

trigger_downstream was a 180-second call to a much heavier process. Under load it kept timing out and returning ok = False. So:

  1. Run at 10:00 — upload the attachments, trigger downstream, it times out, nothing marked done.
  2. Run at 10:02 — the item is still unprocessed (we never marked it), so… upload the attachments again, trigger, time out again.
  3. Run at 10:04 — same thing. And again. And again.

153 runs × ~10 attachments ≈ 1,530 duplicate files. The upload was cheap and reliable; the thing I gated on was expensive and flaky. I had it exactly backwards.

The tell in the logs was brutal once I looked: 151 timeouts, 2 successes for that one item, and an upload-count that climbed in a perfect straight line every two minutes.

The fix is one idea, moved up

The irreversible side effect must be committed to state as soon as it happens, independent of anything downstream. Rewrite:

def run():
    processed = load_state()

    for item in find_unprocessed():
        if item.id in processed:
            continue

        upload_attachments(item)
        # commit "done" immediately — the upload happened, it must never repeat
        processed.add(item.id)
        save_state(processed)
        mark_read(item)

        ok = trigger_downstream(item, timeout=180)
        if not ok:
            log.warning("downstream failed for %s — will NOT re-upload; needs manual retry", item.id)

Now a downstream failure costs you one dropped downstream action that you can see in the logs and retry — not an infinite loop of duplicated side effects. That's the trade you want: a flaky step should fail forward, not backward into the part that already touched the world.

The general rule

Order your steps so the irreversible one is protected by the durable marker, and never gate that marker on a step that can fail for boring reasons (timeouts, rate limits, a service being briefly down).

A cron job is just a function the OS calls again and again. It will be interrupted mid-run, retried, and overlapped. If re-running it repeats a side effect, that's not an edge case, that's Tuesday.

Belt and suspenders

Moving the marker up fixes this specific bug. If you want the side effect itself to be safe under races, make it idempotent so a duplicate run is a no-op instead of a second copy:

  • Deterministic keys. Upload to a path derived from the item, e.g. sender/<message-id>/<filename>, not a fresh timestamped name each run. A repeat overwrites instead of multiplying. (My bug was made worse by timestamped, always-unique filenames — every run was guaranteed to look "new".)
  • Check-before-write when the target supports it: does an object with this key already exist? Skip.
  • A lock so two runs can't overlap. On Linux, wrap the job in flock and a slow run simply skips the next tick instead of stacking:
* * * * * /usr/bin/flock -n /tmp/myjob.lock /usr/local/bin/myjob.sh
  • A dead man's switch. The reason this ran for five hours is that "success" was the only thing that pinged me. A job that also alerts on repetition — "the same item processed N times" — would have caught it in minutes. Alert on the anomaly, not just the failure.

None of these are exotic. They're the difference between a cron job that heals and one that amplifies.

FAQ

Isn't the real bug the flaky downstream call? That's a bug, but it's the wrong one to fix first. Downstream calls will always fail sometimes — timeouts, restarts, rate limits are normal. The design flaw is letting a normal failure feed back into a side effect. Fix the ordering; then, separately, make the downstream call more robust.

Why not just mark the item read at the very start? You can, and for pure side effects it's often right. The subtle point is which action you're protecting. Commit the marker immediately after the step that touches the outside world irreversibly, so no retry can repeat that step. Anything after it is safe to lose and retry.

How do I retro-fit this without re-processing everything? Add the durable marker first (so the bleeding stops), then reconcile once: list what's actually in the target, dedupe by your deterministic key, keep one. I trashed ~1,520 duplicates keeping one of each — reversible via the trash, in case a "duplicate" wasn't.

Does this only apply to cron? No. Anything retried — message queues with at-least-once delivery, webhook receivers, CI steps, a user double-clicking submit. "At-least-once delivery" is the norm, so "safe to run more than once" is a requirement, not a nicety.

If you're wiring up scheduled jobs on Linux, systemd timers instead of cron gives you per-run logging and RemainAfterExit semantics that make this kind of state bug easier to spot in the first place.

Related posts