I run 8 AI agents that manage my solo company — CEO, CFO, COO, Marketing, Accountant, Lawyer, CTO, and an Improver agent that upgrades the others. They share a persistent knowledge graph (JSONL file), consult each other through a central orchestrator, and post content to social media on a 5-minute cron. They've been running for months.
They crash. The interesting question isn't how to prevent that — it's how to make it not matter.
The gnarliest failure I hit: my agents share a knowledge graph through an MCP memory server. When multiple agents fire parallel tool calls (say, create_entities and create_relations in the same batch), you get a classic read-modify-write race. Both operations read the same JSONL state, both write back the full graph plus their additions. Second write obliterates the first. No error, no warning — data just vanishes. Sometimes the write gets interrupted mid-line and you end up with a half-written JSON line that breaks the parser on next load.
My fix was a local fork of the memory server with three things: an async mutex to serialize writes, atomic writes (write to .tmp then rename), and auto-repair on load that skips corrupt lines and deduplicates. But the meta-point is that on the BEAM, this entire class of bug doesn't exist. A GenServer processes messages sequentially from its mailbox — mutual exclusion is the execution model, not something you bolt on with a mutex. Supervision trees restart crashed processes in microseconds. Each process has its own heap, so one agent going haywire can't corrupt another's state.
Erlang/OTP solved this in 1986 for telecom switches that needed 99.999% uptime. The pattern maps almost perfectly to AI agents: many concurrent, stateful, failure-prone processes that need to communicate without taking each other down.
They crash. The interesting question isn't how to prevent that — it's how to make it not matter.
The gnarliest failure I hit: my agents share a knowledge graph through an MCP memory server. When multiple agents fire parallel tool calls (say, create_entities and create_relations in the same batch), you get a classic read-modify-write race. Both operations read the same JSONL state, both write back the full graph plus their additions. Second write obliterates the first. No error, no warning — data just vanishes. Sometimes the write gets interrupted mid-line and you end up with a half-written JSON line that breaks the parser on next load.
My fix was a local fork of the memory server with three things: an async mutex to serialize writes, atomic writes (write to .tmp then rename), and auto-repair on load that skips corrupt lines and deduplicates. But the meta-point is that on the BEAM, this entire class of bug doesn't exist. A GenServer processes messages sequentially from its mailbox — mutual exclusion is the execution model, not something you bolt on with a mutex. Supervision trees restart crashed processes in microseconds. Each process has its own heap, so one agent going haywire can't corrupt another's state.
Erlang/OTP solved this in 1986 for telecom switches that needed 99.999% uptime. The pattern maps almost perfectly to AI agents: many concurrent, stateful, failure-prone processes that need to communicate without taking each other down.
I wrote a detailed post about this with actual code and the full corruption story: https://dev.to/setas/why-erlangs-supervision-trees-are-the-m...