Nobody fully understands their own body. Billions of cells, running processes nobody can yet fully describe, and somehow it works well enough to walk around and pay rent. We accepted that trade-off because we had no choice. Biology doesn’t ask permission.
We’re about to make the same trade-off on purpose. And soon.
AI-generated code, contracts, financial products, etc. all of it is getting complex enough that the people can’t fully explain it. That’s not a guess, it’s already happening. It’s unlikely we can stop this, we won’t. But we should prepare for what to do when these systems break.
What each side actually brings Link to heading
LLMs are relentless at logic. Feed one a tangled system and it will trace every branch, hold the whole thing in its head, and find the inconsistency a tired engineer would miss at 2am. It’s good, and only getting better.
But humans carry something no model has: a lifetime of experience and learning of a world that was never fully written down. Call it an uber context store. You know a decision feels wrong before you can say why. You’ve seen this exact kind of failure rhyme with one from ten years ago. An LLM reasons brilliantly inside the box it’s given. A human notices when the box itself is wrong.
Neither wins alone. The trouble starts when you need an answer now and only one of them is in the room.
The 2am fix that felt like luck Link to heading
I’ve lived this one. Production down, an AI-built service tangled in a concurrency bug nobody wrote down because it seemed obvious at the time. My team is staring at code none of us wrote, on a clock we don’t control.
So we ask the AI. Minutes later, it hands back a real fix, not a patch. It had inferred a constraint that existed nowhere except inside the logic itself, and untangled it faster than any of us could have opened the right file.
It felt like a miracle. It wasn’t. It was a machine doing at speed what would have taken a human hours: reading everything at once and holding it all in mind simultaneously. We shipped the fix, then read it properly once the fire was out.
That trade made sense. Delay was expensive, the fix was cheap to verify, and nothing catastrophic was riding on being wrong for five more minutes. Easy call, in hindsight even easier than it felt at 2am.
The crash nobody in the room can explain Link to heading
Now raise the stakes until they’re unbearable.
Financial instruments are already too complicated for most economists to fully explain. Add AI-designed products on top, ones built to optimize returns across variables no human tracks by hand. Give it two years, maybe ten, and plenty of these will be black boxes even to the people who signed off on them.
Picture the crash. A central bank’s AI-designed instrument starts behaving strangely, then catastrophically. Margin calls trigger other margin calls. Markets seize. The room fills with the country’s best economists, and not one of them can say why it’s happening. They understand pieces. Nobody holds the whole picture.
But the AI has a theory. It says: pull this lever, unwind that position, do it in this order, right now. The people in the room have minutes, maybe less, to decide whether to run with a fix they can’t audit, or hold the line and let humans catch up while the economy keeps falling.
There is no comfortable answer here. Move fast on a fix nobody can check, or wait while things possibly get worse. That’s the version of this problem I lose sleep over, because it’s coming, and I don’t think we’ve agreed on how to answer it.
The rule that actually holds Link to heading
Here’s what separates the 2am fix from the crash: the price of being wrong.
When the failure is cheap, an outage, a bug, a broken build, trust the fix and check your work after. Speed wins, and being wrong costs you an afternoon.
When the failure is catastrophic and hard to reverse, force yourself to slow down, even when every second feels expensive, even when the AI insists it knows. You audit before you act, because being wrong here doesn’t cost you an afternoon. It costs something you can’t get back.
That’s the whole rule. Not “trust AI” or “don’t trust AI.” Match your nerve to the size of the wound if you’re wrong.
We don’t get to opt out Link to heading
This isn’t an argument for slowing everything down, and it isn’t cheerleading for handing over the keys either. It’s a bet that this exact fork in the road is coming for all of us, in code, in finance, in places we haven’t thought of yet.
The only real preparation is deciding, calmly, before the fire alarm goes off, which of your systems can fail cheap and which ones can’t. Do that now, while you still have the luxury of thinking clearly.
Because when the alarm does go off, you won’t have time to figure out which kind of problem you’re standing in front of. You’ll only have time to act on the answer you already gave yourself.