Nobody fully understands their own body. Billions of cells running processes no textbook fully maps, and somehow it works so well you can read this with ease. We accepted that trade-off because we had no choice. Evolution didn’t wait for us to understand it.
We are increasingly making the same trade-offs with our tech. Whether we know it or not, it’s already begun.
AI-generated code, contracts, financial products: all of it is getting too complex for people to fully explain. It’s already happening. We can’t stop it, so we should prepare for what to do when these systems fail.
What each side actually brings Link to heading
LLMs are relentless at logic. Feed one a tangled system and it will trace every branch, and find the inconsistency a tired engineer could easily miss. It’s good, and only getting better.
But humans carry something no model has: decades of tacit, unwritten experience of a world that never made it into any training set. Intuition that is more than just logic.
Today neither human or AI will win alone. The trouble starts when you need an answer now and only one of them is up to speed.
The 11pm fix that felt like luck Link to heading
I’ve lived this one. Production down, an AI-built service tangled in a cloud networking issue nobody could understand. My team is staring at code none of us wrote, clock ticking out of control.
So we ask the AI, and it confidently suggests a fix. It had inferred a constraint that existed outside of the logic itself, and untangled it faster than any of us could have.
It felt like a miracle. It was a machine doing at speed what would have taken a human days, weeks maybe. We shipped the fix & slapped each other on the back. Only the blind trust left me uncomfortable.
At the time it made sense because delay outage was expensive and the fix was cheap, and nothing catastrophic was riding on it not working for a little while longer. An easy call.
The system it fixed was relatively simple; but it, like all of ours, will only ever get more complex, and less understandable.
The crash the room can’t explain Link to heading
Raise the stakes: many financial instruments are already too complicated for most economists to fully explain. Add AI-designed products on top, ones built to optimize returns across variables no human tracks by hand. Give it five years, maybe ten, and plenty of these will be black boxes even to the people who built them.
A central bank sees AI-designed instruments start behaving strangely, then catastrophically. Margin calls trigger other margin calls. Markets seize. The room fills with the country’s best economists, and the chair still can’t say why it’s happening.
But the AI has a theory. It says: pull this lever, unwind that position, do it in this order, right now. The people in the room have do decide quickly, to decide whether to run with a fix they can’t audit, or delay and manage the consequences while humans catch up and the economy keeps plummeting.
No answer will feel comfortable. Move fast on an unknown fix, or wait while things get worse, and that choice is coming for more of us than we’d like to admit.
The rule that holds Link to heading
My rule: I decide based on the cost of being wrong. The LLM reasons well; it just doesn’t have lived context.
When the failure is cheap, an outage, a bug, a broken build, then trust it. Sure, we can increase the likelihood of good outcomes, but we have to be comfortable with the chance of bad ones too.
When the failure is catastrophic and hard to reverse, we need to force ourselves to slow down, even when every second feels expensive, even when the AI insists it knows, because understanding has to come before we act.
There’s no need to be dogmatic about it. Trust isn’t binary — match your nerve to the size of the wound.
We don’t get to opt out Link to heading
This is about matching your response to the stakes, not defaulting to caution or defaulting to trust. It’s a bet that the choice between running with a fix and stopping to understand it is coming for all of us, in finance, in business, in places we haven’t thought of yet.
Science may be demystifying our bodies, but our tech, our world, is only getting more complex.
Decide, calmly, before the fire alarm goes off, which of your systems can fail cheap and which ones can’t. Do that now, while you still have the luxury of thinking clearly.