Why Good Systems Fail Good People

One of the more disorienting lessons of a career spent designing systems is realizing how rarely bad outcomes require bad intentions. Most of the worst software failures I've investigated were not caused by careless or malicious engineers. They were caused by a system where every individual component was behaving exactly as it was locally instructed to, and the sum of all those individually reasonable behaviors produced a result nobody, at any point, actually chose. That lesson has stuck with me far beyond software, because I think it's one of the more useful, underused frameworks for understanding why institutions — governments, large organizations, entire bureaucracies — so often produce outcomes that seem to serve no one, even when most of the people inside them are, individually, decent and trying.

In distributed systems, there's a concept called an emergent failure — a breakdown that can't be traced to any single faulty part, because it arises from the interaction of parts that are each, in isolation, working correctly. A queue that's fine on its own backs up because a downstream service is fine on its own but slightly slower than expected, which causes a timeout in a third service that's also fine on its own, and the whole system grinds to a halt with no single villain anywhere in the chain. I think about this constantly when I try to understand why an institution I'd otherwise expect to work — staffed by people who mostly want to do right — keeps producing results that serve almost no one particularly well.

This isn't a comfortable frame, because it removes the satisfying simplicity of a villain. It's much easier to explain institutional failure by identifying the corrupt official, the incompetent department, the single bad actor — and sometimes that explanation is even accurate. But in my experience studying and building complex systems, the more common and more dangerous failures are structural: incentive misalignments where each part of the system is locally rational and the aggregate is not, feedback loops too slow for anyone to correctly attribute cause to effect, information that degrades in quality every time it passes through another layer of the hierarchy on its way to the people who could actually act on it.

I think this matters for how we hold institutions accountable, because if the failure is structural rather than purely a matter of individual bad actors, then removing the individuals — while sometimes necessary and deserved — doesn't necessarily fix the underlying dynamic. You can replace every person in a poorly designed system and, given enough time, watch it produce roughly the same failure again, because the structure itself is what's generating the outcome, not any specific person occupying a role within it. This is uncomfortable to sit with, because it's a less satisfying story than "fire the person responsible," but I think it's closer to what actually explains why so many institutional problems persist across entirely different generations of leadership.

None of this is an argument for excusing individuals from responsibility — accountability for specific decisions still matters, and structural explanations can become a convenient way to avoid it. But I think the more useful, harder question, the one systems engineers are trained to ask and that I wish got asked more often about institutions, is: even if every person currently in this system were replaced with someone equally well-intentioned, would the structure still produce this same failure? If the honest answer is yes, then the fix has to happen at the level of incentives, information flow, and feedback loops — the actual architecture — not just at the level of who currently occupies which seat.

I don't offer this as a tidy solution to governance, which is a far messier and higher-stakes problem than anything I design professionally, and I want to be honest about the limits of the analogy — institutions involve human values, competing legitimate interests, and questions of justice that a distributed systems framework doesn't capture on its own. But I've found that borrowing this specific lens — looking for structural, emergent causes before reaching for individual blame — has made me both more patient in diagnosing what's actually broken and, somewhat counterintuitively, more precise about where responsibility genuinely does belong, because it stops getting diffused across an entire system when it should be sitting squarely with a person, and stops getting pinned entirely on a person when it should be understood as a structural failure that will keep recurring until the structure itself changes.