Almost every conversation about AI I see right now revolves around orchestration: how to coordinate multiple agents, which framework to use, which supervisor pattern scales best. It is a necessary conversation. I work with a team of agents myself.
I think, though, that we are paying less attention to something that matters just as much or more: human understanding.
An orchestrator can solve complex problems, yet without human understanding, logic errors or a misunderstood intent can slip past unnoticed. With those errors hidden, the system becomes hard to maintain. And as it grows, scaling can turn into chaos.
In my own work, the hardest part is not coordinating the agents, it is continuing to understand what they did. I wrote earlier that the expensive part of an agent is not tokens, it is attention. This post is the flip side of that: what that attention is for. And it ends with a proposal I have not figured out yet.
A paper that helped me put a name to it
A few weeks ago, Why and How to Do Science When AI Is Better at It, by Hoda Heidari and Aydin Mohseni, came out on arXiv. It is a paper about science rather than companies or software, though the argument carries over to other areas that rely heavily on AI, including my work with agents.
They start from a deliberately strong premise: that AI is already more accurate, less biased, and more efficient than human researchers. Even then, they conclude that research must remain human-guided to preserve collective, durable human understanding as the point of science. As a rule for delegating research tasks, they require that the community still be able to comprehend, defend, and question the result on its own, and keep the skills needed to do so in the future. That does not rule out using verified results nobody understands yet, as long as they are marked as such and explaining them is prioritized afterward.
The idea that stuck with me most is the distinction between a verified result and an understood one. Verified means it passed the checks the community accepts, such as replication or a formal proof, without necessarily implying a true result. Understood means a collaborator can also explain it, defend it, and question it.
With that distinction in mind, I see three linked risks:
1. Logic errors nobody sees
A PR with every test green has passed those checks. That does not prove it is right in every case, much less that anyone understands it. Tests check what someone thought to test, and usually the AI itself is the one writing them. A logic error, a misread business rule, or an unwritten assumption slips right through that gap.
Reviewing does not always close that gap either. In Automation Bias in Mammography (Radiology, 2023), Thomas Dratsch and colleagues had 27 radiologists read mammograms with suggestions from a purported AI system. Inexperienced readers got it right roughly 80% of the time when the suggestion was correct, and under 20% when it was wrong. Highly experienced readers dropped too, from 82% to 45.5%. Heidari and Mohseni cite this as evidence of anchoring: seeing the AI’s answer first biases your own evaluation. Even though it is not a code-review study, I worry about a similar risk when we plan a spec or review a PR starting from what an agent says and assume it is right.
2. Later, a system that is hard to maintain
An error nobody caught does not go away. It shows up months later, in production, and someone has to track it down. Doing that requires reading code, forming a hypothesis, and debugging.
In How AI Impacts Skill Formation (2026), Judy Hanwen Shen and Alex Tamkin ran randomized experiments with developers learning a new async library, with and without AI. With AI, conceptual understanding, code reading, and debugging ability came out worse, with no significant efficiency gains on average. Those who delegated all the code did gain some productivity, at the cost of actually learning the library. Patterns that involved cognitive engagement preserved learning even when AI was used.
Using AI was never my worry. What worries me is assuming, consciously or not, that its outputs are correct and, out of time pressure or day-to-day demands, moving on to the next task without a solid grasp of the previous answer.
3. And at scale, chaos
In a small system, an error nobody noticed is just a bug. Once other components start treating that behavior as valid, fixing it is no longer an isolated change: you also have to revisit everything built on top of it. If one agent wrote the code and another reviewed it, and no human took the time to read or debug it, then when it breaks it will be hard first to spot the bug, and second, even if you spot it, to know where to start fixing it. An agent can burn hours reading context just to figure out what happened.
The paper describes that dynamic for science: if AI output grows faster than our capacity to absorb it, the share of verified things that anyone understands drops. And it cites Gradual Disempowerment (2025), by Jan Kulveit and colleagues, for an idea I find most useful when thinking about a company: a society that can act on the world but no longer understands it cannot recognize or correct its own mistakes, and gradually hands control over to the systems it depends on.
I see a similar risk in a company that can no longer explain its own production system. Orchestration can help it produce more, yet it does not guarantee that the team understands what comes out of it, much less that it will make the right calls when scaling or responding to growing demand.
What I still do poorly
To be honest, I do not review everything I delegate to AI. It is very hard, given the volume of information and because the industry demands productivity from us, not understanding. Most of the time I choose to implement and verify that it works first, which is what clients validate, before understanding it completely.
My tool artifacts-manager helps me generate artifacts and diagrams that make understanding easier. Even so, those artifacts and that documentation are also generated by AI, and they are vulnerable to hallucinations and bias.
With that in mind, I decided to trade a bit of productivity for more rigorous, thorough, and above all scalable reviews. The challenge does not end there: we need better tools that help us build our critical thinking. If AI always hands us the answers, when do we exercise certain critical skills?
A bold proposal: let AI ask us the questions
You can check off “Understood” on a checklist or a PR without really having grasped it. A checkbox proves nothing; it only records what I think I know.
That is why I think AI itself should include an evaluator of human understanding. What I am proposing is that, before marking a task done, the AI generates questions about what was implemented that we have to answer to close it: for example, why it was solved that way, what assumptions exist, or how to roll it back. An LLM could act as a judge of those answers.
It will not be perfect. The judge is also a model, with its own hallucinations and biases, and answering well today does not guarantee being able to defend the change later. Still, it is one more filter. It could get us closer to evaluating part of what the paper calls “understood” (being able to explain, defend, and question the change) instead of stopping at verified. Tests check use cases, whereas questions would look for signs that someone can give reasons for what was built.
My bet is that answering questions about the change could introduce into the workflow the cognitive engagement that, according to Shen and Tamkin, preserved learning. At this point, though, that is still a hypothesis inspired by their study.
I have not solved this for myself yet. Open questions remain: how many questions are reasonable before closing every task turns into a chore, what context the judge would need to evaluate answers, how to keep it from approving convincing but wrong explanations, and whether to apply it to everything or only to changes that are expensive to reverse. And there is an uncomfortable objection: I could ask another AI to answer for me. How do you tell my own judgment apart from a generated answer? I have not solved that either.
Both matter
Going back to the topic that started this debate, orchestration: I am not against it. I use it every day, and it lets me do things I could not do with a single agent. Even so, in my work, the limit I worry about most is less about how many agents I can coordinate and more about how much of what they produce we can still explain and fix.
I am going to test this proposal in my own workflow. In a second part, I will share how it went: what worked, what did not, and what I would change. I will also go into more detail on the practices I already use with my team of agents to leave a trail of what they do.
In the meantime, I will leave you with the question I asked myself while reading the paper: in your system, which part is verified yet nobody understands? And what happens if it breaks tomorrow?
