If you run a fintech, you'll know the rule that the person who writes a change can't be the one who approves it. I'm a co-founder of our company, its COO and interim CTO. We're a fintech whose product moves real money, and AI coding agents write its code. Neither Miguel, who builds our interface, nor I have written a line of that code by hand in a year and a half, so the agents are the authors, and the rule has to hold for them too. An agent saying "done" doesn't make anything done here. If an agent's word is enough to close the work on your team, the agent is reviewing its own work.
The agent's own "done"
The rule is right, and an author approving its own work is exactly what it's there to stop. An agent marking its own work done is that same failure. Every time an agent says something is done or fully wrapped up, it never is. Agents do the bare minimum to satisfy the criteria, then tell you it's done when it isn't done to the standard you'd want. In a system that moves money, taking that at face value means real money moving through work nobody proved.
So at our company the agent's report closes nothing. Work isn't marked finished until there's evidence from staging on record. Staging is a working copy of our system that customers never touch, where changes are tested before they go live. The evidence is tests passing there, such as the quick checks that the main paths still work, written down against the piece of work. The list where we track our work gets corrected to match what staging shows.
Who checks the evidence
Miguel or I review all of that evidence, and we also do our own testing and checking of the work by hand.
There isn't a single second in the day when agents aren't building something for us. You may wonder whether that review slows the work down. Review does take our time, but I don't think that time is wasted. Time the agents spend waiting on one of us is where the important decisions are made, and you need a person in the loop to make sure they're the right ones. The alternative is to let the agents run, keep a log of what they decided and review it all at the end, but that could take longer to unwind any wrong decisions they made.
Who decides, and who can stop it
The agents carry out our architecture and our product plans; they don't make business decisions or product decisions. I don't know why you would ever expect an agent to know the right business or product decision. I make the architectural decisions, and Miguel focuses on the front end: the interface and the components it's made of. We often talk through specific features, but our responsibilities are different, and they fit together to get the whole product built and shipped for the people who use it.
One week in October, a bad change was caught in staging, our test environment, and taken back out the same day, before anything reached production and without holding anything else up. For what's live, I've put a considerable amount of work into switches that turn features and services on and off, emergency stops included. It isn't only me, or someone else with developer access to our systems, who can use them: people on our internal teams can too, so protecting the system in an emergency doesn't depend on one person.
A check that never ran
The automated checks are the other place agent-written work can look approved when nothing has proven it. When a check can't run and gets skipped, nothing turns red, so it looks the same as a check that passed. In February I wrote that "tests should provide genuine confidence," and that "performative tests provide false confidence, which is worse than no tests."
At our company a check that can't run fails. Where our system repairs a fault on its own, each switch that does the repair has to declare in advance what it does if the repair fails. If a check that can't run shows up as a pass in your pipeline, your team can approve agent-written work that nothing ever checked.