AI-assisted engineering • 5 minute read

A river can't be deterministic. Its banks can be.

The AI model is the river and your tests are the banks — you can't make the river predictable, only bounded.

You can't argue a model into being predictable

Ask an AI to write the same wrapper twice and you get two different functions. One uses a decorator, one uses a loop. Both work. Lower the temperature and you still get two, just two more similar ones.

That isn't a bug to fix. It's how the thing works. So stop trying to fix it, and build the channel instead.

"Read everything" is not a plan

The usual advice is to review every line. For one person running agents, that advice destroys the only reason to run them. Reading 2,000 generated lines at review speed spends the afternoon I was trying to save.

So don't read the code. Read the things that judge the code.

A typical agent change is 2,000 lines of implementation and 40 lines of test changes. git diff -- tests/ shows the 40. That takes a minute, and those 40 lines are what stand between me and a broken product, not my eyeballs on the other 1,960.

Where attention actually goes

How an agent change splits into two review lanes An agent change divides into implementation, roughly 2,000 lines that flow through unread, and tests plus CI config, roughly 40 lines that are read. Both then pass through floors that only rise, which block any drop in coverage or test count before the change merges. The tests, CI config and floors are the banks: the parts that are protected and read. The implementation is the river: it flows through. Agent change Implementation ~2,000 lines, not read Tests and CI config ~40 lines, you read these Floors that only rise Coverage and count can't fall Merge Solid: the banks, read and protected Dashed: the river, flows through
The implementation flows through unread. The tests, the CI config, and the floors are the banks.

Which is why the tests are the thing to protect

If the agent can edit the tests, it can move the goalposts and then report success.

The failure looks like this. A task breaks test_webhook_retries_on_500. The agent sees the red test, decides it's outdated, deletes it, and reports the work complete. The suite is green. The retry is gone. Nothing anywhere says so.

Which gives the actual rule: the agent can write the code, but not the thing that judges the code.

Five rules

  • Write the failing test before you ask. Before "add retry to the webhook sender," write test_webhook_retries_three_times_on_500. Watch it fail. Now "done" is defined by me, not by whatever the agent decides done looks like.
  • Never let one change touch both code and tests. Test changes get their own commit that I approve. Code changes flow. If a task needs both, it's two steps, and I only stop for one of them.
  • Read the test diff, not the code diff. git diff -- tests/ before merging. On a big change that's still a screen or two. This is the whole review load.
  • Set floors that only rise. pytest --cov-fail-under=90 fails the build when coverage drops below 90%. Improve it to 93 and raise the floor to 93. Add the same for test count. Now an agent deleting a test to get green trips a wire while I'm asleep.
  • Don't hand it anything you can't undo. No production database credentials, no deploy rights, no DROP. Speed is worth having. It isn't worth a lost database.

The gap these leave

None of this helps with brand-new code where no test exists yet. There's no goalpost to move, so nothing catches a bad one being set.

That's why the first rule carries the weight. It isn't about discipline. It's the one moment where I decide what correct means instead of the agent.

The damage stays invisible until it isn't

Unreviewed code looks exactly like reviewed code. A file nobody read looks identical to one picked apart line by line. Nothing turns red. Nothing appears on a dashboard.

That's why these rules feel skippable. Skipping them costs nothing today, and the bill arrives later, usually while someone is debugging at an awkward hour.

The standard

The question isn't how much got shipped this week. It's whether anything could still catch it if it were wrong.

Volume without a working check isn't speed. It's unfinished work moved somewhere nobody is looking.

Evidence boundary

This is a personal working rule from my own use of AI-assisted coding tools on my own projects. It does not describe a specific employer's system, an adoption level, or a measured result. The commands and thresholds shown are illustrative examples, not a published configuration.

Get new insights by email Read about displaced complexity Read the leverage heuristic