Your agent's tests are the cheapest check you have. Don't make them the only one.

I run coding agents on my own projects, unattended. They open changes, the suite goes green, and the change lands. I read the test diff, not the two thousand lines underneath it.

That works right up until the check that was supposed to catch something wasn't watching. Here are five of those, all from code an agent wrote.

The API changed underneath the agent

The agent wired up a payment provider and mocked its response. Then the provider renamed amount_cents to amount. The tests stayed green for weeks, because they never called it:

mockApi.returns({ amount_cents: 500 });   // the agent wrote this response
                                          // then tested against it

One test that actually calls the thing fails the morning it changes:

const body = await (await fetch(`${API}/charges/${id}`)).json();
assert.ok(typeof body.amount_cents === "number");

A mock tests whoever wrote the mock. This tests the API.

The agent had more access than it needed

It wrote a tidy-up job and the WHERE clause was one condition short. Every test passed, because the fixture had four rows. No test protects you here. A missing permission does:

GRANT SELECT, INSERT, UPDATE ON app.* TO agent;
-- no DELETE, no DROP

The job fails on line one instead of finishing successfully and taking the table with it.

Same idea, sharper edge: the credentials it can read. An agent works by loading files into a context window, so a live key in a .env it can open is a live key it has read.

# on my machine, where the agent works
STRIPE_KEY=sk_test_...        # sandbox. worst case is a fake charge.

# the live key exists only as a CI secret,
# injected at deploy, never written to disk

It doesn't have to misuse a credential for that to matter. It only has to read one — and then it can end up echoed in a log, committed to a branch, or sent to a model provider as ordinary context. None of those are attacks. They're just Tuesday.

You can't test your way out of a permission you shouldn't have granted, or a secret you didn't have to share.

The agent changed the build. The site went down.

A build step stopped copying a folder. Every unit test passed and every page returned 404 — of course they did, no unit test ever asked for a URL.

curl -fsS https://staging.example.com/health || exit 1

One line, after deploy, fails the release. The first check that touches what a visitor touches.

Nothing failed, because nothing ran

A refactor moved a nightly job out of the scheduler. No test failed. No error was thrown. Nothing was wrong, because nothing happened at all — and an agent has no way to notice that its own change stopped something from existing.

A job that should report by 3am and hasn't by 6 is a fact, and only something watching production can produce it. This is the gap I ended up building PingStep for — it watches for the ping that doesn't arrive.

The failure is the absence of something, not the presence of a wrong value. Every other check on this page asserts on something that happened. This one is the only one that can notice nothing did.

The agent added an endpoint. A bad value walked in.

An order arrives with quantity: -5. The test used 5, because the agent wrote the test and chose the input:

const order = createOrder({ quantity: 5 });   // it chose this
assert.equal(order.status, "pending");        // so of course it passes

Green. And -5 walks straight in, because nothing was standing at the door.

A schema stands at the door. It's a description of what counts as a valid order, written once — zod here, but any validation library or a typed language does the same job:

const OrderInput = z.object({
  quantity: z.number().int().positive(),
  sku:      z.string().min(1),
});

// at the top of the handler
const order = OrderInput.parse(req.body);

Now -5 never reaches createOrder. Neither does "3", or null, or a missing sku, or the shape some other service starts sending next month.

The test is a sample. The schema is a rule.

Now ask who has to remember

• Contract test — nobody, it's in the suite
• Permissions and secrets — nobody, set once
• Staging request — nobody, it's in the pipeline
• Monitoring — nobody, always on
• A schema per endpoint — me, every single time

Four hold themselves. The one I just showed you is a habit — and a habit is worth nothing when the agent adds endpoint twelve at 2am and I'm asleep. It won't write the schema. Nothing anywhere will notice.

This is the part people miss about handing code to an agent. Advice like "remember to validate your inputs" was always addressed to a person. There's no longer a person at that step.

So don't remember harder. Make the absence fail:

// the only way to register a route
router.post("/orders", withSchema(OrderInput, createOrder));

// withSchema refuses a handler with no schema,
// so the agent cannot register an unvalidated endpoint at all

A forgotten schema is no longer a silent gap. The app won't start.

The rule

A check you have to remember isn't a check. Make its absence fail.

That's all a permission is. That's all a pipeline step is. Anything you have to remember per instance will eventually be forgotten — and when an agent is writing the instances, "eventually" arrives a lot sooner.

If you try one thing: take the check you're proudest of, and ask what happens the next time your agent forgets it. If the answer is "nothing", it isn't protecting you. It's protecting you so far.

Evidence boundary

Personal working preferences from my own projects, where I run coding agents on unattended changes. The five failures illustrate failure modes rather than reporting incidents — no customer impact, outage, or measured cost is claimed. Code shown is illustrative rather than copied from a running system. PingStep is my own product; it is mentioned because this failure mode is what it exists for, not as a recommendation over any alternative.