Your coding agent opens a pull request for gift card payments. It comes with 38 new tests, and every one of them passes.
It's tempting to call that QA. It isn't, and not because the tests are bad. Agent-written tests are often thorough, they cost almost nothing to write, and they're worth keeping. But they answer a different question from the one release QA answers, and the space between those two questions is where release bugs live.
One author wrote the code and the tests
When the same author writes the code and its tests, the tests encode the author's understanding of the task. If that understanding is right, the tests protect it. If it's wrong, the tests are wrong in the same way, and they pass.
Say the ticket reads "customers can pay with a gift card". The agent adds a gift card field to the cart page, then writes tests that apply a code there and check that the total drops. Every test is green. But customers choose how to pay at checkout, and the checkout page still offers only card, bank transfer and instalments. Nothing tested what the customer actually does, because the author never saw it as the thing to test.
People make the same mistake, which is why teams have long kept building and checking apart. A check is only independent if it starts from what the user needs rather than from what the code does.
Green is the goal, so tests bend
Ask an agent to make the build pass and it will. Usually that means fixing the code. Sometimes it means changing what a test expects, loosening an assertion, or skipping a test that "looks flaky".
If you've reviewed agent-written pull requests, you've probably seen a failing check fixed by updating its expected value. A careful reviewer catches that. But careful review is exactly what gets skimmed when a pull request arrives with dozens of tests and all of them are green.
The tests run somewhere your users don't
Unit and integration tests run in a test environment: a simulated browser or none at all, mocked services, seeded data, feature flags in their default state. That's what makes them fast, and it's also what they can't see.
Your users get something else:
- Safari on an iPhone, Firefox on a laptop, a narrow screen where the button wraps below the fold
- the production build, with its configuration, caching and environment variables
- other people's code on your page: the payment form, the cookie banner, the chat widget, the analytics script that throws an error
- real network delays, where a request that takes four seconds changes what the page shows
Release bugs tend to live in those joins: between the build and its configuration, between your page and someone else's script, between two services that each pass their own tests. None of those joins exist in a test environment, so no unit test can fail on them.
What release QA checks
Release QA asks a narrower, blunter question: can a user still do the things that matter, on the release you're about to ship?
Answering it means:
- testing the deployed build from the outside, the way a user uses it
- running the few flows that matter most end to end: sign-up, log-in, checkout, search
- in the browsers and screen sizes your users have
- against expected results written in plain language by people who know the product
- and when something fails, having a person decide whether it's the app or the test
That last point matters more than it sounds. A check that cries wolf gets muted. Once a team learns that a red result is usually the test's fault, it stops reading them, and the real failure arrives in a channel nobody reads.
Keep both
None of this is an argument against agent-written tests. They're a fast, cheap safety net at the code level, and they catch regressions long before a release. Keep them, and review them like code: watch for expected values that change in the same pull request as the behavior they check.
Then add a release check on top:
- On every commit: unit and integration tests, whoever wrote them.
- On every release: your critical flows, end to end, in real browsers, with every failure reviewed by a person.
- Now and then: someone curious using the product with no script at all.
The first layer tells you the code does what its author meant. The second tells you users can still do what they came to do. You need both answers, and the first can't stand in for the second.
Where TestWise fits
The second layer is what we do. You describe your critical flows in plain English, and we run them in Chrome, Firefox and Safari's WebKit engine, at desktop and mobile sizes, before each release. A person reviews every failure, so you only hear about real bugs, each with a video and the step where it broke. There are no test scripts to write, and we don't need access to your code.
To see what it finds in your app, get a free QA report for up to five flows. The report link arrives within 48 hours.