The four engagements Coverage in automated tests
Coverage in automated tests
From two calendar weeks. The calendar depends on the number of flows, agreed after the Assessment. For a team whose releases are gated by a manual pass nobody has time for.
Coverage here is a list of the things that must not break, and whether each of them is checked on every change or only when somebody remembers.
What gets built
An AI system built for your application. Not a generic generator pointed at your screens — a system that knows what each element actually is, where your screens contradict each other, and which traps your application has.
It stays after I leave. Your engineers write the next test case and the automation under it against a system that already knows the application, rather than starting from a blank file.
What it does not do is decide what is worth covering, or whether a result is right. Those stay with me.
How the flows are chosen
By cost of failure, not by ease of automation. Those two lists rarely agree, and the second one is how suites end up covering the settings page and not the checkout.
The number of flows is agreed after the Assessment — you can name them yourself.
The trap catalogue and the tool selection carry on through the build rather than ending with the Assessment: a trap is a property of running code, so the catalogue grows as the suite meets the application, and every entry that costs hours is priced where it is confirmed.
How far down each check goes
End to end means the interface, the API and the database underneath. A green test that only proves the confirmation screen rendered is a test that will pass while the order row is wrong. Checking the row is the part a browser-only tool cannot reach.
What runs, and when
Every change, with an isolated environment per run. Isolation is what makes a red build mean a defect rather than a neighbour, and it is why people act on the suite instead of muting it.
Test data is created and cleaned up by the suite itself, rather than living in a shared account that somebody else edits on a Tuesday. You can check this yourself: run the suite twice in a row, then twice at once, and get the same result both times.
What stops a test that proves nothing
A test that cannot fail proves nothing. Two things stop that. One runs in CI on every pull request and blocks the merge when a test has no assertion, is skipped, or is registered in no CI group. The other breaks the system on purpose and checks that the test goes red and names what broke — that one reports into a scorecard, not into the pipeline, because it needs a live system.
How the two work
The mechanical one runs in CI on every pull request, as a required status check. It catches the literal cases: a test with no assertion in it at all, a skip or a fixme added in the diff, an assertion wrapped in a try/catch so it cannot fail, a tolerance widened to make a comparison pass, and a spec file registered in no CI group — which runs nowhere and so proves nothing wherever it sits. When it fires the merge button goes inactive, and the pull request gets a comment naming the file, the line and which of those it was.
A pre-commit hook is not the gate and is not sold as one: --no-verify removes it in a
single command. The gate is the required check, in branch protection.
The second layer is the one that actually proves a test can tell working from broken, and it is deliberately not a merge gate. The state is broken on purpose against a live environment, and the test has to go red — and the failure message has to name the thing that was broken, not just fail. That needs a running system, a write to the database and a rollback, so it runs as its own pass and reports into a scorecard rather than into a pipeline.
Repository administrators can bypass branch protection. That is true of every gate on GitHub, and it leaves a trace: the organisation's audit log records it, and the pull request timeline says it was merged without waiting for its checks. The bypass exists and it is visible; a silent one does not.
What you get at handover
The suite, in your repository. Tests, page objects, fixtures and CI configuration, written in what your team already runs.
The record. Every element identified with evidence of where it came from, the contradictions between screens named with the reason, and the trap catalogue: the traps found in your application, written down. This is the part that took the time, and it is the part that makes the next flow cheap to add.
The runbook. How to run it, how to add a flow, how to read a failure.
Four numbers, so you can check any of this without me:
- How long a full regression takes.
- How many revenue-carrying flows are covered.
- How many production defects landed in covered ground.
- How long a red run takes to fix.
Measure them again in three months, with somebody who has never met me.
The suite is yours. Open source tools, no licence, nothing of mine you need to keep paying for.
What you provide
A staging environment with database access, by an agreed date. Credentials and VPN, if needed. A sandbox for any third-party service — payments, SMS — that you hold the contract for.
If staging does not exist, scope drops or the calendar moves; I will say which before we start, not on week three.
What can be added
- Team training
- One day on the suite and how to extend it, priced separately. Worth it when your team has never owned automation before. Documentation gets skimmed; a day does not.
- Performance baseline
- Load testing with K6, front-end audits with Lighthouse. Worth it when you have no numbers for "is it getting slower", which is most teams before someone complains.
- Accessibility scan
- axe and Lighthouse across the covered flows. Worth it when you have a public sector client or a procurement questionnaire in front of you.
- Support after handover
Optional, priced with the project. A named owner for the suite while your team learns it: a few hours a month, two working days to respond. Covers breakage and questions; new flows are quoted separately. A fixed three-month term from handover, ending on a date agreed at the start. It does not renew unless you ask for it.
Separate from this: anything that was broken when I handed it over, I fix for thirty days at no charge, as part of the handover rather than a billed add-on.
Show me the application.
Seven questions. No discovery call — the answers here do the same job, and you keep your calendar.
For coverage, questions three and four — what people do in the product, and what it costs when each of those breaks — are the ones the scope is built from.
Only three answers are required — your stack, the list of things people do, and what a failure costs. An incomplete form beats an unsent one.
Not ready to scope anything? Write anyway. I will tell you what I would do first, and you can come back when it matters.
What gets composed — in your mail client, not sent anywhere by this page
This form builds an email in your own mail client — there is no form service behind it, no endpoint, and nothing reaches this server. It needs JavaScript. With scripts off, write to the address below and answer the seven questions above in your own words.
If staging does not exist, say so — it changes the scope or the calendar, and I will tell you which before we start, not on week three.