Test automation is software, and buying it from a vendor means commissioning a second codebase that you will own and maintain for as long as the first one. Most automation disappointments are not about the tests being wrong; they are about nobody agreeing in advance who keeps them working.
A large suite is not the goal and can be the problem
Vendors are incentivised to deliver test count because it is visible and billable. What you actually want is a small, fast, reliable suite covering the journeys that would genuinely hurt if they broke, plus the discipline to add to it carefully. A thousand slow tests that fail intermittently is worse than fifty fast ones that never lie, because a suite people stop believing is a suite that has stopped working while still costing money.
Flakiness has to be a contract term
A test that sometimes fails for no reason is the thing that kills automation programmes, because teams learn to rerun rather than investigate and then miss the real failure. Agree in advance what flakiness rate is acceptable, who is responsible for fixing an intermittent test, and within what period. Require that a test which cannot be made reliable is deleted rather than retried, because an unreliable test has negative value.
What an automated software testing company must hand over
The automation belongs in your repository beside the application code, running in your pipeline, not in a vendor platform you lose access to at the end of the contract. Require that from day one. Require documentation of how to run the suite locally, how test data is created, and how to add a test, written for someone who has not met the vendor. Automation you cannot run or extend yourself is a subscription, not an asset.
Runtime is a design constraint, so state it
If the suite takes an hour, developers will stop waiting for it and it will stop preventing defects. Set the maximum acceptable runtime as a requirement at the start and make the vendor design within it: parallelisation, sensible layering with more fast tests and fewer slow end to end ones, and honest decisions about what does not need automating. A runtime target shapes the architecture, which is why it has to be a requirement rather than a hope.
Questions people ask about automation testing company
How do we judge proposals from software test automation companies?
On suite health rather than suite size: a maximum runtime, an agreed flakiness threshold with named ownership, tests in your repository and pipeline, and documentation written for someone who has not met the vendor.
What do we do about flaky tests?
Make them a contract term. Agree an acceptable rate, who fixes them and by when, and require that a test which cannot be made reliable is deleted. An intermittently failing test has negative value because it teaches people to ignore failures.
Where should the automated tests live?
In your repository beside the application code, running in your pipeline. Tests held in a vendor platform are a subscription rather than an asset, and you lose them at exactly the moment the relationship ends.