For many developers – not just those in the games industry – the idea of paying someone to test software seems like an unnecessary luxury. Test-driven development, live patching and an office full of highly qualified coders can lead some businesses to make strange decisions about the real value of testing.
In the first of a series of articles exploring the issues facing game developers, we welcome Nick Barrett, the founder and CEO of Proper QA to outline the reasons why using an external software testing provider may be the most sensible (and cost effective) choice.
“Why should we pay for QA?”
A question that’s been pondered in countless software environments for decades.
After working in QA for the last 20 years, it’s an easy question to answer: It will cost you more if you don’t.
“Then why bother paying testers when we can check our own stuff… We know it better than anyone else.”
This is a seemingly logical statement. Unfortunately, in reality it is almost never the case. Even if developers were able to test their own code as effectively as a tester, it wouldn’t give you the best results.
Ugly Babies
As creators, we are so enthusiastic about the fruits of our labour that it’s supremely difficult to see the flaws in the precious thing we’ve invested so much time in bringing to life.
If you were to unwisely ask a room full of new parents if any of them would describe their new baby as ugly and you’ll be met with silence and quite possibly, a sudden, violent exit. Ask the same room whether they consider their neighbour’s baby ugly, and the conclusions will almost certainly be different – although it’s unlikely these conclusions will be voiced.
Ask each set of parents in isolation and not only will you receive a truthful, unbiased result, but you can additionally ask them to rate each baby on an ugliness scale of 1-10 and have them provide detailed reasoning for each of their test results!
This is the difference between asking your devs to test their own code, asking them to test their colleagues code, or having an independent tester execute the QA instead.
Direct Cost
Unless you’re doing this wrong, a developer’s time will always be more costly than a tester’s. Essentially, you’d be using an expensive resource, which was hired to create code. They’re unlikely to have the up-to-date training, the relevant qualifications and, most importantly, the mindset that are required of a tester.
Confidence
Developers normally have a degree of faith that their code works. Occasionally this can be a 100% certainty, but certainly most developers believe they’ve built something good. That can be dangerous.
A tester approaches each new piece of software with none of these convictions and is therefore able to test without preconception or bias. This corresponds directly to the level of independence the tester has. An embedded tester who sits among the developers and has the benefit of an almost instantaneous turnaround on feedback, will often ‘go native’. They don’t want to make their colleagues unhappy, or be seen as the voice of doom, so they can often be less reliable than an independent tester.
Perspective
For years there has been a stigma associated with QA. Testing has been seen as an irritant rather than part of the process. It’s something forced upon developers that gets in the way of actual coding. For many developers, QA testing can often be seen as a reprimand, rather than an accepted part of the process. This is exacerbated when developers are asked to test their own or a colleague’s code, because the results are so often in line with their expectations of the outcome: “told you there wasn’t much to find…”
This perception routinely changes when an appropriate QA strategy is implemented and the results become apparent.
After reading an independently produced bug report, the most frequent comment heard from developers is: “But the user isn’t supposed to do that!” This is because the developer is so close to the project, or more commonly a part of the overall project, that they often cannot bring the end-to-end user perspective (or, less kindly, the sheer bloody end-to-end user ignorance),
It’s a matter of perspective. Developers take highly complex tasks and simplify them in inventive ways. The mindset is that of a creator, not a destroyer. A tester’s mindset is the opposite: Writing sophisticated test cases and executing them against simple features to break them in complex ways.
Internal or Outsource
There’s no one-size-fits-all solution. It very much depends what’s best suited for each unique project. Among other criteria, the decision typically depends on the volume, flexibility and frequency of QA required.
Larger companies, producing and delivering code on an ongoing basis can benefit from building an internal test team. As long as they have experienced management and a robust process in which testing is seen as a normal part of the development life cycle, they can produce very reliable and high quality results.
The test team is always busy, always focused and can turn builds around very quickly. The test resource is being used effectively and efficiently.
Companies with a different production schedule, or with a less intensive publishing requirement can often benefit from external support. Building and retaining a test team can be time consuming, and expensive, if they’re not being used on an ongoing basis.
So, why pay for a resource you’re not going to use all the time? Good external testing companies can provide expert, flexible and agile support, which fits into your own management and methodology style. They’re there when you need them and you only pay for the service you need.
Proper QA companies do exist!
A Proper QA company will be able to talk you through the entire process and help to ensure everything is aligned and integrated with your own internal development methods and tools.
If you’re interested in using an external testing company, don’t jump straight to Google. Ask around. Use events, meetups and networking opportunities to find a local company with a good reputation. Find someone who’s worked in a sector the same, or similar, to yours. Use shared contacts to find out more, or ask to speak to existing clients.
