Crowd Sourcing QA and Testing

With the explosion of web enabled computing devices such as netbooks, smartphones and tablets, in addition to the more traditional desktops and laptops, it’s become strategically important for software
companies to have their products and services run on as many platforms as possible. This is in addition to support for the most popular operating systems.Having your application accessible not only from anywhere, but also from any device brings a slew of new problems though. Targeting a large number of different devices brings on increased complexity in the development cycle. Different languages, frameworks, environments and delivery methods all add to the design and development work that needs to be done. But what about QA?

Testing software on the smorgasbord of devices in the ecosystem can be prohibitively expensive – both in time and in money. This is where crowd sourcing testers comes in handy. There’s been a massive increase lately in the use of crowd sourcing for various parts of the software development cycle;  is this a buzzword du jour or can it actually provide tangible benefits to a company? The short answer is :
it depends.
On what you may ask? Well, on many things, and we’ll get to them in a little. Firstly though, for those unfamiliar with the concept of crowd sourcing in the context of software QA, here’s a brief rundown. Essentially, you find a company who has a bunch of testers on it’s books (usually many 1000s). You then pay said company some money (depending on the scale of your testing) and upload your application. You then let the testers loose on it in the hope that they’ll find bugs. Usually, the testers are working “on consignment” i.e you don’t pay them unless they report a bug. Sounds pretty promising doesn’t it? Instead of paying your own QA team (you knew they never do any work anyway…) you can just pay for the bugs that are found. Brilliant! On the face of it, it does sound like an interesting proposition, however, lets take a deeper look at what the reality is, and how you can maximize your return from crowd sourcing.

The Good

As mentioned previously, crowd sourcing your testing has some very convincing economic arguments. You are using the resources of the testers, while not paying them unless they find issues. You can also pay them on a test script basis, but still you’re saving yourself the overhead of 10s or 100s of testers. Additionally, since the testers are globally distributed, localized software can be tested with native speakers with the proper locale.

To maximize the economic benefits of crowd sourcing though, you need to do your homework first. Select which regions or countries you want to concentrate on. Select which devices are most important to you. Select which method of testing you will employ (script, ad hoc, functional, UI, localization).

The more narrow the focus, the better the results will be. You can always run several test runs in parallel if need be.

Give testers access to a known issues database; this could be as simple as a Excel spreadsheet which testers are required to check issues they discover against.

Define what deliverables you want. The testers do not know what information you need to fix the issue. You need to specify what you want from them in each report – screenshots, screen recordings, steps to reproduce, log files, system description. The testers need to know that these are all expected in a bug report.

The Bad

There is no guarantee of the quality of the testers. Some could be better than your internal QA department. Others have no software experience whatsoever, and are hoping to make a quick buck by working at home. It is very difficult to determine which is which, although many sites do provide rankings, and you can see statistics for each tester. Also, since payment is not guaranteed (unless they find a bug), the testers aren’t guaranteed to work. Although this is somewhat mitigated by the fact that given a large enough tester base, there will be at least a few people who do test it.

Of course everyone would like to have access to a huge group of testers who are already familiar with your software and possess the relative domain knowledge. Alas, this is not the case – you either get a lot of average testers, or you get a few expert testers. In crowd sourcing, you get the former, so a lot of time will be spent separating the wheat from the chaff, dealing with repeat bugs, incomplete bug reports, incorrect bug reports and such. Your overall administrative overhead will be slightly increased dure to this additional oversight you’ll have to apply. Overall though, it’s not all that bad. Again, the key is being prepared for it.

The Uselesss
I have to say, there really isn’t much for this section! Apart from the usual people insisting that they have a legitimate issue and arguing every step of the way with you, there’s very little to be worried about. Most crowd sourcing platforms are very well managed, so they deal with arbitration and disputes, and as long as you have a sound reason for not awarding a bug or issue there shouldn’t be too much hassle. Again, preparation is key. Decide the scope of bugs you’ll accept beforehand.

In conclusion, crowd sourcing does offer some great advantages. It is quite different to the usual way of doing things like QA, and it may not be to everyones taste. As I’ve mentioned before, the key is preparation. Do not jump headlong into it without the usual preparation you would do for internal QA (in fact prepare even more!). As long as you know what you need from crowdsourcing, crowdsourcing will help you achieve those aims.