The 2nd Myth of Software Testing - Testing == Automation == Quality

Testing == Automation == Quality

The obvious response to this is to reply with:

‘What exactly do you mean by automation?'

Pretty soon and no doubt after some debate, you might arrive at:

• execution of a test script with no / minimal manual input

• using a mechanism / toolset for executing a set of predetermined steps against a predetermined environment with (usually) predetermined test data….but (usually) != testing. It definitely == checking though

In fairness, I’ve experienced the statement from teams that have been on the receiving end of:

• QA practitioners that are competent at coding but light on QA fundamentals

• Poor application of QA practices prior to the automated test coding effort

• Stakeholders that have read a pro-automation article in middle management weekly that promises to solve all perceived quality issues by judicious application of automated checks leading to Continuous Integration

• Coders that have been roped in to assist with a test effort without any context as to the why

• Toolset vendors looking to hawk their latest silver bullet

As QA practitioners we can directly influence the first 4 points through education.

But automation in and of itself != Quality

The final point would (hopefully) be mitigated by those same stakeholders taking more than a passing interest in the monetised snake oil as a result of education and demonstration…or spend time with the people that have to implement the toolset to find out what they actually need to solve an identified problem.

We demonstrate that QA isn’t just automation by our actions, interactions and most importantly, examples.

But automation in and of itself != Quality

We want to ensure that the idea / user story / requirement has been questioned, the appropriate risks have been identified and mitigation has been understood.

We plan for accurate activities to ensure coverage of the expected functionality, we collaborate with the business to confirm understanding.

We engage the full team in these activities.

But automation in and of itself != Quality

Automated test execution does have undoubted benefits:

• Regression cycles can be significantly shorter

• The feel good factor of your app (outwards to the client and inwards to the team) can be enhanced

• It frees up time to undertake deeper testing, be that exploratory or ad-hoc

We ensure that the automated checks we create have value to the team and by extension, the client;

We do this on a regular basis by reviewing the effectiveness of the test pack, the methods employed, the thinking behind the choices. We prune where required, we enhance where necessary.

But automation in and of itself != Quality

As QA professionals, we need to get better at advocating that what we do matters. Automated test execution is one part of this.

But automation in and of itself != Quality

In the same way that a castle built on sand isn't a solid base for a defensive fort, automation built on a lack of care around quality fundamentals isn't a good baseline for an automation effort.

The problem with common sense is that it isn't that common

As with most things, the single most important question prior to any automated effort is:

What problem is this intended to solve?

In my experience, teams that ask this question at the outset have a far greater chance of achieving a realistic and manageable outcome to any automation efforts.

They also stand a far greater chance of creating artefacts that contribute to the stated quality goal

So, maybe the statement should be posited as:

Targeted Automation == Contribution towards the quality goal

I can live with that.