I was involved in an interesting discussion on Twitter, the other day. It started with a tweet by @testingqa (Guy Mason): “Still of the opinion that ‘Automated Testing’ is a deceptive term,no testing is being performed,it should be called ‘Automated Checking’ #qa“. With that he probably refered to Michael Bolton’s blog that there is a difference between testing and checking.
After that blog lot’s of people, mainly automation sceptics, stated that Automated Testing should be called Automated Checking. Although I acknowledge and agree that there is a difference between Testing and Checking, I don’t think it should be called Automated Checking. I’ll explain below why not, but first the rest of the Twitter conversation:
I responded to @testingqa’s tweet with: “@testingqa but that’s not true either. Automated Testing is more than checks. There are actions as well.”
(@michaelbolton: “. @testingqa I agree, but I think @AdamPKnight expressed things well. He specified checking, so I read “automat[ion assist]ed tests”. #qa“)
@testingqa: “@ArjanKranenburg Yes, actions are taken which verify behaviour is consistent with expectations. But that’s still a check.”
@arjankranenburg: “@testingqa I meant actions before you can start checking. You have to tickle the SUT before verifying the response.”
@arjankranenburg: “. @testingqa and then there are preparations, cleanup, reporting, etc. Automated Tests is so much more than just Checks. #testing #qa”
@michaelbolton: “. @ArjanKranenburg @testingqa Test automation (any use of tools to support #testing) should be much more than checks. Alas, often, it isn’t.”
@arjankranenburg: “. @michaelbolton @testingqa I think it often is. E.g clicking a button is an action, verifying the response is a check. #testing #qa”
@arjankranenburg: “. @michaelbolton @testingqa I understand you don’t want to call anyting automated a test, but it’s more than a check. #testing #qa”
@michaelbolton: “. @ArjanKranenburg Your assertion that I want to call anything automated a check is incorrect. #testing #qa”
@michaelbolton: “. @ArjanKranenburg Something is a check when it doesn’t involve cognitive engagement. Tools can extend cognitive engagement. #testing #qa”
@michaelbolton: “. @ArjanKranenburg #Testing tool use turns into checking when it *displaces* cognitive engagement. #CultOfTheGreenBar #qa”
@michaelbolton: “. @ArjanKranenburg Note that risk identification, design, and programming–the preparation–are #testing activities, requiring sapience. #qa”
@michaelbolton: “. @ArjanKranenburg Verifying *one factor* of the response is a check. Checking focuses on output; #testing on outCOME. Beizer might agree.”
@testingqa: “@ArjanKranenburg Automation does gather info & does allow one to verify response but distinction remains that it only checks to verify…”
@testingqa: “@ArjanKranenburg …if it meets expected outcome (or not). Preparing/cleanup does not reveal new information though and reports based on…”
@testingqa: “@ArjanKranenburg … checks only verifies against expectations still. As @michaelbolton said it can be used for more, but most times do not.”
To summarize Michael’s blog, but please read all his blog-series on testing vs checking because there is more to it, an important difference between testing and checking is that testing requires cognition to interpret the information revealed by one or more checks. But I’d like to extend that as I think testing is more than checking + interpretation.
And this becomes apparent when trying to automate a test. The base of a test consists of actions, checks and interpretation of the retrieved information. Most Systems Under Test (SUTs) need to be tickled before it responses. You need to click a button, send a request, press a key, etc. etc. In theory, the SUT can do things without an external stimulant, but in most cases it doesn’t.
Then the SUT responses and that response can be checked for certain aspects and the revealed information must be interpreted. If you state that Automated Testing should be called Automated Checking because the cognitive part can’t be automated, you’re ignoring the actions that can and often need to be done as well.
And there is more:
- Before starting the actual test, you need to prepare the SUT to make sure it is in the correct state for the test to execute.
- Automated TCs are often run in a batch, so it is good practice to restore the SUT to its original state.
- Since interpretations of the results is still needed, the results need to be presented in a tester-friendly way. What and how you are reporting is very important.
- etc. etc.
All these activities can be automated as well and are often included in the automated test script.
My point is, if you don’t want to use the term Automated Testing, call it Automation Assisted Testing (I like that), but Automated Checking simply doesn’t cover the activities done in Automated Testing.
Hi, Arjan…
Thank you for pointing people to the blog series.
I agree that automated checking doesn’t cover the activities done in automat[ion assis]ted testing, and I agree with your admonition not to call the latter by the name of the former. At the same time, I also counsel the opposite too: I observe that some individuals and groups become fascinated with or dazzled by the checking, and forget the important of the testing activities that surround it. I try to emphasize this in http://www.developsense.com/blog/2009/11/merely-checking-or-merely-testing/.
Again, thank you for the link.
—Michael B.
Hi there!
I’m not an “automate everything” enthusiast neither am I an exploratory testing crusader. I stand for a systems thinking approach.
The first thing to keep in mind while reading “Testing vs. Checking” series is a kind of automation talked about. It is unit testing, mainly with regards to TDD approach. Sure, an executable process which is essentially a calling of a function with a bunch of inputs and checking results it returns perfectly fits definition of automated checking. However, it’s a shame to downgrade unit testing activities to unit checking only. Code review is as much sapient activity as exploratory testing.
I tried to emphasize that aspect in one of the questions I asked.
The main question, that wasn’t answered, is also – why compare code testing with product testing? While sapient testing skills can be (should be) applied at any level, call-checking of code components pursues a different goal than testing of application’s functionalities. One does not substitute another.
Opposition towards using automated tools is called Ludditism. Yes, machinery threatens to scripted types of jobs. For others, it helps increasing productivity or reaching new levels of exploration.
Tools help in different areas. Sometimes they do not substitute manual activity: you can cut one’s body to see what’s inside or you can X-Ray it. Just keep in mind that a tool doesn’t actually see (i.e. perceive) – it brings a picture for a human being to perceive.
You can manually dig an archeological site or you can use electromagnetic scanning and build a 3D picture with a computer. Once again, keep in mind, a program won’t care what is on display.
In software testing, activities where engagement of automation tools helps reaching new levels would be simulated load testing or certain kinds of security testing.
In automation of GUI activities, recording/checking tools occupy a thin niche of regression testing approach, based on the heuristic “if a functionality is not broken then the test passed last time will pass again”. But this heuristic is not really reliable, it doesn’t work too often.
Thank you, Albert Gareev
http://automation-beyond.com/
Hi Arjan
Do not know what happened to my last comment but I will try again
After reading through your article again I can see the point you are making.
I do have a problem where automation is used to ‘test’ a product and a large amount of time and effort is spent putting together a framework for the automation, writing the automated code and then running the automated code.
The actual creation of the automated ‘tests’ requires interaction with a human the running of these ‘checks’ does not. They become static checks not dynamic tests. What then normally happens is the software is changed, the tester then needs to change the automated suite to match the changes in the software instead of looking at the software and exploring what is happening.
IMO automation is valuable as a tool to be used when carrying out testing (exploratory). Automation can be used to:
Create data sets
Set up a test environment
Carry out repetitive steps than must be done so testing can b carried out.
I do love your term ‘Automation Assisted Testing’ and I might have to borrow that (acknowledging source of course) I see automation as a useful tool to aid testing I do not see it as a replacement for testing.
I hope this makes sense since I have had to re-write it.
regards
John
Thanks for your comments.
Just to make this clear, the term Automation Assisted Testing is not my ‘invention’. Credit where credit’s due, I believe it was either Michael Bolton or Adam Knight who came up with the term.
A couple of points:
1) There is such a thing as automated checking; it is the checking that is done by automation (surprised by that?). That is: an observation and a decision rule linked to that observation, such that both can be performed non-sapiently, without human observation, judgement, wisdom, or engagement. Now: that can’t happen without (human) testing activity preceding it, and it is of very questionable value without (human) testing activity following it, but the automated checking part does happen. The mistake that many people make is to make a big deal about the checking part (10,000 automated tests!), and not paying attention to the testing stuff. What quality criteria do the checks cover? What oracles (principles or mechanisms by which the checks help to identify the problem) are being used? What are the context drivers for choosing when and how to implement a particular set of checks? How will the information be reported? Is the testing approach sufficiently diversified? and so forth. Those questions can neither be asked nor answered by automation. Testing rules!
2) I appreciate Arjan giving me credit for the notion of automation-assisted testing, but I’m skeptical that it’s attributable to me. It sounds more like Cem Kaner, James Bach, or Bret Pettichord. In the unlikely event that I was actually the first to utter the words in that order, it’s because they inspired me to do it.
—Michael B.
Hi Michael,
Thanks for your comment.
The number of automated checks, scripts, or cases is not a big deal, but having test automation can be of big value. Like every automation, it should support you in doing your tasks more effectively and efficiently. Like every automation it can be too much in the sense that the costs do not outweigh the benefits and like every automation it remains to be seen if the investment will pay off.
When projects consist of short iterations and increments, more and more these days, the tedious repetitiveness of actions and checks is a sure candidate for automation. Especially tests on method level and components can be useful to automate. As a hobby-programmer, I know it is easy to break something. Having a compiler or test-suite telling you within seconds that something is wrong, or at least different, is of great value in developing software.
The tester can finalize those tests by going quickly through the results and then put all his/her energy in the interesting parts, e.g. the problem areas, the new functionalities, and/or tests on a more broader scope.
But indeed: Testing Rules! Test Automation only serves it!
Hi Arjen (and Michael),
The first time I saw the term ‘Computer Assisted Testing’, which resembles your ‘Computation Assisted Testing’ was when I read “Architectures of Test Automation” from Kem Kaner (http://www.kaner.com/pdfs/testarch.pdf). I really liked the ‘done by humans’ part in it 😉
He some other interesting thought regarding Computer Assisted Testing.
Regards,
Ray
After all this time, there’s something else worth pointing out: checking is often automated, but there is non-automated checking as well. Some people ask humans to check, that is, to follow a specific set of instructions and, basically, act like machines. That’s manual checking. Automated checking therefore
—Michael B.