As an external consultant or expert, I’m often dragged into the stakeholder budget debate about needing to justify the investment in automation. Most recent of these conversations was this week and my response is publishable without editing as side from salutations and other niceties. (Good job me!). So here it is.
Automation and Testing in general is hard (impossible?) to pin a dollar amount to in terms of ROI. It is often better to talk about ROI in terms of time and allowance of better testing.
What I would try to do is identify a number of flows through the application that always need to be tested. These are often the ‘smoke’ or ‘sanity’ tests and have them running out of Hudson (after the unit testing job). You no longer need to do those by hand on each build which means you can can start doing manual exploratory testing faster. Which is very important. Automation succeeds long term when you think about it in these terms rather than as the finder of bugs.
A ‘green’ builds means you can start testing, not that testing is complete.
This of course doesn’t answer the money problem. Often people want to look at how much a bug escape costs the company and then you look at how many the automation finds. But even that is overly simplistic as a lot of the bugs the automation process finds is during the creation of the scripts. In fact, most automation fails to ever find a bug once it is running clean.
By going through the process though, the automation person now knows way more about the system than they did before. Which in turn lets them do better manual testing. See the pattern here?
Michael Bolton splits things between ‘checking’ and ‘testing’. Testing requires a human to determine if something passed or failed whereas checking is something a machine can decide the outcome. So the automation with Se is ‘checking’ not ‘testing’ under this definition. What is interesting is that when you are creating the script you are ‘testing’ but as soon as you call it done it is ‘checking’. This, to me, is an important distinction in terminology.
So I would talk to your boss and explain that Automation ROI is not in terms of measurable dollar amounts but in intangible benefits to the quality of the system.
It seems kinda weasel-y and consultant-y, but I am a tester by origin. And it is a little easier to swallow than throwing out your scripts.
Oh, and I seem to recall hearing the green build comment from Michael as well, but he attributes it to J.B. Rainsberger as When all your unit tests pass, that doesn’t mean your product is done. It means that you’re ready to give it to a real tester to kick the snot out of it.. Again, not something you would put in an email that could float up a chain of command, but bang on.
Element 34
Continuous Delivery & Selenium
Comments 4
Our managers understood the ROI on automated regression tests better when we explained the concept of technical debt. They get it that if our code is supported by tests, we can change it more easily & with confidence. Our velocity is low while we learn to automate tests and do TDD/ATDD, but in the long term, we will have better quality code, spend time on new features instead of prod defects, and have a steady velocity to help them plan.
We can put a $ amount on story points (though it’s not 100% accurate). If we can be fairly sure of doing a certain # of story points each iteration over time, while continuously improving quality, that is worth a lot to the business.
Posted 28 Sep 2010 at 5:32 pm ¶Great post. I was arguing with Dorothy Graham today about this same topic. I felt like she was saying that automation tests save tons of time and thus tons of money. I was claiming that the automation is just a piece of testing that ultimately turns into money save because you can do real testing instead of spending your time in regression tests. We didn’t come to a very productive agreement, she just discounted her metric as a heuristic then sent me on my way. This post very accurately describes exactly what I was trying to say. I hope we can sit and chat while you’re here at Starwest this week.
-Wade-
Posted 28 Sep 2010 at 6:41 pm ¶One other point as too many see the word “automation” and interpret that as if it means autonomous, running with out human input, analysis or upkeep. But Automation in testing isn’t really that way. Yes you might be able to script it strongly enough to get some reporting from it, but like a tree falling in a forest, with testing if noone is around to view, analyze, and adjust the automation as the project grows, then I’d argue that’s not really testing.
Instead I view automation as a computer assist to testing, a way to speed up the repetitive parts of moving through a script/flow/test case with a focus on what is a defect, a failure, fault, bug, etc. To me automation is about being able to get at bugs quicker, especially in highly trafficked areas. However, it still requires Sapience as James and John Bach, and Michael Bolton would say to determine what the outcome of each test was.
Posted 28 Sep 2010 at 7:55 pm ¶I disagree with you a little here Adam.
If you did automation for a year.. and didn’t find any bugs through automated efforts.. I would certainly question its value. Automation is expensive. It needs to prove its value.
That’s not to say I’m against Automation. I’m most certainly not. but i do stay objective.
Most of the time the reason for not finding bugs in automation isn’t because regression bugs aren’t out there. It’s because the Automated scripts are just testing the simple scenario’s. I would challenge the team to enhance their automation scripts beyond just “smoke” tests.
Posted 13 Dec 2010 at 2:48 am ¶Post a Comment