By Trevor Atkins
Introduction
Warning — What you say will be used against you at a later time!
Estimation is a black box process for many organizations and all the more so for their testing activities. The inputs used, the process and end results are therefore open to debate or worse – to un-discussed misinterpretation.
In Steve McConnell's "Software Development's Classic Mistakes 2008" software estimation is mentioned more than once as an area of significant impact. Of the "Top Ten Almost Always", confusing estimates with targets was ranked #5 and shortchanging quality assurance was ranked #3. Overly optimistic schedules was ranked #1 of that list.
Capers Jones noted in Assessment and Control of Software Risks that most projects overshoot their estimated schedules anywhere from 25% to 100%, but that some few organizations have achieved consistent schedule-prediction accuracies within 10% and 5%.
What are these organizations doing different? For one thing, they have most likely taken the time to put in place a formal and repeatable estimation process tailored to the types of projects they undertake – something we all can do.
Challenges in Estimation
Developing your estimation process can be quite straightforward but there are challenges to consider.
Firstly, agreeing to what is an estimate is a necessary first step in defining how to arrive at an estimate. The following definitions provide important cues as to what we need to include in defining our estimation process.
"An approximate calculation of quantity or degree or worth." –
http://wordnetweb.princeton.edu/perl/webwn?s=estimate
"An assessment of the likely quantitative result. Usually applied to project costs and durations and should always include some indication of accuracy (+/- x percent)."
– Project Management Institute, PMBOK 3rd Edition
It is important to realize that estimates are not commitments or promises (targets), but are predictions founded on situational information. Carefully accounting for and describing the situational information allows the estimate to be considered in its proper context and updated as changes occur, with the resulting impacts more easily assessed.
Another challenge is the fact that, although the test effort is directly proportional to the size or scope of the implementation in terms of requirements and behaviours that need to be verified, there are also other factors to include like:
- What is the type of project (New Development, Patch Release, Legacy Modernization, COTS Customization, Enhancement Release, etc.) and therefore its quality requirements and goals?
- Do you have the test lab infrastructure and tools you need?
- Do you have your team assembled? Are their skills and experience applicable?
- What is the project methodology and how does your test approach mesh with it?
However, the biggest influence on an estimate's acceptability is how well it is received by other stakeholders. How many times have you worked on an estimate for the test effort and presented it to the project manager or another senior stakeholder only to have them ask you to change it, or they change it themselves. Maybe you have done the same to someone else.
Often stakeholders want to change the estimate to meet a target of theirs:
"'You've overestimated this project by a third; go away and cut it by a third.' The division head had 3 or 4 useful ways to cut the project down by a third, but I kept saying I couldn't do it and the project was killed. I walked out of there wondering what had gone wrong. I did what was probably the best estimate that had ever been done in that company, and the project was killed because of it." – Daniel Galorath, A CAI State of the Practice Interview
Aside from extraneous things like politics, funding profiles and meeting predefined targets, an estimate may also get changed because of a lack of confidence in the validity of the estimate, the estimation process, the inputs, and/or assumptions upon which the estimate was founded.
In these cases, the stakeholder or approving manager may wish to increase the estimate for the sake of "safety" or may wish to cut it back to remove "unnecessary" padding and put a "healthy" pressure on the team. Any adjustments of this type are likewise made without analytical foundation, but rather with a hunch or "gut feel", collectively resulting in a poor estimate.
Letting poor estimates persist can affect project outcome in two ways:
- Perfecting: If the project is over-estimated Parkinson's Law will apply. The project effort will somehow always expand to use the available schedule and/or budget, even if it could be done for less.
- Rework: The more dangerous scenario is to under-estimate the project. In this case, the project will typically only recognize that it is underestimated at the latter stages in the project cycle. At which time, it is usually too late to make real changes. Often people are added to the project team in an attempt to solve the problem, but adding people to a late project without changing anything else will only tend to make it later.
<
Figure 1: Impact of Poor Estimates
"Principles and Components of Successful Test Team Management"
Good Estimate, Poor Estimate
There can be many approaches to the effort estimation of a given project and some will fare better in different circumstances than others; what is important is to have a strategy that allows you to approach the task in a systematic manner with a defined yet flexible technique – where further research and experimentation will improve the methods used to arrive at increasingly valid estimates.
The following are some of the expected benefits or by-products of a good estimate:
- Providing visibility of options to stakeholders
- Providing a reliable foundation for project planning
- Providing an agreed repeatable process for estimation that moves the debate away from the output to the inputs, e.g. prioritization of features and trade-offs of scope or quality against time or resources.
Figure 2: Standardized Estimation Process
"Principles and Components of Successful Test Team Management"
See Figure 3 for an illustration of how formalized estimation is able to present certain options to decision-makers and clearly highlight the impact on either cost or schedule of choosing the fastest or cheapest. You can use other similar analysis to make visible such things as the impact of involving testing earlier or later in the project cycle versus achievable test coverage or risk to system quality.
Figure 3: Constrained Solutions
"Principles and Components of Successful Test Team Management"
Some tips to keep in mind for your next estimate:
- The response to a request for an estimate should be "Let me get back to you on that." Then, take the time to work through the estimate and get it down on paper, rather than just tossing one out the instant you are asked for it.
- When providing an estimate, present the numbers in a range or as a central point with +/- values based on specific risks and uncertainty. Single-point estimates imply precision where none exists.
- When you de-scope to get your estimated effort to fit within the constraint box, keep the "customer" in the loop even if that is just your manager. You don't want to surprise someone with: "Here is your application on schedule on budget, but we had to reduce the functionality tested by 50%".
- Estimation of testing effort is an iterative process where the test strategy can only be finalized when the proposed effort, schedule and budget estimates are approved. But of course the test strategy has a significant impact on the estimate, so work on these simultaneously, updating each as the project progresses.
- There isn't just one time to do an estimate. As the project progresses, the estimate can be refined for the activities that remain. The estimated effort for those activities may or may not change but the confidence level in the estimate should rise. Make the impact of any changes visible, for good or bad, so the appropriate informed decisions can be made.
- If you can produce one estimate, you can create several, based on different, clearly stated, assumptions or conditions. Review and modify these options to find the optimal solution for the project.
- Have those closest to the work do the estimating to increase accuracy of the estimate and also to create a sense of ownership and commitment in the numbers provided.
- Keep the estimate modular and decoupled so that changes to assumptions, a new input, or project phase can be easily accommodated.
- Leverage historical data in the estimate. Objective reality helps avoid overruns from overly optimistic thinking.
- Brainstorm with the team to avoid overlooking uncommon activities and underestimating the related size of the effort (reviews, rework, SCM).
- Have more than one person do the estimate. Discussion of differences in numbers can make visible and clarify assumptions or advantages of approaches. Compare, contrast and converge to get the best from each.
- Buffer or contingency time helps cope with the unknowns and the unexpected. Base the buffer on quantified risk analysis whenever possible. In the absence of granular detail you can generically buffer against risks, e.g., by scheduling people for only up to 80% of their availability or simply add a factor of 20% to the estimate. As your estimation technique becomes refined you will be able to break this number down and reduce the relative size of this sort of fudge factor.
- Document the estimation process so you and your colleagues can better review/defend your current estimate as well as leverage the process next time.
Conclusion
Estimation process improvement presents a significant opportunity to get more success out of your software projects. And just as it is critical to offer something more than an off-the-cuff answer for the development activities, it is as important to know how to produce a good estimate for the testing effort portion of a project. We can achieve this by leveraging many of the same principles that are applied as best practices in general estimation, while developing a specific framework for predicting test effort.
Don't wait until your next project to work on your estimation model. Start now, and try it on next week's or next month's tasks.
Read Trevor's whitepaper "Estimating Test Effort – From Billiard Balls to Electron Clouds" for more on this topic.
About the Author
Trevor Atkins is Principal Consultant in a new quality assurance consulting practice. Most recently he was a Regional Director of Quality Services with UST Global and before that he was a founder and the Vice-President Operations of QA Labs, the largest independent software testing company in Canada.
Trevor has been involved in all aspects of hundreds of successful software projects for the last 12+ years, and is dedicated to the design and improvement of quality processes for use across projects and organizations.