By Eugene Nizker and Yuri Khramov
Introduction
Without question Agile software development is a ubiquitous term nowadays. One hears companies claiming they have adopted Agile or are hiring managers specifically for their Agile experience etc. etc. The hype is palpable. However, how can we distinguish the hype from the essence or value of Agile.
It is useful to distinguish Agile philosophy from Agile techniques. Several Agile methodologies offer different sets of techniques. XP, Scrum, and Lean seem to be the most popular. Proponents of each of these Agile schools may argue whose approach is more Agile. They may compare their techniques and discuss the benefits of the term "velocity" over the term "performance" or "burn-up charts" over "burn-down charts". And although their arguments may be logical and of utmost importance to their own causes, such discussions are more or less hype and are irrelevant to the larger questions surrounding the value of Agile software development methodologies.
In his 1999 article on the subject of Agile Development, Alistair Cockburn describes his experience on big projects in the late eighties. No matter what he tried, he encountered the same two problems:
- People on the projects were not interested in learning and applying these Agile development methodologies.
- They were successfully able to ignore pleas for the adoption of methodology, and were still delivering software.
In the next ten years, his team observed and reviewed 25 diverse projects using very different methodologies or no methodologies at all. Based on this assessment, he concluded:
- Almost any methodology can be made to work on any given project.
- Any methodology can manage to fail on any given project.
Do these conclusions mean there is no need for methodology? No, not necessarily. It does however mean that we should be aware of other methods besides Agile. In fact, the dichotomy between Agile and Waterfall is a myth. There are many iterative (non-waterfall) methodologies that differ from Agile considerably. One important example is the Open Source development process that relies on contributions of a non-formal group of individual contributors for development and testing.
Agile is based on common sense. It asks a few simple questions and gives very simple answers. Below are some questions and answers based on my extensive software development experience. In the next section, two initial assumptions will be made from which we will attempt to draw an Agile philosophy versus an Agile methodology.
STARTING POINTS
Uncertainty of the Outcome
Do we know what we need to get at the end?
Yes, but in general terms. This is the key question and key answer because together they establish a different paradigm for the software development industry compared to previous methods.
The expectation that a set of instructions can produce results is inherited from the manufacturing and construction industries. Although some of the principles that work well for those industries are applicable to software development, most are not.
Today enterprise software development projects, related to business processes, have a completely different nature.
In manufacturing or construction projects, the vision of the project exists before the project is commenced. In software development a vague and incomplete vision exists at the beginning of the project. In manufacturing or construction the requirements are more or less stable (i.e., we do not expect the number of stories in the building to change half way through the construction process). Conversely, we rarely see stable software requirements.
The majority of key design decisions in manufacturing or construction are made before the commencement of the project, but in software development every software developer makes numerous design decisions every day. Moreover, testing in manufacturing or construction is usually partial and non-destructive, unlike in software development.
Client expectations differ too. While clients are rarely surprised at the end of a construction project, the end user community for a custom software project may often be using the software for the first time at the roll-out phase even though the software has been in development for two years.
This all suggests that software development is indeed a different animal.
Aim for Project Success Not Technical Success
Does technical success guarantee business success?
It may seem obvious to identify a goal at the start of a project to articulate what it is the project is trying to achieve. Quite often the Agile practitioners, especially XP practitioners, set technical success as the primary goal. Bob Martin says in his "Craftsmanship and Ethics" presentation [1], "Our job as programmers is to produce the best code we can", and achieving it should be a development team’s goal. Certainly, a development team should aspire to do the best coding job they can but code quality should not be the ultimate goal. The real goal of the project should be its commercial success – the delivery of a timely solution to an important problem. And, the fact is, the best code does not necessarily ensure the product will be a success. Some years ago, for example, I conducted a review of more than 80 projects in an effort to reveal a link between code quality and project success. The results were presented at the Agile 2006 conference. Among the products and code bases considered, no positive correlation existed between the quality of the code and the success of the project.
In most cases, the product feature list, its usability and even its timely delivery better define the success of an enterprise development project. Being as close as possible to the real stakeholders of the project and its potential end users helps the project team understand and define the "ultimate" goals of the project and thus, better ensures its success and enterprise adoption of the new software.
Deriving the Agile Philosophy
Once there is collective recognition that business success should be the primary goal of a software development project and that software development is at least partially deterministic, the rest is easy.
Huge up front efforts
If we don’t understand the outcome, and if requirements are invariably going to change, what is the value of thick volumes of upfront requirements, big upfront design efforts, and comprehensive upfront estimation and budgeting efforts? Very little, in fact.
The dangers of such an approach however, are self-evident. Indeed, if scope is firmly specified, and time and resources for a project firmly committed to at the start, then any change in requirements will invariably lead to the project missing its delivery date and coming in over budget. This approach clearly sets the stage for an antagonistic atmosphere between the client, who is interested in the changes, and the development team, who is assessed based on the pervasive "on time and on money" metric. Is this conflict detrimental? Ask any developer. Is this conflict necessary? Absolutely not.
Yes, requirements are clearly necessary however, they need only outline the general set of expectations for the future system. These few can articulate the first part of the work effort. Others are going to change before work starts on them anyway.
Yes, a "macro" design of the overall system is needed but not the detailed design of every feature because it, too, is going to change based on shifting requirements.
Yes, it would be good to know "how long and how much" effort is required but knowing that we live in a volatile world, it should also be recognized that these estimations will constantly change. Shifting requirements, unforeseen technical difficulties, resource changes etc., will always pose a challenge and impact the schedule. As a result, ongoing estimation and planning for software related projects should become the norm. And this is what Agile offers. Agile doesn’t want developers and project managers to waste time or to produce something that will not be used. Agile encourages the assessment of the situation regularly and the adjustment to direction and approach as necessary.
Delivering software in one chunk
If we believe in our new approach, then is there a point to going away for a year or two and coming back with the "complete" system? What are the chances that the user still wants what he wanted two years ago? What are the chances that the market situation is still the same, that competition is still the same, and that the management still has the same goals? Virtually none.
Therefore, it may make more sense to take baby steps and deliver something very basic, very quickly and elicit feedback. Then, consider this feedback, modify the direction and deliver the next release very quickly again. Continuing in this manner at least guarantees that the client gets what they want every time. An additional benefit to this approach is that it provides an opportunity to kill the project if it goes in the wrong direction before it actually bleeds the company to death. Remember "fail quickly"?
Short releases are at the core of Agile philosophy.
Producing software in an ivory tower
Techies, are often annoyed with their client. They are seen as not technically savvy, fickle, not receptive to technical reasoning and generally, very demanding. This is often true because the techies are sailing, and the client is on the shore. The solution is of course to invite them aboard! Recognizing that there is no such thing as an "IT project" anymore, business clients must be involved from the get-go. If they are in the same boat, they won’t change the scope "at will", but only when the need is real. The development team would need to accept and apply such changes or risk wasting time and money continuing to develop something no one needs anymore.
Cross-functional teams and involvement by the business client
Applying "best practices"
As the Agile approach gains in popularity, many publications are discussing Agile "coaches" who seem to be concentrating more and more on Agile methodologies rather than philosophy from which it stems. Their recommendations are becoming more and more rigid. This brings us to the subject of Agile hype.
Does one size fit all in software development? Any junior programmer knows the answer. That is why interest is loss at the mention of "best practice". There is no such thing as every project is different, every company is different, and every team is different. Applying "best practices" blindly without adjusting them to the specifics of the current situation is almost certainly counterproductive.
For example, a manager was overheard saying: "Agile tells us that at this point we need to do X. So, let’s do X now". It is not surprising then that some time later that same manager will state, "Agile failed in our company". Was it "Agile" that failed or an attempt to employ Agile hype that was really to blame?
Does this mean that Agile suggests that every team should reinvent the wheel? Of course, but the Agile approach does recommend critical assessments of the situation take place regularly. It is fact that a technique that was useful for Team A may be detrimental to team B, or a technique that worked for team A on project X may be detrimental for the same team A on project Y.
Conclusion
True Agile approach states that software development process should rely upon common sense rather than a set of rigidly defined practices. Every project is different, every company is different, and every team is different. Applying any set of methods blindly without adjusting them to the specifics of the current situation is almost always counterproductive. There is no recipe book in software development.. Get over it. Get creative. Pick and choose what works for your team here and now. Change it or drop it when the situation changes.
About the Authors
Eugene Nizker, Ph.D.
Dr. Nizker is an experienced CIO, CTO, and Agile coach. His technical experience, accumulated during his 30+ years in software development, is augmented by significant business acumen. Eugene is referred to as a “people’s manager” which helps him build exceptionally strong teams that deliver results every time. Eugene’s achievements include: turning under-performing companies into profitable market leaders in a short time frame, creating competitive advantage through technology and implementing Agile approaches to software development with impressive results.
He has received the Victoria Advanced Technology Council's (VIATeC) “Executive of the Year” and “Product of the Year” awards and was a finalist for both Ernst & Young's “CIO/CTO of the Year” and ViaTeC's “Team of the Year” awards. Microsoft has used a number of Dr. Nizker's projects as case studies. He is also an an active contributor to the IT Value Knowledgebase and the Advice blog of “CIO Magazine”.
Yuri Khramov, Ph.D.
Dr. Khramov has over 25 years of experience in the software industry. He has worked at both Microsoft and Apple as a software developer and development manager. He has written code for Adobe PageMaker, Acrobat, InDesign, MacOffice, Corel Web Designer, and several Apple applications. Dr. Khramov is co-founder of three software companies. Dr. Khramov’s most recent company, Schema Software, was acquired by Apple in 2005.
Drs. Nizker and Khramov have worked together for two decades since they co-founded their first company in 1989.