May 13th, 2010 4 Comments
“Quality is more than the absence of bugs”

In recent years, a strong trend has emerged toward the use of “Agile Methods” for development of software and of software-intensive systems. While many forms of Agile exist, they tend to be characterized by values such as “communication, simplicity, feedback, courage, respect” (Extreme Programming [“XP”], Ref.1) and “Interactions and individuals over processes and tools; Working code over comprehensive documentation; Customer collaboration over contract negotiation; and Responding to change over following a plan” (Ref. 2).

The motivation for Agile was the observation that “traditional” (non-Agile, which we will call “plan-based”) methods often delivered software late and over budget, which did not meet current user needs (if indeed it even met what users needed at some earlier time during the development cycle). Taking a broad view, quality is not merely the absence of defects in the code, and meeting user needs is one of the most important aspects of quality. Thus one of the primary objectives of Agile Methods is to improve quality in the sense of better meeting user needs. Additionally, many agile approaches emphasize early, frequent, and automated testing of code, so improvement of quality in that narrow sense is also an objective.

Since plan-based methods were also developed with the objective of improving schedule, budget, and quality performance—primarily through the use of reviewed and approved plans and specifications—one may ask how the quality (broadly defined) of software developed through agile methods compares with plan-based development. A given system is rarely developed twice, once using plan-based and once using agile, so the use of the “scientific method” for comparison is not possible. Nevertheless we can look at products and activities of both types of development and investigate what the expected results might be.

As stated above, many forms of Agile exist (XP, Crystal, Scrum, etc.), and by their nature these methods allow each project team great flexibility in determining and modifying products and activities. Nevertheless some general observations and conclusions are possible.

At the heart of Agile Methods is the realization that it may be impossible to specify the features, cost, and schedule of a large system in advance with any precision. Thus the development team (a) does detailed planning only for one short iteration at a time and (b) has the freedom to adjust scope (feature content) to meet planned delivery dates. Typically features are tentatively assigned to iterations; at the start of each iteration, progress to date in terms of features completed and schedule and budget expended are reviewed, and the feature assignment to the next iteration is adjusted if necessary; then the iteration is planned in detail. In this way Agile resembles many of the iterative, incremental, or cyclic forms of plan-based methods. The primary difference is the reduced formality of the agile methods—the plan-based methods still require a number of documents to be reviewed and approved, even if they are frequently updated. So, we ask what are the key differences with respect to quality and how is the quality of the product affected?

It is widely accepted that agile methods are most effective when a number of conditions are satisfied, e.g.,

· A small, co-located development team

· Close, continuous interaction with the customer/user on requirements, user interface, suitability of code increments, and other subjects

· The developed product can be maintained by the original team

When these conditions are satisfied, the informality of agile methods (e.g., the very informal documentation of requirements, the lack of detailed design descriptions and traceability to enable new staff to maintain the system, lack of documented linkage of test cases to requirements and design) may not be important because the development team knows the information. (Note: as stated, the team has great flexibility in its selection of activities and products, and may choose to provide these documents. However their existence is not mandated as in plan-based methods.) When the conditions are not met, there is great potential for loss of information or for misunderstanding. This potential is increased through the frequent inclination of the development team, per the attributes cited above, to value working code over documentation, and thus to deliver code that meets what the team believes are current needs, but suffers in quality through miscommunication or misunderstanding. It should also be pointed out that management may be uncomfortable with the lack of detailed visibility implied by giving the development team the authority and flexibility that agile methods require, and may subvert the team’s ability to follow the process and respond to customer needs by imposing unneeded activities, and asking for more project reviews than the team planned on.

Ultimately, agile methods rely on informal approaches such as tacit knowledge and understanding, developed among colleagues who converse frequently and in depth (Ref. 3). The difficulties are:

· Scaling: the number of required interactions increases as the square of the number of participants (Ref.4) and becomes unmanageable for large teams or for distributed teams where the closeness of interaction is impractical. Division into closely communicating subgroups is helpful but there may well be some communication between subgroups that fails to take place.

· Objectives: agile methods focus closely on customer requirements, user interfaces, and eliminating defects. Customers are aware of, and can communicate, functional requirements (what the system does) and, to some extent, performance requirements (how fast, how often, how many, how accurately, etc.). But there are important aspects to quality beyond absence of defects and meeting functional and performance requirements—typically the “ilities:” installability, maintainability, efficiency, portability, upgradeability, etc. (Ref. 5, 6). While plan-based methods may use standards or templates that encourage elicitation of these requirements, the less formal approaches to planning and documentation that characterize agile methods are less likely to recognize, and thus to implement, this type of requirement. Customers often don’t think in these terms or have difficulty articulating their needs. More specifically, the methods typically used in agile development to convey user requirements (such as User Stories) are not amenable to encouraging users to think in terms of the “ilities,” nor do they lend themselves to descriptions of such characteristics (Ref. 3). In operations, this deficiency shows up as poor quality in the “ilities.”

1. Extreme Programming Explained: Embrace Change (2nd Edition), Kent Beck and Cynthia Andres; Addison-Wesley, 2004.
2. “The Agile Manifesto,”
3. “Extreme Programming Refactored,”Matt Stephens and Doug Rosenberg, Apress, 2003.
4. Brooks’s Law (’s_law); The Mythical Man-Month: Essays on Software Engineering, Anniversary Edition (2nd Edition), Frederick P. Brooks; Addison-Wesley, 1995.
5. ISO/IEC 9126: Software engineering — Product quality.
6. “Air Force Systems Command Software Management Indicators: Management Insight,” AFSC Pamphlet 800-43, HQ AFSC, Andrews AFB, DC, 31 January 1986.

Be Sociable, Share!


  1. YL says:

    A very good and informative article. I heard A LOT about agile methods, and to some extent involved in mini agile (GUI reviews for new products), but had never really come to a clear understanding of the Pros and Cons, especially comparing with planned-method. This article clarifies for me the essence and the potential issues.

    I hope that you would follow up with more in-depth analysis like this one. Perhaps some success and some failure case studies of Agile?, or perhaps any of the following?

    Is the deficiency in ilicities can be amended in certain style of Agile, or Agile-Planned based hybrid?

    In using Agile, does it imply that the product is not to be ISO 9000 or TL 9000 certified? or does it involve new agreed processes for change management and final product review?

    How would product managers and program managers “manage” in the Agile environment?

  2. Thanks for the post :)

  3. Keep up the wonderful work! I will follow you on RSS.

  4. I was excited to find this site. I need to to thank you for ones time due to this fantastic read!!
    I definitely liked every part of it and I have yyou saved to fav to look at new things in your website.

Leave a Reply