Software Quality Management

1. Be able to define quality in general and software quality in particular.
2. Understand the cost of quality in terms of costs incurred and costs avoided.
3. Be able to express quantitative quality goals for a project.
4. Understand the difference between product quality and project success.
5. Be able to distinguish between the functions of testing, quality control, quality assurance, and process improvement.
6. Be able to create a software quality assurance plan.
7. Be able to plan and implement an effective defect prevention process.

Most new cars come with a 3 year / 30,000 mile warranty. Most new software programs come with a legal disclaimer disavowing any responsibility for the product. Clearly the software industry has room for improvement when it comes to building quality products.

Automobiles weren't always the shining example of reliability and quality which they are today. When automobiles first started appearing on the roads in the early twentieth century, drivers thought nothing of stopping every few hundred miles to fix a flat tire. As the novelty of driving wore off and competition intensified, automobile manufactures had to improve the quality of their products in order to maintain interest and stay ahead of the competition.

Some of the same forces are at work today in the software industry. The first software systems were amazing in part because of their originality. Users were delighted, or at least content, to receive anything that generally worked. During the 1970's and 1980's software was like a dancing bear-- it's not important how well the bear can dance, it's just amazing that the bear can dance at all.

Today the bear is dancing faster--almost desperate for attention--but users expect even more. Software is becoming just another product or service--vital like electricity and clean water, but expected and sometimes taken for granted. Viewed in this light, the expectations are that software should have a level of quality comparable to other everyday products or services. To some extent it doesn't matter that software is more complex than most products or that every software system is a unique intellectual product designed and developed rather than a copy of an existing product manufactured according to a well-defined process. Users increasingly view software as just another product or service and consequently expect software to have a level of quality that is on par with other everyday products and services. Perception becomes reality and so quality is taking an outsized role in product acceptance.

The demand for software quality can be a source of opportunity as well as challenge. The only thing worse than being "just another product" is being a commodity product. Commodity products compete on price alone and the competition is often seen as a race to the bottom. Software providers looking to distinguish their offerings can use quality as a differentiator. Just as some patrons choose a restaurant based on quality, quality can also be the deciding factor in a software purchase decision.

Quality can also be the source of pricing power. Whether it's handbags or golf clubs the provider with the best reputation for quality usually has the power to raise prices when others in the same market segment can't. In a mature market products compete on quality and only the highest quality products can justify a price premium. Not all software is sold commercially but all software has value and its development is justified by the value that it delivers to its customers. Commercial vendors can use quality to justify a price premium. Internal software development groups can use quality to justify their continued existence.

Beyond competition, software quality is vital for some applications. The world is becoming increasingly dependent on software and it's taking on a greater role in life- and mission-critical systems. For these types of systems the costs and risks of poor quality are making quality the number one concern over and above other priorities such as schedule and budget.

17.1 Introduction

Software quality is a broad topic that pertains to many different functions of software development. Figure 2 shows key quality-related topics in the order they typically would be adopted by an organization pursuing higher quality. The figure also shows how key quality-related topics are divided between this chapter and chapter 16, Verification and Validation.

Testing is the most common approach to ensuring quality. Testing is defined as the operation of software under simulated conditions for the purpose of finding defects. No reasonable organization would skip testing, but on the other hand, many haven't evolved beyond a test-only approach to software quality. Testing alone is a narrow and somewhat limited method of avoiding defects because:

Because of the limitations of testing most organizations also adopt some form of product reviews to complement their testing efforts. Reviews vary in type from informal peer reviews, such as asking a colleague to proofread a document, to formal and rigorous inspections with their well-scripted roles and responsibilities. The combination of testing and reviews is particularly effective because reviews complement testing and compensate for many of the limitations of testing. Work products other than code can be reviewed early in the product cycle which allows errors to be found early, closer to their point of origin. The types of errors found during reviews also tend to complement those found through testing. Testing verifies that a product works under certain simulated conditions. Reviews verify that a product has the qualities it needs to work under changing or unanticipated conditions. In terms of product requirements, testing is better at verifying functional requirements and reviews are better at verifying non-functional quality requirements like maintainability, reliability, reusability, etc.

Beyond testing and reviews, when quality is crucial or when there are outside stakeholders that need additional assurance of software quality, organizations will institute some form of quality assurance. The term quality assurance is sometimes used as a synonym for testing or as a reference to an independent test group. The correct use of the term refers to the function which assures management and other project stakeholders that software products have their planned levels of quality [IEEE Std. 610-12-1990]. The quality assurance group (or individual if it is a small project):

It's more difficult to generalize software quality practices beyond quality assurance because there are fewer organizations working at this level from which to generalize. However, software process improvement models [CMMI] and approaches in other disciplines [PMI and Juran] suggest that implementation of a comprehensive system of quality management is the next step on the road to higher software quality. Quality management is an umbrella term that encompasses several quality-related processes including quality planning, quality control, quality assurance, and verification and validation. Each of these processes can be performed individually but performed together they have a synergistic effect. Quality management includes all activities undertaken to establish quality objectives and then manage and control the execution of a project toward the achievement of these objectives. Put another way, quality management is the application of engineering and management techniques toward the achievement of project quality goals. Organizations routinely plan and manage schedule and budget performance. Quality management is about planning and managing quality goals with equal or near equal precision.

The final quality-related practice discussed in this chapter is defect prevention. If poor quality is a disease, testing and inspecting is just treating the symptoms, whereas defect prevention is a cure for the underlying problem. Defect prevention is about identifying the root causes of defects and taking specific actions to remove the causes and prevent the defects from occurring again in the future. Defect prevention has the potential to improve product quality and development productivity. Fewer defects means less time spent finding and fixing errors.

Given the number and variety of quality-related topics it's impractical to cover them all in one chapter. This chapter explores the key concepts and principles of software quality and describes in more detail the following quality-related functions:

These functions rely on and interact with other quality-related functions that are mentioned here but described elsewhere in more detail. Specifically,

17.2 Concepts and Principles

17.2.1 What is Quality?

Quality is an elusive concept. Most people can recognize quality and give examples of quality, but stumble if asked to give a clear and unambiguous definition of the term. In everyday discourse this usually isn't a problem because language is very tolerant of ambiguity. However, before the quality goals of a project can be specified, managed, and assured, we need a good working definition of the term quality.

This first section explores different perspectives on quality in general. These perspectives provide a foundation for the definition of software quality in particular presented in the next section.

1. Transcendent
2. User-Based
3. Conformance to Requirements*
4. Product-Based
5. Value-Based

Figure 5 shows a conceptual depiction of each one of these perspectives on quality.

_________________
*Garvin refers to conformance to requirements as the manufacturing-based view because it is the view preferred by those responsible for the production phase of a manufacturing process. The term conformance to requirements is used here because it is more descriptive of the software development environment where products are developed rather than manufactured.
_________________

Transcendent. The transcendent view of quality equates quality with "innate excellence". According to this perspective, quality can't be described with words, it can only be understood through experience. The transcendent view of quality relies on perception and experience but isn't completely subjective. For example, wine tasting experts generally agree on the relative merits of different vintages of wine which suggests that the activity isn't completely subjective. Robert Pirsig takes on the topic of quality in Zen and the Art of Motorcycle Maintenance and concludes, "Quality is not objective... It doesn't reside in the material world....Quality is not subjective...It doesn't reside merely in the mind....Quality is neither a part of mind, nor is it a part of matter. It is a third entity which is independent of the two." [Pirsig]

The transcendent view of quality is a poor basis from which to develop an operational definition of quality. The most obvious problem is measurability. There is no precise way of quantifying innate excellence. The transcendent view of quality is better suited to products that appeal to the senses. Software is practical and utilitarian and therefore not well suited to the transcendent view of quality.

User-Based. The user-based view of quality is certainly more down-to-earth than the transcendent view. According to the user-based view of quality, quality is the satisfaction of user wants or needs. If product A satisfies more user wants and needs than product B, then product A has higher quality than product B.

There are two potential problems with this definition of quality though. The first problem is that of designing a product that simultaneously meets the needs of a diverse group of users. For example, the popular image editing program PhotoShop is used by average consumers, professional photographers, and high-end animation studios. It would be difficult to design a product that simultaneously meets the needs of this diverse group of customers. What might be satisfying to one customer might be completely unacceptable to another.

The second problem with this perspective is more fundamental. User satisfaction is based on an aggregate of product factors--only one of which is quality. (See figure 6.) Just because a user is satisfied with a product you can't conclude it is exclusively because of high quality. A user might be satisfied with a low quality product if the other factors that influence satisfaction are high enough to compensate for the dissatisfaction attributable to low quality. To say that user satisfaction is a good measure of product quality is to ignore the other factors. For example, there is a robust market for classic Jaguar automobiles from the early 90's. This suggests that these cars are providing user satisfaction. However, older Jaguars are notorious for their electrical and mechanical problems. These cars might be meeting the overall needs of their buyers, but even their most enthusiastic admirers would freely admit that they are not high quality. More likely, it is a combination of product features including their visceral beauty and style that make them appealing in spite of their reputation for poor quality.

There is certainly value in having a definition of quality that defines quality relative to user wants and needs. However, before this perspective can serve as a basis for quality management, there has to be some way of measuring the extent to which product quality is contributing to user satisfaction independent of other product features that might also be influencing overall user satisfaction.

Conformance to requirements. Quality as conformance to requirements is a popular operational definition of quality. Philip Crosby, respected author and evangelist of quality, is probably the best-known advocate of this interpretation [Crosby, Quality is Free, p. 17]. It is also the definition used in the well-known ISO 9000 quality standard.

When quality is defined as conformance to requirements, there is no ambiguity about what quality is. A quality product is one that conforms to specified requirements and design. According to this definition of quality, a meal at McDonalds is high quality as long as it conforms to the specifications of the franchise. By the same token a meal at the Four Seasons could be considered of lesser quality if, for example, the salad fork isn't chilled.

If this definition of quality bothers you it's probably because you are equating quality with luxury or goodness [Crosby]. Viewed in this way, a Lexus has more quality than a Hyundai. A Rolex watch has more quality than a Timex watch. This is a common interpretation of quality but it's incompatible with the interpretation of quality as conformance to requirements. Quality as conformance to requirements is absolute. Quality can't be both conformance to requirements and luxury or goodness. The source of the confusion is in the use of the same term for two separate concepts. A manufacture may plan a product with a certain level of quality (luxury or goodness). In order to deliver on the plan the manufacture will need to maintain high standards of quality (conformance to requirements) during production. Quality as conformance to requirements is a good operational definition of quality because manufactures of all types of products can specify quality goals and control progress towards the accomplishment of these goals.

Another potentially confusing aspect of defining quality as "conformance to requirements" is that it seems to disregard the needs of the user. Quality as conformance to requirements doesn't discount the importance of meeting the needs of the user, it just assumes that these needs will be represented in the requirements. This puts the onus on the requirements process to capture the true requirements in a precise way.

The principle advantage of defining quality as conformance to requirements is that it simplifies the production or implementation phase of the product life cycle. It doesn't make it any easier to identify the right set of product features that will lead to customer satisfaction, but it does make it easier to manage and control the production or implementation phase of the product life cycle. (See figure 7.)

When quality is defined as satisfaction of user needs the whole life cycle is working toward subjective hard-to-measure goals which makes the whole process hard to control. When quality is defined as conformance to requirements, the requirements phase isn't any easier to control, but the design and production phase are working toward an objective easier-to-measure goal--conformance to written requirements. The advantage of defining quality as conformance to requirements is that it makes it possible to define quality goals in specific terms and then measure and control progress towards these goals.

Conformance to requirements is the definition of quality preferred by manufactures because it provides an objective easy-to-measure goal for quality during the production phase of the product life cycle.

Product-Based. With the product-based approach differences in quality are measured by differences in the quantity of some ingredient or attribute of the product. For example, one way to measure the quality of cotton sheets is by thread count. Better quality sheets have a higher thread count.

In practice the quality of a product is affected by multiple attributes with different levels of importance. For example, two attributes that are a measure of a car's quality are horsepower and fuel economy. Each attribute would be weighted differently depending on the buyer's preference for speed and economy.

Car Quality = attribute-weight_hp * horsepower + attribute-weight_mpg * miles-per-gallon

The challenge with defining product quality using this definition is determining the attributes and their weights. If the choice of attributes and weights are subjective there may be no choice of attributes and weights that is optimal for all customers.

Section 17.2.3 describes various models for defining software quality in terms of product attributes.

Value-Based. The value-based approach to quality considers value to be an aspect of quality. Adding features to a product and controlling its defects, beyond a certain point, will likely increase its cost. Higher cost means less value. Tying value to quality means that as the cost of a product goes up, its perceived quality goes down. According to the value-based approach to quality, a quality product is one that has the features the customer desires, limited defects (if any), and is offered at a price that is acceptable to the customer. This approach to quality explains why, when shopping for flooring, linoleum starts to look better than tile after seeing the price of each.

In practice, this view of quality confuses product quality with desire for a product. A customer may marvel at the quality of a pair of shoes, but upon seeing the price may loose interest in purchasing them. The quality of the shoes haven't changed but the customer's desire for owning a pair diminishes because they don't represent good value for the customer.

The relationship between software quality and cost is explored in section 17.2.5.

17.2.2 What is Software Quality?

The perspectives on quality presented above are purposefully general--they apply to any product in any environment. Our interests are more specific. Our goal is a working definition of software quality that can serve as a basis for managing software quality during a project.

The motivation for defining software quality is that it is one of the 4 constraints or variables of a project (along with time, cost and features) that must be planned and managed. You can't plan and manage the quality goals of a project unless there is universal agreement among all stakeholders about what quality is*.

--------------------------------
*Note, we aren't yet talking about specifying the quality goals of a project. Right now we are only interested in defining software quality. Later we will build on this definition and explore techniques for specifying and managing the quality goals or requirements for a project.
--------------------------------

It's interesting to note that of the 4 constraints on a project, quality is the only one that is in danger of being misunderstood and therefore needs to be defined. There's no need to define time, everyone knows what a day, week or month is. There's no need to define cost because money is universally understood. There's no need to define what a feature is because, while stakeholders might argue over the utility of a feature, no one would argue whether or not something is a feature. However, as the multiple perspectives on quality presented above will affirm, there is no universal agreement on the meaning of quality. If the definition of software quality isn't well-defined and understood by all stakeholders, disagreements and misunderstandings are likely to follow.

Another more subtle problem is that if quality isn't well-defined it's likely to be the first of the 4 project constraints to be compromised--irrespective of its true importance to overall project success. Project success is delivering the features and quality the customer desires at a cost and schedule that is acceptable. As figure 8 illustrates, cost and schedule must be balanced with features and quality. Once a project is started, if there isn't enough time and money to deliver the features and quality promised, something has to give. It should be the least important constraint, but in practice it's hard to resist compromising the most convenient constraint. Cost, time and features are well-defined and easily recognizable. If one of these is compromised, the effects will be immediately obvious. On the other hand, if quality isn't well-defined it can be compromised in favor of other more visible project constraints without any immediate consequences. Compromise schedule, budget or features and you have to publicly acknowledge a failure to meet the original project goals. Compromise on quality and it's someone else's problem for another day. The bottom line is: when quality is well-defined, tradeoffs among project constraints are more likely to be based on true long-term priorities rather than short-term conveniences.

Figure 8. Project success is finding the right balance among cost, time, features and quality

Quality is important but it's not everything. As figure 8 shows, product quality is only one component of project success. Project success is delivering desired features at an acceptable level of quality according to the agreed upon schedule and budget. Anyone who declares an unwillingness to compromise on quality probably means well, but isn't considering the bigger picture which is project success. Users may be willing to accept a little less quality if it means faster delivery, less cost, or more features. It's especially hard for engineers to compromise on quality because (1) it's mostly their work and they understandably have pride of authorship (2) engineers are more affected by poor quality because they have to maintain the software into the future, and (3) they are less affected by compromises to the other project constraints. Quality is one component of project success that must be balanced with other dimensions of value such as cost and schedule.

Having laid the groundwork for a definition of software quality, next we consider two definitions from the IEEE--one for quality in general and the other for software quality in particular. The IEEE standard glossary of software engineering terminology defines quality as:

This definition offers what are probably the two most common interpretations of the word quality, (1) conformance to requirements, and (2) measure of user satisfaction. As long as the requirements completely and accurately reflect user needs and expectations, these two definitions are equivalent. When user needs aren't fully reflected in the written requirements, however, there is the potential for disagreements and misunderstandings. For example, at the end of a project the development staff may declare the result to be of high quality because it conforms to the specified requirements. Looking at the same result, the customer may disagree and declare the result to be of low quality because it doesn't meet their needs. According to this definition of quality, both are correct.

The definition for quality above is also broader than most definitions because it explicitly includes software processes as well as the products of these processes. Process quality is important because the only practical way of producing a quality product is with a quality process.

The second definition from the IEEE is from the IEEE standard for quality metrics. It defines software quality as:

Despite its brevity, this definition incorporates both the user-based and product-based views of quality. The word desired suggests a user-based view of quality where quality is relative to the desires and priorities of the users and project stakeholders. The word attributes implies that software quality is best measured by the degree to which certain product attributes are present. The quality attributes of software are the familiar "ilities" of software including usability, reliability, maintainability, etc. Software quality attributes are discussed in more detail in the next two sections.

Building on the aforementioned definitions from the IEEE, software quality is defined here as:

The three parts to this definition aren't three alternative definitions, but rather the three aspects of software quality. Each contributes to the overall quality of the product. Software quality is the degree to which all three of these aspects are met.

Figure 9 illustrates each part of this definition of software quality in the context of the products and process of software development. The written requirements should accurately and completely represent the needs and expectations of the user. The written requirements should include non-functional or quality requirements as well as functional requirements. Non-functional quality requirements may be expressed as a desired combination of attributes (i.e. usability, maintainability, etc.) as suggested in the IEEE definition for software quality.

Because the definition above takes into account the needs and expectations of all stakeholders, it doesn't provide an absolute definition of software quality. What is high quality to one stakeholder may be low quality to another. For example, consider the conceptual diagram in figure 10. From the perspective of marketing, product₁ has high quality. From the perspective of development, the same product is perceived to have low quality because the product's attributes don't reflect the priorities of developers. (This assumes developers are stakeholders with legitimate authority to influence the product.) Product₂ is just the opposite. It is perceived as low quality by marketing and high quality by development.

Disagreements over what is or isn't quality can be avoided by having clear and specific written requirements. The more the written requirements reflect the true needs and expectations of stakeholders, the less ambiguity there is about what is or isn't high quality.

Long-term software quality (i.e. the ability to modify and extend the system over time) is influenced by the development process used to produce the software. There are no guarantees that following certain methodologies will lead to a quality product, but there are generally recognized standards and procedures for professionally developed software. Following these standards and procedures helps to ensure the long-term quality of the resulting software.

Quality software is software that conforms to implicit requirements and implicit expectations of professionally developed software. Just as it's impossible to fully specify product requirements, it's impossible to fully specify engineering standards of performance. For example, it's unnecessary to state that all words in the system dialog should be spelled correctly. Certain professional standards of quality are just assumed. To avoid misunderstandings, it is important that customers and developers have a shared understanding of what are reasonable expectations for professionally developed software.

17.2.3 Software Quality Models

Models are common in software development. There are software life-cycle models (spiral, waterfall), process improvement models (CMM, ISO 9000), and cost estimation models (COCOMO, SLIM). Models are used to manage complexity. They provide an abstraction of a system or phenomenon from a particular perspective for a particular purpose. This section introduces another type of software development model--the software quality model. Software quality models provide a framework for expressing and measuring software quality as a collection of desirable attributes. Just as software life-cycle models help manage the software development process by suggesting activities and their timing, software quality models help manage the specification and measurement of quality in software systems by specifying quality characteristics, their relationships, and associated metrics.

1. McCall's quality model
2. Boehm's quality model
3. The ISO/IEC 9126 Product Quality Standard

These models can be used as-is or as a starting point for creating project-specific or organization-specific software quality models.

All three models have a common hierarchical structure which is illustrated in figure 11. High level quality factors represent external quality characteristics of interest to users, customers and managers. The models relate these high-level quality factors to lower-level subfactors that represent attributes of the software or production process. These subfactors are more meaningful to developers and technical staff. In order to quantify the presence of each factor in a system, metrics are defined for all subfactors. It may also be possible to define direct metrics for some higher level quality factors.

McCall's Quality Model

McCall's quality model is shown in figure 12. It groups software quality characteristics according to the main phases in the software life cycle. The high-level categories of quality characteristics are:

1. Product operation - quality characteristics related to using the product.
2. Product revision - quality characteristics related to maintaining the product in its current environment.
3. Product transition - quality characteristics related to porting the product to a new environment.

Like most software quality models the high level quality factors in McCall's model generally represent external emergent system properties. Emergent system properties are system characteristics not attributable to specific elements of the system but rather those that emerge from the structure and interaction of individual system elements. Low level quality factors in the model refer to more tangible internal properties of the system.

McCall's model links high-level quality characteristics of interest to end users to low-level system properties that developers understand. More generally, it provides a vocabulary of quality factors that users and managers can use when setting the quality goals of a project. For example, end users may decide that usability is an important quality requirement. When quality requirements are stated in terms of high-level quality factors, it's not always clear to technical staff what specific product properties are needed in order to accomplish the desired system qualities. McCall's quality model breaks down high-level quality factors into their constituent components or quality criteria. These low level quality criteria are more familiar to technical staff and represent product properties that can be built directly into work products.

Besides suggesting quality characteristics and their relationships, McCall's model--and the others that will be examined--also suggest metrics for measuring the degree to which quality characteristics are present. McCall's model breaks down high-level quality factors that are difficult to measure into low-level detailed quality criteria which are more tangible and measurable. For each quality criteria, McCall suggests one or more metrics for measuring the degree to which the system possesses or exhibits the associated quality criteria. For example, storage efficiency may have the associated metrics: (1) total disk space required, and (2) average internal memory required while the program is running. Every quality criteria has at least one metric, but metrics may also be associated with some high level quality factors. For example, mean time to failure (MTTF) might be used to measure reliability.

One of the key features of McCall's model is that it provides a quantitative measure of quality. Metric values are expressed in quantitative terms and the relationship between quality criteria and quality factors is formally expressed with a linear equation of the form:

As an example, the quality criteria "self-descriptiveness" which is associated with quality factor maintainability might have the the metric "number of modules with standard headers". If 9 out of 10 modules have a standard header, and historical data suggests that this metric has a 30% influence on the quality factor maintainability, the formula becomes:

Note, that the the number of modules with standard headers is divided by the total number of modules. Metric values should be normalized to remove any bias related to the size of the product.

The formal relationship between metrics and quality factors is captured in the equation's coefficients (c_i in the equation above). McCall showed that for a limited domain significant correlation could be established between predicted quality and actual quality. However, its not clear that a quality model calibrated for one domain is generalizable to other domains. Like other software models calibrated from empirical data, the accuracy of McCall's model depends heavily on tuning and using the model on projects that are similar in type.

One of the ways McCall's software quality model is distinguished from the others that are presented here is that it defines software broadly to include not only code but also requirements and design. A few of the quality characteristics in McCall's model are limited to code (i.e. efficiency and instrumentation), but most are general enough to apply to code and other pre-code artifacts such as requirements and design. For example, traceability and completeness are two desirable characteristics of requirements and design documents.

Boehm's Quality Model

Boehm's quality model is shown in figure 13. It has the same basic categories of quality characteristics found in McCall's model. In Boehm's model, overall system quality (called general utility in the model) is the aggregation of portability, usability (called "as is utility" in the model), and maintainability.

Like McCall's software quality model, Boehm's quality model is hierarchical with metrics defined for lower-level quality attributes. Boehm's model is similar to McCall's but does have a few notable differences. One obvious difference is its factoring of quality characteristics. As an example, in McCall's model maintainability and testability are at the same level which means in McCall's model a product can be maintainable without being testable and vise versa. In Boehm's model, testability is a subfactor of maintainability, which means an increase in testability implies an increase in maintainability. By itself this isn't necessary an important distinction but it does point out the difficulty of having a definitive quality model.

Another way Boehm's model is different from McCall's model is that Boehm's model applies only to code. Boehm's quality model includes 151 metrics which provide measurements of 12 primitive quality characteristics. Each metric is expressed in the form of a question. Stated in this way the metrics form a checklist of good programming practice. Table 1 shows a partial checklist for the quality characteristic self-descriptiveness. Self-descriptiveness is the extent to which a program or module contains enough information for the reader to understand the function of the module and its dependencies. Self-descriptiveness is a measure of a product's testability and understandability.

By expressing quality metrics in the form of checklists, the metrics used for quality measurement can also be used for code reviews and general process documentation and improvement. The metrics developed in Boehm's original work are for Fortran programs, making them mostly outdated by today's standards. However, the quality characteristics and their factorings as shown in figure 13 above are just as applicable today as when they were proposed in 1978, demonstrating that the principles of quality are timeless.

The ISO/IEC 9126 Product Quality Standard

ISO/IEC 9126 is an international standard for defining and measuring software product quality. It comes from the International Organization for Standards (ISO) which is also the publisher of the popular ISO 9000 series of process quality standards (ISO 9000, ISO 9001, ISO 9000-3) ISO/IEC 9126 is a software product quality standard designed to complement the well-known process quality standards offered by the ISO.

ISO 9126 isn't remarkably different from the other software quality models discussed here, but it is distinguished in some important ways. First, it is an internationally recognized standard and a safe choice for any organization looking for a standard way to define and measure product quality. In the 80's there was a popular saying that nobody got fired for buying an IBM computer. Today, it can be said that no one gets fired for choosing to follow ISO standards. A universal standard provides a common language and framework for specifying and measuring software product quality. Having a common definition of software quality makes it easier to compare the quality of one product with another. Just as some organizations rely on internationally recognized ISO process quality standards to ensure a certain level of process quality, ISO 9126 can be used to ensure a certain level of product quality. Second, the ISO has shown a willingness to revise the standard and keep it up-to-date. Quality characteristics are generally timeless, but the metrics associated with quality characteristics depend on current technology and product environments. Since being introduced in 1999, the ISO 9126 standard has gone through 4 significant revisions or updates.

The original version of ISO 9126 introduced in 1991 was rather sparse. The original title of the standard, "Information technology -- Software product evaluation -- Quality characteristics and guidelines for their use", included almost as much information as the actual standard. As the title suggests the original standard offered quality characteristics and a process for applying them. The standard was essentially 6 high level quality characteristics and their definition. The standard claimed that the 6 quality characteristics were sufficient to represent any aspect of software quality. The model was limited because it didn't offer a standard factoring or taxonomy of quality characteristics or associated metrics. Subcharacteristics and metrics were suggested in an annex to the standard but they weren't part of the official standard. Overall the standard identified the elements of a software quality model for specifying and measuring software product quality, but fell short of offering a complete and comprehensive model.

Figure 14 shows the evolution of the ISO 9126 standard since its introduction in 1991. The original standard is now represented in two separate sets of standards. The ISO 9126-x series represents the expanded quality model, and the ISO 14598-x series details a process model for evaluating quality.

The new standard has evolved significantly from its roots. The two main changes are (1) suggested quality subcharacteristics that were informative in the 1991 standard are now normative, and (2) an expended set of suggested metrics have been added for measuring quality characteristics from multiple perspectives.

ISO 9126-1 contains the original quality model and its extensions. The quality subcharacteristics that were specified in an annex in the original standard have been modified slightly and are now part of the official standard. Figure 15 shows the full taxonomy of quality characteristics in the ISO 9126-1 standard. It also illustrates the connection to the other two related standards for external and internal metrics which are defined in ISO 9126-2 and ISO 9126-3, respectively.

The quality model in ISO 9126-1 is the foundation for the other three parts of the ISO 9126 standard which define metrics and techniques for measuring quality from three different perspectives: internal quality, external quality, and quality in use. Figure 16 shows the relationship between these three perspectives.

ISO 9126-2 defines external metrics for measuring subcharacteristics in the base quality model. These are measures of quality from the perspective of the external behavior of the product at its interface. The difference between external quality metrics and quality in use metrics is that external quality metrics measure the external behavior of the product while it is running in a simulated environment. Quality in use metrics measure effects on actual end users while the product is running in its production environment. The difference is analogous to the difference between system testing a product with fabricated data and acceptance testing with real user data in the presence of actual users.

ISO 9126-3 defines internal metrics for measuring subcharacteristics in the base quality model. Some quality characteristics can only be measured with internal product metrics. For example, the maintainability of a solution isn't visible from a product's external interface. Internal quality metrics are typically measurements of static structural properties of the implementation or intermediate forms of the implementation.

ISO 9126-4 defines quality in use metrics that measure the results of using the system in a specific environment. Quality in use is a measure of quality from the user's perspective while using the product in its environment. Quality in use measures the ease and extent to which users can accomplish their goals. As figure 17 shows, quality in use metrics are based on the smaller set of user-oriented quality characteristics: effectiveness, productivity, safety, and satisfaction.

The same basic quality characteristic from the ISO 9126-1 model can be measured in three separate ways depending on the perspective taken. For example, consider the quality characteristic efficiency. Internal efficiency can be measured by analyzing the asymptotic complexity of the algorithms used in the implementation. In terms of efficiency, an O(n) algorithm is more efficient or has higher internal quality than an O(n²) algorithm. External efficiency can be measured by simulation. If a product performs well in a simulated environment it is of high quality. From the quality in use perspective, program efficiency indirectly influences productivity and satisfaction. The product has sufficient quality if performance doesn't negatively impact the productivity or satisfaction of end users.

The ISO 9126 standard defines subcharacteristics and associated metrics but allows for application-specific quality characteristics and metrics. It would be difficult to mandate a specific closed set of quality characteristics or metrics because they are heavily dependent on implementation technology and the application domain. The ISO 9126 standard also doesn't define an absolute measure of quality because, again, the relative importance of each characteristic is application-dependent. For example, reliability is more important for mission critical applications; efficiency for time-critical applications; and usability for interactive applications.

Recent updates make the ISO 9126 standard one of the most current. At the very least it offers another taxonomy of quality characteristics and metrics to consider. The standard will be of special interest to organizations that are already using ISO process standards

17.2.4 Software Quality Characteristics

This section defines some of the key software quality characteristics introduced by the quality models in the previous section. These characteristics, often called the "-ilities" of software, are the foundation for expressing and validating the non-functional requirements of a software system. The definitions in this section are adapted from [ Cavano and McCall '78 via Chow '85 ] [Boehm et. al. '78] [ Ghezzi 2003] [Wiegers 1999].

Correctness. Correctness covers a lot of ground. System requirements are a mix of functional and non-functional requirements. The focus of this chapter is on specifying and managing non-functional quality requirements but the quality of the system is the degree to which functional as well as non-functional requirements are met. Correctness is the degree to which stated and implied functional requirements are met. The word implied makes it clear that quality isn't strictly conformance to written requirements. If a desired feature isn't implemented because it's not included in the requirements, it's a deficiency of the requirements process that also diminishes the overall quality of the product.

Availability. Availability is the probability the system will be available at a certain point in time. Figure 18 below shows a model for the operational life cycle of a system. The system executes for a certain amount of time before a failure is encountered. The average time for this duration is the mean time to failure (MTTF). There is a period of maintenance during which the system is unavailable. The average time for this duration is the mean time to repair (MTTR). Once repaired, the system runs again until the next failure (MTTF). Formally, availability is the percentage of time the system is available for productive use. Using the measurements shown in figure 18, availability = MTTF / MTBF.

In practice availability requirements might be higher or lower during certain periods. For example, an online stock trading system might have high availability requirements during the day while the stock market is open for trading. In the evening the availability requirement may be low or even non-existent.

Reliability. Reliability is the probability that the software will operate without failure during a specific time period. There is a subtle difference between availability and reliability. Availability represents the measure of a software system at a certain point in time. Reliability is the measure of a software system over a duration of time. A user's first concern is availability. "Is the system available when I need it?" Next, the user's concern is reliability. "OK, the system is available, but will it run long enough that I can finish my work?" Using the measurements shown in figure 18 above, reliability is the mean time to failure (MTTF).

As an analogy, a car is available if it starts when you need it to. It's reliable if, once started, it gets you to your destination without failing. It's maintainable if, when it does need repair, the cost of maintenance is reasonable.

Maintainability. Maintainability is a measure of the effort required to correct a defect or make a function-preserving change to software. Software is maintainable if it is easy to understand and test. Maintainability is closely related to flexibility. They both are a measure of the software's ability to accept change. The distinction made here is that maintainability is concerned only with function-preserving changes while flexibility covers functional enhancements. Using the terminology of chapter 17, maintainability is concerned with corrective and preventive changes, and flexibility with adaptive and perfective ones.

There is certainly overlap between maintainability and flexibility (i.e. they both depend on the software being understandable and testable), but it's possible to increase each quality independent of the other. For example, a weak design could still be maintainable if the code is understandable but probably wouldn't be flexible in a way that facilitates extension.

Flexibility. (Also called extendability, evolvability, etc.) Flexibility is a measure of the effort required to extend the functionality of software. Flexibility, like maintainability, depends on software that is easy to understand and test. To be flexible the software should also have a design that supports extension. One way to achieve a flexible design is to anticipate changes up-front. During the requirements phase customers typically have some insight into potential future enhancements. These potential enhancements can be documented along with the requirements and carried over as input into the design phase. Developing iteratively also encourages flexible design since each iteration is essentially product extension.

Robustness. A robust system is one that responds reasonably when confronted with unexpected input or unexpected conditions in the operating environment. Robustness borders on correctness and reliability. Once invalid inputs or operating conditions are identified, the proper responses can be included in the requirements--a violation of which would then be a correctness or reliability issue. (See figure 19.)

Figure 19. Robustness is responding reasonably to unanticipated inputs or conditions

Most systems will encounter unexpected inputs and/or operating conditions sometime during their lifetime. Systems that respond to unexpected inputs or operating conditions in a reasonable way are robust. The robustness of a system is largely determined by how programmers respond when they encounter "impossible" conditions in their code. (See code listing 1.)

Verifiability. (Also called testability.) Verifiability is the measure of effort required to (1) specify the validation criteria for a work product, and (2) verify that the work product meets its validation criteria. For example, a design is harder to verify than a mathematical routine because the criteria for establishing the correctness of a design is harder to define than the criteria for establishing the correctness of a mathematical routine. A mathematical routine is also easier to test because it can be isolated from its environment and tested with known values.

Verifiability is a little more general than testability. While testing is only practical for code, all work products can be verified using static analysis techniques. Dynamic testing is the operation of a system for the purposes of verifying that is works correctly. Static analysis is the analytical review of work products (including code) for the purpose of verifying that the work product conforms to its specifications or requirements. All of the work products of the software life cycle can be verified using either dynamic or static techniques.

One way to enhance the verifiability of code is to instrument it with automated test cases. (Automated testing is discussed in chapter 16.)

Efficiency. A system is efficient if it performs its function without wasting resources such as CPU time, memory, disk space, and bandwidth. Efficiency is mainly a problem when it affects performance. Efficiency is an internal quality, and performance is an outward or external quality. A routine might be inefficient but as long as it doesn't cause unacceptable performance as seen by the end user, it's not a performance problem.

Efficiency is often in conflict with other desirable quality goals such as portability and maintainability. Increases in efficiency usually come at the expense of these other characteristics (and vice versa). Advances in technology make finding the right balance a moving target. Faster microprocessors and cheaper memory make certain inefficiencies more tolerable, but demand for smaller less expensive devices increases the priority of efficiency for many applications.

Portability. Portability is the measure of effort required to move a software system to a new hardware or software environment. Because an operating system and possibly other middleware sit between an application and the underlying hardware, portability between software environments is often the biggest concern. Software dependencies that can limit portability include dependencies to the operating system, database, runtime environment or other services such as message queues, authentication directories, etc. The best way to plan for portability is to modularize around potential dependencies. For example, the first layer in the layered architecture style is often an abstract layer for operating system services. Given such a layer, the changes required to move the program to a new operating system would be limited to a single layer in the design. (For more information on the layered architecture style see chapter 10.)

The portability of a system is often expressed as the percentage of effort required to write the system from scratch in a new environment without the aid of the existing system. For example, a system that has a portability measure of 10% implies that it could be ported to a new environment for 10% of the cost of writing the system from scratch in the new environment without the aid of the existing system. Portability is usually defined relative to a specific target environment [Gilb 88].

Reusability. Reusability is the measure of effort required to reuse a component or artifact in a context other than the one for which it was originally developed. Reusability is another quality characteristic that applies not just to code, but any artifact created during the software life cycle. An organization with a standard development process usually has a collection of software process assets such as templates and sample documents that can be reused across projects. Following a standard development process itself can be considered a form of reuse. Most attention, however, is usually placed on code reuse because it offers the biggest opportunity for leveraging assets. Many organizations promote the creation and use of a common repository of reusable assets. The open source community exemplifies the idea on a larger scale. Simply using the class library associated with a programming language is another form of code reuse.

Developing components for reuse usually costs more than developing for single use. However, if the components can be reused and leveraged across multiple systems, the net result will be a reduction in expenses. Opportunities for reuse are described in more detail in chapter 12.

Understandability. Before a component can be maintained, reused or extended it must be understood. Understandability is the measure of effort required to understand the purpose of a component. What does the component do? How does it do it? What is the relationship between the component and other components? As Boehm's quality model in figure 13 shows, understandability is related to the more primitive quality characteristics: consistency, conciseness, structuredness, self-descriptiveness and legibility. One of the main ways of achieving understandability is with an appropriate use of abstraction as a technique for hiding complexity. There is a staggering amount of detail in even moderately sized work products, but its possible to organize this detail in a way such that it is revealed in layers. Understandability and readability are interchangeable.

Usability. The most common request users have when solicited for requirements is, "make it easy to use". Usability refers to the ease of use or user friendliness of an application. Usability is the measure of effort required to learn and use the application. This includes preparing input and interpreting output. An error message that describes the problem in technical terms might be helpful for the analyst that has to investigate the problem, but it doesn't make the program very usable from the user's perspective.

Usability is maximized by having a user interface (UI) that is consistent and predictable. Usability is reduced when implementation details are allowed to seep into the UI. This can easily happen when developers participate in UI design. They are deeply familiar with the implementation and without even realizing it, may produce designs that depend on this knowledge. For this reason many projects assign the task of UI design to a non-technical domain expert or product champion. This is not to say that developers shouldn't be allowed to participate in UI design. Projects that don't include developers in UI design are missing a trick. Developers have the most knowledge of the capabilities and opportunities provided by the implementation technology. When making design decisions they have to be able to look at the problem from the user's perspective and forget what they know about the implementation. Really good developers can context switch between these two perspectives as needed.

Integrity. Integrity is the extent to which the system is safe from unauthorized access. Integrity is synonymous with security. A system with a high degree of integrity will prevent unauthorized access, data loss, data corruption, service interruption, and virus infections (both accidental and intentional). Integrity is one quality that is absolute and can't be compromised. It would make little sense to have a system quality goal of "some integrity". With just one security problem the system looses all of its integrity. A project might spend more or less time ensuring the system has absolute integrity, but the goal is always 100% integrity. Part of defining integrity requirements for a system is making explicit the authority and level of access available to each type of user.

Interoperability. Interoperability is the measure of the effort required for one system to exchange data and services with another. Interoperability is increased with the use of standard interfaces. Web services promises to increase the interoperability of systems across the Internet.

17.2.5 Cost of Quality

Everyone is for higher quality, but the reality is that expenditures on quality improvement initiatives are economically justified only when the expected savings from higher quality is greater than the initial upfront investment it takes to achieve higher quality. The cost of quality is both a metric and economic model for understanding the cost-benefit tradeoffs of pursuing higher levels of quality. It can be used at the start of a project to justify an investment in quality improvement or at the conclusion of a project to measure the outcomes of such expenditures.

Quality is one of the 4 variables of a project that has to be balanced against the other three which are cost, schedule and features. The following conceptual formula expresses the general relationship between these four variables:

The formula suggests that improving quality requires some combination of higher cost, longer schedule, or fewer features. Holding schedule and features constant, quality correlates with cost:

In practice the true relationship between quality and cost is not so simple. In some cases it's possible to increase quality without increasing costs. To understand why, consider that cost represents not only the money spent on quality improvement (mainly defect prevention and early reviews), but also the money not spent on late stage testing and rework when defects are avoided through additional quality improvement efforts. In other words:

Cost-of-Quality = (cost of defect prevention and early reviews) + (cost of late stage testing and rework)

Increasing the first category of costs (prevention) tends to reduce the second category (rework). When the amount saved by avoiding defects rises faster than the amount invested in order to avoid these defects, quality is free*.

---------------------------------------------
*Philip Crosby was so convinced of this dynamic that he titled one of his books Quality is Free, lest anyone would miss the main point which was that the major costs of quality are the costs associated with poor quality. Crosby maintained that, as the voluntary costs of defect prevention are increased, the involuntary costs of poor quality (mainly rework) decrease by an even greater amount thus making quality free [Crosby].
---------------------------------------------

How much does quality cost? Is quality free? Is there a point of diminishing returns, beyond which an investment in quality is no longer cost-justified? The cost of quality model helps to answer these questions by modeling the relationship between quality and costs*.

--------------------------------
*When discussing the cost of quality, quality is defined as freedom from defects. It explicitly doesn't mean luxury or features. More luxury and features almost always means higher cost.
--------------------------------

Formally the cost of quality is the sum of conformance costs and non-conformance costs. Conformance costs further breakdown into prevention and appraisal costs. Conformance costs are the costs associated with attaining quality. These are the voluntary investment costs for things like defect prevention and early reviews. Non-conformance costs are the costs of dealing with defects. This includes things like debugging and rework resulting from exposed defects. These are involuntary costs and a source of potential savings if defects can be reduced. Non-conformance costs may be internal failure costs (cost of errors found before the product is released), and external failure costs (cost of errors found once the product has been released). Figure 20 shows the taxonomic breakdown of these costs and figure 21 shows when these costs generally occur during the software life cycle.

The following is a closer look at these 4 categories of costs. They are also summarized in table 2 below.

Prevention costs - These are the costs associated with preventing defects from ever occurring. If you can prevent errors from occurring you can avoid the expense of finding and fixing them (not to mention the wasted effort spent making the error in the first place). Prevention costs include the costs of training, early reviews, quality planning, root cause analysis, baseline configuration management, and process improvement initiatives.

Appraisal costs - These are the costs associated with analyzing and testing the product to make sure it conforms to requirements. The most common appraisal costs are the costs for testing, inspections and quality control.

Internal failure costs - These are the costs associated with fixing defects found prior to product release. This includes the cost of rework, retesting, and the administrative overhead of pre-release defect management. It may also include the cost of updating documentation, additional reviews and inspections.

External failure costs - These are the costs associated with fixing defects found after the product has been released. The process of diagnosing and fixing an external failure is much more complicated and involved than fixing an internal failure so costs are higher. There are the regular expenses that go along with internal failures, plus additional expenses that come with field defects. Less tangible external failure costs are the costs associated with possible damage to reputation and the loss of future sales. External failure costs may also include the unpredictable costs of litigation.

The cost of quality is the sum of all conformance and non-conformance costs. The cost of quality is a real expense that shows up indirectly in the budget of every project or directly when the cost of quality is being tracked. It is also an expense that can be controlled by the project manager. In order to control the cost of quality the project manager must understand the complex relationship between conformance costs and non-conformance costs.

There are three notable aspects of the relationship between conformance and non-conformance costs. First, the two categories of costs tend to be inversely related. Spending more on conformance should reduce non-conformance costs. Spending less on conformance will likely increase non-conformance costs. Second, conformance costs are voluntary while non-conformance costs are involuntary. The project manager decides how much to spend on conformance, and this in turn determines how much will be spent involuntarily on non-conformance. Third, conformance costs represent an investment that must be justified by even larger anticipated savings in non-conformance costs. It is an investment that usually has a point of diminishing returns beyond which spending more on conformance won't reduce failure costs enough to justify the initial expense and risk.

The project manager is responsible for determining the optimal amount to spend on conformance activities such that the overall cost of quality (conformance and non-conformance costs) is minimal. Projects vary in size and budget so the amount won't be an absolute number but rather some portion of the total budget. The optimal cost of quality is usually expressed as the cost per good unit of product produced or as a percentage of total development costs.

The problem of finding the optimal cost of quality is made more difficult by the fact that the optimal balance between conformance and non-conformance costs depends on the characteristics of the project and the maturity of the development process being used. There isn't one precise answer, but there are models for understanding in general the forces that affect the optimal balance in different environments. Figure 22 shows two such models. Figure 22.a shows the traditional cost of quality model and figure 22.b shows the contemporary one. The models depict the general relationship between costs and quality for different types of projects.

Both models show the costs of quality in terms of conformance and non-conformance as quality increases (the x axis). In both models the total cost of quality is the sum of conformance and non-conformance costs. Also in both models, non-conformance costs drop to 0 as quality reaches 100 % or zero defects. The models differ in their assumptions about the amount of investment needed in conformance to achieve perfect quality. In the traditional model, a sharp increase in conformance cost is needed to achieve perfect quality. The contemporary model shows a more modest increase in conformance costs as quality rises, thus making it possible to have both perfect quality and minimal cost.

Which model is correct? They are both correct, for the project environments they depict. The conventional cost of quality model in figure 22.a is appropriate for project environments where the cost of failure is low and the effort required to achieve zero defects increases rapidly as product quality approaches 100%. For example, this model accurately depicts the cost of removing all of the errors from a large daily newspaper. The cost of an error is minimal and the cost of removing every single error is prohibitively expensive. It could take as much time to find the last single error as it took to find all of the other errors combined. The economic impact of a reader finding an error isn't worth the extraordinary effort it would take to find all the errors. To paraphrase Crosby, the traditional cost of quality model seems to be saying that quality is free but perfection you have to pay for.

The contemporary cost of quality model in figure 22.b is more appropriate in environments where:

As an example, consider the development process for microprocessors. The cost of putting a defective microprocessors into production is very high thus justifying increased spending on conformance. Modern production processes limit the number and types of errors introduced during development, and automated testing tools in most cases make it economical to perform complete testing during appraisal. This reduces the cost of conformance.

So, which model is most appropriate when the environment is software development? Again, it depends on the specific project, but for the average software development project the traditional cost of quality model is probably more appropriate. Not all of the trends that are making the contemporary cost of quality model more applicable in manufacturing apply when the product is software.

The reason for this rather pessimistic assessment is that first, software development is a human-intensive activity. Humans are fallible, so it follows that software will always have defects as long as humans are in the loop. Second, all but the most trivial software systems are too complex to completely test. There will always be some inputs that haven't been tested. The difficulty of preventing all defects and the inability to test for all inputs, suggests that for most software products the goal of zero defects is either impossible or prohibitively expensive. This implies that for most software projects there is a point of diminishing returns beyond which improvements in quality are not cost-justified.

The above conclusion is mainly a theoretical result--and not a very original one at that. Most software practitioners are well aware that for most software systems complete testing is impractical. What would be more useful is some way of predicting the point of diminishing returns for a software system. Looking at figure 22.a it appears that the rising cost of conformance (principally appraisal costs) is what causes the total cost of quality curve to reverse course and head upward. However, a rapid rise in conformance costs doesn't by itself determine the low point in the cost of quality curve. The cost of quality is conformance costs plus non-conformance costs. If the cost of a failure is high it can justify high, even exponentially increasing, conformance costs. For example, figure 23 shows the cost of achieving ultra high reliability with the primary flight control software on the Space Shuttle. As you might expect, developing near perfect code for the Space Shuttle is an order of magnitude more expensive than developing what might be considered very reliable software (1 defect per KSLOC). In spite of the high conformance costs necessary to achieve near-perfect code, the total cost of quality is still minimal because the cost of failure is even greater. The high cost of a defect in the Space Shuttle software in terms of human life, vehicle expense, and national pride can justify large expenditures on conformance. In general, a rapid rise in appraisal costs alone isn't enough to trigger a point of diminishing returns. Applications that have very high reliability requirements can justify very high, even exponentially increasing, conformance costs.

As the cost of failure rises, increased spending on conformance is justified and the point of diminishing returns in figure 22.a shifts to the right. In other words, a high cost of failure can justify large investments in quality improvement. When calculating the cost of failure the non-obvious costs are often overlooked. These include:

1. The intangible costs of failure, including loss of good will and future sales.
2. The user's cost of failure. Factoring in the user's cost of failure may justify even higher investments in conformance.

The point of diminishing returns is also shifted right by improvements in the standard development process. With an improved development process fewer defects will be injected and conformance activities will be more efficient and effective. This makes higher quality more economical.

In summary, the cost of quality is both a model and metric for planning and monitoring process improvement initiatives. Some practical applications of cost of quality principles are [Plunkett and Dale 1990, Managing Quality]:

17.3 Software Quality Control

The meaning of the term quality control in software development is still evolving. The IEEE offers these candidate definitions:

In everyday discourse the term is typically used as a general reference to any quality-related activity, as in "Our support costs are killing us. We need better quality control."

Outside of the software industry the term has a more precise meaning. In the manufacturing and service industries, quality control refers to a managerial process for maintaining the quality of a result [PMBOK 2000] [Juran] [Gryna]. Once an operation achieves a certain level of performance, quality controls are put into place to maintain that level of performance. For example, consider a customer service call center at a consumer products company. After careful product design and training, the average length of a customer service call might be 3 minutes. If this is acceptable, a quality control process is put into place to maintain this performance. If a 3 minute customer wait is unacceptable, some combination of improved product design and/or employee training is needed to reduce the the average length of a service call. Once the wait time reaches an acceptable level, quality controls are put into place to maintain the capability of the operation.

Quality control is a particular application of the standard process control feedback loop. The basic steps of process control are:

1. Establish plans and standards of performance
2. Perform work according to plan
3. Measure performance relative to plan
4. If actual performance deviates significantly from plan, take corrective action to bring performance back into line with plan

You might recall another application of process control is the control function in the project management process. At the project level, the project manager exercises control over the project by reviewing status reports and taking corrective action as needed in order to keep the project moving forward as planned. Quality control is process control performed on a smaller scale. Quality control is performed by workers as part of their day-to-day routine. They take measurements of results and immediately use these data to make adjustments as needed in order to maintain satisfactory results.

The definition for quality control that will be used here is one that is consistent with the use of the term in the manufacturing and service industries. Here quality control is defined as:

This definition is consistent with the emerging uses of quality control in the practice of software engineering [CMMI] [SPC articles].

Quality control is one of 4 major quality management processes. The other 3 are quality planning, quality improvement and quality assurance. You can gain a better understanding of quality control by studying its function in the context of these other processes. Figure 25, the Juran Trilogy diagram, is a conceptual diagram that illustrates the interrelationships between quality planning, quality control, and quality improvement.

Quality management starts with quality planning. During the planning phase quality goals are set and the means for accomplishing these goals are planned. Quality control activities start during the production or development phase of operation. The diagram shows the changing values of some control subject over time. For example, the control subject might be the number of errors found in different modules during system test. A defined process has a certain inherent capability that characterizes the quality that can be expected from the process. One way to depict this capability is by the amount of variation inherent in the process. The more variation there is, the less capable the process is*. The original zone of control in figure 25 shows a wide variation in the results of the process. This suggests a less capable process. For example, you would expect a poorly defined development process to produce modules of unpredictable quality. The purpose of quality control is to maintain the status quo or the inherent capability of the process. The out-of-range value in the figure indicates a potential problem in the process that has a specific cause which should be investigated. For example, analysis might reveal that fewer errors were found in a particular module because the test cases for that module were mistakenly excluded from those used during the system test. With a quality control process in place, an error such as this can be found and corrected quickly before it has a chance to impact the quality of the product.

--------------------
*In some cases high variability might seem benign as long as the average performance is acceptable. However, the more variation in a process the less predictable it is, and predictability in a process is highly valued.
--------------------

The center part of figure 25 illustrates a change in the process that occurs as a result of some quality improvement initiative. Quality improvement is a change in the process that improves the inherent capability of the process. The figure shows a reduction in variation (zone of quality control is less) as well as performance (average absolute measure of the control subject is less). Note, that after the quality improvement initiative there is a new zone of quality control.

Quality control and quality improvement are both techniques for minimizing error and improving quality. They are distinguished by the types of problems they eliminate. Quality control is a process for controlling variation by detecting and correcting sporadic quality problems that have special or assignable causes. Quality improvement is a process for correcting chronic causes of quality problems. Chronic quality problems have no single identifiable assignable cause. They are the cumulative effect of many small essentially random causes of poor performance. The only way to fix chronic quality problems is by changing the process. (Statistical techniques for distinguishing between sporadic and chronic causes of quality problems are described in more detail in chapter 20.)

The Quality Control Process

Quality control occurs during the production or development stage of a software project. This section presents a 6-step process for performing quality control. A running example is used to illustrate each step. Figure 26 shows a flowchart for the process.

Step 0. The quality control process executes within the larger quality management function. The planning step of the quality control process can begin once project quality goals have been set and activities for accomplishing these goals have been identified and planned. These activities may come from the software development process being followed. For example, reviews, testing and quality assurance are all quality management activities that might be planned for a project.

Step 1. The first step of the quality control process is to identify control subjects--product or process features that need to be controlled in order to achieve the quality goals of the project.

When selecting control subjects, preference should be given to leading indicators. Leading indicators expose quality problems early when corrective action is more effective and economical. One of the tenets of modern quality management is that it's better to build quality into a product than to test errors out. Measuring and responding to leading quality indicators is one way to build quality into a product.

Step 2. Once control subjects have been selected, the next step is to establish metrics or the means of measuring these control subjects. Sometimes a control subject will have more than one candidate measure. For example, product size can be measured in lines of code (LOC) or function points. The metric selected should (1) correlate with the quality characteristic being measured, (2) be trusted by project stakeholders, and (3) require minimal effort to collect. (See chapter x for more information on measuring quality.)

Step 3. The third step of the quality control process is to identify specific performance goals or target values for the control subjects. The target values for control subjects are set with project quality goals in mind. Target values should be sufficient such that meeting them will provide reasonable assurance that the overall quality goals for the project will be met.

Project quality goals are set at the beginning of the project based on business needs and a consideration for the capability of the development process. The project quality goals will dictate the control subject quality goals. When project quality goals are translated into control subject quality goals, the feasibility of the project quality goals starts to become evident. What seemed like realistic project quality goals at the beginning of the project may reveal themselves to be overly optimistic. The capability of the development process and experience on past projects suggest what are reasonable quality goals for control subjects. If project quality goals require control subject quality goals that are unreasonable for the existing development process, changes in the development process will be needed. Possible changes include adding more reviews, practicing defect prevention, reusing existing components, etc.

Steps 4-6. The first three steps of the quality control process are preparatory. The remaining steps form the feedback loop which is performed at regular intervals during the execution of the project. Quality control is an ongoing activity. Values for control subjects are measured over time to detect deviations from plans and to identify trends. In steps 4-6, results are measured and compared with plans. If actual results deviate significantly from plans, corrective action is taken to bring performance back into line with plans.

Limits of Quality Control in Software Development

Having just discussed the union of software development and quality control, it's time for a reality check. Quality control is a regular practice in manufacturing, but software development is not manufacturing (see sidebar). The mass production of identical physical items is fundamentally different from the design and development of software systems. Software development is primarily an intellectual design activity where every product is unique. This makes it difficult to achieve the degree of precision and control during software development that is commonplace in manufacturing.

Software Development is not Manufacturing

Manufacturing is sometimes used as a metaphor for software development. The metaphor is useful only where the similarities and differences between the two disciplines are well understood. Software development is not manufacturing, so practices common in manufacturing don't necessarily apply to software development.

What follows is a brief comparative look at the manufacturing and software development product cycles. The comparison will reveal why software development is less predictable and controllable than manufacturing and why practices such as quality control, which are common in manufacturing, don't automatically apply to the products and processes of software development.

The manufacturing product cycle, illustrated in figure 28.a, consists of three distinct phases: (1) a product is designed according to customer specifications, (2) an efficient production process is established for creating the product according to the design, and (3) the production process is transferred to an operations phase where the process executes continuously turning out essentially identical items. The production process is specific to the product and is worked out in great detail in advance. It is highly predictable, repeatable and controllable.

Figure 28. Manufacturing Compared to Software Development

The software development product cycle has rough analogs to the manufacturing product cycle. (1) a product is designed according to customer specifications, (2) the development phase, including design and implementation, is planned and executed, (3) the source code is compiled and linked into a finished product. Creating another instance of the product is as simple as duplicating a CD-ROM or copying a file.

The obvious difference between software development and manufacturing is that software development is purely design and development. It doesn't have a production phase*. Or rather, the production phase is completely automatic and trivial from the developer's perspective. This is the source of the problem with the software development as manufacturing metaphor. The comparison is often made between software development and the production phase of manufacturing, when a more apt comparison would be between software development and the product development phase of manufacturing. It's important to make the distinction so as not to expect the same predictability and control during software development that's routine during the production phase of manufacturing. For example, quality control works well during the production phase of manufacturing because the production process is well-defined and repeatable. Quality control doesn't work so well during software development because design and development is not a deterministic process.

-------------------------------
*Some might argue that the analog to the production phase in manufacturing is the coding stage of software development. Coding might be more routine than design but it is not as predictable and repeatable as the production phase in manufacturing. It's a poor analogy principally because the design stage in manufacturing is not the same as the design stage in software development. The design for a manufactured product is fully specified. The design for a software product is a high-level abstraction that is validated and refined during the implementation and testing phases. In software development, a significant amount of design and innovation occurs after the design phase proper. In terms of the manufacturing analogy, "the source code is the design" and not the product of software development [Reeves, What is software design? 1992, reprinted in Robert Martin 2003]. The product of software development is the output from the compiler.
-------------------------------

Controlling the quality of manufactured products is based on two principles. First, most products are aggregates of more primitive components. The quality of a product is determined by the quality of its constituent components. Second, the quality of a component is inversely proportional to the amount of variation in the attributes of that component. Low variation means high quality and vise versa. For example, ball bearings are one of the constituent components of a motor. The quality of a motor is determined in part by the quality of the ball bearings used in its construction. High quality ball bearings are ones with measurements that vary little from their specification. (See figure 29)

It's possible to control the quality of manufactured products because (1) the variation in components of a manufactured product can be controlled with precision, and (2) there is a direct link between controlling variation in these components and the quality of the products built from these components.

Quality control is much less certain when the product is software. The main problems are:

During manufacturing it's possible to control the quality of a product by controlling the quality of its constituent components. The process of software development is less predictable and controllable. There is less control over the evolving elements of a software system and no guarantee that controlling the quality of these elements will ensure the quality of the resulting system.

Most of the discussion up until this point has been about quality control at the product level--controlling the quality of the evolving products of software development. Quality control may also be used at the project level for controlling the execution of the project [Kan]. For example, specific goals for the number of defects injected and detected at each phase can be set and tracked. Goals can be set based on past experience with similar types of projects and the expected capability of the process being used. Actual results provide feedback on the state of the project. When actual results are significantly different from plan, corrective action can be taken to bring results back in line with plan. This is quality control at the project level. Variation in the execution of the project is tracked and controlled to ensure the project is performed as expected.

Statistical Methods of Quality Control

Quality control is often performed with the aid of statistical methods. Statistical methods of quality control include statistical process control (SPC), statistical sampling, and design of experiments [Montgomery]. Chapter 20 describes SPC in more detail. The others are beyond the scope of this text. The rest of this section shows the connection between the process aspects of quality control described here, and the statistical techniques of quality control discussed in chapter 20.

The example used here to introduce the idea of statistical quality control is a non-technical one. Consider the use of quality control during the production phase of a manufacturing process that packages breakfast cereal. One of the quality goals is to be able to control the amount of cereal that goes into each box. There is a certain amount of variation in any process so the amount of cereal will vary between boxes. Any less than the advertised weight and the customer will be dissatisfied. Any more than the advertised weight and profits are reduced. The quality control process as described in this section can be used to keep the production process performing to plan. For this example assume the following quality control parameters:

During operation samples are selected and measured for conformance to the stated quality control goals. Figure 31 shows the results in the form of a run chart:

The results look good. The run chart in figure 31 shows that the weight of selected samples is well within the specification limits. Because the recorded values for all measurements fall within the specification limits, there is no need to intervene with corrective action. This is the most that can be expected from the process of quality control as described in this chapter.

The quality control process described in this chapter keeps performance on track but it doesn't maximize the capability of the process. Statistical methods go beyond what can be expected from the process of quality control alone. Statistical methods quantify the data in ways that reveal new information which can be used to maximize the capability of the process. For example, here are three specific ways statistical methods of quality control can enhance the results of the example above:

Statistical process control is a collection of tools for performing statistical quality control. One of the primary tools of statistical process control is the control chart. Figure 32 shows what a control chart for the example above might look like*. A control chart has control limits rather than specification limits. The control limits are calculated based on a stable set of measurements from the process. They define the capability of the process. Values outside of the control limits are statistically likely to be the result of some specific identifiable cause that can be corrected as part of the quality control process. Variations within the control limits are likely to be due to the random noise in the process and are better addressed by fundamental changes to the process.

-----------------------------------
*The graph shown is more of a pseudo or conceptual control chart. The data values on a real control chart are the average values for a small number of samples and not the values of individual samples that appear to be shown in the control chart in this example.
-----------------------------------

17.4 Software Quality Assurance

Software quality assurance means different things to different people. Some people use the term as a synonym for testing or as a synonym for the group responsible for testing, as in "the product must pass through quality assurance before we can ship it." Quality assurance is not the same as testing though. Testing is about detecting defects. Quality assurance is about preventing them and assuring others that they have been prevented. Quality assurance may use the results of testing but it is primarily an oversight function. The goal of testing is to identify defects. The goal of quality assurance is to assure stakeholders, especially those not directly involved with day-to-day operations, that the product has achieved a certain level of quality.

Quality assurance is a common practice in manufacturing and the service industry. Consider the following examples of quality assurance in everyday activities:

These are all examples of quality assurance in action. In all cases there is a person or organization responsible for ensuring that goods or services are being delivered according to approved standards and procedures. In all cases the person doing the inspection has no direct control over the quality of the product or service. The inspector's job is only to verify that the product or service being delivered meets established standards.

Software quality assurance is the practice of quality assurance applied to the products and processes of software development. The IEEE Standard 12207 defines software quality assurance as,

Expanding on each element of this definition provides a deeper understanding of the term quality assurance:

Figure 33 outlines the key activities of SQA roughly in the order they would occur during the software life cycle.

At the beginning of a project SQA works with the project staff to identify and plan appropriate standards and procedures of software development. These standards and procedures typically come from the organization's standard process. The organization's standard process may be owned and maintained by a separate organizational entity such as the software engineering process group (SEPG). If so, the SEPG usually has responsibility for defining and maintaining the standards and procedures while SQA ensures that they are planned and followed.

During a project SQA monitors development processes for conformance to procedures and evaluates products for conformance to standards. Process monitoring and product evaluation are performed through reviews and audits.

The results of process monitoring and product evaluations are reported to management and other stakeholders. When there are deviations from plan, SQA may recommend actions for bringing performance back into line with plan. This reporting function provides another source of information on project status and progress, especially with respect to the quality goals of the project. Management uses this information to make decisions about the project such as when to transition from one phase to the next.

The SQA function plays an important role in overall project management. Recall that the four variables of a project that must be managed are: schedule, cost, features and quality. Compared to the other three variables of a project, quality is difficult to evaluate from outside of the project. The SQA function makes the quality of the evolving product visible outside of the project. (See figure 34.) It externalizes the quality of the evolving product, putting quality on equal footing with the other 3 variables of a project.

Standards and Procedures of SQA

Standards and procedures provide structure and consistency to the process of software development. Following standards and procedures make software development more predictable, repeatable and manageable. As a point of definition:

SQA is responsible for enforcing an organization's standards and procedures for software development.

Most standards apply to either code or documentation--the two main products of software development. Coding standards include naming conventions, format conventions, constructs that are encouraged/discouraged, etc. Document standards typically specify format and desired content for written documents such as the requirements document, architecture document, verification and validation plans, test case specification, etc.

GNU Coding Standards

Coding standards are usually specific to an organization, product or language. The GNU coding standards [http://www.gnu.org/prep/standards/] are a good example of a collection of standards for a particular product, in this case the GNU open source operating system. Coding standards are especially important in open source software development because:

There is a large diverse group of contributors or programmers.
Open source's raison d'etre is to provide a foundation for others to build upon and extend. Coding standards promote consistency which improves readability, maintainability and extensibility.
In general, the turnaround time for bug fixes and feature enhancements with open source products is much shorter than with commercial products. Having a common consistent coding style decreases the time it takes to read and understand code and improves the depth of understanding that's possible when working within tight schedule constraints.

The GNU coding standards have been evolving for more than a decade. They specify such things as:

Naming standards for variables, functions, etc.
Standards for commenting code.
Guidelines for selecting a programming language.
Ways to avoid intellectual property challenges.

A process is a sequence of steps for transforming inputs into outputs. Procedures are the specific steps to be followed when performing a process. All processes should have defined procedures. Most software development processes include procedures for:

Organizational Independence

The organizational placement of the SQA function is key to its effectiveness. SQA is an oversight function, and to be effective in this role SQA should be organizationally separate from the staff they are monitoring. This gives SQA the freedom to provide objective unbiased assessments of process and product quality. SQA may report preliminary results of evaluations to the project manager, but the allegiance and formal reporting relationship of those performing the function should be with an organizational entity above the project manager. (See figure 35.)

Rational for SQA

SQA is primarily an oversight or audit function. Many people are wary of audits and may even resent having their work checked. The monitoring may be perceived as a sign of mistrust or lack of confidence.

Resistance to SQA can be overcome by educating staff on the role and importance of SQA. When workers understand the role and value of SQA, they are usually more willing to have their work audited. It's easier to accept auditing as part of software development when you consider that auditing is an essential and accepted practice in other areas of life. For example,

In software development, product quality is difficult to accurately measure from outside the project. External project stakeholders need reliable information about the quality of the evolving product in order to make decisions that affect the project. The development staff are the best source of information on product quality, but the motivations and priorities of developers are sometimes in conflict with those of external stakeholders. SQA helps balance the structural relationship between developers and stakeholders for the long-term benefit of all.

Oversight plays an important role when performance is critical or when independent assessment is needed to balance certain structural relationships. For life- or mission-critical software systems SQA can provide additional assurance that the system produced will meet its intended requirements. The quality of the final product is more certain when it is verified by the activities of SQA.

SQA Planning

SQA activities must be planned. Planning for SQA may be included in the main project plan or specified in a separate Software Quality Assurance Plan (SQAP). This section describes the key elements of a SQAP. The elements presented here are derived from and are a subset of those found in the IEEE standard for software quality assurance plans [IEEE 730-2002]. The characteristics of the system and project dictate the level of rigor and formality needed in the SQAP. Only the essential elements of a SQAP are presented here. The elements presented here may not be sufficient for safety- or mission-critical systems*. These systems may require the full details of the IEEE standard.

-------------------------------
* Another reason to consider following the full details of the IEEE standard is to avoid being found legally negligent. One of the criteria for establishing legal liability when work results in a catastrophic failure is whether or not the work was performed according to standards established by a representative consensus of industry professionals. It is an open question whether or not software development practices are mature enough to be standardized [IEEE Computer, 27(9) 1994, Evaluation of Software Engineering Standards], but to the extent that they are, the collection of IEEE standards arguably represent the closest work we have to standards established by a representative consensus of industry professionals.
-------------------------------

Purpose

This introductory section of the SQAP describes the motivation and justification for performing the SQA function. What is to be achieved? What are the characteristics of the system or project that justify the additional assurance that SQA provides? Is the system safety critical? Does the project have inherent risks?

This section may also include specific quality objectives. Quality objectives may be process objectives, defect objectives (expected number of defects injected and detected during each phase of the software life cycle), or desired quality characteristics of the system (usability, maintainability, robustness, etc). Some example quality objectives are:

Notice that each quality objective is specified in a way that is measurable and verifiable. The priority of each objective should also be clearly stated in case there are conflicts between objects or resource constraints that make it impossible to accomplish all objectives.

Standards and Procedures

This section of the SQAP should list the standards and procedures that will be used during the course of the project. These are the standards and procedures that SQA will make certain are adequate, planned and followed. In most cases this section will contain references to standards and procedures defined elsewhere. Any planned deviations from or tailoring of external standards and procedures should also be documented here.

Standards and procedures are the basis for assessing compliance. They should be well-defined and documented because technical staff have the right to know the criteria for success.

Reviews and Audits

Reviews and audits are the primary tools used by SQA during a project to evaluate product quality and process performance. An audit is a formal type of review performed by independent reviewers. This section should specify what is to be evaluated (which products and processes), the type of evaluation (review or audit), the timing or frequency of the evaluation, and the specific acceptance criteria. Criteria may be specified directly or referenced if available in other external documents or standards. The following table gives an example of what data are expected in this section.

Management

The software quality assurance function is like a mini project with all the planning and management needs of a full project. At a minimum the SQAP should specify an organization structure, schedule and budget.

The organizational structure defines the organizational elements responsible for SQA. The SQA function may be centralized or decentralized. The recommended approach is to have an independent organizational element responsible for SQA. The organizational structure should also define the relationship between SQA and the other organizational elements, especially those responsible for the work that will be reviewed. Associated with each organizational element are roles with their assigned authority and responsibility. Because SQA is an oversight function, it's important to clearly define SQA's authority to monitor and evaluate the processes and products of development.

Part of defining roles is defining the expectations of each role. SQA is responsible for evaluating and reporting on the quality of products, but what responsibility do they have for the actual quality of the products they monitor? If a product is found to be of poor quality and they accurately detect and report this, do they bear any responsibility for the defects in the product? The theory of authority and responsibility suggests no. The theory holds that because SQA doesn't have any authority or control over the quality of the product, they shouldn't be held accountable for any deficiencies in the product. If, on the other hand, SQA becomes an active participant in reviews and takes some responsibility for finding defects then they may be held partially accountable for the quality of the resulting products.

The activities of SQA need to be broken down into tasks which are scheduled, budgeted, and coordinated with tasks and milestones in the main project plan. (See chapter 4 for guidelines on defining, scheduling and budgeting tasks.)

Problem Reporting and Corrective Action

This section should describe the procedures for reporting, tracking and resolving problems or issues identified by SQA. Attempting to first resolve problems locally engenders trust between SQA and those performing the work monitored. Problems that can't be resolved locally are usually escalated to senior management for resolution.

17.5 Software Quality Management

The four main constraints on a project are: time, cost, features and quality. Over optimizing one is a wasted opportunity with another; under performing on any leads to customer dissatisfaction. Key to project success is setting agreeable targets for each constraint at the beginning of a project and then managing the execution of the project to deliver on these targets. Chapter 4 describes how to manage time, cost, and features. This section describes how to manage project quality with the same clarity and precision.

Managing quality goes beyond the familiar procedural approach to quality control [Jalote 2000/2004]. With the procedural approach to quality control there are defined procedures for verification and validation. Test cases are written to verify specific requirements, and reviews are planned for selected work products. (See figure 36.) The assumption is that if tests and reviews are performed according to their defined procedures, the resulting product will have adequate quality. No attempt is made to assess the effectiveness of testing and reviews or verify that a certain level of quality has been achieved with the product.

The quality management approach to quality control builds on the procedural approach adding more control and predictability. Quality management starts with a quantitative quality goal for the project. From this goal, intermediate quality sub goals are defined for various stages of the project. (See figure 37.) These intermediate quality sub goals are quality milestones or control points for managing project quality. They motivate action and provide a baseline for measuring progress at various stages of development. With the procedural approach to quality control, performance is measured by the number of test cases executed and reviews performed. With the quality management approach to quality control, performance is measured by the degree to which quantitative quality goals are satisfied. The idea is to make product quality as predictable and controllable as the project's schedule and budget.

The basic principles of quality management are much like those that underlie time or resource management: you set goals, make plans, track progress, and take corrective action when actual progress deviates significantly from plans [Humphrey 1990]. The key to effective quality management is having explicit numeric quality goals that can be measured objectively. What gets measured gets done. If a project has accurate measures for schedule and budget but not quality, the project is likely to meet its schedule and budget goals at the expense of its quality goals.

This section starts with a brief look at the dynamic nature of software quality and then examines the main activities of quality management in more detail.

Dynamic Nature of Software Quality

There are a number of different factors that collectively influence the perceived quality of a software system. In order to better understand these factors it's helpful to consider the hierarchical nature of software quality [Humphrey 2005]. A user's first impression of a system is influenced by the number and severity of defects present in the system. A defect is any behavior that fails to meet written requirements or the user's expectations. If a system has any major defects or too many minor ones, it will create an instant negative impression in the mind of the user and any other positive features of the system will go unnoticed.

Once defects are under control, the perceived quality of a system is shaped by the degree to which desirable quality characteristics such as usability, maintainability, robustness and so forth are present. (See figure 38.) The idea is similar in concept to Herzberg's theory on employee motivation which is discussed in chapter 4. Recall that Herzberg discovered that the factors of work that cause dissatisfaction are different from those that cause satisfaction. For example, if there isn't enough light in a room people will be dissatisfied but once there is adequate light you can't create more satisfaction by adding more light. In the case of product quality, controlling the severity and number of defects is necessary to avoid a negative perception of the product, but controlling defects alone isn't enough to create a positive impression. The system must also possess the quality characteristics that are important to the end user.

This hierarchical nature of software quality explains the often repeated adage "you can't test quality into a product." Simply removing defects isn't enough to ensure quality. Removing defects only reveals the inherent quality of the product. If all the defects are removed but what is left doesn't have the quality characteristics that are important to the customer, it won't be considered a quality product. As an analogy, having a lush garden isn't simply a matter of removing all the weeds. Removing weeds is a necessary first step, but if there isn't a well-planned and nurtured selection of plants underneath, just removing weeds won't yield an attractive garden.

While quality is more than just the absence of defects, the focus of this section is on controlling the number of defects in the evolving product. The assumption is that quality characteristics are non-functional requirements which are specified and verified during the normal requirements and testing process. (Chapter 5 describes the elicitation and specification of non-functional quality requirements.) Quality management is indirectly concerned with quality characteristics to the extent that failure to properly identify or implement a non-functional quality requirement will result in a reported defect.

Setting the Project Quality Goal

The quality management process is driven by the project quality goal which is decided during the planning phase of the project at the same time schedule and budget goals are being determined. A de facto metric for expressing quality goals is defect count [Fenton 1997]. The project quality goal is typically expressed as the maximum number of defects that can be tolerated during late stage testing or during the first year of operation. Results from late stage testing are available early but results during the first year of operation provide a more accurate account of the project's true quality.

Defect count is just one metric for specifying and tracking system quality. Other potential metrics are cost of quality, defect density and test case coverage. Managing project quality by defect count is common because the data are relatively easy to obtain, it's available early in the development process, and it correlates reasonably well to quality as experienced by the customer.

Planning to make any errors might seem like defeatism or even condoning mediocrity, but it's unrealistic to expect no errors. As long as software engineering remains a human activity it will be an error-prone activity. With current development technology, perfect software is impossible or prohibitively expensive for most projects. The project quality goal may be ambitious to motivate action, but it must also be realistic since it is the basis for planning and control actions.

The feasibility of a particular quality goal can be assessed by comparing it to quality performance on past projects and the capability of the existing development process. If the desired quality goal is less than what can be expected from the existing development process, appropriate process improvement actions must be planned in order to close the gap. Keeping quantitative data on completed projects is the best way to know what level of quality can be expected from a development process and what effect process improvement actions are likely to have.

The project quality goal is based on customer needs but the customer may not be able to fully quantify their quality requirements. If the customer has experience with an existing system from the same development team, the quality goal for the new system can be expressed relative to the known quality of the existing system. For example, a customer might request a new system with quality that is equal to or better than that of the existing system. Assuming defect data has been kept on the existing system, developers can translate a request like this into a quantitative quality goal, such as: "The new system shall have at most 30 moderate defects and no sever defects during the first year of operation."

The following example illustrates the main considerations when setting the quality goal for a project.

Example. In this hypothetical example a customer has requested a new system, ACME-2. This request comes a little more than a year after a similar system, ACME-1, was delivered to the same customer. The customer is satisfied with the quality of ACME-1 and has requested that the quality of the new system be as good or better than that of the existing system. Table 4 shows a summary of the defect data collected for ACME-1. According to the data, ACME-1 had 93 defects reported against it during its first year of operation. If the perceived quality of the new system ACME-2 is to be as good or better than that of ACME-1, the new system should have no more than 93 defects reported against it during its first year of operation.

The data in table 4 not only shows the absolute number of defects found in ACME-1, but also defect density which is a measure of the quality of the development process used to create the system. Defect density is the number of defects found per unit of code during a certain time period. The defect density for ACME-1 after 1 year of use is 17 defects per KLOC, with 11% of these defects being found in production. If the same development process is used for ACME-2, similar results can be expected. Table 5 shows the estimated defect profile of ACME-2 assuming no changes in the development process. Notice that the defect density is predicted to remain the same at 17 defects per KLOC, but since ACME-2 is estimated to be 10% larger than ACME-1, the number of production errors is also expected to increase by 10%.

Table 5. Estimated quality profile of ACME-2 based on experience with ACME-1

It should be pointed out that the performance of the development process during the ACME-1 project is just one data point and may not be representative of the true quality potential of the development process. Performance on a single project can be corroborated by comparing it with data on the capability of the development process. Process capability is the range of results expected from a process. For the purposes of this example assume that the data in table 6 represents the quality capability of the development process used to create ACME-1. The data represents the aggregate performance of the development process experienced over several similar projects. Comparing project performance in table 4 with process capability in table 6 shows that the performance of the development process during the ACME-1 project is in line with the capability of the development process. This makes the quality profile of ACME-1 a reasonable baseline from which to estimate the quality performance of future projects.

Because the new system is larger than the previous system, the development process will need to be improved just to maintain the same level of quality from the customer's perspective. This illustrates why it is important to track both the quality of the development process as well as product quality from the customer's perspective. Most organizations strive for continuous process improvement. This means improving the quality of their development process as well as release-to-release product quality. If the system size grows between releases, it's important to focus on release-to-release quality. If the system size should decrease between releases, the development team will want to focus on the quality of the development process (i.e. defect density) in order to meet internal quality improvement goals.

The quality goal for the ACME-2 project is 93 or fewer defects reported during the first year of operation. Assuming no changes in the development process the new system is estimated to have 103 defects during the first year of operation. The next section on planning for project quality explores process improvement options for meeting the desired project quality goal.

Planning for Project Quality

Preparation and planning are the keys to success in any complex endeavor. The quality plan specifies the course of action for accomplishing the project quality goal.

Planning for project quality is much like planning for other project goals. For example, most projects start with a desired target completion date and then work backwards defining intermediate tasks or milestones throughout the project for tracking and controlling progress toward the goal of competing the project on time. Planning for project quality follows a similar course of action. Once the quality goal is set, intermediate quality sub goals are defined for tracking and controlling progress towards the desired project quality goal. Intermediate quality goals motivate action and provide milestones for measuring progress toward the project quality goal.

The quality plan may be a separate document or integrated with the main project plan or quality assurance plan.

The example in the previous section introduced the ACME-2 project and derived the following quality goal for it: 93 or fewer defects during the first year of production. The example that follows develops the main elements of a quality plan for accomplishing this project quality goal.

Example. Recall from the previous example that, based on the estimated size of the new project and expected defect density of the development process, process improvement initiatives need to be planned in order to meet the desired project quality goal.

Planning for process improvement starts with a thorough understanding of the existing process. For this example, performance during the ACME-1 project is being used as a baseline for understanding the existing development process. Table 7 shows the quality profile of the ACME-1 project. It shows the defect injection and removal rates for various stages from requirements through the first year of operation. Residual defects are latent defects that exist in the product at the start of a stage but have not yet been identified. The number of residual defects at the start of a stage can of course only be calculated after a certain duration of time has passed. When a defect is found it becomes a residual defect in all phases where it was present but not yet detected. Another metric in table 7, removal efficiency, is especially revealing. Removal efficiency is the percentage of existing defects that are removed by a removal activity. For example, a removal activity that is 33% efficient means that 2/3^rds of the defects that exist in the product at the start of the removal activity are not detected by the activity. The formula for calculating removal efficiency is:

Note that removal efficiency is a dynamic number--it goes down over time as latent defects are discovered. The number of defects detected during a stage is fixed at the conclusion of the stage but residual and injected defects go up as latent defects are detected and traced back to their point of origin. Consequently, when comparing removal efficiency (and other dynamic metrics) between projects, it's important that the measures be for the same time period. For example, if application A has been in production for a year it is unfair to compare its removal efficiency to application B that is just entering production. Application A has been exposed to more rigorous testing that has probably had the effect of lowering its removal efficiency.

Also notice that table 7 breaks out data separately for testing and review activities. Research shows that inspections are the most efficient method of defect removal [Jones 97]. It is nearly impossible to have high development efficiency without also having high inspection efficiency. Knowing the efficiencies of different defect removal activities provides insights as to where process improvement efforts should be focused.

Once the existing development process is well-understood you can begin to look for ways to improve it. Opportunities for process improvement can be found by comparing the performance of the existing development process with organization and industry norms. Areas of the existing process that compare unfavorably are good candidates for process improvement initiatives.

Table 8 shows data on the capability of the existing development process. This data represents the aggregate performance of the development process calculated over several projects. Comparing process performance (table 7) with process capability (table 8), reveals that inspection efficiency during the ACME-1 project is slightly below the organization average--62% vs. 65%. Looking closer at removal efficiency by stage in both table 7 and table 8, it appears that low removal efficiency during code inspections accounts for a significant portion of the difference (especially when you consider the number of defects injected during the coding phase). Still more analysis is needed in order to determine if poor efficiency is due to the number of inspections performed or the effectiveness of individual inspections. For the purposes of this example assume that the problem was simply a matter of inspections not being performed. Assume that the 54% removal efficiency during the coding phase of ACME-1 was achieved by inspecting only 20% of the code base. Based on this analysis one tactic for improving the performance of the development process during the ACME-2 project is to increase the number of code inspections performed.

Now that we have identified an opportunity for process improvement we can begin to construct the quality plan for the ACME-2 project. The quality goal for the ACME-2 project is 93 or fewer defects during the first year of production. The estimated size of ACME-2 is 55 KLOC. Assuming that defect density remains the same at 17 defects per KLOC, development efficiency will need to increase from 89% to 90% in order to accomplish the desired quality goal of 93 or fewer defects during the first year of production. Without any other changes in the development process, this can be accomplished if removal efficiency during the coding phase is increased from 50% to 56%. Some analysis is required to determine what portion or percentage of the ACME-2 code base needs to be inspected in order to reach this goal. Table 9 shows the estimated quality profile for the ACME-2 project assuming the development process is changed to include more code inspections. The data in table 9 also provides quality sub goals for each phase of development for tracking and controlling progress towards the project quality goal. For example, according to the data the development team should expect to find 58 defects during the requirements phase. If significantly more or fewer defects are found this is an indication that the team is not on track to accomplish the quality goal for the project.

Another option for improving the existing development process is to take action to prevent defects from occurring. This will lower the defect density and, assuming removal efficiency doesn't go down, improve overall product quality.

Data on defect distribution in tables 7 and 8 show that about half of all defects are introduced during the coding phase. This makes the coding phase an attractive target for defect prevention actions. Table 9 shows that without taking any action to reduce the number of defects injected during the ACME-2 project, 477 defects are expected to be injected during the coding phase. If this can be reduced by 20% the quality goal for the ACME-2 project can be accomplished through defect prevention alone. Table 10 shows the estimated quality profile for the ACME-2 project assuming a 20% reduction in the number of defects injected during the coding phase. Just making the estimate doesn't make it so though. The quality plan must also justify the estimate and include detailed plans for achieving it. The defect prevention process is described in more detail in the next section.

The examples in this section showed defect detection and defect prevention activities being applied separately. In practice, the most effective approach to quality improvement is implementing some combination of both.

Tracking and Controlling Project Quality

Tracking and control are universal management functions just as important for quality management as for schedule, budget and scope management. Tracking and control are used collectively to ensure that a project, once started, makes steady progress towards established goals. The general concept of tracking and control is discussed on chapter 4. In the context of quality management, tracking and controlling consist of:

Tracking and control at the project level is similar in concept to quality control at the product level. The difference is one of scope. Tracking and control at the product level drives tracking and control at the project level.

17.6 Defect Prevention

The benefits of finding errors early are well known. The cost of finding and fixing a defect goes up dramatically the longer it remains in the product. The ultimate in early defect detection though is defect prevention, that is preventing errors from ever occurring in the first place. Defect prevention is a systematic process for avoiding errors by identifying the underlying causes of errors that have occurred in the past and taking specific actions to eliminate these causes in order to prevent the errors from reoccurring again in the future.

Defect prevention and detect detection are complementary approaches to achieving better quality products. Defect detection looks for defects in work products and defect prevention looks for ways to improve the development process so that errors are avoided. Errors that can't be prevented must be detected and removed. (See figure 39.)

Defect prevention improves product quality and development productivity through incremental improvements to the underlying development process. This makes defect prevention primarily a method for improving the development process with the expected benefits being improved product quality and development productivity.

This section on defect prevention is divided into two parts. The first half explores the benefits of defect prevention in terms of process improvement, quality improvement and productivity improvement. The second half describes the activities of the defect prevention process in more detail.

Process Improvement

Defect prevention is an effective method of process improvement because process changes that come from defect prevention activities are systematic and driven by actual defect data. In most development environments there is no shortage of ideas for improving the development process. However, most ideas are based on personal perceptions and experiences. There is no assurance that these individual ideas for process improvement are more broadly applicable. Process improvement changes initiated by the defect prevention process are based on a statistical analysis of all problems so they are more likely to address the most urgent deficiencies in the development process.

Quality Improvement

In order for defect prevention to improve the quality of products it must prevent errors that otherwise would not have been found during the testing and review process*. Defect prevention can improve the quality of delivered software because testing tends to remove a certain percentage of errors. If defect prevention causes the product to enter the test phase with fewer inherent errors there will be fewer residual errors in the product after testing. Figure 40 illustrates this concept. The stroke pattern in both images depicts the fact that for any non-trivial system it's impractical and sometimes impossible to test all paths or operating conditions. Figure 40.b illustrates a reduction in defect density (fewer dots) that can be expected from using defect prevention. Because the characteristics of an error that make it preventable are independent of the characteristics that make it detectable, the defects prevented are just as likely to be among those detected as those not detected. The net result is that defect prevention reduces the overall number of residual errors that escape testing because some of the errors prevented will be errors that would not have been detected during inspection and testing.

_______________
*Notice the argument here is strictly for improving the quality of products. If defect prevention avoids an error that would have been found with testing, it doesn't improve the quality of the product but it still can improve overall productivity because it avoids the wasted effort of finding and fixing the defect and making any collateral changes.
---------------------------

The assertion that defect prevention tends to remove a certain percentage of all errors can be empirically tested by comparing the reduction in the number of errors found during development to the reduction in the number of errors found in production. If defect prevention does in fact remove a certain percentage of errors, then it should reduce both development and production errors by equal percentages.

The quantitative data in the literature seem to support the idea that defect prevention removes a certain percentage of all errors. Reporting on the results of a defect prevention program at IBM, Mays notes a 54% decrease in defects during development and preliminary estimates of a 60% decrease in field defects [Mays 1990]. Humphrey reports on a different project that experienced a 50% reduction in defects found during development and an even greater 78% reduction in field defects. Humphrey goes on to explain why defect prevention might eliminate a greater percentage of post-release defects than pre-release defects. He surmises that when fewer errors are injected during development, inspections (and possibly testing) are likely to be more efficient. The defect removal efficiency of inspections is likely to go up when there are fewer errors in the product being inspected. As an example, consider a small document with 70 or more errors in it. At the very least the large number of errors creates a distraction that makes it harder to do a thorough review. Reviewers are likely to give up in frustration before finding all of the errors that are easily detectable. If instead, the document starts with just 10-15 errors, reviewers have a better chance of finding most or all of the errors.

Productivity Improvement

Defect prevention improves productivity because it reduces the amount of effort it takes to deliver a certain output*. Preventing defects tends to increase productivity primarily because it takes less effort to avoid a mistake than it does to make a mistake, detect it, and then correct it. In other words, when the added burden of rework is taken into account, it usually takes less overall effort to do things right the first time.

___________
As mentioned in the previous section, defect prevention also improves product quality. Because productivity is both the amount and quality of output per unit of effort, defect prevention can also increase productivity even when effort isn't reduced. Increasing product quality for the same amount of effort also increases productivity. It's much harder to measure the effects of improved quality on productivity though.
--------------------

As section 17.2.5 explains, the cost of quality curve is U shaped for most applications. This implies that for most applications there is a point of diminishing returns beyond which additional investments in defect prevention aren't paid back with lower failure costs. This implies that it is possible to overspend on quality improvement. One of the characteristics of the DP process that mitigates this concern is that defect prevention is a controlled process. It's not an all or nothing approach to defect prevention. The actions with the highest predicted return on investment can be implemented first. Additional corrective actions can be implemented as justified based on results achieved.

Another way that preventing defects improves productivity is that, when combined with testing, defect prevention makes it possible to achieve high levels of product quality more efficiently than with testing alone. Recall from section 17.2.5 that with testing alone, as the percentage of defects detected approaches 100%, it becomes exponentially more expensive to find additional errors. (See figure 42.)

The last few defects in a product are expensive to remove because they are difficult to detect. They might be difficult to detect but they aren't necessarily difficult to prevent. For example, consider an application that is close to its point of diminishing returns when testing reaches 90% efficiency. Finding the last 10% of defects with testing alone is likely to be very expensive. However, another option for improving the quality of the product is defect prevention. Preventing just 40% of the errors that would have been injected into the product is likely to remove 40% of the remaining residual errors. It would be much more expensive to find these additional errors with testing alone.

For most applications there is a point of diminishing returns for both defect detection and defect prevention. The best value for your quality dollar is when defect prevention is combined with defect detection and both processes are allowed to work in their "linear zone". For a given amount of effort the overall level of quality achieved is greater than what either method could accomplish with the same amount of effort working alone. Figure 43 shows a conceptual example. The combined effort for defect prevention and defect detection is less than the effort required to get the same level of removal efficiency with either defect prevention or defect detection alone.

The Defect Prevention Process

The defect prevention process is a systematic process for improving the quality of delivered software through incremental improvements to the underlying development process [Jones]. Figure 44 illustrates the main steps in the defect prevention process which are:

Defect prevention is a planned and systematic form of continuous process improvement. Causal analysis of defects drives changes in the underlying development process. The changes are implemented and the cycle starts again on the changed process.

Defect prevention is a supporting process. A prerequisite to performing defect prevention is the existence of a well-defined software development process. Most of the ideas for preventing defects are process improvement ideas. A well-defined process provides the structure and framework needed for implementing and institutionalizing process changes. Figure 45 shows how the defect prevention process integrates with the software development process. The solid lines represent elements of a standard software development process and the dashed lines represent the additional elements of defect prevention.

Figure 45. Defect prevention relies on the framework of an existing development process

Planning for Defect Prevention

Defect prevention is a simple process but it still requires some up-front planning. Like any other planning exercise you must decide who does what when.

One of the first decisions in planning for defect prevention is deciding when and how often to hold stage kickoff, causal analysis and action team meetings. A kickoff meeting is normally held at the beginning of every task or stage of development. The timing of the causal analysis and action team meetings depends on the needs of the project and the life cycle model being followed. One option is to hold causal analysis meetings after every stage of development and action team meetings at regular intervals [Mays 1990 IBM Systems Journal]. (See figure 46.) This arrangement is probably best suited to projects following the waterfall life cycle model where a substantial amount of work is performed during each phase. When developing software in short frequent iterations, distinct stages may be difficult to discern and the amount of work performed during a stage of development may be insufficient to justify a causal analysis meeting.

A more appropriate timing for defect prevention activities during a project following an iterative life cycle model is to schedule causal analysis and action team meetings after each iteration. (See figure 47.) The amount of work performed during an iteration is usually sufficient to justify a causal analysis meeting. In general, causal analysis should be delayed until there are enough significant defects to make the activity worthwhile--typically 15-20 errors. On the other hand, the sooner causal analysis is performed and corrective actions are implemented, the greater the quality and productivity will be because corrective actions will have had longer to affect the current project.

A third option for scheduling defect prevention activities is by percentage of work complete. At Infosys the general guideline is to perform causal analysis after 20% of the system has been coded, reviewed and unit tested, and again after 50% of the system has been coded, reviewed and unit tested [Jalote, 2002]. (See figure 48.) Twenty percent into a project is late enough for substantial problems to emerge, but early enough that corrective actions will have a chance to contribute to the quality and productivity of the current project. Performing another round of defect prevention at the 50% mark provides the chance to verify that corrective actions are having their intended effect as well as plan additional corrective actions for the second half of the project.

Another convenient time for performing causal analysis is a few months after the product has been released. Errors that make it all the way to the customer are the most important to prevent because they are the most expensive to fix and the most damaging to the reputation of the product. When an error reaches the customer it indicates a problem in the testing process as well as the development process.

The two main groups of individuals that share responsibility for the defect prevention process are the causal analysis team and the action team. The causal analysis team is comprised of the technical staff responsible for development. Programmers shouldn't test their own code but they should perform causal analysis on their own errors. The person who makes an error probably understands the error and contributing factors better than anyone else. As May observes, "direct causal analysis by the developer making the error results in a more accurate determination of the cause of the defect and more relevant preventive actions." [May] The action team is responsible for ensuring that suggested actions are implemented. The action team is staffed with individuals that have the authority and resources to effect changes in the development process.

Both the causal analysis and action teams should be trained in defect prevention. In addition, the causal analysis team should be trained in statistical techniques such as Pareto charts and cause-effect diagrams. (Statistical tools for quality control are discussed in chapter 20.)

Stage Kickoff Meeting

A stage kickoff meeting is held at the beginning of every task or stage of development. The individuals performing the work meet to review the upcoming activities including feedback from earlier rounds of defect prevention. The meeting is used to increase awareness of quality issues right before the work begins. Many errors are caused by oversight or lack of awareness. When developers are reminded of the most common errors made during a task right before they perform the task, they are less likely to repeat the errors.

The stage kickoff meeting is the primary mechanism for introducing feedback from causal analysis into the development process. Recommended changes from causal analysis are typically first introduced during the stage kickoff meeting of a single project. Those that prove helpful locally are later migrated to the standard development process and made available to all projects.

Collecting Data on Defects

To aid later causal analysis, data on defects are gathered as defects are detected and corrected. It's important to collect as much information as possible up front while the details of the problems are fresh in the minds of those dealing with them.

Most organization getting started with defect prevention will already have a system in place for tracking and reporting defects. In most cases this system can be made to work for defect prevention with the addition of a few extra fields. The fields important for defect prevention, that might not be included in a general defect tracking database are:

More information on defect data tracking and reporting can be found in chapter 16.

Causal Analysis

Causal analysis meetings are held periodically during a project, usually at the end of a development phase after the work products of that phase have been verified and all defects found in the phase have been corrected and documented. A causal analysis meeting is similar to a formal inspection and has some of the same sensitivities found at formal inspections. The participants are those directly responsible for the work just completed. These are the individuals responsible for the errors which are being analyzed. As with inspections, the focus should be on the errors and how to improve the process rather than the faults of the individual. To encourage open and honest discussion of defects, managers generally don't attend.

During a causal analysis meeting participants analyze recent defects, determine their root causes, and suggest ways of avoiding them in the future.

The focus of causal analysis is on error prevention, but for errors detected in a stage later than the stage in which they were introduced, some consideration may also be given to suggestions for finding the error earlier, closer to the phase it was introduced.

The product or output of causal analysis is a prioritized list of specific suggestions or actions for preventing the defects that were analyzed. These preventive actions are forwarded to the action team for implementation.

To clarify the concepts, here is simple example that illustrates the key activities of causal analysis.

Preliminaries. For this example, assume that 22 errors were found during the implementation phase of one iteration. The errors have been corrected and data on each error has been recorded in a database.

Review Defects. The first order of business during causal analysis is to decide which errors to review in more detail. It may not be feasible or worthwhile to perform detailed causal analysis on all errors. Priority should be given to errors with high failure costs and those more likely to occur again in the future. A Pareto chart showing the distribution of errors among a set of common causes helps to identify the categories with the most errors. Figure 49 shows what a Pareto chart might look like for the 22 hypothetical errors assumed for this example. The chart suggests that errors caused by oversight and insufficient technical education are more prevalent and therefore deserve the most attention.

Root Cause Analysis. The best preventive actions come from thoughtful consideration of specific causes. Just knowing the common cause category of a defect is not enough to suggest useful preventive actions. Table 11. shows a portion of the defect data record for a specific error from the oversight category. The specific error is exceeding the number of database connections that can be open at one time. During root cause analysis additional information is usually drawn out. Did the programmer really forget to close database connections as specified in the defect record, or was the programmer unaware that database connects must be closed. If it was the latter this suggests a different common cause category--insufficient technical education--which would dictate a different response. For the rest of this example, assume the root cause was simply forgetting to close a connection after use.

A useful tool for analyzing defects with multiple apparent causes is a cause-effect diagram (also called fishbone or Ishikawa diagram). It is used to identify and organize the possible causes of a defect. (Cause-effect diagrams are described in more detail in chapter 20.)

Preventive Actions. Once the root causes of defects are understood, attention can shift to looking for ways of preventing them from occurring again in the future. There are a couple of ways to avoid the problem of programmers forgetting to close database connections. First, a reminder to close database connections can be added to the common errors list for the implementation phase. Updating the common errors list for a particular development phase is one of the most common changes made to prevent defects. Second, and potentially more costly, the interface onto the database can be redesigned--made more abstract--so that programmers don't have to be concerned with opening and closing database connections.

Output of the causal analysis phase are specific suggestions for preventing defects. When there is more than one response to a particular defect, the suggested actions should be prioritized. These recommended actions are usually documented in the defect record for the error they are meant to address. Table 12 shows the updates to the suggested actions field of the record for the defect being analyzed in this example.

The Action Team

The causal analysis team generates ideas for preventing defects and the action team implements them. The action team meets regularly to review, prioritize and implement suggested actions from causal analysis. The action team is staffed with individuals with the authority to authorize work in the areas most likely affected by action items such as the development process, tools and education.

1. Review and prioritize action items. Typically there are more ideas for preventing defects than there is time and resources available to implement them. The action team considers each suggestion and selects the most promising ones for implementation. Factors considered in the decision process are the consequences of not addressing the cause, the cost of implementing the suggestion, and the expected impact or effectiveness of the proposed solution.

2. Assign responsibility for implementing each action item. Typically action items are divided among the members of the action team for implementation. If there is someone outside the team better qualified to implement an action item, the item is assigned to this individual and a member from the action team takes responsibility for tracking the item to completion.

3. Track and report status of action items. During each meeting the action team not only considers new action items but also monitors the progress of open items. It is the team's responsibility to ensure that all items selected for implementation are completed. There are no dependencies on defect prevention action items which makes it easy to put off their completion. The interest the action team shows in open items motivates progress.

4. Provide feedback. Developers performing causal analysis are encouraged and energized to come up with new ideas for defect prevention when they see their suggestions being taken seriously. The final step in the implementation of an action item should be to notify the individual submitter and organization of its completion. When a suggestion is not selected for implementation the submitter should be notified of the rational for the decision.

Category	Definition	Example
Prevention	Costs associated with preventing defects.	Training, early reviews, quality planning, tools, process improvement initiatives.
Appraisal	Costs associated with analyzing and testing the product to ensure it conforms to specifications.	Inspections, testing, audits, quality control.
Internal Failure	Costs associated with fixing defects found prior to release.	Repair, retesting, updating documentation.
External Failure	Costs associated with fixing defects found after release.	Technical support, defect reporting and tracking, field updates, loss of future sales.

Product or Process	Scheduled Review (R) or Audit (A)	Acceptance Criteria
Requirements Document	Upon completion (R)	Includes business objective or purpose for developing the system Includes prioritized list of planned features Identifies desired non-functional quality requirements and their priority Includes any constraints on implementation (i.e. "the database must be open source") Each requirement is uniquely identified to facilitate traceability Each requirement is specified in a way that can be verified objectively
Architecture Document	Upon completion (R)	The architecture identifies system components, subcomponents, their relationships and interfaces.
Verification and Validation Plans	Before system test (R)	<See external document standard xyz>
Product documentation	Before acceptance test (R)	<See external document standard xyz>
Change Control Procedure	Random assessment (A)	<See external document standard xyz>
Code Inspections	SQA will verify that all inspections are performed and audit 10% of code inspections (selected at random) for adherence to standards.	<See external document standard xyz>

Error:	Exceeded number of database connections that can be open at once
Cause Category:	Oversight
Cause:	Programmer forgot to close database connections after use
Suggested Actions:

Error:	Exceeded number of database connections that can be open at once
Cause Category:	Oversight
Cause:	Programmer forgot to close database connections after use
Suggested Actions:	(1) (Priority: high) Update common errors list for implementation phase to include reminder to explicitly close database connections after use. (2) (Priority: low because of cost) Redesign the data access layer to be more abstract so that clients don't have to be concerned with connections.

17. Software Quality Management

Objectives