«Software Fault Reporting Processes in Business-Critical Systems Jon Arvid Børretzen Doctoral Thesis Submitted for the partial fulfilment of the ...»
1. Introduction An important goal for most software development organizations is improving software quality, typically reliability. There are several ways to go about this, but it is not a trivial task, and different stakeholders have different views on what software quality is. In addition, the application domain of the actual software will influence what quality attributes we consider to be most relevant. For many organizations, their routinely collected data is an untapped source for process analysis and possible process improvement. Indeed, leaving collected data largely unused can demotivate the developers and reduce data quality. Our conjecture, and supported by previous research, is that fault analysis can be an effective approach to software process improvement (Grady, 1992).
The Business-Critical Software (BUCS) project (Børretzen et al., 2004) works to develop a set of techniques to improve support for analysis, development, operation, and maintenance of business-critical systems. Business-critical systems are systems that we expect will run correctly and safely, even if the consequences are mainly of a “mild” economic nature. In these systems, the software is critical, and the main target for developers will be to make systems that operate correctly and without serious consequences in case of failures (erroneous behaviour vs. specifications). One important issue when developing these kinds of systems is to remove possible causes for failure, which may lead to wrong operation of the system.
In two previous studies (Børretzen and Conradi, 2006; Børretzen and DyreHansen, 2007), we have investigated fault reports in nine business-critical industrial software projects. These studies were quantitative studies based on fault report analysis, where we mainly studied fault types and severity of faults. Building on the results of the most recent study, we want to gain a better understanding of the results this study gave us. A considerable share of the reported faults were of a type that are associated with early software development phases, indicating flaws in the quality control in these phases. This paper presents the results from interviews done with representatives from some of the projects studied in (Børretzen and Dyre-Hansen, 2007), as well as two workshops with further representatives on fault reporting and fault classification.
The rest of this paper is organized as follows. Section 2 gives our motivation and related work. Section 3 describes the research, research questions and procedure.
Section 4 presents the results from the interviews and workshop feedback, and Section 5 presents a discussion of the work and the results. The conclusion and further work is presented in Section 6.
2. Background and Motivation The motivation for the work described in this paper is to expand on the knowledge gained from a previous quantitative study on fault reports from five projects in a software development organization. By performing an additional qualitative investigation with representatives from some of the same projects, we sought to better understand the reasons for the results we saw in the quantitative study, as well as receiving input from practitioners on how a more thorough fault management can be part of a software process improvement initiative. “Value-based” software engineering say that developing models and measures that focus on value received, makes us able to perform trade-off decisions which helps us concentrate on the right issues in process improvement (Biffl et al., 2006). There are several motivations for preventing and correcting faults as early in the process as possible, but the main issue will usually be of an economical nature.
When considering quality improvement through use of fault analysis, there are many related topics to consider. Several issues about fault reporting are discussed by Mohagheghi et al. in (Mohagheghi et al., 2006). General terminology in fault reporting is one problem, validity of use of fault reports as a means for evaluating software quality is another. Important issues to consider are how to describe the fault by “what” – the cause of the fault, “where” – the location of the fault, and “when” – the detection phase of the fault. One of the conclusions in (Mohagheghi et al., 2006) is that “There should be a trade-off between the cost of repairing a fault and its presumed customer value. The number of faults and their severity for users may also be used as a quality indicator for purchased or reused software.” By using fault report analysis, one can get a step closer to understand the cost of repairing faults of various categories.
Fault reports is the term describing the information that is recorded about faults that are discovered during development, testing and field use of a software system.
These reports can contain a range of different information, examples some common fault report attributes are fault description, fault severity, fault type and fault location.
Their most obvious function is to be the link between fault discovery and fault correction, but they are also valuable when performing fault analysis of a system.
One way to improve the quality of developed software is to reduce the number and severity of faults introduced into the system during development. Faults are potential flaws in a software system, that later may be activated to produce an error. An error is the execution of a "passive fault", leading to a failure. A failure results in observable and erroneous external behaviour, i.e. inconsistent system state. Faults that have been introduced into the system during implementation can be discovered either by inspection before the system is run, by testing during development or when the application is run on site. The discovered faults are then reported in a fault reporting system, and will normally be fixed later. Faults are also commonly known as defects or bugs, and also under the more extensive concept anomalies.
Orthogonal Defect Classification – ODC – is a way of studying defects in software systems, and is mainly suited to design and coding defects. (Chillarege et al., 1992; El Emam and Wieczorek, 1998) are some papers on ODC and using ODC in empirical studies. ODC goes along with a scheme where the semantics of each software fault can be captured quickly and easily.
Avizienis et al. (2004) state that the fault prevention and fault tolerance aim to provide the ability to deliver a service that can be trusted, while fault removal and fault forecasting aim to reach confidence in that ability by justifying that the functional and the dependability and security specifications are adequate and that the system is likely to meet them. Hence, by working towards techniques that can prevent faults and reduce the number and severity of faults in a system, the quality of the system can be improved in the area of dependability.
There are different perspectives and motivations for working to prevent and correct faults in software, but the most important motivation is that of cost. Correcting faults is costly and in many instances is nothing but redoing the work that should have been done correctly in the first place.
2.2 Previous work
Previously, we had conducted a fault report analysis of five projects in a software development organization. We studied the projects with regard to reported faults, through analysing fault reports from the development of the applications. Looking at descriptions of the individual faults, as well as other data reported about the faults, we classified the faults into different fault types. By grouping faults into fault types, we tried to find indications on where reported faults originated in the development process.
The analysis was based on ODC, and we categorized faults into fault types based on that technique. Table 1 shows the fault types that were used in that study. The fault analysis of the five projects and the results are described further in (Børretzen and Dyre-Hansen, 2007).
The results of the fault analysis were presented to the interviewees, and the basis for the interviews was therefore our analysis of projects in their organization as well as their own experiences on fault reporting in the organization. Quantitative results from the fault report study indicated that a large share of faults reported originated from early phases of development, and we wanted to explore this area further with qualitative feedback from people involved in the studied projects. An earlier study performed by us in four other companies had shown the same tendency of high numbers of early phase faults (Børretzen and Conradi, 2006). Another example of results in a related study is the work done by Vinter and Lauesen (2000). This paper used a different fault type taxonomy, but reports that in their studied project close to a quarter of the faults found were of the similar type “Requirements and Features”. In addition to building on the quantitative results, we wanted to learn about practitioners’ opinions and knowledge about fault reporting and management.
Following on to our two previous qualitative studies on fault reports; we wanted to continue studying the projects explored in (Børretzen and Dyre-Hansen, 2007) using qualitative methods to further explain our quantitative results. Feedback from the developers’ organization would aid us understand the source of these results, and help us suggest concrete measures for process improvement in the organization.
2.3 Study Context
The development organization we have studied is a part of a large company developing and maintaining business-critical applications in the financial sector. It has been involved in the user-driven EVISOFT research and development project concerning industrial process improvement since 2006 (EVISOFT, 2006). The organization under study is distributed over several locations, and it employs hundreds of software developers. Our study has mainly involved representatives from two locations in Norway.
This organization had been working on process improvement concerning their fault management routines for some time, and some changes in the way faults were reported and handled were on the verge of being introduced when our first quantitative study started. They had an existing fault categorization scheme, although this was mostly focused on issues concerning the test environment, and did not capture the semantics of specification, design and coding faults in a detailed manner. This existing scheme is shown in Table 2. The feedback they received from our study (Børretzen and Dyre-Hansen, 2007) prompted some further proposals for change, which were to be implemented in a pilot project. The organization used a common process for reporting and managing faults, with minor customizations for individual projects. Test leaders were responsible for dealing with fault reports and reporting the faults back to developers for fixing.
2.4 The Grounded Theory Approach
Grounded Theory is a systematic research methodology originating from the social sciences developed by the sociologists Barney Glaser and Anselm Strauss emphasizing generation of theory from data. When the principles of grounded theory are followed, a researcher using this approach will formulate a theory about the phenomena they are studying that can be evaluated. The grounded theory approach is a “qualitative research method that uses a systematic set of procedures to develop an inductively derived grounded theory about a phenomenon” (Strauss and Corbin, 1998). The methodology is designed to help researchers produce “conceptually dense” theories that consist of relationships among concepts representing “patterns of action and interaction between and among various types of social units” (Strauss and Corbin, 1998). Potential sources of data for developing grounded theory include interviews, field observations, documents, and videotapes.
At the heart of the grounded theory methodology, are three coding procedures of open coding, axial coding, and selective coding (Strauss and Corbin, 1998). These codes are generated and validated using the constant comparison method, and coding, at each stage, terminates when theoretical saturation is achieved with no further codes or relationships among codes emerging from the data.
Open coding involves immersion in the data and generation of concepts with dimensionalized properties using constant comparison. This is done by “breaking down, examining, comparing, conceptualizing, and categorizing data”, often in terms of properties and dimensions (Strauss and Corbin, 1998). The examination of data in order to fracture it and generate codes could proceed “line by line”, by sentence or paragraph, or by a holistic analysis of an entire document. The open coding process, while procedurally guided and promoting a realist ontology, requires researchers to “include the perspectives and voices of the people” whom they study. Data, for open coding, is selected using a form of theoretical sampling known as “open sampling.” Open sampling involves identifying situations/ portions of the transcripts that lead to greater understanding of categories and their properties.
Axial coding refers to the analytic activity for "making connections between a category and its sub-categories" developed during open coding, that is, reassembling fractured data by utilizing "a coding paradigm involving conditions, context, action/interactional strategies and consequences" (Strauss and Corbin, 1998).
Selective coding involves the identification of the “core category” (central phenomenon that needs to be theorized about) and linking the different categories to the core category using the paradigm model (consisting of conditions, context, strategies, and consequences). Often, this integration takes the shape of a process model with the linking of action/interactional sequences.
Although Grounded Theory is not the most common research method in computer sciences, there are several studies showing that this way of building theory and drawing conclusions from qualitative data is highly applicable in this field as well as in the social studies (Bryant, 2002; Hansen and Kautz, 2005; Sarker et al., 2000).