«Software Fault Reporting Processes in Business-Critical Systems Jon Arvid Børretzen Doctoral Thesis Submitted for the partial fulfilment of the ...»
RQ.S6.b: How does the distribution of fault types found in the fault analysis compare to the one found in the PHA?
RQ.S6.c: Does the PHA technique identify potential hazards that also actually appear as faults in the software?
Validity comment. Being a study of just one system, external validity would be weak.
Another concern was construct validity, as we would be making a comparison of hazards and faults, which are two different concepts.
3.3.7 Study 7: Fault management and reporting This study does not have explicit research questions, but is a compilation of lessons learned over the course of studying fault management and fault reporting in several different organizations. This was based on our experience from collecting and analysing fault reports as well as from literature studies and feedback from the organizations involved in our studies.
Validity comment. The main validity concerns is that our experience comes from a limited number of organizations, and our main means of validating the lessons learned is comparison with literature review.
3.4 Overview of the studies In Table 3-2, an overview of the studies is given, together with short description of the type of study.
This chapter summarizes the research results for each of the studies. The results are reported in more detail in the papers in Appendix A, but this chapter also includes some results of work that so far have not been reported in papers.
In order to learn more about the way business-critical software projects are being executed, we sought out a few companies and conducted short interviews with representatives from these companies. Eight interviews were conducted in eight different companies. The companies were picked partly for being representative among Norwegian IT industry, partly because of our suspicions of their relevance to the business-critical topic, and also partly because of convenience with respect to geographic location and general availability. The companies were represented by persons in different positions in the company structure, from directors to project managers and developers. The interviews lasted 30-45 minutes, and each interview was performed by one researcher taking notes. The questions, or topics, had been worked out beforehand. They were partly taken from literature studies, and dealt with areas we felt were important to solicit answers to this early in the project. After the interviews, the researchers compiled and wrote up an internal BUCS technical report, for use as future reference for the BUCS project members [Stålhane03].
The main results to be extracted form the interview sessions were the following:
• The industry defines the term ‘business-critical’ as something that is related to their economy, their reputation and their position in the market.
• RUP or some variant of it, is common among companies who actually employ some specified process.
• Business-critical software development is a very common activity among software development companies.
• A typical problem in development of business-critical software is communication, both within the company and towards the customer.
• The companies generally do not consider the technical risk aspects of a project in detail, perhaps mainly due to a lack of an instrument for this.
Contributions of Study 1:
The purpose of these interviews was to elicit knowledge about how the situation in Norwegian IT industry was with respect to development of business-critical software.
As this was the first investigation of the BUCS project, the goal was in the main part to get an overview and a general impression of the situation. Also, it was intended as a basis for further work, both for further empirical studies, and as a tool to help us focus future research.
This study was the first step towards the main contribution C1: “Describing how to utilize safety criticality techniques to improve the development process for businesscritical software.”
4.2 Study 2: Combining safety methods in the BUCS project (Paper P1) Study 2 was carried out by doing a literature review of software engineering practices and safety criticality analysis methods. We wanted to propose a way to combine these into a more unified tool set.
P1. Jon Arvid Børretzen, Tor Stålhane, Torgrim Lauritsen, and Per Trygve Myhrer:
"Safety activities during early software project phases"
to P1: This paper describes how methods taken from safety-critical practises can be used in development of business-critical software. The emphasis is on the early phases of product development, and on use together with the Rational Unified Process.
One important part of the early project phases is to define safety requirements for the system. This means that in addition to satisfying the need for functional system requirements, non-functional requirements about system safety must also be included.
By using information that already is required or produced in the first phases of RUP together with some suitable “safety methods”, we are able to produce a complete set of safety requirements for a business-critical system before the system design process is started.
In P1, we showed how the Preliminary Hazard Analysis, Hazard and Operability Analysis and Safety Case methods can be used together in the RUP inception phase, to help produce a safety requirements specification, this is illustrated in Figure 4-1. The shown example is simple, but demonstrates how the combination of these methods will work in this context. By building on information made available in an iterative development process like RUP, we can use the presented methods to improve the process for producing a safety requirements specification. The paper also emphasizes that early development phases are prime candidates for efficient safety analysis work.
Contributions of Study 2:
The contribution of this study was showing possible integration of a common software development method with techniques taken from development of safety-critical systems. This study thus supports the main contribution C1.
4.3 Study 3: Fault report analysis (Papers P2, P3, P5) The work and results of Study 3 has been presented in three papers P2 (main), P3 and P5. The basis was a quantitative study of fault reports in four companies.
P2. Jon Arvid Børretzen and Reidar Conradi: "A study of Fault Reports in Commercial Projects" Abstract to P2: Faults introduced into systems during development are costly to fix, and especially so for business-critical systems. These systems are developed using common development practices, but have high requirements for dependability. This paper reports on an ongoing investigation of fault reports from Norwegian IT companies, where the aim is to seek a better understanding on faults that have been found during development and how this may affect the quality of the system. Our objective in this paper is to investigate the fault profiles of four business-critical commercial projects to explore if there are differences in the way faults appear in different systems. We have conducted an empirical study by collecting fault reports from several industrial projects, comparing findings from projects where components and reuse have been core strategies with more traditional development projects.
Findings show that some specific fault types are generally dominant across reports from all projects, and that some fault types are rated as more severe than others.
P3. Parastoo Mohagheghi, Reidar Conradi, and Jon A. Børretzen: "Revisiting the Problem of Using Problem Reports for Quality Assessment” Abstract to P3: In this paper, we describe our experience with using problem reports from industry for quality assessment. The non-uniform terminology used in problem reports and validity concerns have been subject of earlier research but are far from settled. To distinguish between terms such as defects or errors, we propose to answer three questions on the scope of a study related to what (problem appearance or its cause), where (problems related to software; executable or not; or system), and when (problems recorded in all development life cycles or some of them). Challenges in defining research questions and metrics, collecting and analyzing data, generalizing the results and reporting them are discussed. Ambiguity in defining problem report fields and missing, inconsistent or wrong data threatens the value of collected evidence. Some of these concerns could be settled by answering some basic questions related to the problem reporting fields and improving data collection routines and tools.
P5. Jingyue Li, Anita Gupta, Jon Arvid Børretzen, and Reidar Conradi: "The Empirical Studies on Quality Benefits of Reusing Software Components" Abstract to P5: The benefits of reusing software components have been studied for many years. Several previous studies have concluded that reused components have fewer defects in general than non-reusable components. However, few of these studies have gone a further step, i.e., investigating which type of defects has been reduced because of reuse. Thus, it is suspected that making a software component reusable will automatically improve its quality. This paper presents an on-going industrial empirical study on the quality benefits of reuse. We are going to compare the defects types, which are classified by ODC (Orthogonal Defect Classification), of the reusable component vs. the non-reusable components in several large and medium software systems. The intention is to figure out which defects have been reduced because of reuse and the reasons of the reduction.
Paper P2 was the main paper for this study, and it presented some results of an investigation on fault reports in industrial projects. The main conclusions of this paper
• When looking at all faults in all projects, “functional logic” faults were the dominant fault type. For high severity faults, “functional logic” and “functional state” faults were dominant. This is shown in Tables 4-1 and 4-2.
Also, we saw that some fault types were rated more severe than others, for instance “Memory fault”. However, the fault type “GUI fault” was rated as less severe for the two projects employing systematic software reuse in development, this is illustrated in Figure 4-2.
0,0 % 20,0 % 40,0 % 60,0 % 80,0 % 100,0 % Figure 4-2 Percentage of high severity faults in some fault categories The main conclusions of P3 were the following: We identified three questions that define a fault: what- whether the term applies to manifestation of a problem or its cause, where- whether problems are related to software or the environment supporting it as well, and whether the problems are related to executable software or all types of artifacts, and when- whether the problem reporting system records problems detected in all or some life cycle phases. We also described how data from problem reports may be used to evaluate quality from different quality views, as shown in Figure 4-3, and how measures from problem or defect data is one the few measures used in all quality views.
Finally, we discussed how data from problem reports should be collected and analyzed and what is the validity concerns using such reports for evaluating quality.
Figure 4-3 Quality views associated to defect data, and their relations In P5, we presented the research design of an on-going empirical study to investigate the benefit or cost of software reuse on software quality. By analyzing the defect reports of several software systems, which include both reusable and non-reusable components, we planned to deepen the understanding on why reuse improves the quality of software.
This paper also described the future work; to collect data from project with different contexts, such as application domains, technologies, and development processes, in order to find the common good practices and lessons learned of software reuse.
Contributions of Study 3:
In paper P2, the contributions were the description of the most typical and severe faults found by analyzing fault reports, which was related to main contribution C3: “Improved model of fault origins and types for business-critical software”.
In paper P3, we described our experience with using fault reports for quality assessment, and in answering three questions about what, where and when faults are and how they are discovered, we showed that improvements in how faults are described and worked with are needed. This was related to the main contribution C2: “Identification of typical shortcomings in fault reporting”.
The contribution in P5 was using fault categorization to compare defect types of reused and non-reused components, which was related to main contribution C3.