«Software Fault Reporting Processes in Business-Critical Systems Jon Arvid Børretzen Doctoral Thesis Submitted for the partial fulfilment of the ...»
5.1.1 Contributions related to BUCS goals The relation between our contributions and the BUCS goals as defined in Section 1.1
are now considered:
BG1 To obtain a better understanding of the problems encountered by Norwegian industry during development, operation and maintenance of business-critical software.
Regarding BG1, we have found that early development phases like specification and design are a source of a high number of faults in software. Lack of communication and adequate tools and processes for describing development difficulties in these phases seem to be the main problem. It is thought that the work in this thesis advances the state-of-the-art of software engineering of business-critical software as defined by our contributions C1-C3. Better understanding of problems encountered by Norwegian industry is achieved, as is reflected in contributions C1 and C3.
BG2 Study the effects of introducing safety-critical methods and techniques into the development of business-critical software, to reduce the number of system failures (increased reliability).
Our studies on fault reports suggest concrete measures to reduce the largest group of faults found in studies of business-critical software in our contribution C3. In addition, we have found that lightweight hazard analysis such as the PHA method is useful in eliciting hazards that could be avoided to reduce the number of faults originating in early development phases, from our contribution C1.
BG3 Provide adapted and annotated methods and processes for business-critical software.
Although the goal BG3 has not been an explicit focus of this thesis, we describe how fault report analysis and certain hazard analysis methods can be used to improve the development process, related to C1 BG4 Package and disseminate the effective methods into Norwegian software industry.
Most results are published, or planned on being published, and presented at interational and national conferences and workshops. During this thesis work, several Masters students have directly or indirectly been involved in activities, project work or Masters theses concerning business-critical systems and the BUCS project. Furthermore, the knowledge gained from our studies in commercial organizations have been disseminated back to them through reports and internal workshops. This relates to all contributions C1-C3.
5.2 Contribution of this thesis vs. literature In this section we present how our results and contributions compare with state-of-theart.
Looking at the wide perspective, our research on business-critical systems and software has shown not to be directly comparable with much of the literature on software engineering. This is something we were aware of from the start of the BUCS project.
The introduction of safety related methods into “regular” software engineering is not common, many of the methods are still regarded as resource-hungry and rigid methods, and this is difficult to combine with the emergence of agile and other lightweight methods [Beck99]. On the other hand, there are many types of systems that demand a more rigorous development process to ensure reliability and related qualities (e.g.
financial systems), and for these types of systems we have contributed both on a process level and with techniques that could be applicable.
In our work, we have proposed a novel method for doing fault inspections of specification and design documents [P6]. This adds to the existing literature on inspections, for instance that of Basili et. al concerning perspective-based reading [Basili00, Shull00].
The results of our quantitative studies on fault reports [P2][P4] show that in many systems, faults originating from specification and design phases constitute a major part of the total number of faults being found in testing. This is in line with the findings of Vinter et al. [Vinter00], but in contrast to findings by Eldh et al., where a common type of early process fault (function) was not very frequent [Eldh07]. Our fault report study of a small frame simple system in [P7] did, however, show that systems have different fault profiles.This may be as a result of both the type of system and the development method used when designing and implementing the system.
Further, we have discussed the need for improving fault reporting as a support tool for process improvement. Several sources present fault management processes as useful for such improvement in a software organization, among others [Grady92, Chillarege92].
We support this stance and suggest how to better utilize the available fault information [P3][P4][P6][P8].
5.3 Revisiting the Thesis Research Questions, RQ1-RQ4
In answering our four research questions, we have the following:
RQ1. What is the role of fault reporting in existing industrial software development?
a) Fault reporting seems to generally be underused and undervalued. Our experience is that the recorded data is often not of high quality, which not only makes any analysis hard, but also diminishes the usefulness of the fault reports for fixing faults.
b) All software developing organizations have a fault reporting system in operation, but its use differs substantially. The most basic fault reporting system is only used as a means to document faults that have been found and that are to be corrected, but more advanced use of the available data can easily be arranged.
c) Even where fault report data is thoroughly recorded and stored, it is not systematically used as a tool for software process improvement. A lot of detailed information is stored in the fault management systems of software organizations, but is never used beyond the simplest applications.
RQ2. How can we improve on existing fault reporting processes?
a) Developers should be more conscious about the potential for improvement by analyzing fault reports. Only through feedback on quality/fault data can an organization “learn from their mistakes”.
b) We need more formalized reporting schemes, and clearly defined procedures for reporting faults.
c) Introduce updated fault reporting schemes (fault type, severity, priority, effort, location etc) for the organization’s needs, so that the correct and complete information is reported. There is a need for a process looking at the requirements and possibilities in each organization.
RQ3. What are the most common and severe fault types, and how can we reduce them in number and severity?
a) P2 and P4 show that the most common fault type is the “function” fault type, i.e.
faults related to faults in the specification and design phases of development.
“GUI” faults are also numerous, and can in many cases also be related to specification and design phases.
b) Our studies on safety-critical analysis techniques have shown that the PHA technique is a useful tool for eliciting hazards that can be related to the fault types that are most common [P1] [P6].
RQ4. How can we use safety analysis techniques together with failure report analysis to improve the development process?
a) In P6, we have found that the PHA technique is useful for eliciting hazards that can be related to faults that are commonly introduced in early development phases.
5.4 Evaluation of validity For the validity of the work in this thesis, there are some overall issues to be discussed.
Initial validity concerns of the individual studies are discussed for each study in Section 3, as well as in each individual paper.
To improve validity of the studies seen as a whole, some possible actions can be
1. Replication of studies, both over time and in other organizations. This applies especially to the quantitative studies, in order to track development over and also to ensure that the results are generalizable. Example: our fault report studies on projects from five different organizations show very similar main results for most projects [P2, P4].
2. Using different research strategies to triangulate the research results. By using different research methods for the same study objects etc., we increase the validity of the results. Examples: Fault report study combined with interview sessions on the topic of fault report management [P4,P6], combining a qualitative study and a quantitative one in the DAIM study [P7].
Wohlin et al. define four main categories of validity threats [Wohlin00], which are further discussed in the next section, for different types of studies performed.
5.4.1 Quantitative studies: construct, internal, conclusion and external validity Studies 3, 4 and 6 used quantitative methods, and were mostly concerned with analyzing fault report data. These data were collected from existing fault report collections made by the organizations’ internal measures. Our contribution was the categorization of faults in the data where this had not been performed. Some threats to
the validity of quantitative studies and how this was handled is described here:
• Construct validity: In study 6, the main threat to construct validity is the conceptual difference between hazards and faults. We had to perform a conversion of the hazards found to potential fault types. It should be verified whether this type of hazard to fault type conversion is consequently correct, but during hazard analysis, there was a discussion of how each hazard could influence the system, and in many cases a software fault was proposed.
• Internal validity: In study 3, the greatest threat to internal validity is missing data in the fault reports. Many fault reports were not described well enough to be categorized and had to be left out. In certain fault reports, the fault had been classified by the developers, and they may have had a different opinion of the fault types than we had. In addition, with respect to severity of faults, it is not certain that the developers reporting the fault necessarily reported the true severity. In study 6, the hazard analysis sessions were time limited, so only the most obvious hazards were taken into account. Also, these sessions were performed over a period of time, so some maturation in the form of better understanding of the system being analyzed can have occurred.
• Conclusion validity: One possible threat to conclusion validity in study 3 and 4 is low reliability of measures, because of some missing and ambiguous data.
Because categorization of faults into fault types is a subjective task, it was important that the data we based the categorization on was correct and understandable. To prevent mistakes, we added an “unknown” type to filter out the faults we were not able to confidently categorize. The subjective nature of categorization is also a threat to conclusion validity.
• External validity: Especially in study 6 where we only studied one project, but also partly in studies 3 and 4 one threat to external validity is the relatively low number of projects studied. In study 6 we were not able to gain access to system documentation of more systems where we could also have fault report data. The projects under study may also not necessarily be the most typical businesscritical systems, but this is hard to verify in any way.
5.4.2 Qualitative studies: internal and external validity.
Studies 1, 2, 5 and partly 6 are qualitative studies, mostly explorative and descriptive in nature. The collected data is mainly from interviews and other subjective techniques (PHA) and are subject to interpretation. Here we have identified internal and external validity threats as the most serious.
• Internal validity. For Study 5, the main internal validity threat is that the same person performed interviewing, transcribing and information coding, which may introduce bias to how responses have been interpreted. By having workshops as feedback sessions after the interviews, we feel bias have been reduced.
• External validity. In Study 1, the main validity threat was the low number of organizations interviewed, and in Study 5 all interviews were performed with representatives from the same organization, although this is explained by us having to interview the people who had been involved in certain projects we had studied earlier.
5.5 Industrial relevance of results As many of our studies involved industrial data, our results were interesting not only to us, but also to the organizations the data was collected from. As such, we were able to present our results to the organizations and receive feedback both in terms of the results of the studies and how we should interpret the results.
In general, the organizations received general reports on the results, but also a specific report concerning the results from their organization. After Study 4, a workshop was held in order to convey our results to the organization as well as to receive more feedback.
5.6 Reflection: Research cooperation with industry Both the BUCS and the EVISOFT research projects are based on cooperation with industrial partners. Whereas EVISOFT had a number of industrial organizations involved from the project start, the BUCS project had no formal connections to any industrial partners as the project got under way. This meant that some effort had to be made in order to initiate contact and agreement with organizations in order to collect research data.
The hardest part of industrial cooperation was establishing contact and an agreement about what was going to be performed. In Study 3, the first fault report study, we initially contacted over 40 different organizations developing business-critical software.