«Software Fault Reporting Processes in Business-Critical Systems Jon Arvid Børretzen Doctoral Thesis Submitted for the partial fulfilment of the ...»
5.1 Implications of the results The results found in this study coincide with the results of the previous fault study we performed with different development organizations. In both studies the “function” faults have been the most numerous, both in general and among the faults rated as most severe. As “function” faults are mainly associated with the design process phase, as stated by Chillarege et al. in  and also by Zheng et al. in  as shown in Table 6, this indicates that a large number of faults had their origin in early phases of development. This is a sign that the design and specification process is not working as well as it should, making it the source of faults that are demanding and expensive to fix, as “function” faults will generally involve larger fixing efforts than pure code errors like “checking” and “assignment” types of faults. This means that we can recommend the developers in the projects that have been studied to increase the effort used during design in order to reduce the total number of effort demanding faults in their products.
This finding is also similar to the one from the study of Vinter and Lauesen , where “Requirements and Features” faults were the dominating fault type.
Table 6. ODC fault types and development process phase associations 
When looking at each fault type in Figure 3, we see which fault types that tend to produce the most severe faults. One observation here is that although “function” faults dominate the picture for critical severity faults in Figure 2, it is the “relationship” and “timing/serialization” fault types that consist of the most critical severity rated faults.
It can therefore be argued that the fault types “relationship” and “timing/serialization” fault types are important to prevent, as it is likely that these types of faults have greater consequences than those of for instance “GUI” and “data” type faults. “Function” faults show themselves to be important to focus on preventing due to the sheer number of them, both in general and for the “critical” severity rated faults. Although “function” faults do not stand out as a fault type where most faults are rated as “critical”, it is still the biggest contributor to “critical” severity rated faults.
When informing the organization involved of the results of this study, the feedback was anecdotal confirmation of our findings, as they informed us that they were indeed having issues with design and specification, even though their own fault statistics showed most faults to be coding faults. We would like to study this issue further in our future work on the subject.
In many cases, fault reporting is performed with one goal in mind, to fix faults that are uncovered through inspection and testing. Once the fault has been corrected, the fault report information is not used again. The available information can be employed in a useful fashion as long as future development projects are similar to, or based on previous projects. By reusing the information that has been accumulated during fault discovery through testing and during production, we are able to learn about possible faults for new similar projects and further development of current projects.
Measuring quality and effects on quality in a software system is not a trivial matter. As presented in Section 2, the opinion on how and if this can be done is divided. One of the means Avizienis et al. suggests for attaining better dependability in a system is fault removal in order to reduce the number and severity of faults . By identifying common fault types, developers can reduce a larger number of faults by focusing their efforts on preventing these types of faults. Also, identifying the most severe fault types makes developers able to focus on preventing those faults that have the biggest detrimental impact on the system.
5.2 Further issues concerning fault reporting in this organization
In addition to our quantitative study results, we were able to identify some points of possible improvement in the studied organization's fault reporting. Two attributes that we found lacking, which should be possible to include in fault reporting are Fault Location and Fault Fixing Effort. The location of a fault should be readily known once a fault report has been dealt with, as fault fixing must have a target module or software part. This information would be very helpful if the organization wants to investigate which software modules produce the most serious faults, and they can then make a reasoned argument if these modules are of a particularly critical type (like infrastructure or server components), or if some modules are simply of a poorer quality than others.
Including fault fixing effort into the fault reports is also an issue that could be of great benefit when working to improve fault prevention processes. By recording such information, we can see which fault types that produce the most expensive faults in terms of effort when fixing them. These are issues that will be presented to the organization under study. Their current process of testing and registering faults in a centralized way hinders the testers and developers from including this valuable information from the fault reports. The testers who initially produce the fault reports do not necessarily know which software modules the fault is located in, and developers fixing the fault do not communicate the location it was found in after it has been found and fixed.
5.3 Threats to validity
When performing an empirical study on industrial projects, it is not possible to control the environment or data collected as we would do in an experiment. The following is a short presentation of what we see as the main validity threats.
Internal validity. An issue here might be factors affecting the distribution of fault types. When the fault data was collected the intention of use was solely for fault fixing, it was not intended to be studied in this way. The coarse classification given by the developers could have been biased. Such bias or other inconsistencies were hopefully reduced by us classifying the fault reports with new fault types.
External validity. The small number of projects under investigation is a threat to external validity. However, the results of this study support the findings of a previous similar study of fault reports from other software development organizations. The projects under study may also not necessarily be the most typical, but this is hard to verify in any way.
Conclusion validity. One possible threat here is the reliability of measures, as the categorization of faults into fault types is a subjective task. To prevent categorizing faults we were unsure of into the wrong category, we used a type “unknown” to filter out the faults we were not able to confidently categorize.
6. Conclusion and future work
In this paper we have described the results of a study of fault reports from five software projects from a company developing business-critical software. The fault reports have been categorized and analyzed according to our research questions. From the research questions we have found that "function" faults, closely followed by "GUI" faults are the fault types that occur most frequently in the projects. To reduce the number of faults introduced in the systems, the organization should focus on improving the processes which are most likely to contribute to these types of faults, namely the specification and design phases of development. Faults of the fault types "documentation", "relationship", "timing/serialization" and "interface" are the least frequent occurring fault types.
The fault types that are most often rated as most severe are "relationship" and "timing/serialization" faults, while the fault types "GUI" and "documentation" are considered the least severe. Although “function” faults are not rated as the most severe type of fault, this fault type still dominates when looking at the distribution of highly severe faults only.
In additions to these results, we observed that the organization’s fault reporting process could be improved by adding some information to the fault reports. This would facilitate more effective targeting of fault types and locations in order to better focus future efforts for improvement.
In terms of future work, we want to continue studying the projects explored in this paper, using qualitative methods to further explain our quantitative results. Feedback from the developers’ organization would aid us understand the source of these results, and help us suggest concrete measures for process improvement in the organization.
Acknowledgements The authors would like to thank Reidar Conradi for careful reviewing and valuable input. We also thank the organization involved for their participation and cooperation during the study.
1. R. Grady, Practical Software Metrics for Project Management and Process Improvement, Prentice Hall, 1992
2. J. A. Børretzen; T. Stålhane; T. Lauritsen; P. T. Myhrer, “Safety activities during early software project phases”. Proceedings, Norwegian Informatics Conference, 2004
3. J. A. Børretzen; R. Conradi, ”Results and Experiences From an Empirical Study of Fault Reports in Industrial Projects”. Proc. 7th International Conference on Product Focused Software Process Improvement (PROFES'2006), 12-14 June 2006, Amsterdam, Pages: 389P. Mohagheghi; R. Conradi; J.A. Børretzen, "Revisiting the Problem of Using Problem Reports for Quality Assessment", Proc. the 4th Workshop on Software Quality, held at ICSE'06, 21 May 2006, Shanghai, Pages: 45-50
5. ISO, ISO/IEC 9126 - Information technology - Software evaluation – Quality characteristics and guide-lines for their use, ISO, December 1991
6. J.-C. Laprie, “Dependable computing and fault tolerance: Concepts and terminology”, Twenty-Fifth International Symposium on Fault-Tolerant Computing, 1995, ' Highlights from Twenty-Five Years', June 27-30, 1995
7. B. Littlewood; L. Strigini, “Software reliability and dependability: a roadmap”, Proceedings
of the Conference on The Future of Software Engineering, Limerick, Ireland, 2000, Pages:
175 - 188
8. N. Leveson, Safeware: System safety and computers, Addison-Wesley, Boston, 1995
9. IEEE, IEEE Standard Classification for Software Anomalies, IEEE Std 1044-1993, December 2, 1993
10. K.A. Bassin; T. Kratschmer; P. Santhanam, “Evaluating software development objectively”, IEEE Software, 15(6): 66-74, Nov.-Dec. 1998
11. K. Bassin; P. Santhanam, “Managing the maintenance of ported, outsourced, and legacy software via orthogonal defect classification”, Proceedings. IEEE International Conference on Software Maintenance, 2001, 7-9 Nov. 2001
12. K. El Emam; I. Wieczorek, “The repeatability of code defect classifications”, Proceedings.
The Ninth International Symposium on Software Reliability Engineering, 1998, 4-7 Nov.
1998 Page(s):322 – 333
13. R. Chillarege; I.S. Bhandari; J.K. Chaar; M.J. Halliday; D.S. Moebus; B.K. Ray; M.-Y.
Wong, “Orthogonal defect classification-a concept for in-process measurements”, IEEE Transactions on Software Engineering, Volume 18, Issue 11, Nov. 1992 Page(s):943 - 956
14. R.R. Lutz; I.C. Mikulski, “Empirical analysis of safety-critical anomalies during operations”, IEEE Transactions on Software Engineering, 30(3):172-180, March 2004
15. R.A. Paul; F. Bastani; I-Ling Yen; V.U.B. Challagulla, “Defect-based reliability analysis for mission-critical software”, The 24th Annual International Computer Software and Applications Conference, 2000. COMPSAC 2000. 25-27 Oct. 2000 Page(s):439 - 444
16. D. Hamlet, “What is software reliability?”, Proceedings of the Ninth Annual Conference on Computer Assurance, 1994. COMPASS '94 'Safety, Reliability, Fault Tolerance, Concurrency and Real Time, Security', 27 June-1 July 1994 Page(s):169 – 170
17. A. Avizienis; J.-C. Laprie; B. Randell; and C. Landwehr, “Basic Concepts and Taxonomy of Dependable and Secure Computing”, IEEE Transactions on Dependable and Secure Computing, vol. 1, no. 1, January-March 2004
18. O. Vinter; S. Lauesen, “Analyzing Requirements Bugs”, Software Testing & Quality Engineering Magazine, Vol. 2-6, Nov/Dec 2000.
19. B. Beizer, Software Testing Techniques. Second Edition, Van Nostrand Reinhold, New York, 1990
20. J. Zheng; L. Williams; N. Nagappan; W. Snipes; J.P. Hudepohl; M.A. Vouk, “On the value of static analysis for fault detection in software”, IEEE Transactions on Software Engineering, Volume 32, Issue 4, April 2006 Page(s):240 – 253
AbstractThe benefits of reusing software components have been studied for many years. Several previous studies have concluded that reused components have fewer defects in general than non-reusable components. However, few of these studies have gone a further step, i.e., investigating which type of defects has been reduced because of reuse. Thus, it is suspected that making a software component reusable will automatically improve its quality. This paper presents an on-going industrial empirical study on the quality benefits of reuse. We are going to compare the defects types, which are classified by ODC (Orthogonal Defect Classification), of the reusable component vs. the nonreusable components in several large and medium software systems. The intention is to figure out which defects have been reduced because of reuse and the reasons of the reduction.
1. Introduction Software reuse is a management strategy for software evolution, in terms of development for and with reuse. Development for reuse refers to the generalization of components towards reuse, while development with reuse has to do with the inclusion of these reusable components in new and future development . Understanding the issues related to software reuse, involving its purpose and promises, has been a focus since the 1970s. The focuses have been on how to develop for/with reuse, technical/managerial/organizational aspects, measuring reuse in terms of quality and productivity, as well as reporting success and failures of reuse practices. Although some studies have found that reusable software components have less defect density than nonreusable components -, few studies have studied why software defects have been reduced because of reuse.