«Software Fault Reporting Processes in Business-Critical Systems Jon Arvid Børretzen Doctoral Thesis Submitted for the partial fulfilment of the ...»
They generally agreed that the results from the quantitative study were valid and seemed relevant for their individual projects as well. Results also showed that experience in the organization was in line with our previous findings, that there were weaknesses in the early development phases, specification and design, that were origins for faults being introduced in the software. This is a similar conclusion as presented by Vinter and Lauesen (2000).
In RQ2, we wanted to draw on their experience to hear if they thought a fault type classification scheme could be helpful towards improving their development processes.
The response was that introducing a better and more structured fault classification scheme would be a good idea, as long as it was a scheme that was clearly defined and useable for the developers. El Emam and Wieczorek (1998) claims that as long as fault type classes are well understood, developers are able to correctly categorize faults into fault types.
In RQ3, we also wanted to hear their opinions on increasing effort in data collection and fault report analysis in order to improve their software development processes. The respondents were all very positive about introducing fault report analysis for process improvement into their fault management process. By using fault report analysis more actively, they could be able to produce better metrics for process improvement. As most software developing organizations produce large quantities of information about the faults in the system they develop, it makes a lot of sense to utilize this information further than the simplest and most obvious way that is purely using fault report data as a fault reporting and correction log.
Lastly in RQ4, we wanted to ask where they thought there was most potential of improvement in their fault management system. This was to elicit areas that they felt were lacking in their current fault reporting process. The answers indicated that information flow between testers and developers should be improved, and there were areas of improvement on the information stored in fault reports. This is an issue we have touched upon in our previous studies (Børretzen and Conradi, 2006; Børretzen and Dyre-Hansen, 2007).
Concerning the validity of this study, we have to consider internal validity - i.e., credibility, believability, plausibility of findings and results and external validity - i.e., generalizability or applicability of the study's findings, results and conclusions to other circumstances. Also, we touch upon an issue concerning construct validity.
The main internal validity threat we have identified is that the interview, transcribing and information coding were all performed by the same person. This may introduce bias to how certain responses have been interpreted. The reason for having just one person performing all the tasks in the study has been due to resource limitations. By using feedback from the workshops to augment the interview responses, we feel that the potential bias has been reduced. In addition, we had a dialogue with some of the interviewees to clear up some questions we had after the transcription of the interview recordings. Also, as Sarker et al. (2000) states, “grounded theory coding and sampling must never be delegated to hired assistants, but must be done by the researchers who have a stake in the theory emerging from the project.” The main threat to external validity is that interviews have all been carried out with interviewees from the same organization. This means that we have only investigated the opinions of people using the same form of fault management and reporting system. The reason for this was that we wanted to base our interviews on information gathered in the previous quantitative study, and we therefore had to interview the people who had been involved in the actual projects.
As for construct validity, one threat was in the truthfulness of the responses in the interviews. The topics of faults in software and fault management are delicate ones, as it touches upon aspects concerning quality deficiencies in the product an organization delivers. The interviewees might feel that they were being evaluated, and present a better picture than what was actually true. Nevertheless, we found the interviewees to be truthful and open minded, and do not suspect that they were holding information back or presenting a more attractive situation in their organization than what is actually the case. A more likely issue could actually be participants being more positive to the ideas when presented with them in interviews than what they would actually feel about them in real life.
5.2 Relevance of results for the studied organization
The organization involved in the study were already working to improve their fault management processes, which meant that our suggestions and findings based on their fault report data would be welcome. Our previous quantitative fault report study lead to a specific suggestion of new fault types being introduced into a pilot project, and the results of these interviews and workshops will most likely be used to fine tune the selection of fault types used. Compared to their original fault type classification, the new classification scheme should be better suited for process improvement work.
6. Conclusion and Future Work
We have performed qualitative interviews with representatives of a software developing organization regarding fault reporting and fault classification. This has expanded our knowledge and is built on results from a quantitative study of five projects in the same organization. Our main contribution is showing that practitioners are motivated to use their existing knowledge of software faults in a more extensive manner to improve their work practices. By triangulation of both qualitative and quantitative methods, we have
increased the validity of our studies. Our main findings are that:
• The interviewees agreed with our conclusions from the previous quantitative study (Børretzen and Dyre-Hansen, 2007), i.e. that the early phases in their development process had weaknesses that lead to a high number of software faults from early development phases.
• They also expressed a need for better fault categorization in their fault reports, in order to analyze previous projects with intention of improving their work processes.
• The proposed ODC fault types were seen as a useful basis for introducing a better fault classification scheme, although simplicity was important.
• They were positive to using fault report analysis feedback to improve development processes, although introducing such analysis for regular use would have to be done carefully in the organization.
• Finally, they revealed some areas in their fault reporting scheme that could be improved in order to improve analysis usefulness, for instance including attributes like fault finding and correction effort and component location of fault. The knowledge was present, it was just not recorded formally.
In terms of future work, we would want to perform a second series of interviews in the organization after the new fault categorization scheme has been in use for some time. Through this we would be able to ascertain how this initiative has worked in the organization, and how it influences their project analyses and development process. We would also like to expand the generalizability of the study by including other software developing organizations using similar fault management processes.
Acknowledgements The author would like to thank Reidar Conradi for careful reviewing and valuable input.
We also thank the organization involved for their participation and cooperation during the study.
References Avizienis, A., Laprie, J.-C., Randell, B., Landwehr, C., 2004. Basic Concepts and Taxonomy of Dependable and Secure Computing. In: IEEE Transactions on Dependable and Secure Computing. (1)1, January-March 2004.
Biffl, S., Aurum, A., Boehm, B., Erdogmus, H., Grünbacher, P., 2006. Value-Based Software Engineering, Springer, Berlin Heidelberg.
Bryant, A., 2002. Grounding systems research: re-establishing grounded theory. Proceedings of the 35th Annual Hawaii International Conference on System Sciences, (HICSS’02). IEEE Computer Society, pp. 3446-3455, Big Island, Hawaii, 7-10 January 2002.
Børretzen, J.A., Stålhane, T., Lauritsen, T., Myhrer, P.T., 2004. Safety activities during early software project phases. Proceedings of the Norwegian Informatics Conference, pp. 180Stavanger, Norway.
Børretzen, J.A., Conradi, R., 2006. Results and Experiences From an Empirical Study of Fault Reports in Industrial Projects. Proceedings of the 7th International Conference on Product Focused Software Process Improvement (PROFES'2006), pp. 389-394, Amsterdam, 12June 2006.
Børretzen, J.A., Dyre-Hansen, J., 2007. Investigating the Software Fault Profile of Industrial Projects to Determine Process Improvement Areas: An Empirical Study, Proceedings of the European Systems & Software Process Improvement and Innovation Conference 2007 (EuroSPI07), pp. 212-223, Potsdam, Germany, 26-28 September 2007.
Chillarege, R., Bhandari, I.S., Chaar, J.K., Halliday, M.J., Moebus, D.S., Ray, B.K., Wong, M.Y., 1992. Orthogonal defect classification - a concept for in-process measurements. IEEE Transactions on Software Engineering, (18)11, pp. 943 – 956, Nov. 1992.
El Emam, K., Wieczorek, I., 1998. The repeatability of code defect classifications. Proceedings of The Ninth International Symposium on Software Reliability Engineering, pp. 322-333, 4-7 November 1998.
Grady, R., 1992. Practical Software Metrics for Project Management and Process Improvement, Prentice Hall.
Hansen, B.H., Kautz, K., 2005. Grounded Theory Applied - Studying Information Systems Development Methodologies in Practice. Proceedings of the 38th Annual Hawaii International Conference on System Sciences (HICSS'05), IEEE Computer Society, 10 p., Big Island, Hawaii, January 3-6 2005.
Hove, S.E., Anda, B., 2005. Experiences from conducting semi-structured interviews in empirical software engineering research. 11th IEEE International Symposium on Software Metrics, 10 pages, 19-22 Sept. 2005.
Leveson, N., 1995. Safeware: System safety and computers, Addison-Wesley, Boston.
Mohagheghi, P., Conradi, R., Børretzen, J.A., 2006. Revisiting the Problem of Using Problem Reports for Quality Assessment. Proceedings of the 4th Workshop on Software Quality, held at ICSE'06, pp. 45-50, Shanghai, 21 May 2006.
Sarker, S., Lau, F., Sahay, S., 2000. Building an inductive theory of collaboration in virtual teams: an adapted grounded theory approach. Proceedings of the 33rd Annual Hawaii International Conference on System Sciences, pp. 1-10, Hawaii, 4-7 Jan. 2000.
Seaman, C.B., 1999. Qualitative Methods in Empirical Studies of Software Engineering. IEEE Transactions on Software Engineering, (25)4, pp. 557-572, July 1999.
Strauss, A., Corbin, J., 1998. Basics of Qualitative Research, Sage Publications, London, UK.
Vinter, O., Lauesen, S., 2000. Analyzing Requirements Bugs. Software Testing & Quality Engineering Magazine, Vol. 2-6, Nov/Dec 2000.
P7. Using Hazard Identification to Identify Potential Software Faults: A Proposed Method and Case Study
Abstract When designing a business-critical software system, early analysis with correction of software faults and hazards (commonly called anomalies) may improve the system’s reliability and safety, respectively. We wanted to investigate if safety hazards, identified by Preliminary Hazard Analysis, could also be related to the actual system faults that had been discovered and documented in existing fault reports from testing and field use.
A research method for this is the main contribution of this paper. For validation, a small web-based database for management of student theses was studied, using both Preliminary Hazard Analysis and analysis of fault reports. Our findings showed that Preliminary Hazard Analysis was suited to find potential specification and design faults in software.
1. Introduction When developing a critical software system, much effort is put into ensuring that the system will have as few critical anomalies (faults and hazards) as possible in the context of its environment and modes of operation. Despite this effort, critical systems are still failing due to software faults, i.e. reducing reliability and possibly safety. A goal for the research community is to develop and introduce processes and techniques to reduce the number of critical faults and hazards in software. In this paper we present a novel method, results and lessons learned in a study where we compared the findings from Preliminary Hazard Analysis (PHA) with findings by traditional analysis of system testing and field use fault reports, both applied to the same system. PHA is a review technique for safety-critical systems, and used in early stages of development.
This paper is organized as follows. Section 2 gives our motivation and related work.
Section 3 describes the research method, research questions and procedure. Section 4 presents the proposed method, the results from the hazard analysis and the fault report analysis. Section 5 presents the interpretation of these results and evaluation of the work. The conclusion and further work is presented in Section 6.
2. Motivation and Background When proposing a method in the border area between reliability and safety, we need to clarify some of the terminology. A fault is an incorrect part of the system (program, hardware, even “data”), i.e. where possible later execution will violate stated requirements and cause a system failure (reliability dimension). A hazard is a state or set of conditions of a system or an object that, together with other conditions in the environment of the system or object, may lead to an accident (safety dimension) . In this paper, we will investigate whether the hazard analysis technique PHA can help us to identify not only hazards, but also faults and in turn failures, thereby reducing the reliability of the product stemming from these failures.