«Software Fault Reporting Processes in Business-Critical Systems Jon Arvid Børretzen Doctoral Thesis Submitted for the partial fulfilment of the ...»
According to Votta et al., the goal of empirical software engineering is to construct a “credible empirical base... that is of value to both professional developers and researchers” [Votta95]. They argue that empirical software engineering inherits most of the methodological approaches and techniques of social sciences, since its goal is to examine complex social settings, contexts where the human interaction is the most critical factor determining the quality and effectiveness of the results being produced. In particular, empirical work is accomplished through the execution of empirical studies.
This entails observations of specific settings, where the purpose is collecting and analysing/deriving useful information on their behaviour and attributes.
Empirical studies can be classified in three categories, according to the increasing
degree of rigor and confidence in the results of the study [Wohlin00]:
(1) Surveys (2) Case studies (3) Controlled experiments This is in line with the classification made by Votta et al. where the different types of investigations are said to be anecdotal studies, case studies and experiments [Votta95].
It can be argued that surveys and case studies should swap positions, as case studies do tend to be hard to replicate and very open ended, while surveys can be made very rigorous and are definitely suitable for replication.
Zelkowitz et al. describe different ways of studying technology [Zelkowitz98], and states that the empirical method is the following: “Empirical method: A statistical method is proposed as a means to validate a given hypothesis. Unlike the scientific method, there may not be a formal model or theory describing the hypothesis. Data is collected to verify the hypothesis.” Zelkowitz et al. describe 12 different ways of studying technology, in three dimensions: Observational, Historical and Controlled. The different ways are described in detail in [Zelkowitz98], and the the models they propose for validating technology are shown in table 2-3. In addition to these, there are a few other techniques commonly used in empirical software engineering. Firstly, as previously mentioned Wohlin et al. discusses surveys in [Wohlin00]. Secondly, the Action Research method is presented by Avison et al. This method involves researchers and practitioners acting together on a set of activities, including problem diagnosis, action intervention and reflective learning [Avison99]. Finally, a research method that is becoming more used in studies of software developing organizations is Grounded Theory which emphasizes generation of theory from data. This method originates from the sociologists Barney Glaser and Anselm Strauss and is presented in [Strauss98].
2.8.1 Research strategies in Empirical Research There are three main types of research strategies, each with distinct approaches to empirical studies, and they may all be used for Empirical Software Engineering [Wohlin00][Seaman99]:
• Quantitative approaches are mainly used to quantify a relationship or comparing groups, with the aim being to identify cause-effect relationships, verifying hypotheses or testing theories.
• Qualitative approaches are observational studies with the aim of interpreting a phenomenon based on information collected from various sources. This information is usually subjective and non-numeric.
• Mixed-method approaches are used to overcome limitations in the two strategies above, by triangulation of data and combining the advantages of the two former ones.
Table 2-4 gives an overview of empirical research approaches and examples of strategies for each. This is taken from [Moghaghegi04b] and [Creswell03].
The boundaries between these approaches are not sharp. For instance, case studies can combine qualitative and qualitative studies, and although case studies are often classed as qualitative in nature Yin states that case studies do not by their nature equal qualitative research [Yin03].
2.9 Main challenges in business-critical software engineering The following is a short list of some of the general challenges in software engineering today. They are sometimes cited as reasons for difficulties in software engineering, e.g.
• Poor Requirements
• Rising Complexity
• Ongoing Change
• Failure to Pinpoint Causes In our work, we concentrate on issues dealing with reduction of product risk, improving requirement specifications, coping with complexity, and helping with pinpointing causes for failures. This adds up to a main challenge for business-critical software
• Developing methods that help reduce product risk, without increasing costs too much compared to the received benefit from these methods.
What this means is that we need to introduce low cost methods and techniques that focus on important areas, so as to spend just a little more effort to reduce the largest problems. This is in line with Boehm’s notion of value-based software engineering, where the main agenda is developing an overall framework to better focus effort where it is needed [Boehm03].
As far as empirical software engineering research is concerned, an important challenge is that of following up research and technology change proposals with continued observation and measurement in the field when practitioners put theory into practice.
Much of the research being performed concerns the software development process up to a point, but does not follow the eventual implementation of these results further.
3 Research Context and Design In this chapter, the project context is presented with more details. The overall research design is presented, and it combines quantitative and qualitative studies. Finally a more detailed description of each study is given.
3.1 BUCS Context In the last decade, computers have taken on a more important role in several areas of commercial business. As an effect of this, many of the functions required by industry and services depend on software and computer systems. Failures in such systems can have serious consequences for businesses that depend on these systems for their livelihood. As in many related areas, there will be substantial savings by discovering, reducing or removing these potential failures early in the system’s life-cycle. In fact, most potential failures should be possible to reduce very early in the process of the system’s development.
The main goal for the BUCS project is to better understand and to improve the software technologies and processes used for developing business-critical software. Much of the information about current practises and possible problems was collected at Norwegian IT companies. It was important that the relationship between the BUCS project and the involved companies and organizations was based on mutual profit for both parties.
The BUCS project have – through literature reviews, controlled experiments, historical data analysis and case studies – investigated methods from the area of safety-critical software. These methods include Preliminary Hazard Analysis (PHA), HazOp, Fault Tree Analysis (FTA), Cause-Consequence Diagrams (CCD), and Failure Mode and Effect Analysis (FMEA). We have also studied important standards in this area, such as IEC 61508 – a standard for functional safety. The effects of both methods and standards have been studied using controlled student experiments and through industrial case studies. All the above-mentioned methods are rather general. They can be applied to both local and distributed systems, they can be used on hardware, software and “wetware” (people). This is specially important when we are dealing with a problem as wide and diverse as business criticality.
Important techniques that we have sought to adapt from the development of safetycritical software are mainly PHA, HazOp, and FMEA.
As well as the industrial focus from the BUCS basic R&D project, some of the studies in this thesis also cooperated with organizations that were involved in the EVISOFT project, an industrially-driven research project [EVISOFT].
3.2 Research Focus When deciding the focus areas of this thesis, the input was the BUCS project context, less-exploited research areas, and available sources for research data. During our literature studies and after contact with Norwegian IT-companies some key areas to
focus our work in this thesis was decided:
• Business-criticality in terms of software faults
• Fault report analysis
• Fault reporting in software development
In terms of goals for the research, we formulated the following research questions:
• RQ1. What is the role of fault reporting in existing industrial software development?
• RQ2. How can we improve on existing fault reporting processes?
• RQ3. What are the most common and severe fault types, and how can we reduce them in number and severity?
• RQ4. How can we use safety analysis techniques together with failure report analysis to improve the development process?
To obtain answers to these research questions, we decided on common metrics for our studies. We started broadly, including attributes like structural fault location, functional fault location and fault repair effort. When we received actual data from industrial projects, we had to reduce the scope somewhat. This was due to lack of complete information in the data material, and great variation between organizations on what data they stored.
The main metrics we identified for fault report studies were:
• The number of detected faults is an indirect metric, attained simply by counting the number of faults of a certain type or for a certain system part etc.
• The metrics that are used directly from the data in the fault reports are the reported type, severity, priority, and release version of the fault.
The reasons why we decided to focus on software faults were several:
• The BUCS project is concerned with business-critical systems. A recurring theme in the definition of business-criticality is that the major threat to such systems is failures that stop or limit the use of the system. As described in section 2.4, “Faults are potential flaws in a software system, that later may be activated to produce an error. An error is the execution of a "passive fault", leading to a failure.” This means that by working to reduce the number and criticality of faults in the software, we would also reduce the number or frequency of failures.
• As Avizienis et al. suggests, one way of attaining better dependability in a system is fault removal to reduce the number and severity of faults [Avizienis04]. By working to identify critical or numerous fault types, developers can eliminate a larger number of faults by focusing on preventing such fault types.
• As far as industrial data, fault report data is abundant in most software developing organizations. Thus we had a wide array of potential industrial partners to collect data from.
As shown in Figure 1-1, the studies we have performed are all connected on the topic of business-critical software and fault report analysis, and have been performed in sequence. A short description of the studies is given in Table 3-1, and each study is elaborated in Section 3.3.
Study 2 Literature review: Software Criticality Techniques, Fault (2004) reporting and management literature.
Study 3 Historical data mining: Fault report analysis of industrial (2004-05) projects from four organizations.
Study 4 Historical data mining: Fault report analysis of industrial (2006-07) projects.
Study 5 Structured interviews: Exploring the results from study 4 further, (2007) regarding fault report analysis and fault reporting processes.
Study 6 Case study: Comparison of hazard analysis and fault report (2006-07) analysis in a practical setting.
Study 7 Lessons learned from our experiences with fault report studies.
(2004-07) 3.2.1 Data collection Before starting to plan and conduct our empirical studies, we decided on the goals of our studies, which types of studies we were going to perform, and which data sources we were going to need to complete them. Data collection was split up in three phases.
First, there was a pre-study phase of initial data collection and pre-analysis of this to narrow down research areas and questions. Second, we would focus the data collection on more deeper issues that seemed the most relevant. Finally, there would be an analysis phase to summarize and reflect and to collect lessons learned.
As the BUCS project is aimed at supporting business-critical systems, and part of the BUCS goals was close cooperation with Norwegian IT industry, we chose early on to focus on empirical studies of Norwegian commercial projects and organizations. This meant that we had to contact and select relevant organizations developing businesscritical applications. This raised the issue of which organizations we wanted to study.
As it turned out, the sampling of companies was mostly done out of convenience, because of apparent reluctance to disclose sensitive information about quality data and processes, as well as many organizations simply being “too busy”.
Another issue was whether data collection should be performed in pre-implementation phases or post implementation ones. Pre-implementation studies are better for working with possibilities for improvement initiatives, but a problem is knowing which data to collect. In this case the data would be more of a qualitative nature, and thus harder to analyse. Post-implementation studies, on the other hand, would be better for obtaining quantified data, but then there is the question about the data being relevant for investigation.
Once the studies to be performed were tentatively planned, the actual data collection depended on the available projects and their data, i.e. which companies we were able to cooperate with, and which processes those companies were willing to let us participate in. The employed data collection methods were interviews, surveys, and field/case studies, as well as historical data mining and analysis of reports or logs on relevant issues.