FREE ELECTRONIC LIBRARY - Dissertations, online materials

Pages:     | 1 |   ...   | 24 | 25 || 27 |

«Software Fault Reporting Processes in Business-Critical Systems Jon Arvid Børretzen Doctoral Thesis Submitted for the partial fulfilment of the ...»

-- [ Page 26 ] --

[15] J. A. Børretzen, R. Conradi: “Results and Experiences From an Empirical Study of Fault Reports in Industrial Projects”, Proceedings of the 7th International Conference on Product Focused Software Process Improvement, pp. 389-394, Amsterdam, 12-14 June 2006.

[16] Jon Arvid Børretzen, Jostein Dyre-Hansen: “Investigating the Software Fault Profile of Industrial Projects to Determine Process Improvement Areas: An Empirical Study”, Proceedings of the 14th European Systems & Software Process Improvement and Innovation Conference, pp. 212-223, Potsdam, Germany, 26-28 Sept. 2007.

[17] O. Vinter, S. Lauesen: “Analyzing Requirements Bugs”, Software Testing & Quality Engineering Magazine, Vol. 2-6, Nov/Dec 2000.

–  –  –

Abstract: This report describes our experiences with fault reports and their processing from several organizations. Data from investigated projects is presented in order to show the diversity and at times lack of information in the reports used. Also, we show that although useful process information is readily available, it is seldom used or analyzed with process improvement in mind. An important challenge is to describe to practitioners why a standard description of faults could be advantageous and also to propose better use of the knowledge gained about faults. The main contribution is to explain why more effort should be put into codifying of fault reports, and how this information can be used to improve the software development process.

1. Introduction In all software development organizations there is a need for some minimum fault logging and follow-up to respond to faults discovered during development and testing, as well as claimed fault reports (really failure reports) from customers and field use.

Such reports typically contain fault attributes that are used to describe, classify, analyze, decide on and correct faults. There are many standards for this kind of information, although the original fault reports will be more of an ad hoc character than of a specified standard. A software development organization will, in addition to a fault report scheme, have defined own customized metrics and processes related to fault management.

Systematic fault management is often also motivated by certification efforts, for instance ISO 9000. Software Process Improvement (SPI) and Quality Assurance (QA) initiatives can also be a motivation for fault management improvement work.

Despite of this, in most organizations there is still much underused or even non-used data, either from lack of knowledge about the subject or from lack of procedures to assist in using the available data. As Jørgensen et al. states, “no data is better than unused data” [Jør98]. This is because collection of data that is not used, leads to waste of effort during data collection, poor data quality and even possibly a negative attitude to any kind of measurements during development or other SPI and QA activities.

In this investigative pre-study, we are reporting earlier experience from case studies and data mining in 8 Norwegian IT organizations and also an Open Source Software community, where fault data has been under-reported and/or under-analyzed. That is, poor or wrongly coded classification of faults, missing fault information for affected program module, no effort registration and so on. There is also the issue of fragmented data representation, partial fault reports, Software Configuration Management logs, or merely comments in code.

This paper describes our experiences with fault reports and fault reporting from working with fault reports from several different organizations. Results from these studies have been published papers like [Bor06, Bor07, Con99, Moh04, Moh06], but there is also a need to describe what we have learned from these studies in a descriptive manner.

In this field of study, different terminology is used in various sources. For this paper, we use the term fault in the same meaning as bug or defect. That is, a fault is the passive flaw in the system that could lead to an observable failure (vs. requirements) when executed. For fault report, other terms that are used are problem report and trouble report.

2. Metrics Our studies have given us insight and knowledge about the practice and information available from fault report repositories in several commercial organizations. This section presents these organizations and some attributes of these repositories. Such information gives a quick insight into how fault reporting is performed and what possibilities are available in termas of analysis and process improvement.

Table 1 shows an overview of the 8+1 involved organizations. Because of nondisclosure agreements with some organizations, their identities have been anonymized (O1-O8). We compare with the Open Source organization Gentoo [Gen].

–  –  –

For each of these organizations, we have studied and analyzed fault reports in one or more development projects. From this, we have selected some relevant fault report attributes, and report the situation for each of the organizations. The attributes are the


• Fault report description: Whether the initial description of the fault is long or short, this indicates how well the fault has been described when found.

• Fault severity: How many levels of fault severity does the organization use to discern their faults?

• Fault type categorization: Does the organization classify faults according to type?

• Fault location: Does the organization describe where the fault is located, either structurally (i.e. which component) or functionally (what user function the fault relates to)?

• Release version of fault: Does the organization register in which release of the software the fault was found?

• Correction log: Does the organization keep a correction log for each fault, where developers can enter information relevant to the identification of fault cause and correction?

• Solution description: Does the organization record what the solution of the problem was and how the fault was corrected?

• Correction effort: Is information recorded about the effort needed to find and correct the fault?

• Mandatory completion: Are all fault report entry fields mandatory for completion?

• Specialized fault report system or change reports: Is the fault reporting system a separate entity, or is it used in combination with all change reports?

• Standard or custom fault reporting system: Does the organization use a standard available fault reporting system, or do they use a custom made system?

These attributes are shown for each organization in Table 2.

Table 2 shows that there is a wide range of information used in the fault reports of these organizations. For instance the amount of information recorded differs from well described faults with correction log and solution description, to cases where the faults are scantily described and the only information about correction or solution is whether it has been solved or not.

–  –  –

3. Process Using fault report information to support process improvement can be a viable approach to certain parts of software process improvement [Gra92]. Some organizations have used this approach actively, while others have not. For the most part, organizations have done little work in this area until other researchers start studying (by data mining) their fault report repositories. For each organization, we describe the level of fault report use beyond fault correction, i.e. if any analysis work has been performed by the organization itself, followed by what have been performed by us as researchers. This is shown in Table 3. As we see, as external researchers we have been able to exploit the available data in the companies in a much larger degree than the organizations themselves.

–  –  –

One example of research results is from organization O1, where one conclusion of the work performed by the external researchers was performing software inspections was cost-effective for that organization. Inspections found about 70% of the recorded defects, but only cost 6-9% of the effort compared with testing. This yielded a saving of 21-34%. In addition, this study showed that the existing inspection methods were based on too short inspections. By increasing the length of inspections, there was a large saving of effort compared to the effort needed in testing. Figure 1 shows that by slowing down inspection rate from 8 pages/hour to 5 pages/hour, they could find almost twice as many faults. Calculations showed that by spending 200 extra analysis hours, and 1250 more inspection hours, they could save ca. 8000 test hours!

In O7, external researchers concluded through analysis of fault reports and fault types that the organization’s development process had definitive weaknesses in the specification and design phases, as a large percentage of faults that were found during system testing were of types that originated mainly in these early phases. Additionally, this external research lead the organization to alter the way they classified faults in a pilot project in order to study these issues further.

–  –  –

Other results we have drawn from several studies, is that the data material is not always well suited for analysis. This is mostly because of missing, incorrect or ambiguous data.

It is apparent that since the organization generally does not use this data after recording it, the motivation for recording the correct data is low. In O3, for instance, 97% of the fault reports were classed as “medium” severity faults. This was the default severity when recording a fault, and was rarely altered even if the fault was actually of a more severe character.

There are some interesting fault report attributes that are not in wide use, even if the information is most likely available. Such types of information could be very useful in process improvement initiatives and the cost of collecting and analyzing this data is

marginal. Some examples of such attributes are the following:

• Fault location: This attribute addresses where the fault is located, either as a functional location or structural location. When the functional location of a fault is reported, this is mainly from the view of the user or testers. It tells us which function or functional part of the system where the fault is discovered through a failure. In case of structural location, the fault report points to a place (or several) in the code, an interface or to a component where the fault has been found. For analysis purposes, the structural location is often the more useful information.

• Fault injection/discovery phase: The fault injection phase describes when in the development process the fault has been introduced. Sometimes faults are injected in the specification phase, but the most common faults are introduced in design and implementation. Even during testing faults can be introduced, if you include test preparation as part of the system implementation. The fault discovery phase describes in which phase the fault has been discovered, and the gap between injection and discovery should preferably be as small as possible, because the longer a fault is present in the system, the more effort it will take to remove it.

• Fault cost (effort): This shows how much effort has gone into finding and/or correcting a fault. Such information shows how expensive a fault has been for a project, and may be an indication on fault complexity or areas where a project needs to improve their knowledge or work process.

By introducing and implementing a core set of fault data attributes (i.e. a metric) to be recorded and analyzed, we could make a common process for fault reporting. Already, several schemes for recording and classifying faults exist, like the Orthogonal Defect Classification scheme [Chi92], or the IEEE 1044 standard [IEEE 1044]. A core process could be customized for organizations who want a broader approach to analysis of fault reports. Some organizations use custom made tools for fault reporting, but a great deal do use standard commercial or open source tools. Introducing a core set of fault report attributes in these tools would help encourage organizations to record the most useful information that can be used as a basis for process improvement. Many tools already have functionality for analysis of the data sets they contain.

4. Use of data repositories With industrial data repositories, we mean contents of defect reporting systems, source control systems, or any other data repository containing information on a software product or a software project. This is data gathered during the lifetime of a product or project and may be part of a measurement program or not. Some of this data is stored in databases that have facilities for search or mining, while others are not. Zelkowitz and Wallace define examining data from completed projects as a type of historical study [Zel98].

As the fields of Software Process Improvement (SPI) and empirical research have matured, these communities have increasingly focused on gathering data consciously and according to defined goals. This is best reflected in the Goal-Question-Metric (GQM) paradigm developed first by Basili [Bas94]. This explicitly states that data collection should proceed in a top-down way (i.e. designing research goals and process before examining data) rather than a bottom-up way (i.e. designing reseach goals and process after seeing what data is available). However, some reasons why bottom-up

studies are useful are (taken from [Moh04]):

1. There is a gap between the state of the art (best theories) and the state of the practice (current practices). Therefore, most data gathered in companies’ repositories are not defined and collected following the GQM paradigm.

2. Many projects have been running for a while without having improvement programs and may later want to start one. The projects want to assess the usefulness of the data that is already collected and to relate data to goals (reverse GQM).

3. Even if a company has a measurement program with defined goals and metrics, these programs need improvements from bottom-up studies.

Pages:     | 1 |   ...   | 24 | 25 || 27 |

Similar works:

«Town of Shrewsbury – Spag’s/Building 19 PDA Analysis Purpose of Analysis The Spag’s/Building 19 site on Route 9 in Shrewsbury was identified as a Priority Development Area in the recently completed 495/MetroWest Development Compact project. Priority Development Areas (PDAs) are areas within a town that have been identified as capable of supporting additional development or as candidates for redevelopment. These are areas on which a town is focusing its energy to promote thoughtful...»

«Taxation and Corporate Financial Policy Alan J. Auerbach University of California, Berkeley and NBER February 2001 This paper has been prepared for a forthcoming volume of the Handbook of Public Economics, edited by Alan Auerbach and Martin Feldstein. I am grateful to Kevin Cole for research assistance, to the Burch Center for Tax Policy and Public Finance for research support, and to Doug Bernheim, John Graham, Jim Hines, Vesa Kanniainen, Hans-Werner Sinn and Jan Södersten for comments on an...»

«MINISTRY OF NATIONAL EDUCATION THE ANNALS OF THE UNIVERSITY OF ORADEA ECONOMIC SCIENCES TOM XXIII 1st ISSUE / JULY 2014 ISSN 1222-569X (printed format) ISSN 1582-5450 (electronic format) The publication of the papers in the Journal “The Annals of the University of Oradea. Economic Sciences” Tom XXII, 2013, ISSN 1582-5450 (in electronic format on CD-ROM), ISSN 1222-569X (in printed format) a journal listed CNCSIS category B+ and indexed in RePec, Doaj, EBSCO and CABELLS PUBLISHING SERVICES...»

«Essays in Finance by Pengcheng Wan A Dissertation Presented in Partial Fulfillment of the Requirements for the Degree Doctor of Philosophy Approved April 2015 by the Graduate Supervisory Committee: Oliver Boguth, Co-Chair Yuri Tserlukevich, Co-Chair Ilona Babenka ARIZONA STATE UNIVERSITY May 2015 ABSTRACT In the first chapter, I develop a representative agent model in which the purchase of consumption goods must be planned in advance. Volatility in the agent’s portfolio increases the risk...»

«This is an Author's Accepted Manuscript of an article published in Applied Economics, Vol. 43, Issue 26, October 2011, pp.3775-3788, [copyright Taylor & Francis], available online at: http://www.tandfonline.com/doi/abs/10.1080/00036841003724411. Determinants of ICT adoption: Evidence from firm-level data Stefanie A. Haller and Iulia Siedschlag † Economic and Social Research Institute, Dublin Abstract We analyse factors driving interand intra-firm diffusion of ICT using data from Irish...»

«THEMATIC STUDY THE DEVELOPMENTAL EFFECTIVENESS OF UNTIED AID: EVALUATION OF THE IMPLEMENTATION OF THE PARIS DECLARATION AND OF THE 2001 DAC RECOMMENDATION ON UNTYING ODA TO THE LDCS VIETNAM COUNTRY STUDY Adam McCarty, Alexander Julian, and Daisy Banerjee Mekong Economics Ltd. 389 Thuy Khue, Hanoi, Vietnam www.mekongeconomics.com adam.mccarty@mekongeconomics.com December 2009 i Acknowledgements The research team on untied aid at Mekong Economics Ltd. would like to thank all the organisations and...»

«Consumer Based Brand Equity Conceptualization & Measurement: A Literature Review George Christodoulides* Lecturer in Marketing Birmingham Business School The University of Birmingham University House Edgbaston Birmingham B15 2TT United Kingdom Phone: + 44 (0)121 414 8343 Fax: + 44 (0)121 414 7791 Email: G.Christodoulides@bham.ac.uk Leslie de Chernatony Professor of Brand Marketing Università della Svizzera italiana, Lugano and Aston Business School, Birmingham Phone: +44-(0)7905088927 Fax:...»

«GEOTHERMAL TRAINING PROGRAMME Reports 2003 Orkustofnun, Grensásvegur 9, Number 2 IS-108 Reykjavík, Iceland LECTURES ON GEOTHERMAL RESOURCES AND UTILIZATION IN POLAND AND EUROPE Beata Kępińska Polish Academy of Sciences Mineral and Economy Research Institute, Geothermal Laboratory Krakow POLAND labgeo1@tatry.net.pl Lectures on geothermal energy given in September 2003 United Nations University, Geothermal Training Programme Reykjavík, Iceland Published in September 2004 ISBN 9979-68-148-9...»

«Judith A. Swanson March 19, 2015 Department of Political Science Boston University 232 Bay State Road Boston, MA 02215 U.S.A. Voicemail: (617) 965-5229; E-mail: jswanson@bu.edu Education B.A. (Political Science major), magna cum laude The Colorado College, June 1979 (January-June 1977 in France) Advisors: Fred Sondermann (1976-77), Timothy Fuller (1978-79) M.Sc., in History of Political Thought, with Distinction The London School of Economics and Political Science, October 1980 Tutors: Robert...»

«Overview of Contract Farming in Thailand: Lessons Learned Songsak Sriboonchitta and Aree Wiboonpoongse July 2008 ADB Institute Discussion Paper No. 112 Dr. Songsak Sriboonchitta is an associate professor in the faculty of Economics at Chiang Mai University, Thailand. Dr. Aree Wiboonpongse is a professor in the department of Agricultural Economics, in the faculty of Agriculture at Chiang Mai University, Thailand. The views expressed in this paper are the views of the authors and do not...»

«This PDF is a selec on from a published volume from the Na onal Bureau of Economic Research Volume Title: Fiscal Policy a er the Financial Crisis Volume Author/Editor: Alberto Alesina and Francesco Giavazzi, editors Volume Publisher: University of Chicago Press Volume ISBN: 0‐226‐01844‐X, 978‐0‐226‐01844‐7 (cloth) Volume URL: h p://www.nber.org/books/ales11‐1 Conference Date: December 12‐13, 2011 Publica on Date: June 2013 Chapter Title: Fiscal Rules: Theore cal Issues and...»

«  Department of Economics Working Paper Series The Political Economy of Soda Taxation Sriparna Ghosh Joshua C. Hall Working Paper No. 15-50     This paper can be found at the College of Business and Economics Working Paper Series homepage: http://be.wvu.edu/phd_economics/working-papers.htm The Political Economy of Soda Taxation Sriparna Ghosh ∗ Joshua C. Hall West Virginia University West Virginia University College of Business and Economics College of Business and Economics PO Box 6025 PO...»

<<  HOME   |    CONTACTS
2016 www.dissertation.xlibx.info - Dissertations, online materials

Materials of this site are available for review, all rights belong to their respective owners.
If you do not agree with the fact that your material is placed on this site, please, email us, we will within 1-2 business days delete him.