«Abstract Large IT projects may not deliver what they promise. Often they are late. Often they are over budget. Often what is eventually delivered is ...»
Why Large IT Projects Fail
School of Electronics and Computer Science
University of Southampton, SO17 1BJ, UK
Large IT projects may not deliver what they promise. Often they are late. Often they are
over budget. Often what is eventually delivered is not either what was originally speciﬁed nor
what is actually needed. There are many reasons why this happens. We discuss some of those
reasons here and argue that many of them are a consequence of failure to specify requirements anything like adequately. We introduce the notion of requirements drift as a root cause of failure and show, by the use of models, that the often quite large discrepancies between expectations and reality can be the consequence of quite modest amounts of requirements drift.
This preprint is available at http://www.ecs.soton.ac.uk/˜ph/LargeIT.pdf Keywords: Large IT Projects, Requirements, Failure, Open Systems, Complexity, Models 1 Introduction Large IT projects are those that involve hundreds of software engineers and require many years to complete. Such projects often fail. Failure can be in various forms. Sometimes the delivered system fails to provide signiﬁcant requirements to the standard of quality which makes it possible to actually deploy the system. Sometimes the anticipated cost for the project is exceeded by a large factor. Sometimes the anticipated delivery date is exceeded by a large factor. Often, failure involves a combination of all three of these eﬀects.
The main problem is to determine the causes of such failures, since this must precede the recom- mendations for mitigating their consequences. We will enumerate the accepted reasons for large project failure and show that an underlying cause of all of them is what we shall call requirements drift.
The accepted reasons why large IT projects fail are many and various, but include all the following 1 2 Why Large IT Projects Fail - http://www.ecs.soton.ac.uk/˜ph/LargeIT.pdf
1. There is too much optimism as to the potential beneﬁts of IT and as to the cost of delivering those beneﬁts
2. There is too little investment at the beginning.
3. Suﬃcient investment is made eventually, but is too late.
4. There is not enough technical know-how in the project team.
5. There are opposing Human Factors aspects - including insuﬃcient consideration of teams, project management and risks.
6. The project does not plan incremental roll-out and fails to anticipate the eﬀect of big-bang on the organisation.
7. The project tries to match IT to the existing Business Processes of the organisation, rather than mandating that the Business Processes change to match those implicit in the IT.
8. Initial underinvestment leads to an investment legacy, where the project has invested in bad decisions and doesn’t have the courage to retreat.
9. There are many management disaster scenarios such as (a) parent/child governance scenario - where authority within the project remains with senior management in the customer.
(b) the enthusiastic builder scenario - where the supplier has a vested interest in prolonging the project since payment is related to time-on-the-job rather than delivery of results.
10. And of course, because of insuﬃcient planning and simply because projects take a long time from inception to completion, there is a requirements explosion during the lifetime of the project which means that what is eventually required is signiﬁcantly diﬀerent from what was originally anticipated In fact, because of the impossibility of accurately capturing requirements (where supplier and customer absolutely have the same understanding) before a project begins, and because an organisation learns over time, requirements are subject to constant change, a phenomenon we call requirements drift. Each of the reasons listed above can be shown to have an underlying cause which is explained by requirements drift.
We have built a series of models of large projects, based on our understanding of how real projects perform [Chatters 1998], and used these models to explain how failure could be anticipated if requirements drift were considered from the outset and throughout. Our results demonstrate the quite remarkable butterﬂy eﬀect of modest changes in the rate of drift. Small changes in that can lead to signiﬁcant changes in time, cost and quality, the principal measures of success.
1.1 Outline of Paper First we characterise what we mean by large IT projects and what it means for them to fail. Then we put some ﬂesh on the bones of the ten reasons listed above for project failure. These reasons are, of course, interrelated, so there is no best order in which to present them, since whatever we do we have to make forward reference to reasons that we have not yet elaborated. It is a conjecture of this paper that everything is related to the (in)ability to keep control of requirements, so every reason will make forward reference to that.
Next we introduce the notion of requirements drift which captures the observation that as a project proceeds, the requirements change. Many of these changes are actual, but many are simply a consequence of the organisation learning more about what the original requirements actually involved.
In this section we argue that the ten reasons previously discussed can all be seen as an aspect of requirements drift, or can be shown to be exacerbated by requirements drift.
Then we introduce a model that we can use to explain the relationship between requirements drift and project metrics. We show, by the use of this model, the eﬀect of modest requirements drift on project cost and schedule and on the quality of the delivered product.
Finally we discuss how this insight can be used to better control projects and to mitigate the consequences of project failure.
2 The Characteristics of Large IT Projects and their Failures
2.1 The Characteristics of Large IT Projects There are a number of ways in which a large project is measured. Perhaps the most obvious is the cost. However, this measure cannot be considered in isolation. For example, one way to lower the cost of a product, is simply to deliver something of lower quality. Or, what amounts to the same thing, to deliver it too early. Consequently, when discussing large projects we need to consider a number of dimensions. Here, we distinguish Size, Quality, Cost, Schedule and Complexity. Before we discuss their relationship, let us ﬁrst deﬁne them.
Large IT projects are dominated by the need to deliver software, in all its many diverse forms.
Normally, contemporary projects involve a much greater proportion of existing code, that simply needs to be conﬁgured, than they do original code. The software engineers spent their time modifying some of this code, in particular setting parameters and writing scripts that will specialise the existing code to the speciﬁc requirements of this project. Thus, the quantity of delivered software may be measured in millions of lines of code, but only a few thousand of those lines will be new.
Nevertheless, the larger ﬁgure is an essential measure of size, since in many repects it quantiﬁes the amount of work that the engineers need to do in order to understand the functionality delivered by a given component before they can accurately specialise it. There are established means of calculating how much work will be involved in producing code [Boehm 1981] which have been reﬁned in recent 4 Why Large IT Projects Fail - http://www.ecs.soton.ac.uk/˜ph/LargeIT.pdf years to handle this contemporary case of building largely from existing components [Boehm 2000].
These methods require good historical data from similar projects, ideally those delivered by the same supplier. This data is seldom available. Other measures of size are necessary.
Once a project is complete, a reasonable measure would be the number of (hundreds of) engineeryears required to complete it. Clearly, projects continually try to estimate this ﬁgure from their inception, since it is a major component of the delivery cost. Where software is concerned, such measures are often wildly inaccurate. The Mythical Man-Month [Brookes 1979] tells us that, for many reasons, the delivery of a software product can require up to ten times the eﬀort of producing a proof-of-concept prototype. This ﬁgure is seldom believed by managers and is frequently the reason that initial estimates of cost are substantially too low. This can be thought of as an inability to estimate the size of the product accurately.
One conjecture is that the best measure of size for a project is to count the number of requirements.
Clearly the more requirements there are, the more work there will be to do, even if it is only to conﬁrm that an existing component does indeed deliver a given requirement. Of course, this begs the question of how big is a single requirement, and even whether that question is well-formed. But, assuming we can enumerate all requirements, collect them in a document and break them down into unit requirements (say, for example, something anticipated to require one engineer-month), then we would have a reasonable prediction of the size of a project. After all, this is exactly what we do in project management when we construct a Gantt chart and agree a delivery schedule.
Now we come to the measure of what it means for a delivered IT project to be of acceptable quality. When project progress is reported informally (in the press, for instance) it is often the case that the perceived end-user quality is unacceptable. The system may not deliver the anticipated functionality, it may be very slow or it may be very diﬃcult to use. It may even still break in the users’ hands, where residual bugs expose themselves too readily.
Considering only this extrinsic quality, we can imagine the end-user sitting down with a list of requirements, and analysing the system to see which of those requirements are delivered to their satisfaction. If we imagine them ticking those requirements that they ﬁnd acceptable, then we would eventually have a crude measure of (their perception) of the delivered quality. Optimistically we might hope that they had ticked 90%, 99% or even 99.9% of those requirements. These would be signiﬁcantly diﬀerent measures of quality, perhaps only the last of those actually being acceptable in a realistic application and often higher levels of quality are expected, especially in a safety-critical context. Moreover the cost of going from 90% to 99% can exceed the cost of getting to 90% in the ﬁrst place, and then getting to 99.9% can, in turn, exceed that. These diminishing returns are issues that we will return to later.
2.1.3 Cost Large projects cost large amounts of money. The recent project “National Programme for IT” in the UK NHS [NHS 2005] was originally costed at around £6,000m, whereas a recent oﬃcial report estimates that the eventual cost may be double that and some reporters have even predicted £20,000m before delivery is as originally planned [NAO 2006]. These are very large numbers, but the eﬀect is not uncommon, an initial estimate that doubles, then trebles and worse. Many large public works go over budget by similar factors, For example, in his report on the inquiry into the cost of the Scottish Parliament building, Lord Fraser states the inquiry was set up to evaluate a project “which is now two and a half years behind schedule with costs running approximately ten times more than the original estimate of 40m.” [Holyrood 2004]. While going over budget is not unique to the IT industry, it is increasingly familiar in respect of the size of the discrepancy between initial estimate and ultimate cost [Humphrey 2004].
In the case of large IT projects, the cost is largely for staﬀ. The procurement aspect of large projects is usually dominated by the cost of employing suﬃcient staﬀ. This in turn is estimated on the basis of perceived size and complexity of the building task.
Large projects are however normally the subject of competitive tendering and the initial cost proposals are clearly dominated by the need to win the bid. Purchasers of large IT will try to mitigate their risk by tying the supplier in to some punitive form of damages but often this just causes one or more of the consortium to collapse. The customer is left with a legacy investment and a partially completed project. The money has gone. The choices are stark. Invest more to continue (with new suppliers) or go without the solution. Eﬀectively the project starts over. A cynic might say that large projects require two or three such restarts until the organisation learns suﬃciently about the problem in hand, until more realistic objectives are established [Cambridge 2001].
Delivery on time is also unusual for large IT projects. Many of the steps in a development process are necessarily sequential. Software can’t be tested until it is built. Integrated systems can’t be built and tested until their components are available. Non-functional requirements (performance, human factors etc) can’t be tested until a realistic subset of the functionality is available and a large scale evaluation can be established.
The project will anticipate that many of these things can be overlapped. In theory, that will be true. In practice, it will be diﬃcult to achieve. Often we have a project that has been planned with dependencies on a number of key components (e.g. promised functionality in a new version of an operating system). Late arrival of the key components result in delays and reworking, adding to schedule slippage and overall cost.
Again, the Mythical Man-Month [Brookes 1979] tells us that person-months are not subject to arithmetic laws. A task may well require 12 person-months, but it may (for example) be that this necessarily is in the form of 2 persons for 6 months. Adding a third person will not reduce the schedule. Three people still require 6 months. Indeed, they may require longer, if the added person needs to be trained, since this calls time away from the initial two. This is sometimes quoted as “adding more people to a late project makes it later”.