«Abstract Large IT projects may not deliver what they promise. Often they are late. Often they are over budget. Often what is eventually delivered is ...»
The issue here is really that two inconsistent models are being reconciled. If both are subjected to modest change, rather than one being forced to ﬁt the other, the overall complexity of the problem to be solved will be minimised. This minimisation may be essential to success. The organisational learning required (on the part of the project, but in particular on the part of the customer) before this issue can be addressed may be the reason that large IT projects often undergo two or more “restarts” (or “stumbles”). The adjustments made at these events are often to persuade the customer to change their business processes to match those more readily supported by the IT.
If this had been accepted as necessary at the outset, then considerable cost and delay might have been avoided. This is particularly true for Open Systems, where maintaining the integrity of the interfaces, necessary for future evolution, often requires accepting that it is better to move the business processes to match the IT, than the other way around.
2.3.8 Legacy Initial underinvestment leads to an investment legacy, where the project has invested in bad decisions and doesn’t have the courage to retreat.
An issue that arises as a consequence of initial underinvestment is that, by the time that money is spent, there is a partial solution in existence. It may be that the optimal solution now is to throw that away and start again. But it is diﬃcult, if not impossible, to make such a decision. The partial solution becomes a legacy, in the derogatory sense of that word, which means that the eventual solution is constrained to include those bad decisions that led to problems in the ﬁrst place.
2.3.9 Senior Management There are many management disaster scenarios...
At the highest levels of project management, governance is always an issue. There are many projects which ﬁnd themselves in inappropriate governance structures, and the problems of delivery lower down the organisation are exacerbated by the uninformed decisions made at this level.
want cannot be had at the price they have allocated. This scenario is often seen in large government departments and similar publicly-funded organisations.
Another scenario, also common, we refer to as the enthusiastic builder scenario, where the supplier has a vested interest in prolonging the project, since payment is related to time-on-the-job rather than delivery of results. The supplier will be enthusiatic about agreeing anything with the customer, especially variations in the contract, since that will lead to increased income for the supplier. Where the supplier has a professional responsibility to warn the customer that the requirements they have speciﬁed may need to be revisited in future, since they are under-(or over-)speciﬁed, they may defer to the customers’ authority at the early stage since future renegotiation may lead to further work.
2.3.10 Requirements Explosion There is a requirements explosion during the lifetime of the project And of course, because of insuﬃcient planning and simply because projects take a long time from inception to completion, there is a requirements explosion during the lifetime of the project, which means that what is eventually required is signiﬁcantly diﬀerent from what was originally anticipated.
This requirements explosion is well known and is attributed to many things, some positive such as realising new potential in the IT that was not anticipated in the initial contract. Such things can be set aside until they are aﬀordable. More pernicious is the slow change in requirements throughout a project that we call requirements drift. We introduce this phenomenon in the next section, where we show that it can be seen as common to all of the reasons for project failure that we have introduced in this section
3 Requirements Drift
At the outset of a large IT project, even before contracts are agreed, there is a list of requirements stated by the eventual customer. The early stages of the project are all about reﬁning these requirements, at least to the point where teams can be organised and work can begin. A project plan enumerates the requirements and assigns eﬀort and timescales.
But the requirements are poorly understood. Even when the customer signs oﬀ a requirements document, it is unlikely that the customer’s understanding of what they will get coincides with the supplier’s understanding of what they will deliver. It won’t be until quite late into the project that a prototype is available and both sides can see the extent of this discrepancy. They will expect it, but they won’t be able to deﬁne it until that late stage.
As the project proceeds, the organisation, both customer and supplier, learns. One learns about the real potential of the IT, the other about the real critical business processes of the customer.
As they learn, they reﬁne the requirements. As they reﬁne them, the requirements drift. Existing requirements change, probably quite subtly. The text of the formal agreement may not change at all, but the understanding of what is meant will shift. New requirements are stated, either because they were overlooked at the outset or because some new aspect of the IT or the customer’s business processes is realised. Again, the addition of a single new requirement (when the list of requirements 13 Why Large IT Projects Fail - http://www.ecs.soton.ac.uk/˜ph/LargeIT.pdf may have thousands of elements) almost goes unnoticed. The requirements have drifted a little further from their original agreement.
Both of these phenomena, revising (our understanding of) existing requirements or adding new ones we refer to as requirements drift, choosing a word which reminds us that the individual changes are almost trivial in the context of the whole.
In the next section we discuss a simple simulation model which (if the simulation can be trusted) shows that a modest drift, where each requirement has a chance of changing of only 0.1% each day can nevertheless lead to a doubling of the cost and the schedule.
Before we look at those models however, let us revisit each of the reasons we have previously given for IT project failure and show how they relate to requirements drift. Then we will argue that, while these reasons will remain, approaching a project knowing that requirements are drifting all the time will help us to mitigate the problems.
Optimism The optimism with which a project begins is based on a (unsafe) conviction that the requirements are better understood than they actually are. As requirements drift slowly, this optimism is not diminished. Each day is not too diﬀerent from the previous day, so why should we be less optimistic.
Investment Because of our conviction that our grasp on the requirements is solid, we are conﬁdent that the initial investment is appropriate. We honestly believe that we can deliver (what we understand of) the requirements for this price. Requirements drift does not impact our belief that this investment is adequate for quite some time.
Procrastination But even when we realise part way into the project (for example, in the roll out of a ﬁrst prototype) that our optimism was misplaced and the initial investment was inadequate, we renegotiate a new investment (and a new schedule) and make the same mistake again.
Now we really believe that we have the project under control. But the requirements continue to drift.
Technical Know-how Lack of technical knowledge exacerbates the problem of believing that we understand the requirements. Indeed, requirements drift may go largely unnoticed simply because the observer isn’t suﬃciently technically competent to realise the extent to which a drifted requirement diﬀers from an old one.
Human Factors This naive view is actually encouraged by project planning and risk analysis.
Neither can be performed competently unless we believe we understand the requirements.
Moreover, project planning assumes that the requirements won’t change.
Incremental Delivery Of course, well managed projects plan incremental delivery because they understand that it is only when the customer begins to actually evaluate the product that the organisation begins to learn at anything like the necessary rate. Except we don’t plan incremental delivery. Because that would be too expensive. Why deliver one component every three months when we can deliver six in a year. And we can do this because we understand the requirements and they are ﬁxed (it says so in the contract). So we go for the big bang.
But by the time we do that, the requirements have drifted so far that, even if our product would pass muster against the original requirements, it won’t be acceptable because what the customer now wants is not what we thought they wanted at the outset.
14 Why Large IT Projects Fail - http://www.ecs.soton.ac.uk/˜ph/LargeIT.pdf Inertia Maybe it is the requirements that are wrong. The initial agreement was to use the IT to support the customer’s original business processes. But that is too hard. It would be easier to have the customer adjust their business processes to match what the IT can (and probably does) support. One of the directions of requirements drift is towards this obvious solution.
The customer is persuaded that doing things the IT way is a better option (in the sense that they will actually get something) than trying to automate the processes they previously followed.
Legacy When a large IT project goes through a reset, because it has realised that it needs to renegotiate cost and schedule, it inherits a legacy of all the design and building that has gone before. The drift in requirements more-or-less ensures that this legacy is far from an appropriate starting point. So now we have a new problem. How can that legacy be reshaped to supply the new (i.e. the drifted) requirements. It takes courage to be brutal with the legacy. We seldom have that courage. If we were more conﬁdent that the requirements drift has been inevitable, then that courage might be easier to ﬁnd.
Senior Management Good governance in the project is vital to its progress. Where the authority lies determines the direction the project takes. Where senior management believes it understands the requirements, it can compound the problems that arise from all the other factors. Senior management often make political decisions based on a weak understanding of the technology and the requirements. The fact that the requirements are drifting all the while goes even more unnoticed at this level.
Requirements Explosion Because IT seems to oﬀer so much potential, there is always the wishlist of new requirements that would be nice to have. These are the requirements that are properly considered and properly costed. If they are added to the project, the additional resources to deliver them are also added. These are not what we mean by requirements drift.
All requirements change subtly over time (even if it is only our late realisation of what was originally intended) so that, while we may end up with only modestly more requirements than we began, most of them will have changed suﬃciently that the cost of implementing them greatly exceeds what was originally planned. This is drift. It is drift that we fail to cope with, not explosion.
So, our argment will be that, while we must deal with all the above reasons why large IT projects fail, if we do so always bearing in mind that we have drifting requirements, we will be more cautious about how we proceed and will be more successful. Recognising the existence of requirements drift will mean that a major aspect of the project will be the continual attendance to (our understanding) of the requirements and our constant eﬀort to detect these small changes.
4 Models that support our argument
To study this phenomenon of requirements drift we have built a series of models using various simulation methods, in particular System Dynamics [Wolstenholme 1990], Analytical Models [Britton 2003] and Monte Carlo methods [Grey 1995]. These models have been of varying degrees of complexity, but all have shown similar behaviour which correlates reasonably with our practical experience [Chatters 1998].
15 Why Large IT Projects Fail - http://www.ecs.soton.ac.uk/˜ph/LargeIT.pdf
The model we will present here has the beneﬁt of being very simple. It shows the basic behaviour that we claim is seen in all large IT projects, but is suﬃciently simple that the underlying parameters can be explained without too much mathematical background. The method of simulation is basic Monte Carlo, where the data we present has been averaged over a suﬃciently large number of experiments. We have chosen only two sets of parameters to illustrate our point, since that is all we need. All our experiments with this model have shown behaviour that is reasonably represented by these two choices.
The model is shown in Figure 2. We have two signiﬁcant variables E and D, which respectively refer to the requirements that are expected and delivered. These variables are both lists, indexed by the identity of each requirement. They record whether or not that requirement is expected by the customer and whether or not it has been delivered.
We allow Di and Ei to change with probability, respectively, p and q.
The variable D is modiﬁed on each step with probability p. Only if an individual requirement is expected, will it be delivered and successful delivery will have probability p. The simulation we will illustrate assumes there are 500 of a possible 1000 requirements initially expected. For example if p is 1%, as it is in one of our scenarios below, and if we have around 500 requirements expected, we should deliver about 5 of these with each step, and it should take around 100 steps to deliver all of them (ignoring drift).
We have however built in a very elementary law of diminishing returns [Britton 2003] where the probability of successful delivery is modiﬁed by a factor determined by the way requirements to be implemented are chosen. All requirements are chosen, whether they have been implemented or not.
This models the fact that changing requirements may involve reimplementation. The net eﬀect, however, is that the number of unimplemented but expected requirements goes down. As it does, so does the expected number that will be successfully implemented at each step.