«A rapid evaluation of SHREWD: the Single Health Resilience Early Warning Database Erica Wirrmann Gadsby Linda Jenkins Stephen Peckham Centre for ...»
The interrelationships between different organisations, and the way pressure moves through the economy as a whole, were discussed by several interviewees who were interested in potentially seeing whether there are any patterns in the way pressures move – for instance, whether Medway sees pressure rise in a particular area a day or two before West Kent. This sort of analysis would be helpful in predicting and perhaps mitigating against the rising of pressures within organisations or local areas in the future, but would only be possible using the data captured over time within SHREWD.
Problems with or disadvantages of using the system
Interviewees were asked to identify any problems with or disadvantages of using SHREWD. No actual (as opposed to potential) disadvantages of using it were identified. However, a number of issues were raised. Many of these were ‘teething problems’ associated with rolling out a brand new system, and these were felt to be quickly overcome and largely not problematic. These include occasional difficulties logging in, and a few ‘niggles’ and ‘glitches’ often attributed to system changes or upgrades.
Most other issues relate not to the system itself, but to the implementation of it. They point to some important areas of consideration for future improvement or further roll out (see section 4.4).
Interviewees at the operational and tactical levels raised a number of issues that can be grouped into four broad areas. The first was concerned with the importance of, and difficulties with, getting the indicators and trigger levels right. It was clear that having the right indicators and trigger levels within SHREWD was crucial to achieving the benefits of the system, and important in creating ‘buy-in’ from all partners. The process of identifying and agreeing indicators and triggers is lengthy, but cannot be underestimated. It has been described as an important process in itself, but it is also important to get right.
In Medway, there was general agreement on the indicators and trigger levels, and they appeared to be working well. In other areas, they were not yet right. In one organisation, indicators had become less relevant as the organisation had changed over time. In another, it was felt that more indicators were needed to give a clearer picture. In primary care, one interviewee said that more detailed information is needed, and more primary care indicators would be extremely helpful. For example, he explained that it is not particularly useful just to know the proportion of GP practices currently closed; it is important to know whether they are from one particular area (which would have profound consequences), or are spread across the patch (which would not be so significant). However, this is difficult to achieve because of the number of sites involved, and because of the nature of the independent contractual relationship with GP practices, which means they are limited with regards to what information they can get from all practices.
Getting the trigger points right is also important. Since the RAG system is such a simple and visual one, it can easily give the wrong impression if it is not right. It is clear that at the time of our evaluation, trigger levels were still not right in some organisations. This led to one interviewee (tactical level, acute site) ‘switching off’ from the system, and repeatedly stating that he did not use the system because he felt that the data was unreliable (although when questioned further, it was clear that it was not the data that he did not trust, it was the RAG status automatically attributed to that data). He was irritated that SHREWD was giving people the wrong indication of what was going on in his organisation.
An interviewee in a non-acute organisation also commented that hospital trusts frequently ‘declare black’, and that this frequent occurrence can lead to a ‘desensitisation’ to it. He stated the importance of being able to distinguish between ‘real crises’ and ‘a bit of pressure’. Several interviewees stressed the importance of not only getting the trigger levels right internally, but also of having comparability across organisations – so that red in one organisation meant the same as red in an equivalent organisation. One interviewee felt that the issue with the trigger points was the biggest problem with SHREWD, because it “camouflages everything else”.
The second broad area concerns the collection of regular and routine information. Several interviewees remarked that the system is only as good as the information in it. There were some clear problems with data not being entered regularly and reliably, although interviewees frequently stated that the automation process will address this, providing the process is robust.
Strong leadership from executive (strategic) level has also improved engagement with the system at tactical and operational levels. Whilst the system is not providing real time information, one interviewee remarked that it is closer to real time than they have ever been.
The third broad area concerns accessing and using the system. Whilst access to the system was felt to be good, it does depend on an internet connection, which isn’t always available. In addition, the networks in the NHS are sometimes slow, which can hamper access, and once or twice the system has been ‘down’. The majority of interviewees found SHREWD easy to access and use. Only one interviewee found the application to be slightly ‘clunky’ at times, with a few ongoing ‘glitches’.
The final grouping of issues is associated with ‘data overload’. Several interviewees referred to other information sharing databases, and their overlaps and potential links with SHREWD. The roll-out of SHREWD across Kent occurred at roughly the same time as the implementation of QlikView in one acute trust. QlikView is being used as an internal application to analyse and report on live data from wards and A&E to help manage resources and demand. Several interviewees also commented on the Ambulance Service’s data dashboard which enables them and hospitals to track ambulance activity. Capacity Management System (CMS) and the GP Management Information System were also mentioned as tools that provide information on resources, demand, and patient pathways. The linking of these systems was seen to be important and, whilst it is not usually technically difficult, it was recognised that organisational and professional hurdles often need to be overcome. The issue is complicated further when organisations work across borders, or when organisational boundaries do not match.
Just one potential disadvantage of using the system was raised by one interviewee (strategic level, ambulance), who felt that there was a potential danger in seeking to reduce the amount of discussion that occurs during conference calls. He placed great value in communication, and felt that some understanding or detail might be missed if people become overly reliant on the dashboard.
The impact of SHREWD
Since SHREWD is not yet fully utilised, its impact so far will only be partial. Indeed, within interviews at the strategic level there was a general sense that the potential of SHREWD had not yet been realised, and interviewees tried to explore why that might be. All attributed this to the way the system was being used (or not being used), rather than to the system itself. One interviewee was frustrated by the apparent reluctance from some people to engage with it, despite a general positive response regarding the principle of having near real time data in the system. It was suggested that at the moment, people have mainly been “going along with it”;
not really seeing what they’re getting out of it, but engaging with it because there is “a three line whip from the PCT”. Two interviewees pointed out that it is not until people have used it for a while, and really engaged with it, that they will be able to see what is and isn’t working,
and will work to improve it and get it right:
“People aren’t using it as they should do because they’re not used to it, and they’re not getting used to it because they’re not using it” (strategic level).
One interviewee pointed out that SHREWD is a different way of working for people. To date, it has been reliant on being driven by the commissioner, who hosts the conference call and produces the minutes. It is therefore reliant on a high level of aptitude, and a strong engagement with SHREWD, from the person in that position.
It was felt by several interviewees that the system can only have an impact if everyone believes the data in it, and understands and agrees that the data is reliable. This is not yet the case (at least in Kent), although this might change rapidly as the automation of data input for key indicators is realised. One interviewee (at strategic level) felt that managers in different organisations are suspicious of each other and are reluctant to trust each other’s data. This can act as a barrier to accepting SHREWD and what it is trying to achieve. An alternative opinion, however, was that organisations are now “past that”, and that SHREWD was consequently believed to give them “one version of the truth”.
It is clear that SHREWD is not yet providing real time information, but the information it provides is closer to real time than ever before. This is also changing rapidly as the rollout of automatic updates progresses, and deserves to be re-assessed in a few months’ time. The range of indicators within SHREWD are broadly agreed by partners, although some individuals would like to see more indicators or different indicators. It was recognised, however, that there is a balance to be made regarding the level of detail and the usefulness of the dashboard.
SHREWD is clearly accessible to a range of people at all levels, remotely via the use of mobile technology (one interviewee described how she chaired a teleconference call from her parked car mid-way through a journey). However, the information that is accessible varies in usefulness. Repeated observations of the dashboard found that information in some organisations was often not updated for a week or more. There were also indications that some information could be misleading, particularly regarding the RAG status of organisations that had not yet identified the right trigger levels. During the winter planning exercise, it was observed that a few participants had difficulty logging in to the system, and/or had difficulty navigating round the system once logged in (although for some people, it was the first time they had viewed SHREWD).
This evaluation found that the process of multi-agency conference calls had been streamlined considerably in Medway, where the system was most effectively used. In other areas, it was less clear, either because interviewees were not familiar with the process pre-SHREWD, or because they were not using the teleconferencing facility or the dashboard in the way it was intended.
In one observed teleconference call, although SHREWD was theoretically being used, participants read out all their data without referring to the information on the dashboard (which in some cases was different). The chair of this call, however, felt that they are only now starting to use the system, and that their use of it is changing weekly as they become more confident of the data, and as they become more comfortable with the system.
At this stage, it is not clear that SHREWD is providing a proactive view of stress within a health economy, individual organisation or area of practice, although there are indications that in Medway it is starting to show potential for this.
In the remainder of this section, we discuss the extent to which SHREWD has contributed to improvements in the quality of co-ordination and planning processes, and made it easier to manage pressures within the system.
More specifically, we consider whether SHREWD has:
- made the sharing of information easier
- improved users’ ability to monitor the situation on a day to day basis
- helped users to think more strategically about how to manage system pressures
- helped users to reflect on the way emergencies/system pressures are handled.
Information sharing There is general agreement that the SHREWD dashboard is a better way of sharing information than exchanging information over a phone call. However, whilst some organisations stand to gain from this access to information, others don’t particularly feel any benefit. Whilst it is easier to see the information, this doesn’t necessarily have a great impact for all organisations; it doesn’t eliminate the need for follow up phone calls to, for instance, confirm that a bed is indeed available, or to organise the transfer of a patient.
One provider felt that the data is more meaningful on the dashboard since they get a sense of what the numbers mean – for instance, whether 3 beds remaining is good or bad. Several interviewees felt that a start had been made, that some small benefits could be seen, but that in some areas, more information, or more useful information, was needed. It was felt by some that the information presented by SHREWD can be a bit superficial, particularly if you don’t understand the business of the provider, and what each indicator and trigger actually means.
Several interviewees felt that their own business wasn’t necessarily completely understood by other people viewing the dashboard. They indicated that unless there is an understanding of implications, having access to the data has limited impact: “why do I want to know if the ambulance trust is red? What does that mean to me?” (tactical level, acute). It was clear that SHREWD cannot replace the ‘human component’.
Several interviewees felt that SHREWD provides a consistent and standardised approach to information sharing. This has worked well in some areas, but one interviewee felt that this ‘regimental’ approach is not as suitable for the needs of other economies.
One interviewee in primary care indicated that data is not only shared amongst those who view the dashboard; he said that when he sees on SHREWD that hospitals are under pressure, he can then inform GP practices, so they are more aware of what is going on, and to remind them to use alternative pathways.
There is a general sense from interviewees at all levels that they are using SHREWD to share information within the health economy, and that SHREWD has made this sharing of information easier.