«A rapid evaluation of SHREWD: the Single Health Resilience Early Warning Database Erica Wirrmann Gadsby Linda Jenkins Stephen Peckham Centre for ...»
Methods This evaluation was conducted by two researchers experienced in health systems research, working with the Director of CHSS, over a period of 23 days in November-December 2012. We collected data using recorded semi-structured qualitative interviews. In addition, we made a number of observations of the SHREWD dashboard, of teleconference calls and minutes using the SHREWD system, and of a SHREWD-based winter planning exercise. For each of these observations, we made notes, and used the findings to supplement and cross-examine the interview data.
In planning interviews, we developed a purposive sampling frame to ensure we collected data from a range of organisations within the health economy, and from a range of people at different levels across the organisations who were involved in: the development of SHREWD;
inputting data to SHREWD; using SHREWD for co-ordinating purposes; and using SHREWD for strategic decision making. We refer to three broad levels as being: operational (including those who input data into SHREWD), tactical (including those responsible for winter planning and managing seasonal pressures) and strategic (including those who operate across organisations and across a larger geographical area). In addition, we made a further distinction between sectors, and categorised people as belonging to: commissioning, acute care, community care, primary care and ambulance services. We hoped to achieve a good spread of interviewees across each of these levels and sectors, and individuals were selected and approached on this basis. However, a number of individuals (particularly within the acute care sector) did not respond. Given the tight time-scale of this project, we were unable to chase further or select alternative interviewees. Additional interviews in each of the sectors, but perhaps particularly in acute care, undoubtedly would have enriched our data and may have led to further conclusions.
Interviewees were asked a set of open-ended questions guided by the three research questions (see Annex A for interview guide). For some interviewees (particularly those at the operational level who only used SHREWD to input data), not all questions on the guide were relevant. A total of 21 interviews were conducted with SHREWD users (see table 1), which ranged in length from 15 minutes to 90 minutes and were either face-to-face, or over the telephone. All interviews (except one) were digitally recorded to aid note taking.
Findings General experiences of using Shrewd There was a range of experience of using SHREWD across the research area. There are clear differences between Medway, where the system was first developed over two years ago, and where it has been used routinely for over a year, and the rest of Kent. This section starts by describing implementation and use of SHREWD, followed by interviewees’ comments on the benefits and problems from using it.
SHREWD was developed in Medway, an area with coterminous organisational boundaries, a unitary council and a single acute hospital, to meet their identified needs, and to match their ways of working. The development team at Transforming Systems have worked closely with the users in Medway to continue to refine and improve the system. In Medway, data is routinely uploaded by a number of individuals in each organisation to the SHREWD dashboard. At the time of the evaluation, data input for key indicators for the ambulance trust and the acute trust was also in the process of being automated. The system is fully used in regular teleconferences during winter (twice weekly, or more if necessary) and some staff have continued to use it throughout the year for monitoring. The teleconference calls are chaired by the urgent care commissioner and involve middle to senior managers of the local council (social services), the acute trust, the ambulance trust, the community trust and any relevant neighbouring organisations. At times of pressure a once a week executive level conference also takes place to pick up issues over a wider geographical area.
The economies of North, West and East Kent cover a population that is four times that of Medway, with three acute trusts (and three district general hospitals in East Kent alone) and the need to take into account pressures across boundaries with London, Surrey and Sussex as well as Medway. SHREWD was introduced to the Kent economies as a partly developed system, with some key indicators pre-identified. There were further discussions across Kent to agree indicators and trigger levels, but these areas have had less involvement in the system’s development as a whole. The process of implementation in Kent and Medway differed considerably. Whilst there was an expectation that SHREWD would be rolled out across Kent relatively quickly once senior executives had bought into it, actual implementation was much slower. Consequently, there has been less time to gain experience of using SHREWD and there is not the same degree of ownership of the system in Kent compared to Medway. At the time of the evaluation, organisations in all three Kent economies were uploading data manually onto the SHREWD dashboard on a reasonably regular, though not routine basis. As in Medway, the process of automating data input for the acute trusts and ambulance trust was being rolled out during the period of evaluation. Only one of the three Kent economies was making use of the SHREWD teleconferencing facility.
Key benefits of using the system
Interviewees across all levels and sectors identified many benefits of using SHREWD, most of which impact on users at tactical and strategic levels. However, the system is in its infancy in the sense that it has only been in use for a short period of time (particularly within Kent), but also in the sense that it is still evolving (a new version, SHREWD version 2 was rolled out at the start of our evaluation, and another is expected soon). Consequently, SHREWD (or at least the latest version of it) has not yet been used on a regular or routine basis across Kent and Medway.
Some of the benefits identified therefore are potential, rather than actual. It was very clear that interviewees unanimously felt that SHREWD has good potential benefit, but many mentioned various caveats to this: for example, if data within it is reliable and accurate; if data is updated regularly; if people trust the data; if people use the system proactively, to foresee and prevent the worsening of system pressures down the line; if people use the system for teleconferencing and recording minutes and actions; and so on. Given that these issues cannot yet be taken for
granted, most actual benefits were identified as ‘hit and miss’. One commissioner commented:
“If all the information on there is accurate, it’s like a balance sheet. It’ll tell you exactly how the system is at any one given moment. So you see if it’s red, green, black – in a second, you can see how bad your system is. I can’t underestimate how valuable that is.
Without SHREWD, we would have that, but we would have it through ringing around individually, and eventually coming to that assessment. So that single point of information is brilliant.” (Tactical level commissioner).
It is important to get a sense of what people at tactical and strategic levels within the Kent and Medway health economy had without SHREWD. In most interviews, it was clear that SHREWD had added value to some degree. One strategic level interviewee, who had been involved in helping to implement the system, felt that before SHREWD, it was difficult for commissioners to
get any idea of how to improve things when pressure was building in the system:
“By the time they got the information, it was out of date, and it needn’t necessarily reflect how it was going to be tomorrow. They got a sense that some provider organisations were having a hard day, but by the time they got any real sense of the issues, it was too late to do anything – they were always 24 hours behind. With SHREWD, the data sharing, and discussion of the issues, is quicker. In Medway, it is clear they have been able to move more into talking about real time, and making a difference.” (Strategic level interviewee).
One of the key benefits of using SHREWD was clearly around having the information on a dashboard that was accessible and easy to interpret. Interviewees appreciated the broad overview of information on their own organisations, and on other organisations within the
“Having the basic information from all the providers in one place helps to understand what the pressures are in the system.” (Tactical level commissioner).
A strategic level interviewee who appreciated that all data in all areas cannot yet be trusted entirely, commented that even when you don’t simply trust all the data on SHREWD, it is a good prompt for further questions.
Those who don’t need to have this information themselves (such as operational level interviewees who only used SHREWD to input data) saw the benefit of having the information available to others. One operational level interviewee in Medway felt that using the system had reduced the number of phone calls and the sense of panic within her area of the organisation, as there is more understanding around issues such as delayed discharges.
A further benefit related to the way in which the information was presented; the visual representation, with arrows to indicate changes in pressure within indicators, was felt to be useful. The RAG rating (the colour coding of each indicator) also helped to put numbers into context – to understand whether a particular situation was worrying or not. This seemed particularly useful for users in non-acute sector organisations who are less familiar with bed numbers and A&E targets.
Other benefits related to what can be done as a result of having the information presented on SHREWD. There were several examples of how having the information on their own and other organisations had proved useful. SHREWD has, for instance, helped a manager in social services to spot things like problematic sickness affecting their staff; a manager in community health care said it has helped to see potential points of concern and to allocate staff accordingly; and it has helped a manager within the ambulance service to think about things they could do around patient movement to help relieve pressure. Similar discussions were observed during the winter planning exercise and the observed teleconference call where, after viewing the dashboard, participants were making suggestions about how they could help relieve pressure, mitigate escalation, or improve patient flow.
Having the information collated and presented on the dashboard has also streamlined the conference call process – particularly in Medway, where they are more used to using this function. Several interviewees noted that having the data on the dashboard in advance of the phone call is “undoubtedly helpful”; it is no longer necessary to write down the figures by hand whilst on the phone, and it enables people to focus on the issues, and discuss causes and solutions. One interviewee (from Medway) suggested that a knock-on effect of having conference calls that are shorter, more organised and “less of a pain”, is that that more people are now reliably dialling in to them: “these calls have tended to bring people together, so we build those relationships” (tactical level, primary care).
Having the information available on SHREWD also allows users who log-in to see when there is a problem that might escalate. A tactical level manager in social care commented that he can see from the dashboard when a hospital trust is “stuffed to the gunnels”, and they can actually get on to the actions that need to follow from this. An interviewee from an ambulance trust commented that using SHREWD “gives her a head start” to knowing which hospitals are getting into trouble, so rather than having a phone-call when the issue has arisen, it allows pre-planning of ambulance movements to avoid a problem.
In addition to the presentation of data, some of the other functions within SHREWD were pointed out as being useful – for instance, one interviewee liked the fact that the system will send text messages alerting you of a potential problem (like an organisation escalating to black).
Several interviewees (in Medway) valued the fact that when the teleconference function within SHREWD was used, it enabled the minutes from those meetings to go out quickly, containing all the most important information, giving people a record of the organisations’ pressures, and related actions, within hours of the teleconference meeting. The fact that these minutes, with clearly allocated actions, are all recorded and stored within the system, was also thought to be useful, since it can provide an ‘audit trail’.
There were some benefits of using the system that were slightly less tangible, but which would be interesting to explore further. For instance, one interviewee from within a community hospital felt that having her organisation’s data visible on the system gave her site “more credibility”: if they are ‘black’, others can see this, and this is accepted. There was also a feeling that use of SHREWD has helped to create an awareness of the whole system, by giving a more rounded picture of what is going on. A tactical level interviewee in primary care felt that this puts the economy at a “better state of readiness”, and can also help people to see knock-on effects – for example, if an A&E department is busy, patients are being diverted, and there is an impact on social care in terms of staffing levels and allocation. An interviewee from within the ambulance trust felt that SHREWD gives her an idea of how pressures elsewhere will impact on their service.
There was evidence that individuals who did not necessarily need to look at the data in their own and other organisations and economies often did so anyway – sometimes described as ‘being nosey’, but also credited for giving them a sense that they are ‘not alone’ in feeling pressured. The general sense of being one small part of a much wider system might have implications which need to be explored further.