«A rapid evaluation of SHREWD: the Single Health Resilience Early Warning Database Erica Wirrmann Gadsby Linda Jenkins Stephen Peckham Centre for ...»
- When SHREWD does not include all the organisations in winter pressures discussions (eg London ambulance)
- When indicators are not seen as satisfactory or trigger levels are not correct
- Not using SHREWD completely or consistently
This rapid evaluation took place in November and December 2012, speaking to the developer of SHREWD and interviewing 21 people involved with SHREWD in Kent and Medway across a range of organisations and categories of staff. It is inevitably limited by the small number of informants and where there was under-representation (mainly from acute trusts) if interviews could not be arranged in the available time.
SHREWD was developed in Medway and then rolled out to Kent, so informants to the evaluation had differing experience with SHREWD ranging from almost no exposure to being involved for three years since its inception.
Key findings We found that SHREWD had met many of its key objectives since it was easy to use, it required little training, and it presented key information in a simple and clear manner. Users found it easy to access the dashboard from any location, and because all parties could view the same data it reduced the number of phone calls they made. Using SHREWD during teleconference calls reduced the length of calls, improved the quality of discussion and allowed minutes and actions to be recorded. Seeing the SHREWD dashboards allowed greater understanding of pressures elsewhere in the system and the impact of decisions on other organisations. It was described as being very helpful at times of pressure but had not been used in an emergency.
The cost of development, both in terms of actual expenditure on the system and the development time provided was not quantifiable and particularly in Medway, staff have invested substantial time in supporting development. Savings in staff time have been made as using SHREWD approximately halved the length of teleconference calls, and freed up time to focus on and have more effective discussions on managing pressures. Small gains were also reported at other times as people at all levels could view the dashboard to check the situation and that reduced the number of phone calls to ask for or check the data. Few respondents referred to resource issues and focused more on the use of the system.
Negative user experiences were more likely to be voiced in the roll-out areas, where SHREWD had not yet been fully integrated into existing ways of working, and was sometimes felt to be partially duplicating other information systems. The dashboard was also criticised due to data not being updated regularly, when indicators and trigger levels were not seen as correct, and when it did not cover all the information they would like.
A number of aspects seem important to successful implementation and use of SHREWD. These include having central direction, assured funding and having responsive IT support. Spending time discussing what information is to be displayed and establishing the correct trigger values for red/amber/green/black rating is essential, and a substantial amount of time has to be allowed for this part of the process. Having accurate and timely data is also fundamental as SHREWD becomes less useful when data are out of date.
SHREWD is not yet being used to capacity and can deliver further benefits, for example when uploading of data has been fully automated (even though manual uploading requires only a small amount of effort), and as data builds up over time it can be used for more strategic purpose such as reflection and audit, and for prediction and planning.
Within this evaluation it was not possible to explore (i) how much it costs to implement SHREWD, and how that balances against gains in efficiency and productivity and improvements in patient care and experience, (ii) where the benefits arising from using SHREWD are most felt, and (iii) which of the identified benefits can be evidenced by real examples, and which are hypothetical.
Further issues and questions
This study has raised a number of questions relating both to operational and evaluative aspects of SHREWD. On the operational front it is not yet clear how well SHREWD will fit into the new NHS architecture, who will be the lead organisation responsible for implementing and maintaining SHREWD, and how SHREWD licensing, development and associated costs will be met.
Questions for further evaluation are:
- In implementing SHREWD, who are the most important actors?
- What factors limit people’s engagement with SHREWD?
- Does SHREWD lead to more efficient sharing of information?
- How reliable / accurate is SHREWD data? To what extent is it ‘real time’?
- How is the information on SHREWD being used, and by whom? What are the impacts of this?
- To what extent and how have communications between key users of SHREWD changed?
- Does SHREWD enable more informed decisions to be made at times of system pressure?
- What actual benefits are there, and who (which organisations, which levels) is benefiting most/least?
- What are the key aspects of added value (entailing some comparison with how things were done before SHREWD)?
- Is SHREWD enabling lessons to be learned over time?
- Is SHREWD being used proactively, to foresee and prevent the worsening of system pressures down the line?
There is a need to carry out more detailed evaluative work, for example a productivity assessment to understand the impact of SHREWD on planning and managing winter pressures.
An analysis of benefits and quality improvements seems more valuable than gathering and analysing detailed costs. Further consideration should also be made of the issues around successful implementation, since the evaluation showed clear differences between the site where SHREWD was developed and other areas when it was rolled out. Other issues around successful implementation include central direction, local buy-in, and time to agree satisfactory indicators and trigger levels. A detailed evaluation is also needed of data quality on SHREWD, covering the frequency, timing and accuracy of data uploaded.
In summary this evaluation found that two years after its implementation, SHREWD had achieved many of its key objectives and there are indications of an improvement in the quality and efficiency of managing winter pressures in the site where it was developed. It was thought that there were further gains to be achieved over time. The evaluation uncovered a range of issues when SHREWD was rolled out to larger and more complex geographical areas. Further evaluative work would be worthwhile to understand the reasons behind these findings more fully.
Appendix: Interview Guide Research questions
- What issues have arisen from the implementation and operation of SHREWD?
- To what extent and how has SHREWD contributed to improvements in the quality of coordination and planning processes?
- What are the cost/resource implications of implementing/using SHREWD?
1. How do you use SHREWD?
When did you first start using it?
How was it introduced? (training? Etc) How have you found using it?
Have you experienced any benefits of using it?
Have you come across any problems or disadvantages of using it?
Are you aware of how others use the system? (and what impact it has had?)
2. Do you think using SHREWD has made the sharing of information easier? How? (example?)
Has it improved your ability to monitor the situation on a day to day basis?
Has it helped you to make decisions when responding to an emergency?
Has it helped you to think more strategically about how to manage system pressures?
Has it helped you to reflect on the way emergencies / system pressures are handled?
3. Are there any local costs incurred through using SHREWD? (annual fees, IT costs, etc.?) How has using the system impacted on your role (or that of others you know of)? (time spent doing tasks?) Are you aware of any ongoing cost or resource issues? (any training, development inputs, software/hardware needs? etc?)
4. What further developments do you think are needed to the system?
Any other comments (3 key benefits and 3 key problems/concerns – if not already covered above)?