Article Text
PDF
Abstract
Objectives The current UK healthcare workforce crisis is particularly severe in community services. A key limitation with traditional service-delivery models is the reliance on practitioners with levels of training and experience to enable them to operate independently. This paper describes a real-world evaluation of the implementation of digital health technology designed to provide remote, real-time support and task delegation in community palliative care services. It explores the ability of technology to support sustainable community workforce models and reports on key indicators of quality and efficiency.
Methods The study was a mixed-methods, theory-driven evaluation, incorporating interviews, observations and analysis of routine data. The focus of this paper is the reporting of findings from pre–post implementation comparison and interrupted time series analysis. Data include community hospice service visits, hospital use by hospice patients and patient reported experiences.
Results The digital health intervention allowed the service to include a more junior workforce (p<0.001, Cramer’s V=0.241), requiring fewer joint visits (p<0.001, Cramer’s V=0.087). No negative changes in hospitalisation were observed and patient reported experiences improved (p=0.023). Changes in hospital non-emergency bed days were inconclusive. However, emergency department admissions reduced significantly (−76.9 /month at 17 months, p=0.001). The cost per hour for visits reduced from £16.71 to £16.23 and annual savings of £135 153 are estimated for reduced emergency admissions.
Conclusions The evaluation demonstrates the value of digital innovation to support programmes of service redesign and begin to address the healthcare workforce crisis, while having a positive economic effect and indicating an improvement to patient experiences.
- hospice care
- hospital care
- service evaluation
- clinical decisions
- communication
Data availability statement
Data are available on reasonable request. The total monthly admissions data used for the interrupted time series analysis are available on reasonable request from the lead author: SMBA, s.ariss@sheffield.ac.uk ORCID ID: http://orcid.org/0000-0002-5557-4613.
This is an open access article distributed in accordance with the Creative Commons Attribution 4.0 Unported (CC BY 4.0) license, which permits others to copy, redistribute, remix, transform and build upon this work for any purpose, provided the original work is properly cited, a link to the licence is given, and indication of whether changes were made. See: https://creativecommons.org/licenses/by/4.0/.
Statistics from Altmetric.com
Key messages
What was already known?
The workforce crisis in community health services is compounded by the lack of access to real-time medical and senior decision-making and the requirement for nursing staff to have a wealth of experience and training in order to work autonomously.
What are the new findings?
This study demonstrated that digital health technology, designed for remote working, can support flexible, scalable models of community health services, involving a more junior workforce. Concurrent improvements in patient experience and reduced use of emergency services were observed.
What is their significance?
Clinical
The clinical significance of these findings is that specific digital technology can support a larger and more diverse community health workforce, while improving key outcomes, such as patient experience and more appropriate secondary care.
Research
This study provides a framework for real-world evaluations of the technology in a single setting. However, the intervention would benefit from further research, focused on implementation in other settings, wider outcomes and comprehensive economic impacts.
Background
The increasingly central role of UK digital health technologies (DHTs) was reinforced in a ‘The future of healthcare: our vision for digital, data and technology in health and care’ (2018), which outlines ambitions to maximise the impact of DHTs.1 Alongside this ambition, NICE (UK National Institute for Health and Care Excellence) evidence standards support methods to implement and appraise them.2
With increasing importance, clinical nurse specialist roles complement traditional medical and nursing structures.3 Community and hospital liaison services for palliative medicine rely on specialist nurses working under senior guidance,4 while site-specific cancer nurses provide an essential contact for service users.5 Comparable roles have been established in most other clinical fields.3
However, one challenge in community services is the reliance on practitioners acting independently. Furthermore, the dependence on medical visits, the high banding of senior nursing staff, coupled with extensive investment in training for independence, means care is; costly, inflexible, difficult to scale and inefficient. More generally, community services are struggling with recruiting, funding and retaining staff.6 7
The Topol report suggests that technologies will enhance healthcare professionals in 90% of all jobs.8 To begin to address these challenges, we report the implementation of a new model of care (Enhanced Community Palliative Support Services (EnComPaSS)). In this model, technology aims to support the workforce of a specialist community palliative care service to become more efficient, sustainable, scalable and able to generate systemic cost savings.9
The model uses eShift, to incorporate a delegated model of support into patient care.9–11 We report on a mixed-methods real-world evaluation of eShift at a specialist community palliative care service in England. This paper focuses on the key quantitative results from the impact evaluation. The process evaluation and qualitative findings will be published separately. Implementation was funded by the Nursing Technology Fund (National Health Service, NHS England) and the evaluation was funded by the NIHR (National Institute for Health research) CLAHRC (Collaboration for Leadership in Applied Health Research and Care) for Yorkshire and Humber.
The study population consists of adults (>18) referred to the service from primary and secondary care. The majority (approximately 80%) have a diagnosis of cancer. Prior to the intervention, care was delivered via clinical nurse specialists and medical staff (consultants in palliative medicine, specialty doctors and trainees under supervision); visits would be planned in advance, paper records used in the field, and telephone calls made for senior support. Accessing or updating electronic records was undertaken ‘at base’ before or after visits.
The eShift technology11 is a cloud-based platform, which facilitates structured, directed and supported activities using standard remote devices (such as smart phone, laptop or tablet) for accessing and updating clinical information in the field. It uses forms, text and audio communication to allow; sequencing of routine tasks, and instantaneous multidisciplinary (MDT) decision-making and task-allocation. In addition, eShift allows professionals to seek advice from senior medical and nursing colleagues at base in a ‘delegating’ role, while maintaining a clear electronic record of interactions. Delegating clinicians can follow the progress of members of staff in the community as, for instance, they arrive for the visit, complete assessments, review medications and care-plans; additional tasks can also be requested. The service in question has incorporated the Outcome Assessment and Complexity Collaborative (OACC) suite of patient-reported outcome measures into the system.12
Aims
The aim of this study was to describe and evaluate changes occurring around the implementation of the eShift digital health system at a specialist community palliative care service.
Methods
The technology was implemented as part of the EnComPaSS project.9 A real-world, mixed-method, theory-led evaluation was undertaken; exploring change over time of key variables hypothesised to be affected by the intervention. The evaluation team consisted of external evaluators (SMBA, DF and SM), a clinical academic with joint posts in the evaluation team and the service (PT) and the Medical Director of the service (SK). The data manager for the service assisted in data retrieval, cleaning, screening and configuration.
The study developed hypotheses about why and how specific outcomes might be expected, and used qualitative and quantitative evidence to test hypotheses and demonstrate generative and correlational13 causation. Potential confounding factors and unintended consequences were explored through extensive interviews, observations and analyses.
As a service evaluation of wholesale change at a single site, neither randomisation nor external control groups were feasible. Therefore, pre–post comparison and interrupted time series (ITS) analyses was undertaken. Fisher’s exact test was used to examine the significance of the association between pre- and post-intervention categorical measures.
ITS analysis is useful where random allocation is not possible or desirable.14 This approach controls for confounding variables and regression to the mean through comparison of a number of preintervention and postintervention measures. It also detects changes in the slope of a measure as well as changes in the level, thereby indicating immediate and gradual effects.
The service covers a population of 575 000. Anonymised, aggregated, routinely collected hospital data for the time-periods of interest were extracted for analysis in two stages. First, pre–post analysis provided interim findings. This was followed by an ITS analysis, based on a 36-month period. Data were collected from June 2014, transition to the new model began in October 2015 with changes to MDT meetings. A 3-month small-scale implementation (with four nurses) began in January 2016, with a fully ‘live’ system from March 2016. Data collection continued to the end of May 2017 (illustrated in figure 1).
Implementation and data collection processes. T1 and T2 refer to the data collection periods for the pre–post analysis of community visits, views on care and pre–post descriptive hospital admissions. ITS1 and ITS2 refer to data collection periods for the ITS analysis. ITS, interrupted time series; MDT, multidisciplinary task.
Variables for analysis were proposed in advance (see online supplemental data). The starting point was the development of a ‘benefit tracking’ spreadsheet for the project funding scheme (Nursing Technology Fund). Certain outcomes were unlikely to be altered as a result of the intervention, and were designated as low priority. The analyses for this paper focuses on four key components of the evaluation framework; (1) impact on type and volume of visits to community patients, (2) impact on patient-reported quality of care (3) impact on hospital admissions and (4) impact on costs of hospital admissions and community care. These were chosen through a process of theory testing, based on benefit tracking, implementation chains and information requirements of key stakeholders.
Supplemental material
This study was undertaken as a service evaluation using anonymised routine data.
Analysis plan
The study population are hospice community service users for the time periods under observation. The hospital admissions data for all service users were retrieved from hospital records of the three adult hospitals in the city following their first admission to the community service.
‘Visit’ data refers to visits by all members of staff in the community service. The data cover 20 months in total (T1=March to December 2015, T2=March to December 2016). This presents comparable data from sequential years, excluding the workforce transition from January 2016 to March 2016.
The pre–post hospital admissions and views on care (VoC) interim analyses were conducted to monitor for negative changes in care quality and patient experiences. These cover two adjacent 12 month time periods (T1=October 2014 to September 2015, T2=October 2015 to September 2016). The preimplementation and postimplementation hospital admission data are presented descriptively.
Patient experience is measured using the VoC questions; part of the (OACC) suite of measures.12 15 The OACC collaborative (led by the Cicely Saunders Institute) has collected a suite of outcome measures for palliative care services. The VoC module assesses patient’s quality of life, the impact of services and overall well-being.
To account for natural variation in absolute quality of life data, we focus on the VoC question ‘Taking everything into account, do you think the palliative care team is making a difference to how things are going for you at present?’ which returns a five-point scale . We include a subset, recording only the second visit for each spell of care, to minimise skewed data from spells with a large number of scores. Data are aggregated into positive or negative responses to undertake a Fisher’s exact test.
Hospital admission data were also used for a segmented ITS analyses. This involved a regression model of outcome values against time, to analyse data preceding and following the intervention. By comparing values and rates of change of outcomes either side of the intervention, ITS addresses the potential impact of natural fluctuations in data that might otherwise be attributed to the intervention, or lead to over or under-estimation of effect.
The ITS analyses focused on: (1) total admissions to hospital emergency department (ED), (2) non-ED total bed-days and (3) hospital admissions via the ED as a percentage of all admissions for community service-users. These were carried out using AutoRegressive Integrated Moving Average (ARIMA) methods, with SPSS (Statistical Package for the Social Sciences, V.25) software. Data management, analysis and reporting were carried out according to the Cochrane Effective Practice and Organisation of Care resources.16 The data were grouped into monthly time intervals. Positive autocorrelation was found between subsequent observations of all admissions (Durbin-Watson test=1.239). Therefore, the models were adjusted by setting the non-seasonal autoregressive values to 1 (ARIMA (1, 0, 0)). There were 36 months of data; 19 months of data pre- intervention (from June 2014) and 17 months postintervention (from January 2016). Statistical comparisons are presented as p values and 95% CIs and were considered significant at the alpha level of 5%.
Results
Pre-post analysis
Pre–post analyses of community palliative care visits (table 1) are shown below. Notably, between T1 and T2 there is a shift away from senior medical and nursing visits (97.3%–83.4%) towards increased junior doctor and junior nurse visits (2.7% to 16.6%). There is also a reduction in joint visits from 16.0% to 10.0%, partially accounting for the reduced total staff contacts. Fisher’s exact tests show that the changes in seniority of visits and joint visits are significant with low strength of association (p<0.001, Cramer’s V=0.241; p<0.001, Cramer’s V=0.087, respectively).
Community visits
Tables 2 and 3 show results from interim analyses of balancing measures to monitor for any concerning increases in hospitalisations or negative patient experiences. Between the two periods, caseload size remains comparable (1521 and 1501) and no negative changes in hospitalisation are observed. The statistical significance of changes to ED admissions and total bed-days is explored through ITS analysis.
Hospital admissions are reported for all community palliative care patients once they become patients of the community service for 2×12 month periods for T1 (October 2014–September 2015) and T2 (October 2015–September 2016)
Patient reported Views on Care (VoC3) scores for 2×12 month data for T1 (October 2014–September 2015) and T2 (October 2015–September 2016)Improvement in scores for second visits is significant on Fisher’s exact test (p=0.023)
Table 3 reports on quality of care, recorded through VoC3 to determine whether quality was maintained after the change period. These findings illustrate an increase in positive responses with a corresponding reduction in negative responses. When focused on second visits, a Fisher’s exact test demonstrates that the observed changes are statistically significant (p=0.023).
ITS analyses
The ITS analyses covered three aspects of hospital use. These were ED admissions, bed-days for in-patient stays and the proportion of all admissions that presented via ED. This describes the frequency and duration of contacts with secondary care and highlights whether admission routes were appropriate.
For each variable, six parameters are used. Preslope represents the change in that variable per month prior to the intervention. Slope difference represents the additional change in that variable occurring after the intervention. The 3, 6, 12 and 17-month level effects highlight the absolute ‘step’ change occurring at each of these time periods. The p values describe how well the model describes the data, and CIs indicate 95% confidence that repeated observations would fall between these parameters. Detailed findings can be found in the online supplemental data.
Admissions to the ED
Figure 2 shows that the absolute number of admissions to the local ED was already decreasing prior to the intervention at a rate of 2.86 ED admissions per month. The slope difference demonstrates an accelerated rate of decrease after the intervention by an additional 2.56 ED admissions per month. Therefore, following the intervention, ED admissions were decreasing with a trend of 5.42 admissions per month. At 17 months postintervention the level effect is a reduction of 76.09 ED admissions per month (p=0.001; 95% CI −117.75 to –34.43). All level effects demonstrate statistically significant change.
ITS analysis of total number of ED admissions per month for community palliative care patients. Total 36 months from June 2014–May 2017 (19 months preintervention, 17 months postintervention). At 17 months postintervention the level effect is a statistically significant reduction of 76.09 ED admissions per month (p=0.001). The slope difference indicates a statistically significant additional monthly reduction of 2.56 admissions per month (p=0.026). ED, emergency department; ITS, interrupted time series.
Non-ED total bed-days
Figure 3 illustrates that prior to the intervention, time that these patients were spending in hospital was decreasing by 46.96 days per month. Following the intervention, this was decreasing with a trend of 84.98 days per month. At 17 months postintervention the level effect is a reduction of 203.22 days per month. The value for the preintervention slope is statistically significant (p=0.001; 95% CI −71.69 to –22.24). However, the value for the postintervention slope is not statistically significant (p=0.052; 95% CI −76.35 to 0.32).
ITS analysis of total hospital bed-days per month for community palliative care patients. Total 36 months from June 2014–May 2017 (19 months pre-intervention, 17 months postintervention). The 17-month level effect is not statistically significant (p=0.565). The difference between the preintervention and postintervention slope is also not statistically significant (p=0.052). ITS, interrupted time series.
The level effects show an increase of total bed-days during early implementation, followed by a steady decrease; the width of confidence intervals indicate that findings are not statistically significant. For example at 3 months p=0.101; 95% CI −68.2 to 726.301, and at 17 months p=0.565; 95% CI −916.503 to 510.071. A possible explanation for these findings (and a relevant observation for the study) is the marked variation in bed-days from month-to-month. It is also likely that this metric is not immediately responsive to change, being dependent on prevalent hospital inpatients and admissions related to prior treatments and care plans.
Admissions via ED as a percentage of all admissions
Figure 4 shows the percentage of admissions to in-patient wards that involved the ED. Before the intervention the trend was a non-significant decrease of 0.030% per month in the proportion of admissions taking place via ED (p=0.668; 95% CI −0.17, 0.11). The difference between the preslope and the postslope is −0.148% proportion of ED admissions per month (p=0.168; 95% CI −0.36 to 0.07) resulting in a combined reduction in the proportion of ED admissions of 0.178% per month. This suggests that the very slight month-by-month decline in the ratio of ED admissions prior to the intervention was replaced by a marginally steeper month-on-month decline after the intervention, but this finding is not statistically significant.
ITS analysis of the % of community palliative care patients being admitted to hospital via the ED. Total 36 months from June 2014–May 2017 (19 months preintervention, 17 months postintervention). At 17 months post-intervention the level effect is a statistically significant reduction of 7.5% of all admissions via ED per month (p=0.001). The slope difference does not indicate a statistically significant additional monthly reduction (p=0.168). ED, emergency department; ITS, interrupted time series.
In contrast with the predicted slopes, the level effects (the changes occurring around the time of the intervention), show significant change. The 3-month level effect in the proportion of ED admissions is −5.78% (p<0.001; 95% CI −21.93 to –7.92), and 17-month level effect is −7.50% (p=0.001; 95% CI −53.51 to 6.47). These results indicate that the proportion of admissions occurring via ED reduced almost immediately following the implementation of eShift, and that this was maintained. The pattern can readily be seen in figure 4.
Value assessment
Potential economic impact on hospital admissions
The observed reduction in bed days was not statistically significant. However, there is strong evidence that the intervention resulted in reduced ED admissions. Lengths of stay attributed to the ED were usually very short, with a maximum of 2.14 days and a mean of 0.2 days (<5 hours). Therefore, these are treated as single episodes of £148 each.17 This only indicates hospital costs; there are potentially additional savings, such as ambulance conveyance. The 95% CIs for the 17-month level effect indicate a reduction of between 117.8 and 34.4 ED admissions per month. Potential savings are shown in table 4.
Estimated cost savings from reductions in ED admissions per annum, based on its analysis and PSSRU Hospital unit costs
This figure for estimated savings (£135, 153) is purely indicative and relies on a number of assumptions, for instance that the PSSRU (Personal Social Services Research Unit) average unit costs are applicable for these palliative care patients.
Potential economic impact on community visits
There are four key mechanisms by which the new staffing model could reduce visit costs. First, by reducing workforce seniority, thereby reducing the average hourly visit cost and second by reducing the need for joint visits. These mechanisms are only made possible by the new, technology-supported model and real-time senior support.
The third potential area for cost reduction is changes in the overall numbers of visits, which reduced between T1 and T2. Qualitative investigation indicated that real-time decision-making reduced the need for repeated visits, for instance, by reducing the need for additional assessments following team meetings. A fourth potential mechanism, visit durations remained stable (T1=58.55 min, T2=59.51 min).
Changes in hourly staff costs due to changes in seniority of the workforce can be calculated using mean hourly salary costs for each staff grade (see table 1), to compare the average cost per patient-contact hour. For T1, the mean cost was £16.71 /hour; for T2 this was £16.23 /hour (2.9% or £0.49 reduction). These costs are based on mid-range Agenda for Change18 and British Medical Association (BMA)19 reported salaries. Agenda for Change is the current pay and grading system for the NHS, and doctors’ pay is regulated by the BMA.
Table 5 shows changes in visiting hours between T1 and T2, broken down by pay grades. By applying the mean hourly salary for each grade, overall changes in staff costs for all visits can be estimated.
Comparison of T1 and T2 hours spent visiting patients and associated costs broken down by pay grades and whether the visits are joint or solo visits
Findings indicate that increases in visiting hours of junior nurses and junior medical staff (particularly solo visits) are more than offset by decreases in hours of senior members of staff. This represents a potential saving of £14 701 (for 10 months), or £17 642 per annum.
These comparisons indicate that introducing junior grades by providing remote support, thereby reducing the need for joint and repeat visits, potentially reduces the cost of care. However, this would need to be offset by the cost of senior staff at the base for live support and delegation and technology costs. This is a conservative estimate, as savings related to travel and administration, National Insurance contributions and other costs associated with visits are not included.
Discussion
Nationally, in the last year of life, hospital care accounts for the majority of total care costs, and emergency hospital admissions are responsible for the majority of hospital costs. According to PSSRU data ‘they accounted for 85% of hospital costs in the final month’.20
This study explored the impact of a new digital health system in a specialist service. Previous Canadian implementations have been evaluated (mostly qualitative21). However, this study explored the intervention, following redesign for the UK. This focused on addressing workforce shortages and expanding the service in a city-setting, whereas Canadian services were primarily concerned with geographical challenges. Therefore, the relevance of previous research is substantially limited.
The study demonstrated that introducing the technology allowed a more sustainable workforce model, in which junior members of staff are supported to work in the community and fewer joint visits are required.
An interesting finding from this study is the immediate reduction in ED admissions. The parallel process evaluation and qualitative investigations found no confounding factors. Whereas, the programme theory development and testing indicated plausible causal links between the intervention and these outcomes.
Previously, medical decisions were largely made at weekly MDT meetings and could trigger visits for further assessments. Decisions about changes to care or preparations for hospital admissions could take days or weeks and depended on sufficient time to discuss cases. Following the implementation, staff members reported that the platform allowed instantaneous multidisciplinary decision-making; reducing the need for decision-making at meetings. Potential breakdowns in family care or rapid health deteriorations could be responded to immediately to prevent emergency unplanned admissions.
Based on improvements in working and admission patterns, some key costs arising from community visits and hospital admissions across the wider service have reduced following the intervention. Alongside this introduction, the quality of the service, as reported by patients and carers, appears to have been maintained or improved.
A common implementation challenge was confronted in the integration of an additional clinical record system.1 While these issues were being addressed, some duplication of record-keeping was experienced, reinforcing the need for technology providers to facilitate integration between systems.22
Conclusion
The technology has the potential to facilitate a transition of community care to more sustainable, flexible and scalable workforce models. Importantly, this can be achieved while maintaining quality of care, improving efficiency and ensuring all visits have multi-disciplinary input and senior oversight; with associated reductions in the need for senior members of staff to attend visits.
There is compelling evidence for associated reductions in emergency admissions and service costs. Interventions such as this could make a substantial impact on the NHS workforce crisis by narrowing gaps between workforce skills and the population needs identified in the NHS long-term plan.6 7 23 24 The technology (as developed for the UK setting) could be spread to other community palliative care contexts with little adaptation. However, as a complex intervention, replicability of findings is dependent on context and organisational implementation.
These findings are useful for researchers undertaking evaluations of similar technology, and professionals wishing to understand the potential of changes. The findings suggest that the intervention would be beneficial for community palliative care services seeking to develop sustainable workforce models or expand their services. However, in order to support decision-making, further research on the economic and financial impacts of the intervention is needed. An additional area for further study is understanding optimum processes for the adaptation and implementation of the technology in other services.
Data availability statement
Data are available on reasonable request. The total monthly admissions data used for the interrupted time series analysis are available on reasonable request from the lead author: SMBA, s.ariss@sheffield.ac.uk ORCID ID: http://orcid.org/0000-0002-5557-4613.
Ethics statements
Patient consent for publication
Ethics approval
Ethical review was conducted by the Research Ethics Committee at University of Sheffield, School of Health and Related Research (Ref. 002550) and the project was approved by the Healthcare Governance Groups of the participating services.
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Twitter @ariss_s
Contributors SK, DF and SM were involved in the conception and design of the intervention. SMBA led the conception and design of the evaluation, supported by the other authors. SMBA, PT and SK were responsible for data acquisition. SMBA and PT led the analysis and interpretation of data, assisted by the other authors. All authors were responsible for drafting, critical revision and final approval of the manuscript. SMBA is responsible for the overall content and accepts full responsibility for the work and/or the conduct of the study, had access to the data, and controlled the decision to publish.
Funding The technology and implementation costs were funded by the NHS England Nursing Technology Fund (Ref: VSL/NTF2-1). This research was funded by the National Institute for Health Research Collaboration for Leadership in Applied Health Research and Care Yorkshire and Humber (NIHR CLAHRC YH), now recommissioned as NIHR Applied Research Collaboration Yorkshire and Humber.
Disclaimer The views expressed in this publication are those of the authors and not necessarily those of the NIHR or the Department of Health and Social Care.
Competing interests PT is working clinically at the service in question (although was not during the period of the evaluation). SK is the medical director at the clinical service in question. A discounted version of eShift was provided to the service by the manufacturers for the pilot.
Provenance and peer review Not commissioned; externally peer reviewed.