Article Text
PDF
Abstract
Objectives To determine the accuracy of predictions of dying at different cut-off thresholds and to acknowledge the extent of clinical uncertainty.
Design Secondary analysis of data from a prospective cohort study.
Setting An online prognostic test, accessible by eligible participants across the UK.
Participants Eligible participants were members of the Association of Palliative Medicine. 99/166 completed the test (60%), resulting in 1980 estimates (99 participants × 20 summaries).
Main outcome measures The probability of death occurring within 72 hours (0% certain survival−100% certain death) for 20 patient summaries. The estimates were analysed using five different thresholds: 50/50%, 40/60%, 30/70%, 20/80% and 10/90%, with percentage values between these extremes being regarded as ‘indeterminate’. The positive predictive value (PPV), negative predictive value (NPV) and the number of indeterminate cases were calculated for each cut-off.
Results Using a <50% versus >50% threshold produced a PPV of 62%, an NPV of 74% and 5% indeterminate cases. When the threshold was changed to ≤10% vs ≥90%, the PPV and NPV increased to 75% and 88%, respectively, at the expense of an increase of indeterminate cases up to 62%.
Conclusion When doctors assign a very high (≥90%) or very low (≤10%) probability of imminent death, their prognostic accuracy is improved; however, this increases the number of ‘indeterminate’ cases. This suggests that clinical predictions may continue to have a role for routine prognostication but that other approaches (such as the use of prognostic scores) may be required for those cases where doctors’ estimates are indeterminate.
- prognosis
- palliative care
- clinical decisions
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Introduction
Palliative care doctors are frequently asked to make predictions about the future health outcomes for their patients; this is what they base treatment decisions on. One such prediction is about whether a patient is imminently dying. This has been shown to be a difficult task.1–3 Towards the end of life, accurate recognition of the dying phase can enable a ‘good death’ to occur, with the patient’s final wishes being achieved, surrounded by loved ones and harmful interventions ceased.4–6 These wishes cannot be fulfilled if the clinical team are not able to recognise accurately that the person is dying. Although clinician predictions of survival are frequently inaccurate and overoptimistic,7–9 they are still quite well correlated with actual survival and so are still recommended for use in routine clinical practice.10
The ‘More Care: Less Pathway’ report into the limitations of the Liverpool Care Pathway (LCP) described, among other things, the dissatisfaction of relatives with the information they were given about how long their relative was expected to live. A particular issue arose when patients, who were not in fact imminently dying, were cared for using the LCP. The report included a recommendation that more research was needed into the best way to communicate prognostic uncertainty. A recent report from the Royal College of General Practitioners highlighted how difficult doctors find it to initiate a discussion about dying with people who have a terminal disease.11 The report describes many barriers to this discussion including the fact that some doctors feel unconformable discussing uncertainty. For these reasons, it is vital to improve doctors’ ability to recognise when patients are imminently dying in order to initiate conversations about prognosis, including the appropriate communication of uncertainty.
There is a lack of evidence on the final days of life on which to base clinical practice1 2 and little is understood about how doctors recognise dying.12 Furthermore, it has been recognised that uncertainty might be an inherent part of predicting dying.13 14 For these reasons, we wanted to examine how certain a doctor needs to be that a death is likely to be imminent in order to have the highest diagnostic accuracy.
In a previously reported study, we asked palliative care doctors across the UK to participate in an online prognostic task.3 The task presented 20 patient summaries of real patients who had been referred to specialist palliative care, of whom half died within 72 hours and half survived beyond 72 hours. As part of this task, the doctors were asked to read the summaries and provide a probability estimate of that patient dying within the next 72 hours.
For the purpose of this paper, we analysed the results of the prognostic task to explore the following:
Do palliative care doctors use the full range of probability estimates from 0% (certain survival) to 100% (certain death) when asked if a patient is imminently dying?
What is the effect on prediction accuracy of using different probability thresholds to predict which patients will or will not die imminently.
Methods
Study design
This is a secondary analysis of data collected as part of a prospective study to investigate how palliative care doctors recognise imminently dying patients.3 This paper follows Standards for Reporting Diagnostic accuracy studies (STARD) reporting guidelines (see online supplementary file 1).
Supplemental material
Participants
All medical doctors who were members of the Association of Palliative Medicine (APM) were invited to participate. Recruitment took place between 22 April 2016 and 1 July 2016.
Patient and public involvement
Members of the South West London Cancer Research Network, consisting of patients and bereaved relatives, reviewed the study documentation for the wider prospective observational study.3
Procedure
An email was distributed via the association secretariat to all APM members containing a web link to an online prognostic test. Consent was sought via a tick box prior to participation. Participants were asked to provide some demographic information about themselves before completing the study.
The prognostic test contained 20 patient summaries, presented as ‘vignettes’. Each summary was about a patient who had been referred to specialist palliative care services, in either a hospice or hospital setting. In addition to a narrative summary of the patient’s condition, supplementary information was presented: blood test results, medications and observations such as temperature or oxygen saturation level (where available). All patients were monitored to record if death occurred within 72 hours so that this actual outcome could be compared with the doctors’ probability estimates. An example of a patient summary can be seen in online supplementary file 2 to show the level of detail presented.
Participants were asked to read through each patient summary and provide an estimated percentage probability (any number between 0% and 100%) that the patient would die within the next 72 hours.
Test methods
The prognostic test of interest was the percentage estimate of the likelihood of death in the next 72 hours provided by the participants. Previous literature has suggested that 50% is an estimate often used in place of ‘I don’t know’.15 Therefore, we started by assuming that every prediction about the probability of imminent death made by a clinician indicated either that the patient was likely to die (if the estimate was greater than 50%) or that the patient was likely to survive (if the estimate was less than 50%). Estimates of exactly 50% were regarded as being ‘indeterminate’.
Then, we investigated the effect of classifying patients with scores in between two more extreme values as having an ‘indeterminate’ prognosis. For instance, we investigated how the accuracy of predictions would be affected if only estimates of imminent death of ≥60% were taken to indicate that a patient would be likely to die; an estimate of imminent death of ≤40% indicated that a patient would be likely to survive; and an estimate between 41% and 59% was regarded as being ‘indeterminate’. This process was repeated for different threshold values. Thus, we also looked at using threshold values of ≤30%, 31%–69% and ≥70%; values of ≤20%, 21%–79% and ≥80%; and finally at values of ≤10%, 11%–89% and ≥90%. In total, we looked at five different ways of grouping the clinicians’ survival probabilities. The 30%/70% cut-off range has previously been suggested as the most clinically meaningful grouping.16 17
Analysis
The demographics of the respondents and of the patients described in the vignettes were summarised using mean (SD) and number (%). For each of the 20 patient vignettes, the clinicians’ predictions of the probability of imminent death were summarised across all participating clinicians using median and IQR. The clinicians’ predictions were then compared with the known patient survival outcomes. The accuracy of their predictions under these different threshold values was summarised in a series of 2×2 tables, showing true positives, false positives, true negatives and false negatives (where ‘positive’=death within 72 hours, and ‘negative’=survived 72 hours). The indeterminate predictions were omitted from this analysis. The positive predictive value (the proportion of patients who died when the clinician predicted dying) and the negative predictive value (the proportion of patients who survived when the clinician predicted survival) were calculated.
Results
A total of 166 participants started the test, of whom 99 (60%) provided survival probabilities for all 20 case histories. Data from participants who did not complete all of the cases were excluded from the analysis because they were judged to have lost interest in the task and/or not treated it sufficiently seriously. The majority of non-completers (41/67) stopped the test after reading the first case history. All but two other participants completed less than half of the assessments. Thus, there were 1980 estimates provided by the participants (99 doctors each provided 20 estimates) included in the analysis. Figure 1 shows the flow of participants in the study, and the number of predictions of death/survival within 72 hours according to each of the five threshold value groupings. In this figure, ‘positive’ refers to the number of estimates predicting death within 72 hours, and ‘negative’ refers to the number of estimates predicting survival. Table 1 shows the demographics of respondents. Table 2 presents a brief demographic background of the patients presented to the participants within the case summaries.
Study flow diagram.
Demographics of participants
Characteristics of the patients presented within the prognostic test
Use of percentage probability estimates
Figure 2 shows the median and IQR of the probability estimates given by the 99 respondents for each of the 20 vignettes included in the prognostic test. This does not represent the order in which they were presented to the participants. Vignettes 1–10 are those where the patient died within 72 hours. Vignettes 11–20 are those where the patient survived for over 72 hours. For 6 patients, the median predicted probability of dying was between 90% and 100%, 2 patients had a median estimate between 0% and 20% of dying and 12 patients had a median estimate between 20% and 80%
Boxplot of responses to each vignette.
Participants used the full range of the scale (0%–100%) in their survival predictions. There was one vignette in which the median response by the participants was greater than 90% probability of dying and the patient survived beyond 72 hours.
The assessment of accuracy
Table 3 shows the accuracy of clinicians’ predictions about the probability of imminent death at each of the five thresholds. The resulting number of ‘indeterminate’ scores is also presented. The positive predictive value of the clinical prediction (ie, the proportion of estimates of imminent death that correctly predicted that the patient would die imminently) improved from 62% (at the 50% threshold) to 75% at the 10%/90% threshold. The negative predictive value of the clinical prediction (ie, the proportion of estimates of survival beyond 72 hours that correctly predicted the outcome) improved from 74% (at the 50% threshold) to 88% at the 10%/90% threshold.
Accuracy of clinician probability estimates using different double thresholds (including indeterminate values)
Indeterminate cases
Applying the simplest threshold of <50% or >50%, 96 estimates (5%) were ascribed a probability of exactly 50% and were regarded as being indeterminate. The number of indeterminate cases increased steadily with each change in the thresholds applied until at the ≤10 %/≥90% threshold, 1231 estimates were deemed to be indeterminate (1231/1980; 62% of all estimates). In these 1231 indeterminate estimates, 700 (57%) patients survived and 531 (43%) died.
Discussion
The principal finding from this study is that when using a <50% or >50% threshold, and ignoring predictions of exactly 50%, clinical predictions of imminent death are not very accurate (accuracy 66%). However, accuracy could be improved by adjusting the threshold values that are taken to indicate ‘likely death’ or ‘likely survival’. The best way to maximise accuracy was to only regard those predictions with a probability of ≥90% as indicating imminent death, and only those predictions with a probability of ≤10% as indicating likely survival beyond 72 hours, with all other values in between being regarded as ‘indeterminate’. The drawback with this strategy was that a relatively large proportion of clinician estimates would have been regarded as indeterminate. In our study 14/20 vignettes were ascribed a median probability estimate in the range between 10% and 90%, suggesting a majority of predictions would be regarded as uninformative using this approach.
It makes logical sense that the higher the probability estimate provided by the doctor, the more likely it is that the outcome will occur. However, these results illustrate how challenging it is to predict the imminent future, even by specialist palliative care doctors. The results suggest it is important to acknowledge and discuss uncertainty whatever the level of the clinician’s estimate, as is illustrated by one of the patient summaries used in the prognostic task. This patient survived for longer than 72 hours despite the median probability ascribed to his or her death being greater than 90%. Perhaps, however, clinical estimates in the range of 11%–89% should be regarded with greater circumspection.
Ultimately, the decision about which is the ‘best’ threshold to use is a qualitative and subjective one. What this analysis does is to quantitatively illustrate the effect of using different thresholds on the accuracy of clinicians’ predictions. Even when using the highest threshold level (≥90%), clinicians’ predictions of imminent death were only correct on 75% of occasions. Accuracy gradually increases for every rise in threshold value but at the extreme values the number of indeterminate cases escalates rapidly.
The European Association for Palliative Care (EAPC) has recommended that clinical predictions of survival should still hold an important role in prognostication, while accepting that, taken in isolation, clinical predictions of survival are not very accurate.10 Other means to estimate prognosis in palliative care patients have been developed16 18–20 and are still being evaluated.21–23 One question that therefore arises is when and how to combine clinical predictions of survival with prognostic scores, multidisciplinary team (MDT) decisions or algorithms. Our study suggests that clinical predictions of survival are at the most accurate when the thresholds of ≤10% and ≥90% are employed; however, there are important implications for the number of ‘indeterminate’ estimates that arise as a result of this.
Strengths and limitations
This is the first study to measure the accuracy of multiple doctors’ predictions about the imminent death of the same group of patients, thus allowing a direct comparison of their prognostic accuracy.
The prognostic test could be criticised for lacking some face validity as the doctors did not get to review the patients at a face-to-face assessment. Nonetheless, the vignettes all concerned real cases and included a detailed descriptive summary of the patient including routinely available blood results and other clinical findings. A further limitation of this study is that the findings are based on these specific 20 case histories, which may not be ‘representative’ of all the patients who will be observed in palliative care. Half of the vignettes in this sample died, and this might not be representative of survival times in palliative care. Further research is required to see if a different set of 20 summaries would produce similar results.
Conclusion
Our results show that even when patients are very (>90%) confident that a patient will die within the next 72 hours, these estimates are only correct on 75% of occasions (and at the expense of a high number of indeterminate cases). It is therefore important for clinicians to convey this level of uncertainty in their communications with patients and relatives and not to forget that even their most confident predictions are inaccurate 25% of the time. This suggests that while clinical predictions will continue to have a role for routine prognostication, other approaches (such as the use of prognostic scores) may be required for those cases where doctors’ estimates are indeterminate.
Ethics statements
Patient consent for publication
Acknowledgments
We would like to thank Dr Adam Harris at University College London (UCL) for his input in this study. We would also like to thank the specialist palliative care doctors who participated in the study.
Footnotes
Contributors NW developed the research concept, designed the data collection tools, monitored and collected the data for the study, wrote the statistical analysis plan, cleaned and analysed the data and drafted and revised the paper. She is the guarantor. FR and VV assisted in the study design and the statistical analysis of the data. PH and PS developed the research concept and oversaw the data collection and analysis. CT developed the study website to collect the data and monitored the data collection. All authors reviewed and provided comments on the drafted paper. All authors approve the final version.
Funding This research was part funded by a UCL PhD Studentship. Also by Marie Curie I-CAN-CARE Program grant (MCCC-FPO-16-U). Professor Stone is supported by the Marie Curie Chair’s grant (MCCC-509537).
Disclaimer The funders had no role in trial design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests CT reports personal fees from University College London during the conduct of the study. All other authors have no competing interests.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement All data relevant to the study are included in the article or uploaded as supplementary information.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.