Table 3

Accuracy of individual studies

First authorSample sizeSurprise question resultsDiagnostic test results
Surprise Question responsesTotalDeadAliveSensitivity (95% CI)Specificity (95% CI)Positive predictive value (95% CI)Negative predictive value (95% CI)Accuracy (95% CI)
Rice J.301Not surprised191621340.646 (0.541–0.741)0.544 (0.485–0.602)0.316 (0.276–0.360)0.825 (0.779–0.863)0.569 (0.518–0.619)
Surprised19434160
Ikari T.1411Not surprised8472326150.820 (0.770–0.863)0.455 (0.425–0.48400.274 (0.259–0.289)0.910 (0.886–0.929)0.528 (0.502–0.554)
Surprised56451513
Ikari T.1411Not surprised11796365430.944 (0.923–0.960)0.263 (0.232–0.297)0.539 (0.528–0.551)0.836 (0.786–0.877)0.588 (0.562–0.614)
Surprised23238194
Dogbey D.M.43Not surprised211830.667 (0.460–0.835)0.813 (0.544–0.960)0.857 (0.676–0.945)0.591 (0.446–0.721)0.721 (0.563–0.847)
Surprised22913
Moor C.C.140Not surprised3919200.679 (0.477–0.841)0.821 (0.738–0.887)0.487 (0.372–0.604)0.911 (0.856–0.946)0.793 (0.716–0.857)
Surprised101992
Gonzalez- Jaramillo V.174Not surprised8317660.850 (0.621–0.968)0.571 (0.489–0.651)0.205 (0.166–0.250)0.967 (0.911–0.988)0.603 (0.527–0.677)
Surprised91388
Tripp D.381 (30 days)Not surprised192170.125 (0.016–0.384)0.953 (0.927–0.973)0.105 (0.029–0.318)0.961 (0.954–0.968)0.919 (0.887–0.944)
Surprised36214348
365 (365 days)Not surprised10838700.469 (0.357–0.583)0.754 (0.699–0.803)0.352 (0.285–0.425)0.833 (0.801–0.861)0.690 (0.640–0.738)
Surprised25743214
Flierman I.234Not surprised13559760.808 (0.699–0.891)0.528 (0.448–0.607)0.437 (0.389–0.486)0.859 (0.788–0.909)0.615 (0.550–0.678)
Surprised991485
Kim S.H.130 (7 days)Not surprised207130.467 (0.213–0.734)0.887 (0.815–0.938)0.350 (0.204–0.531)0.927 (0.888–0.954)0.839 (0.764–0.897)
Surprised1108102
130 (21 days)Not surprised5437170.529 (0.406–0.649)0.717 (0.586–0.826)0.685 (0.579–0.775)0.566 (0.493–0.636)0.615 (0.526–0.699)
Surprised763343
130 (42 days)Not surprised10087130.821 (0.734–0.889)0.458 (0.256–0.672)0.870 (0.821–0.907)0.367 (0.242–0.512)0.754 (0.671–0.825)
Surprised301911
Ouchi K.10 737Not surprised332468526390.433 (0.409–0.458)0.820 (0.813–0.826)0.206 (0.196–0.217)0.931 (0.928–0.933)0.782 (0.776–0.788)
Surprised12 89989612 003
ErmersD.J.M.379 (SQ1)Not surprised188103850.873 (0.799–0.927)0.674 (0.614–0.731)0.548 (0.501–0.594)0.922 (0.879–0.950)0.736 (0.689–0.780)
Surprised19115176
188 (SQ2)Not surprised10542630.408 (0.312–0.509)0.259 (0.170–0.365)0.400 (0.339–0.465)0.265 (0.196–0.348)0.340 (0.273–0.413)
Surprised836122
Yarnell C.417 (AP – 365 days)Not surprised2981431550.894 (0.835–0.937)0.397 (0.337–0.460)0.480 (0.452–0.508)0.857 (0.789–0.906)0.588 (0.539–0.635)
Surprised11917102
Not surprised2501321180.8250.5410.5280.8320.650
417 (SMR – 365 days)Surprised16728139(0.757–0.881)(0.478–0.603)(0.490–0.565)(0.777–0.876)(0.602–0.696)
288 (AP - admission)Not surprised5321320.636 (0.451–0.796)0.875 (0.828–0.913)0.396 (0.303–0.498)0.949 (0.922–0.967)0.847 (0.800–0.887)
Surprised23512223
288 (SMR –admission)Not surprised4716310.485 (0.308–0.665)0.878 (0.832–0.916)0.340 (0.242–0.455)0.930 (0.904–0.949)0.833 (0.785–0.875)
Surprised24117224
Van WijmenM.P.S.57Not surprised191810.692 (0.482–0.857)0.968 (0.833–0.999)0.947 (0.720–0.992)0.790 (0.677–0.870)0.842 (0.721–0.925)
Surprised38830
Ramer S.J.377Not surprised12445790.672 (0.546–0.782)0.745 (0.693–0.793)0.363 (0.307–0.423)0.913 (0.881–0.937)0.732 (0.684–0.776)
Surprised25322231
Lai C.-F.401Not surprised3418160.529 (0.351–0.702)0.956 (0.930–0.975)0.529 (0.388–0.667)0.956 (0.939–0.969)0.920 (0.889–0.945)
Surprised36716351
Rauh L.A.231 (MD – UPMC)Not surprised9036540.706 (0.562–0.825)0.700 (0.627–0.766)0.400 (0.334–0.470)0.894 (0.845–0.929)0.701 (0.638–0.760)
Surprised14115126
168 (RN – UPMC)Not surprised6131300.689 (0.534–0.818)0.756 (0.671–0.829)0.508 (0.417–0.599)0.869 (0.810–0.912)0.738 (0.665–0.803)
Surprised1071493
199 (APP – UPMC)Not surprised8135460.796 (0.647–0.902)0.703 (0.625–0.774)0.432 (0.364–0.503)0.924 (0.870–0.956)0.724 (0.656–0.785)
Surprised1189109
78 (MD – UVA)Not surprised3919200.864 (0.651–0.971)0.643 (0.504–0.766)0.487 (0.392–0.584)0.923 (0.805–0.972)0.705 (0.591–0.803)
Surprised39336
130 (RN – UVA)Not surprised8229530.725 (0.561–0.854)0.411 (0.308–0.520)0.354 (0.297–0.414)0.771 (0.658–0.855)0.508 (0.419–0.596)
Surprised481137
22 (APP – UVA)Not surprised17891.000 (0.631–1.000)0.357 (0.128–0.649)0.471 (0.376–0.568)1.0000.591 (0.364–0.793)
Surprised505
309 (MD –combined)Not surprised12955740.753 (0.639–0.847)0.686 (0.623–0.745)0.426 (0.371–0.483)0.900 (0.857–0.931)0.702 (0.648–0.753)
Surprised18018162
298 (RN –combined)Not surprised14360830.706 (0.597–0.800)0.610 (0.541–0.676)0.420 (0.368–0.473)0.839 (0.786–0.880)0.638 (0.580–0.692)
Surprised15525130
221 (APP –combined)Not surprised9843550.827 (0.697–0.918)0.675 (0.598–0.745)0.439 (0.378–0.501)0.927 (0.874–0.959)0.710 (0.646–0.769)
Surprised1239114
Verhoef M.- J.245Not surprised203172310.891 (0.839–0.931)0.404 (0.270–0.549)0.847 (0.815–0.875)0.500 (0.373–0.628)0.788 (0.731–0.837)
Surprised422121
Maes H.190 (AcuteGeriatric Unit)Not surprised6631350.674 (0.520–0.805)0.757 (0.679–0.825)0.470 (0.384–0.557)0.879 (0.826–0.918)0.737 (0.668–0.798)
Surprised12415109
189 (CardiologyUnit)Not surprised6323400.622 (0.448–0.775)0.737 (0.659–0.805)0.365 (0.285–0.453)0.889 (0.840–0.924)0.714 (0.644–0.778)
Surprised12614112
Yen Y.-F.21 098Not surprised262079918210.456 (0.432–0.479)0.906 (0.901–0.910)0.305 (0.291–0.319)0.948 (0.946–0.950)0.868 (0.864–0.873)
Surprised18 47895517 523
193Not surprised10344590.7860.5690.4270.8670.632
AaronsonE.L.Surprised901278(0.656–0.884)(0.482–0.654)(0.371–0.486)(0.794–0.916)(0.560–0.700)
Schmidt R.J.749Not surprised173611120.604 (0.502–0.700)0.827 (0.796–0.856)0.353 (0.302–0.407)0.931 (0.913–0.945)0.797 (0.766–0.825)
Surprised57640536
Lakin J.R.1448 (Nurses)Not surprised3521122400.526 (0.457–0.595)0.806 (0.783–0.827)0.318 (0.282–0.356)0.908 (0.895–0.919)0.765 (0.742–0.786)
Surprised1096101995
1163 (Physicians)Not surprised4521433090.794 (0.728–0.851)0.686 (0.656–0.715)0.316 (0.291–0.343)0.948 (0.932–0.961)0.703 (0.675–0.729)
Surprised71137674
VeldhovenC.M.M.292 (SQ1)Not surprised161241370.923 (0.749–0.991)0.485 (0.424–0.547)0.149 (0.130–0.171)0.985 (0.944–0.996)0.524 (0.465–0.583)
Surprised1312129
161 (SQ2)Not surprised2210120.417 (0.221–0.634)0.912 (0.852–0.954)0.455 (0.289–0.631)0.899 (0.864–0.926)0.830 (0.772–0.892)
Surprised13914125
Burke K.325 (majority vote– 90 days)Not surprised3615210.833 (0.586–0.964)0.932 (0.897–0.957)0.417 (0.310–0.531)0.990 (0.971–0.996)0.926 (0.892–0.952)
Surprised2893286
306 (majority vote– 365 days)Not surprised10625810.833 (0.653–0.944)0.707 (0.649–0.760)0.236 (0.195–0.282)0.975 (0.946–0.989)0.719 (0.665–0.769)
Surprised2005195
238 (100% agreement - 90 days)Not surprised211470.933 (0.681–0.998)0.969 (0.936–0.987)0.667 (0.488–0.808)0.995 (0.970–0.999)0.966 (0.935–0.985)
Surprised2171216
175 (100% agreement – 365 days)Not surprised5721360.955 (0.772–0.999)0.765 (0.689–0.829)0.368 (0.302–0.441)0.992 (0.945–0.999)0.789 (0.721–0.847)
Surprised1181117
290 (75–100% agreement – 90 days)Not surprised2614120.824 (0.566–0.962)0.956 (0.925–0.977)0.539 (0.391–0.679)0.989 (0.969–0.996)0.948 (0.916–0.971)
Surprised2643261
236 (75–100% agreement – 365 days)Not surprised8024560.923 (0.749–0.991)0.733 (0.668–0.792)0.300 (0.250–0.355)0.987 (0.953–0.997)0.754 (0.694–0.808)
Surprised1562154
122 (Neurology –365 days)Not surprised287210.875 (0.474–0.997)0.816 (0.732–0.882)0.250 (0.173–0.347)0.989 (0.937–0.998)0.820 (0.740–0.883)
Surprised94193
27 (Oncology – 365 days)Not surprised2212101.000 (0.735–1.000)0.333 (0.118–0.616)0.546 (0.456–0.632)1.0000.630 (0.424–0.806)
Surprised505
73 (Congenital –365 days)Not surprised244200.667 (0.223–0.957)0.702 (0.577–0.807)0.167 (0.093–0.282)0.959 (0.882–0.987)0.699 (0.580–0.801)
Surprised49247
Ouchi K.207Not surprised10234680.773 (0.622–0.885)0.583 (0.503–0.660)0.333 (0.282–0.289)0.905 (0.844–0.943)0.623 (0.553–0.690)
Surprised1051095
Liyanage T.187Not surprised8030500.714 (0.554–0.843)0.655 (0.572–0.732)0.375 (0.309–0.446)0.888 (0.829–0.928)0.668 (0.596–0.735)
Surprised1071295
Mitchell G.K.2840 (Intuition)Not surprised154321220.337 (0.243–0.441)0.956 (0.947–0.963)0.208 (0.159–0.268)0.977 (0.973–0.980)0.935 (0.925–0.944)
Surprised2686632623
1525 (ST)Not surprised179251540.532 (0.381–0.679)0.896 (0.879–0.911)0.140 (0.107–0.181)0.984 (0.978–0.988)0.885 (0.868–0.900)
Surprised1346221324
Ebke M.236 (Neurorehabilitation Physicians)Not surprised4517280.500 (0.324–0.676)0.861 (0.806–0.906)0.378 (0.273–0.496)0.911 (0.879–0.935)0.809 (0.753–0.857)
Surprised19117174
236 (Palliative CarePhysicians)Not surprised8323600.677 (0.495–0.826)0.703 (0.635–0.765)0.277 (0.219–0.344)0.928 (0.887–0.955)0.699 (0.636–0.757)
Surprised15311142
Mudge A.M.100Not surprised5216360.889 (0.653–0.986)0.561 (0.447–0.670)0.308 (0.249–0.374)0.958 (0.860–0.989)0.620 (0.518–0.715)
Surprised48246
Salat H.488Not surprised171561150.644 (0.534–0.744)0.713 (0.666–0.757)0.328 (0.281–0.378)0.902 (0.874–0.925)0.701 (0.658–0.741)
Surprised31731286
Malhotra R.208 (180 days)Not surprised203101930.769 (0.462–0.950)0.010 (0.001–0.037)0.049 (0.037–0.065)0.400 (0.109–0.785)0.058 (0.030–0.099)
Surprised532
189 (365 days)Not surprised162131490.650 (0.408–0.846)0.118 (0.074–0.177)0.080 (0.059–0.108)0.741 (0.580–0.855)0.175 (0.123–0.236)
Surprised27720
Hadique S.500 (Derivation cohort)Not surprised238148900.822 (0.758–0.875)0.719 (0.666–0.767)0.622 (0.577–0.665)0.878 (0.839–0.908)0.756 (0.716–0.793)
Surprised26232230
543 (Validation cohort)Not surprised204139650.739 (0.671–0.801)0.817 (0.773–0.856)0.681 (0.628–0.730)0.856 (0.822–0.883)0.790 (0.753–0.824)
Surprised33949290
Gomez- Batiste X.1059 (365 days)Not surprised8372685690.937 (0.902–0.962)0.264 (0.233–0.297)0.320 (0.309–0.332)0.919 (0.877–0.947)0.446 (0.416–0.476)
Surprised22218204
1059 (730 days)Not surprised8373734640.914 (0.883–0.940)0.287 (0.253–0.324)0.446 (0.432–0.460)0.842 (0.792–0.882)0.529 (0.498–0.559)
Surprised22235187
Javier A.D.388Not surprised137331040.635 (0.490–0.764)0.691 (0.638–0.740)0.241 (0.196–0.292)0.924 (0.894–0.946)0.683 (0.634–0.729)
Surprised25119232
Amro O.W.201Not surprised5022280.550 (0.385–0.707)0.826 (0.759–0.881)0.440 (0.336–0.549)0.881 (0.839–0.913)0.771 (0.707–0.827)
Surprised15118133
Lakin J.R.1737Not surprised11423910.205 (0.135–0.292)0.944 (0.932–0.955)0.202 (0.143–0.277)0.945 (0.940–0.950)0.896 (0.881–0.910)
Surprised1623891534
Hamano J.2361 (7 days)Not surprised9312826490.847 (0.804–0.884)0.680 (0.659–0.700)0.303 (0.287–0.320)0.964 (0.955–0.972)0.704 (0.685–0.722)
Surprised1430511379
2361 (30 days)Not surprised185110667850.956 (0.942–0.967)0.370 (0.343–0.397)0.576 (0.565–0.587)0.904 (0.876–0.926)0.647 (0.627–0.666)
Surprised51049461
Feyi K.178Not surprised5837210.726 (0.583–0.841)0.835 (0.758–0.895)0.638 (0.535–0.730)0.883 (0.828–0.923)0.803 (0.737–0.859)
Surprised12014106
Moroni M.231Not surprised12687390.837 (0.751–0.902)0.693 (0.605–0.772)0.691 (0.629–0.746)0.838 (0.768–0.890)0.758 (0.697–0.811)
Surprised1051788
O'CallaghanA.501 (180 days)Not surprised9956430.727 (0.614–0.823)0.899 (0.866–0.926)0.566 (0.487–0.641)0.948 (0.926–0.963)0.872 (0.840–0.900)
Surprised40221381
501 (365 days)Not surprised9967320.626 (0.527–0.718)0.919 (0.887–0.944)0.677 (0.593–0.751)0.901 (0.876–0.920)0.856 (0.823–0.886)
Surprised40240362
Da SilvaGane M.3896Not surprised9382816570.496 (0.455–0.538)0.803 (0.789–0.816)0.300 (0.278–0.323)0.904 (0.896–0.911)0.758 (0.744–0.772)
Surprised29582852673
Pang W.-F.367Not surprised10927820.614 (0.455–0.756)0.746 (0.695–0.793)0.248 (0.196–0.308)0.934 (0.907–0.95400.730 (0.682–0.775)
Surprised25817241
Haga K.138Not surprised12039810.886 (0.754–0.962)0.138 (0.076–0.225)0.325 (0.297–0.355)0.722 (0.497–0.873)0.377 (0.296–0.463)
Surprised18513
Fenning S.172Not surprised386320.353 (0.142–0.617)0.794 (0.721–0.854)0.158 (0.084–0.277)0.918 (0.886–0.941)0.750 (0.678–0.813)
Surprised13411123
Moss A.H.826Not surprised13153780.747 (0.629–0.842)0.897 (0.873–0.918)0.405 (0.346–0.466)0.974 (0.962–0.983)0.884 (0.860–0.905)
Surprised69518677
Cohen L.M.450Not surprised7139320.379 (0.285–0.480)0.908 (0.872–0.936)0.549 (0.447–0.648)0.831 (0.808–0.852)0.787 (0.746–0.824)
Surprised37964315
Moss A.H.147Not surprised3410240.455 (0.244–0.678)0.808 (0.728–0.873)0.294 (0.189–0.427)0.894 (0.851–0.926)0.755 (0.677–0.822)
Surprised11312101
Ros M.M.3140 (ICU stay)Not surprised15398550.363 (0.306–0.423)0.981 (0.975–0.986)0.641 (0.568–0.708)0.942 (0.937–0.947)0.928 (0.918–0.937)
Surprised29871722815
3140 (Hospital stay)Not surprised2521481040.378 (0.329–0.428)0.962 (0.954–0.969)0.587 (0.531–0.641)0.916 (0.909–0.921)0.889 (0.878–0.900)
Surprised28882442644
3140 (365 days)Not surprised6093632460.509 (0.471–0.546)0.899 (0.886–0.910)0.596 (0.562–0.629)0.862 (0.852–0.871)0.810 (0.796–0.824)
Surprised25313502181
GuliniJ.E.H.M.B170Not surprised8941480.820 (0.686–0.914)0.600 (0.507–0.688)0.461 (0.398–0.524)0.889 (0.813–0.936)0.665 (0.588–0.7350
Surprised81972
Haydar S.A.6122Not surprised9181078110.682 (0.603–0.754)0.864 (0.855–0.873)0.117 (0.104–0.130)0.990 (0.988–0.992)0.859 (0.850–0.868)
Surprised5204505154
Carmen J.49Not surprised207130.778 (0.400–0.972)0.675 (0.509–0.814)0.350 (0.234–0.487)0.931 (0.796–0.979)0.694 (0.546–0.818)
Surprised29227
Barnes S.231Not surprised9511840.786 (0.492–0.953)0.613 (0.545–0.678)0.116 (0.087–0.153)0.978 (0.942–0.992)0.623 (0.558–0.686)
Surprised1363133
Straw S.114 (Consultants)Not surprised6433310.846 (0.695–0.941)0.587 (0.467–0.699)0.516 (0.441–0.590)0.880 (0.774–0.940)0.675 (0.581–0.760)
Surprised50644
128 (JuniorDoctors)Not surprised6533320.750 (0.597–0.868)0.619 (0.507–0.723)0.508 (0.428–0.587)0.825 (0.734–0.890)0.664 (0.575–0.745)
Surprised631152
89 (Heart FailureNurses)Not surprised6027330.900 (0.735–0.979)0.441 (0.312–0.576)0.450 (0.388–0.514)0.897 (0.740–0.963)0.596 (0.496–0.698)
Surprised29326
123 (Staff Nurses)Not surprised5029210.659 (0.501–0.795)0.734 (0.623–0.827)0.580 (0.475–0.678)0.795 (0.715–0.856)0.707 (0.619–0.786)
Surprised731558
119 (2 or more)Not surprised6226360.8180.5770.5000.8600.659
Surprised57849(0.673–0.918)(0.465–0.683)(0.429–0.571)(0.761–0.922)(0.570–0.740)
129 (3 or more)Not surprised5431230.705 (0.548–0.832)0.729 (0.622–0.820)0.574 (0.475–0.667)0.827 (0.748–0.885)0.721 (0.635–0.796)
Surprised751362
129 (4 ‘no’s’)Not surprised181440.318 (0.186–0.476)0.953 (0.884–0.987)0.778 (0.551–0.909)0.730 (0.687–0.769)0.736 (0.652–0.810)
Surprised1113081
129 (all ‘no’)Not surprised3523120.523 (0.367–0.675)0.859 (0.766–0.925)0.657 (0.514–0.777)0.777 (0.716–0.827)0.744 (0.660–0.817)
Surprised942173
32 (Diabetes)Not surprised209110.900 (0.555–0.998)0.500 (0.282–0.718)0.450 (0.339–0.566)0.917 (0.621–0.987)0.625 (0.437–0.789)
Surprised12111
82 (no diabetes)Not surprised4424200.828 (0.642–0.942)0.623 (0.479–0.752)0.546 (0.450–0.638)0.868 (0.743–0.938)0.695 (0.584–0.792)
Surprised38533
35 (IHD)Not surprised201460.824 (0.566–0.962)0.667 (0.450–0.867)0.700 (0.539–0.823)0.800 (0.577–0.922)0.743 (0.567–0.875)
Surprised15312
79 (no IHD)Not surprised4419250.864 (0.651–0.971)0.562 (0.424–0.693)0.432 (0.352–0.516)0.914 (0.784–0.969)0.646 (0.530–0.750)
Surprised35332
18 (eGFR<30)Not surprised171070.909 (0.587–0.998)0.000 (0.000–0.410)0.588 (0.542–0.633)0.0000.556 (0.308–0.785)
Surprised110
96 (eGFR>30)Not surprised4723240.821 (0.631–0.939)0.647 (0.522–0.759)0.489 (0.399–0.580)0.898 (0.796–0.952)0.698 (0.596–0.788)
Surprised49544
Mahes A.301Not surprised136251110.807 (0.625–0.926)0.589 (0.528–0.648)0.184 (0.152–0.220)0.964 (0.928–0.982)0.611 (0.554–0.667)
Surprised1656159
Lin C.A.840Not surprised214321820.615 (0.483–0.748)0.768 (0.739–0.798)0.149 (0.101–0.196)0.968 (0.954–0.982)0.759 (0.730–0.788)
Surprised62620606
Um Y.W.300Not surprised11825930.833 (0.653–0.944)0.662 (0.603–0.718)0.212 (0.176–0.253)0.973 (0.942–0.988)0.679 (0.623–0.731)
Surprised1825177
Gaffney L.191Not surprised5620360.571 (0.394–0.737)0.769 (0.695–0.833)0.357 (0.270–0.455)0.889 (0.844–0.922)0.733 (0.664–0.794)
Surprised13515120
  • Sensitivity (the ability of the prompt to successfully identify those patients who were dying); specificity (the ability of the prompt to successfully identify those who were not dying); positive predictive value (the proportion of patients who died when the respondent predicted death); negative predictive value (the proportion of the patients who survived when the respondent predicted survival); accuracy (the proportion of correct predictions among all cases).