Objective
• Validate existing patient-level risk prognostic prediction scores for pediatric
30-day mortality which we had previously rated as having low risks of bias per
the PROBAST methodology
– 2 scores:
–Pediatric Risk Assessment Score (PRAm) (Nasr et al. 2017)
–Intrinsic surgical risk score (Nasr et al. 2019)
Pediatric Risk Assessment (PRAm)
Risk scores
Intrinsic surgical risk score
Pediatric Risk Assessment (PRAm)
Risk scores
Intrinsic surgical risk score
Methods
1. Calculated published risk scores for each patient
2. Updated (re-estimated) models using MPOG data
For both scores:
– Discrimination: ROC, precision recall curve
– Calibration: calibration plot
– Clinical utility: decision curve
Methods
1. Calculated published risk scores for each patient
2. Updated (re-estimated) models using MPOG data
For both scores:
– Discrimination: ROC, precision recall curve
– Calibration: calibration plot
– Clinical utility: decision curve
Results
•
Cumulative incidence of 30-day mortality: 0.14%
• N=822 / 608,488 cases clustered within 56 hospitals
8
Overall
(N=608,488)
Non
-
survivor
(N=822)
Survivor
(N=605,666)
Neonate (age <=28 days)
6,266
167 (2.7)
6,099 (97.3)
Weight < 5 kg
16,030
253 (1.6)
15,777 (98.4)
Female sex
258,883
335 (0.1)
258,548 (99.9)
ASA Physical Status
I
200,396
18 (> 0.0)
200,378 (100.0)
II
293,575
57 (> 0.0)
293,518 (100.0)
III
102,884
241 (0.2)
102,643 (99.8)
IV
9,094
381 (4.2)
8,713 (95.8)
V
531
125 (23.5)
406 (76.5)
Model discrimination
• Precision-recall curve
• Random performance =
0.0014
0.008
0.085
0.031
0.094
Model discrimination
• Precision-recall curve
• Random performance =
0.0014
0.008
0.085
0.031
0.094
Model calibration
Calibration plots
Model calibration
Calibration plots
Assessing clinical utility
• Decision curve analysis
Vickers, Andrew J., Ben van Calster, and
Ewout W. Steyerberg. "A simple, step-by-
step guide to interpreting decision curve
analysis." Diagnostic and prognostic
research 3.1 (2019): 1-8.
Summary of findings
• Performance inferior compared to original studies
• The intrinsic surgical risk score exhibited better discrimination than the PRAm
• Both scores resulted in large numbers of false positives
• 30-day mortality was over-estimated at higher probabilities in all models except for the
recalibrated intrinsic surgical risk score
• Decision curve analyses showed limited benefit to using either score in clinical practice
Study conclusions
• Why the overall poor performance in this external validation compared to the
original studies?
– Accounting for differences between MPOG and NSQIP databases
– Difficulties in defining measurements with MPOG data
• What is the utility of a risk score if it relies on another risk score as a
predictor?
• Sicker patients have a great influence on model performance
– aOR for ASA PS V in original study: 297; in updated model: 2471!
•
Is using a risk score to predict the rare outcome of 30-day mortality in
children clinically useful?
Study conclusions
• Why the overall poor performance in this external validation compared to the
original studies?
– Accounting for differences between MPOG and NSQIP databases
– Difficulties in defining measurements with MPOG data
• What is the utility of a risk score if it relies on another risk score as a
predictor?
• Sicker patients have a great influence on model performance
– aOR for ASA PS V in original study: 297; in updated model: 2471!
•
Is using a risk score to predict the rare outcome of 30-day mortality in
children clinically useful?
LESSONS LEARNED: USING THE MPOG
DATA FOR EXTERNAL VALIDATION
Types of external validation studies
• Temporal validation: more recent patients
• Geographic validation: multisite testing
• Fully independent validation: other investigators at other sites,
different definitions of predictors, study patients differently selected
Types of external validation studies
• Temporal validation: more recent patients
• Geographic validation: multisite testing
• Fully independent validation: other investigators at other sites,
different definitions of predictors, study patients differently selected
Lessons learned: differences in databases
• Conducting an external validation using an entirely different database from the
original study
• Comparison between NSQIP and MPOG databases:
NSQIP
MPOG
Available
measures
No 1:1 concordance with MPOG
variables
No 1:1 concordance with NSQIP
variables
Population
Sample of high
-risk surgical
procedures
All procedures with anesthesia
Data
collection method
Manual data collected by trained
research nurse
Data collected directly from EMR
Outcome
Comprehensive outcome data
available
Less comprehensive outcome data
available
Specific
to this
validation
study
Has data on preop CPR & preop
meds (with associated timing)
Less comprehensive data if not
documented on anesthesia record
Lessons learned: differences in databases
• Conducting an external validation using an entirely different database from the
original study
• Comparison between NSQIP and MPOG databases:
NSQIP
MPOG
Available
measures
No 1:1 concordance with MPOG
variables
No 1:1 concordance with NSQIP
variables
Population
Sample of high
-risk surgical
procedures
All procedures with anesthesia
Data
collection method
Manual data collected by trained
research nurse
Data collected directly from EMR
Outcome
Comprehensive outcome data
available
Less comprehensive outcome data
available
Specific
to this
validation
study
Has data on preop CPR & preop
meds (with associated timing)
Less comprehensive data if not
documented on anesthesia record
Lessons learned: Differences in procedure text documentation
MPOG: Nasr et al. (2019):
@laparoscopy,
appendicocecostomy, robot assist (wrvu
19.91) (n/a flank); modifier
robot,davinci xi (n/a )
cecostomy laparoscopic
[phi] tube with 5mm
instruments;exam
under anesthesia (eua) with injection of
botox;removal
rectal obstruction;appendectomy
laparoscopic;dental
extractions
right laparoscopic appendectom
intestine, laparoscopy creation
cecostomy;skin level -peds
(n/a ); appendix, laparoscopy appendectomy
-peds (n/a );
rectum, remove fecal impaction
-peds (n/a )
intestine, laparoscopy creation
cecostomy;skin level -peds
(n/a ); appendix, laparoscopy, unlisted procedure; appendix
(n/a )
intestine, laparoscopy creation
cecostomy;skin level -peds
(n/a ); rectum, remove fecal impaction
-
peds (n/a ); appendix,
laparoscopy, unlisted procedure; appendix (n/a )
laparoscopic assisted
appendicostom
Lessons learned: Takeaways
• Why conduct an external validation with the MPOG data?
– Comprehensive and representative: Census of anesthesia records
– Data directly from EMR
• Roadblocks to validation with the MPOG data
– Not all desired data points are captured in the anesthesia record
– Measures not necessarily equivalent (or absent in some cases)
Thank you/
Dank aan:
Weill Cornell:
Jurgen de Graaff
Hugh Hemmings
Kane Pryor
Zachary Turnbull
Silis Jiang
Samson Obembe
Erasmus MC:
Sanne Hoeks
Robert Jan Stolker
Michigan:
Sydney Brown
Sachin Kheterpal
Mike Mathis
David Clark
