AKI-01 Risk Adjustment Methodology

Summary

AKI 01 was risk adjusted using the characteristics commonly associated with increasing perioperative risk of AKI.  MPOG data overlapping with these characteristics include:

    1. Age, sex, BMI, race, ASA PS emergency status
    2. Pre-existing renal disease, hypertension, diabetes, congestive heart failure, pulmonary disease, liver disease, pre-eclampsia
    3. Procedure type and institution

Based on these characteristics, each case was predicted to have AKI or No AKI.  Risk adjustment models have significant limitations and need to be updated at regular intervals.  MPOG AKI 01 risk adjusted performance will be shared in both QI Reporting Tool and DataDirect.

Risk Adjustment Details

Background

Risk adjustment in health care quality measurement can be important to accurately compare health care outcomes across different providers and settings.  It can account for differences in patient populations, such as age, gender, severity of illness, and other factors that can affect health outcomes.

Risk adjustment can isolate the impact of the health care provider’s performance from the influence of patient characteristics. This can help ensure that performance scores reflect the quality of care provided rather than the underlying health status of the patients2 and enable the identification of areas needing improvement by highlighting true performance differences rather than differences due to patient mix.

Methodology

AKI 01 (percentage of cases complicated by acute kidney injury) identifies cases where AKI is not an expected outcome of the procedure.  Our risk adjustment process started with identifying plausible variables that may affect whether a patient is at greater risk for postoperative AKI.  These include:

    1. Pre-existing elevated Creatinine / CKD
    2. Age
    3. Male sex
    4. Race
    5. Hypertension
    6. Active CHF
    7. Pulmonary disease
    8. Insulin dependent diabetes
    9. Peripheral vascular disease
    10. Ascites
    11. BMI
    12. Pre-eclampsia (for OB cases)

We reviewed the MPOG database to determine how these variables are represented in MPOG and confirmed that missing data for these variables would not affect the risk adjustment model. Characteristics of our AKI risk adjustment model training dataset include:

    1. Data from all MPOG sites from 1/1/2023 to 12/31/2024 (2 years of data)
    2. The starting dataset included all cases included in the AKI measure (case exclusions listed here). The starting population was 1,008,585 cases.
    3. Additional cases were excluded based on missing, unknown, or invalid BMI, eGFR, ASA PS, body region, or sex data. Cases with the body region “Other Procedures” were also excluded.  The final population consisted of 818,856 cases.
    4. MPOG variables used in the model included:

Summary statistics

We summarized the data and reviewed the frequencies and distributions of each of these variables using R version 4.4.1.  AKI was determined in 4.3% of cases in this dataset. We reviewed the balance between cases with and without AKI across each of the variables using standardized differences, with a standardized difference greater than .2 indicating lack of balance between the two groups.

Risk Adjustment Model Used

A generalized mixed effects model (GLMM) was developed to predict how likely a case was to have AKI.  Using this model accounts for differences in institutional practice by including institution in the model as a random effect.  We tested the model for multicollinearity with Variance Inflation Factors (VIF).   VIFs for all covariates were less than 5, indicating our model did not have high multicollinearity. We measured discrimination with area under the receiver operating characteristics curve (AUC = .7009).  The model generated a predicted value of AKI for each case. 

Using the Predicted Value to Generate a Predicted Result of AKI

We then had to determine which predicted value would serve at the Cutoff Value to predict AKI for a specific case.  All values above the Cutoff Value would be assigned a result of “Predicted AKI = yes”.  All results at or below the Cutoff Value would be assigned a result of “Predicted AKI = no”. 

Three methods of cutoff values were considered to find an optimal cutoff for predicting AKI:

    1. Cutoff value specified by Youden’s index, which is the point that maximizes the value of sensitivity + specificity – 1
    2. A cutoff by percentile (e.g. the 90th percentile of the predicted value of AKI)
    3. A cutoff that would yield the same number of predicted AKI as there were actual AKI in the dataset (4.3%).

We selected a cutoff that would yield the same number of predicted AKI as there were actual AKI in the dataset (4.3%) for the following reasons:

    1. AKI is a relatively rare outcome, and we wanted to find a cutoff resulting in an adequate sensitivity, that is, not missing too many true positives
    2. We did not want to predict too many false positives and result in inadequate specificity.
    3. In our analysis, the number of predicted AKI using Youden’s Index was not clinically plausible. Additionally, the 90th percentile cutoff generated an unacceptably high 10% predicted positive rate, with our actual positive rate being only 4.3%.  This also meant predicting a relatively high number of false positives.

The actual Cutoff Value used was 0.1127162295.  Cases with values greater than 0.1127162295 were assigned a result of “Predicted AKI” and cases with values less than or equal to 0.1127162295 were assigned a result of “Predicted No AKI”.  The table and image below display key information used to determine the cutoff value for predicted AKI.  The graph shows the number of predicted AKI for each cutoff value, represented by the total number of cases to the right of each vertical bar.

AKI Risk Adjustment Model Results

Predicted AKI for each cutoff value selection method

Limitations

There are several important limitations that should be carefully considered when reviewing AKI risk adjusted performance:

    1. The model is limited by data accuracy. While this is true of all MPOG analyses, this is worth specifically mentioning as we will be sharing benchmarking data (anonymized) in our QI Reporting Tool.
    2. The cutoff value for Predicted AKI was determined by current AKI rates across MPOG. While we are confident this was the best available methodology for determining the cutoff value, it may not be representative of what the Predicted Rate should be, based on a more generalized population.
    3. The model will become stale over time, as new institutions join MPOG, practice changes, and data quality improves. MPOG will need to update the model in regular intervals (likely annually) to mitigate this.
    4. As only existing active sites with data can be included in the model, new sites will have to wait until the next model update cycle to view risk adjusted data

Application of AKI Risk 01 Adjustment

We plan to share risk adjusted data at a site level (not provider level) in the QI Reporting Tool.  Variation in case counts across providers could make the model inaccurate for individual providers.  Practice leaders will be able to view site actual and risk adjusted performance for AKI, and comparison data across anonymized MPOG hospitals.  Additionally, sites will be able to query for risk adjustment values for individual cases using DataDirect.

References

Index for rating diagnostic tests (Youden’s Index)
MPOG AKI 01 Measure Specification
Perioperative AKI
Risk Adjustment | CMS
Determine Risk Adjustment | The Measures Management System
Receiver operating characteristic curve: overview and practical use for clinicians – PMC
SAS Global Forum 2012: A unified approach to measuring the effect size between two groups using SAS
GLMM in SPSS
Multicollinearity in Regression Analysis: Problems, Detection, and Solutions
CMS – Risk Adjustment and Risk Stratification in Quality Measurement

Odds Ratio for AKI Risk Adjustment Variables

For any inquiries please email

mpog-admin@med.umich.edu