Skip to main content
Download PDF

Construction and validation of machine learning models for sepsis prediction in patients with acute pancreatitis

BMC Surgery volume 23, Article number: 267 (2023) Cite this article

Abstract

Background

This study aimed to construct predictive models for the risk of sepsis in patients with Acute pancreatitis (AP) using machine learning methods and compared optimal one with the logistic regression (LR) model and scoring systems.

Methods

In this retrospective cohort study, data were collected from the Medical Information Mart for Intensive Care III (MIMIC III) database between 2001 and 2012 and the MIMIC IV database between 2008 and 2019. Patients were randomly divided into training and test sets (8:2). The least absolute shrinkage and selection operator (LASSO) regression plus 5-fold cross-validation were used to screen and confirm the predictive factors. Based on the selected predictive factors, 6 machine learning models were constructed, including support vector machine (SVM), K-nearest neighbour (KNN), multi-layer perceptron (MLP), LR, gradient boosting decision tree (GBDT) and adaptive enhancement algorithm (AdaBoost). The models and scoring systems were evaluated and compared using sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), accuracy, and the area under the curve (AUC).

Results

A total of 1, 672 patients were eligible for participation. In the training set, 261 AP patients (19.51%) were diagnosed with sepsis. The predictive factors for the risk of sepsis in AP patients included age, insurance, vasopressors, mechanical ventilation, Glasgow Coma Scale (GCS), heart rate, respiratory rate, temperature, SpO2, platelet, red blood cell distribution width (RDW), International Normalized Ratio (INR), and blood urea nitrogen (BUN). The AUC of the GBDT model for sepsis prediction in the AP patients in the testing set was 0.985. The GBDT model showed better performance in sepsis prediction than the LR, systemic inflammatory response syndrome (SIRS) score, bedside index for severity in acute pancreatitis (BISAP) score, sequential organ failure assessment (SOFA) score, quick-SOFA (qSOFA), and simplified acute physiology score II (SAPS II).

Conclusion

The present findings suggest that compared to the classical LR model and SOFA, qSOFA, SAPS II, SIRS, and BISAP scores, the machine learning model-GBDT model had a better performance in predicting sepsis in the AP patients, which is a useful tool for early identification of high-risk patients and timely clinical interventions.

Peer Review reports

Background

Acute pancreatitis (AP), an inflammatory disease of the pancreas, is the leading cause of hospital admissions for gastrointestinal diseases worldwide [1, 2]. The worldwide incidence rate of AP is 33.74 per 100,000 person-years, with a gradual increase in incidence [3, 4]. Approximately 10–20% of patients with AP have complicated systemic inflammatory response syndrome (SIRS) and multiple organ dysfunction syndrome, which can lead to the development of severe AP with a mortality rate of 10–15% [5]. Sepsis is a life-threatening SIRS caused by the host’s dysregulated response to infection, which ultimately leads to septic shock and multiple organ failure and is the main cause of health loss all over the world [6]. Up to 40–70% of patients with AP will develop an infection related to pancreatitis in the late stages, or sepsis in severe cases [7, 8]. The progression of AP to sepsis is associated with higher mortality rates and a poor prognosis [9]. Therefore, early identification of AP patients who are likely to develop sepsis is of great significance for reducing mortality and disease burden.

Several scoring systems have been identified to predict the severity and prognosis of AP and sepsis, including the SIRS score, bedside index for severity in acute pancreatitis (BISAP) score, sequential organ failure assessment (SOFA) score, quick-SOFA (qSOFA), simplified acute physiology score II (SAPS II) [10,11,12]. However, poor performances of scoring systems in predicting sepsis have been observed [13]. The predictive performance of the logistic regression (LR) model based on conventional clinical indicators in predicting sepsis among patients with AP was also moderate, with the area under the receiver (AUC) of the operating characteristic curve (ROC) value being 0.73 [9]. Advanced machine learning algorithms can model nonlinear relationships, analyze complex high-order interactions, and robustly handle multicollinearity among the predictor variables [14]. Machine learning has been widely used in the diagnosis/risk prediction of sepsis, and the prognosis of sepsis. A database study conducted a machine learning approach to predict 30-day mortality for patients with sepsis, the AUC of the model was 0.857 [15]. A study conducted in the Chinese population used a machine learning model for accurate prediction of sepsis in intensive care unit (ICU) patients, the established machine learning-based model showed good predictive ability with AUC being 0.91 [16]. In addition, the machine learning model also showed excellent predictive value for severe AP and concurrent acute kidney injury (AKI) risk in AP [17, 18]. However, to the best of our knowledge, no study has reported the application of machine learning in predicting the risk of sepsis in patients with AP. The early detection and prediction of patients who may develop sepsis are essential to improve the adverse consequences of AP.

Herein, this study aimed to (1) construct predictive models for the risk of sepsis in patients with AP using machine learning methods and validate the predictive performances; (2) select the optimal machine learning model and compare it with the LR model and scoring systems. This study may help to identify the risk of sepsis in patients with AP at an early stage and assist in the clinical treatment of AP and the prevention of sepsis.

Methods

Data design and study population

This study was a retrospective cohort study. Data were collected from Medical Information Mart for Intensive Care III (MIMIC III) database (https://mimic.mit.edu/docs/iii/) between 2001 and 2012 and the MIMIC IV database (https://mimic.mit.edu/docs/iv/) between 2008 and 2019. MIMIC-III includes data from more than 58,000 admissions to Beth Israel Deaconess Medical Center in Boston from 2001 to 2012, including 38,645 adults and 7,875 neonates [19], and MIMIC-IV includes 524,740 admissions for 382,278 patients at this center from 2008 to 2019 [20, 21]. The included criteria were (1) aged ≥ 18 years; (2) diagnosed with AP upon intensive care unit (ICU) admission. Excluded criteria were (1) patients with a length of ICU stay less than 24 h; (2) diagnosed as sepsis upon ICU admission. he requirement of ethical approval for this was waived by the Institutional Review Board of Tianjin Medical University General Hospital, because the data was accessed from MIMIC III database and MIMIC IV database (publicly available database). The need for written informed consent was waived by the Institutional Review Board of Tianjin Medical University General Hospital due to retrospective nature of the study. All methods were performed in accordance with the relevant guidelines and regulations.

Data extraction

Data collected from the database including (1) baseline characteristics: age (years), gender (male), Race (Black, White, and other), insurance (government, private, and unknown), marital status (divorced, married, separated, single, widowed, and unknown), interventions (vasopressors, mechanical ventilation), and effusion; (2) vital signs: heart rate (bpm), respiratory rate (breaths/minute), temperature (°C), SpO2 (%), systolic blood pressure (SBP, mmHg), diastolic blood pressure (DBP, mmHg); (3) scoring systems: SOFA score, qSOFA score, SAPS II score, BISAP, SIRS, Glasgow Coma Scale (GCS), charlson comorbidity index (CCI), International Normalized Ratio (INR); (4) laboratory values: white blood cell (WBC, K/uL) count, platelet count (K/uL), hemoglobin (g/dL), red blood cell distribution width (RDW, %), hematocrit (%), bilirubin (mg/dL), blood creatinine (mg/dL), prothrombin time (PT, sec), partial thromboplastin time (PTT, sec), blood urea nitrogen (BUN, mg/dL), glucose (mg/dL), calcium (mg/dL), sodium (mEq/L), chloride (mEq/L), and total bicarbonate (mEq/L). All the data were extracted from the data generated within the first 24 h after the patient entered the ICU.

Variable definition

Patients diagnosed with AP were determined by using the International Classification of Diseases (ICD) (ninth edition, code 577.0 or 10th version, code K 85.0) codes. Sepsis was diagnosed according to the sepsis-3 criteria [22]; in brief, patients with documented or suspected infection and an acute change in total SOFA score of ≥ 2 points were considered to have sepsis. Infection was identified from the ICD code.

SOFA score calculated the dysfunction of six organ systems and the severity of the dysfunction, including the respiratory, coagulation, liver, cardiovascular, kidney, and nervous systems with a score of 0–4 for each item and a total score of 0–24 [23]. qSOFA score: calculated by the presence of changes in mental status, respiratory rate > 22 breaths per minute, and preoperative systolic blood pressure < 100 mmHg [22]. The SAPS II score (0-163) consists of 17 variables composed of 12 physiological variables, age, type of admission, and three different underlying disease variables [24]. Components of the BISAP scoring system included BUN > 25 mg/dl, impaired mental status, SIRI, age > 60 years, and pleural effusion [25]. SIRS was defined as two or more out of the following four: temperature > 38.0 °C or < 36.0 °C, heart rate > 90 beats/minute, respiratory rate > 20 breaths/minute, leukocytosis > 12,000/dL, or leucopenia < 4,000/dL [26].

Outcome and follow-up

The outcome of the study was the risk of sepsis. Follow-up was conducted during hospitalization in the ICU and the end point of follow-up was sepsis or discharge from the ICU. The mean follow-up time was 3.64 (1.93–9.70) days.

Construction and performances assessment of the machine learning models

The patients were randomly divided into two groups, of which 80% were used as the training set and the remaining 20% as the testing set. Based on the predictive factors selected, 6 machine learning models were constructed including support vector machine (SVM), K-nearest neighbor (KNN), multi-layer perceptron (MLP), LR, gradient boosting decision tree (GBDT), and adaptive enhancement algorithm (AdaBoost). The models were evaluated and compared by sensitivity, specificity, positive prediction value (PPV), negative prediction value (NPV), accuracy, and the AUC of the ROC.

Sample size calculation for predictive models

Our sample size calculation aimed to ensure a precise estimation of model parameters while minimizing the potential of overfitting. In order to achieve the goal of an average absolute prediction error (MAPE) of 0.05, as suggested by Riley et al. [27], 478 samples would be sufficient for a maximum of 13 predictors, a statistically determined risk prediction model.

Statistical analyses

Variables with more than 20% missing values were excluded from further analysis. Random forest imputation was used to deal with missing data below 20%. Random forest imputation is a nonparametric algorithm that accommodates nonlinearities and interactions and does not require the specification of a specific parametric model [28]. Supplementary Table 1 shows the variables with missing values below 20%. Sensitivity analysis was performed to compare the data before and after imputation (Supplementary Table 2). Means ± standard deviations (SD) was used to describe the distribution of normally distributed measurement data, and T-test was used to compare the differences between the two groups. Medians and quartiles were used to represent measurement data that conformed to a normal distribution, and rank-sum tests were used for comparisons between groups. Count data were expressed as the number of cases and composition ratio (%), and the chi-square test was used for comparison between groups.

The least absolute shrinkage and selection operator (LASSO) (“LassoCV” method in Sklearn) regression plus 5-fold cross-validation were used to screen and confirm the predictive factors and selected the best alpha = 0.0075 when one standard error of the minimum mean squared error (MSE) was used as a screening criterion. In order to select the optimal model from the 6 machine learning models, Delong’s test was used. Comparing the performance of the optimal machine learning model with LR, scoring systems (SOFA, qSOFA, SIRS, SAPS II, and BIASP). Clinical benefit was assessed using Decision Curve Analysis (DCA). A P < 0.05 was considered statistically significant. Python 3.9.0 (Python Software Foundation) and R (version 4.2.2) were used for all analyses.

Results

Basic characteristics of the study population

A total of 1,930 participants diagnosed with AP were screened from MIMIC III and MIMIC IV databases; of these 1,930 patients, 256 were excluded due to the length of ICU stay less than 24 h, and 2 were excluded due to the age < 18 years. Finally, 1, 672 patients were eligible for participation, with 1,338 patients in the training set and 334 patients in the testing set. The flow chart of the participants’ selection is depicted in Fig. 1. In the training set, 261 AP patients (19.51%) were diagnosed with sepsis. In the training set, the mean age of the AP patients with sepsis was 58.43 (16.50) years, 58.2% of the AP patients with sepsis were male, 42.9% of the AP patients with sepsis were married, 67.8% of the AP patients with sepsis were on vasopressors, and 95.8% of the AP patients with sepsis were on mechanical ventilation. The mean GCS score of AP patients with sepsis was 9.73 (4.48), the mean heart rate of AP patients with sepsis was 102.91 (23.15) bpm, and the mean respiratory rate of AP patients with sepsis was 22.57 (6.86) breaths/minute. There were significant differences between AP patients with and without sepsis in insurance, marital status, vasopressors, mechanical ventilation, GCS, heart rate, SBP, respiratory rate, SpO2, WBC, RDW, blood creatinine, BUN, bicarbonate, SOFA, qSOFA, SAPS II, SIRS, and BISAP (each P < 0.05). All baseline characteristics of the study population are summarized in Table 1.

Fig. 1
figure 1

The flow chart of the participants selection

Full size image
Table 1 Basic characteristics of study population
Full size table

Predictive factors selection for the risk of sepsis in AP patients

After LASSO regression selection with 5-fold cross-validation via minimum criteria, 13 variables remain as the predictive factors for the risk of sepsis in AP patients: age, insurance, vasopressors, mechanical ventilation, GCS, heart rate, respiratory rate, temperature, SpO2, platelet, RDW, INR, and BUN. Figure 2 shows the loss curves for the MSE loss with different Lambda. The SHAP plot (Fig. 3) shows the relationship between the value of features and their impact on the model prediction. Each row represents the SHAP value distributions of a feature, and the x-axis refers to the SHAP value, where the value of SHAP > 0 shows that the prediction favors the positive class, and a value < 0 indicates that the prediction tends to be the negative class. The color of sample points in Fig. 3 indicates the corresponding feature value: redder points mean higher feature importance values, while bluer points indicate lower feature values. The features are sorted according to the sum of SHAP values incorporating all the samples in the dataset.

Fig. 2
figure 2

The loss curves for the MSE loss with different Lambda

Full size image
Fig. 3
figure 3

The SHAP plot of the relationship between the value of features and their impact on the model prediction

Full size image

Construction and performance validations of machine learning models

Based on the predictive factors, 6 machine learning models were constructed. The AUC value in the training set of the GBDT model was 0.994 [95% confidence interval (CI): 0.988 to 1.000], higher than the AUC value of the LR model (0.890, 95% CI: 0.860 to 0.920), AdaBoost model (0.918, 95% CI: 0.894 to 0.941), SVM model (0.912, 95% CI: 0.888 to 0.936), KNN model (0.908, 95% CI: 0.883 to 0.933), and MLP model (0.948, 95% CI: 0.929 to 0.967). In the testing set, GBDT had the highest AUC value (0.985, 95% CI: 0.966 to 1.000), thereby, GBDT was selected as the final predictive model. The ACU of the GBDT model (0.985, 95% CI: 0.966 to 1.000) was higher than the LR model (0.896, 95% CI: 0.841 to 0.951), achieving statistical significance (P < 0.001). Comparisons of predictive performances among machine learning models are shown in Table 2.

Table 2 Construction and performance validations of machine learning models
Full size table

Comparisons of the predictive performances of the GBDT model with LR model, SOFA, qSOFA, SAPS II, SIRS, and BISAP scores

In the testing set, the GBDT model achieved the best performance with an AUC of (0.985, 95% CI: 0.966 to 1.000) compared with qSOFA score (AUC: 0.780, 95% CI: 0.709 to 0.852, P < 0.001), SAPS II score (AUC: 0.625, 95% CI: 0.529 to 0.720, P < 0.001), SIRS (AUC: 0.552, 95% CI: 0.461 to 0.64, P < 0.001), SOFA score (AUC: 0.745, 95% CI: 0.660 to 0.829, P < 0.001), and BISAP score (0.566, 95% CI: 0.472 to 0.660, P < 0.001). Comparisons of the predictive performances of the GBDT model with SOFA, qSOFA, SAPS II, SIRS, and BISAP scores. Comparisons of the predictive performances of the GBDT model with SOFA, qSOFA, SAPS II, SIRS, and BISAP scores are shown in Table 3. Figure 4 shows the ROC curve comparison between GBDT and LR models and scoring systems. The net benefit for predicting sepsis in AP patients using the GBDT model was greater than the LR model and scoring systems at different threshold probabilities (Fig. 5).

Table 3 Comparisons of the predictive performances of the GBDT model with LR model, SOFA, qSOFA, SAPS II, SIRS, and BISAP scores
Full size table
Fig. 4
figure 4

The ROC curve comparison between GBDT and LR models and scoring systems

Full size image
Fig. 5
figure 5

The net benefit of GBDT model, LR model, and scoring systems at different threshold probabilities for predicting sepsis in AP patients

Full size image

Discussion

In this retrospective study, we developed and validated machine learning-based models for predicting sepsis in AP patients. In the training set, 261 AP patients (19.51%) were diagnosed with sepsis. The results of this study showed that the GBDT model had an excellent performance in the prediction of sepsis in patients with AP, with the AUC in the testing set at 0.985. Furthermore, the GBDT model achieved better predictive performance for sepsis prediction in AP patients compared with the LR model, and scoring systems.

Advanced machine learning methods are good at dealing with high-order interactions and fitting complex nonlinear relationships, and can be used to integrate large amounts of data from electronic health records (EHRs). The application of machine learning to data-driven analysis shows promise for improving predictive performance in healthcare [29,30,31]. A large retrospective study developed and validated a machine learning tool within 48 h after admission for predicting which patients with AP [32]. A retrospective study enrolling patients with AP from multiple centers explored a machine learning model for early identification of severe AP (SAP) among patients hospitalized for AP, and the model showed evident clinical practicability [17]. The study by Qiu et al. developed and validated three machine-learning models for predicting multiple organ failure in moderately severe and severe AP [33]. A systematic review included 47 machine learning predictive models for AP, with 10 studies reporting severity prediction, 10 studies complication prediction, 3 studies mortality prediction, 2 studies recurrence prediction, and 2 studies surgery timing prediction [34]. The study by İnce et al. evaluated the success of artificial intelligence for early prediction of severe course, survival, and ICU requirements in patients with AP [35]. A meta-analysis suggested that the machine learning approach had a better performance than the existing sepsis scoring systems in predicting sepsis [36]. A systematic review and meta-analysis showed that individual machine learning models can accurately predict sepsis onset ahead of time [37]. A machine learning model for prediction of sepsis in ICU patients showed good predictive ability in Chinese sepsis patients [16]. However, there have been limited studies that constructed predictive models for the risk of sepsis in patients with AP using machine learning methods. This study used machine learning methods to construct predictive models for the risk of sepsis in patients with AP and validated the predictive performance.

The results of this study showed that the GBDT model had an excellent performance in predicting sepsis in AP patients. The GBDT model has been applied to diagnose and predict the outcomes of several diseases. A study that developed and assessed machine learning models for predicting recurrence risk after endovascular treatment in patients with intracranial aneurysms found that the GBDT model showed an optimal prediction performance for predicting recurrence risk in patients with intracranial aneurysms after endovascular treatment in 6 months [38]. Lee et al. established machine learning models for predicting the risk of end-stage renal disease among chronic kidney disease patients who survive sepsis, and the GBDT algorithm yielded an accuracy as high as 0.879, as measured using the AUC [39]. Furthermore, we compared the performance of models, the traditional LR model, and scoring systems to predict sepsis in AP patients in the early stage. The result showed that the GBDT model achieved the best performance in predicting sepsis in terms of the predictive performance. Similarly, a previous study suggested that compared to the classical LR model, machine learning models using features that can be easily obtained at admission had a better performance in predicting AKI in AP patients [40]. A retrospective temporal validation study suggested an interpretable machine learning model performed significantly better than LR and outperformed conventional severity scores in predicting in-hospital mortality among sepsis patients and varying subgroups [31]. The high AUC of the GBDT model, compared to traditional models and scoring systems suggested that machine learning models can be used frequently as an adjunct to clinical decision making and provider’s intuition regarding patient prognosis and ideal next steps in care. Early and effective identification of high-risk patients with sepsis in AP patients can prevent further deterioration of the patient’s condition. This study helps clinicians to develop individualized treatment plans for patients, reducing the disease burden on patients and facilitating the rational allocation of medical resources.

GBDT is an ensemble algorithm widely used for regression and classification tasks. The GBDT algorithm creates multiple weak learners or individual trees by bootstrapping training samples and integrates their outputs to make predictions. The GBDT algorithm is less sensitive to hyperparameters, less prone to overfitting, and easy to implement. For the practical applicability of the GBDT model in a clinical setting, an example of how SHAP can be used locally to explain individual prediction was provided (Supplementary Fig. 1). The GBDT model is a promising approach for sepsis prediction in AP patients, but further research is still needed to evaluate its generalizability to other tasks and its computational efficiency.

This study suggested that the basic characteristics of patients (age, temperature, and insurance) and vital signs (heart rate, respiratory rate, and SpO2 were associated with the risk of sepsis in AP. A study by Hong et al. indicated that age may be useful for predicting the development of persistent organ failure in patients with AP [41]. According to the study by Miller et al., an ED-SAS score that incorporates factors including SpO2 and age provides a rapid method for predicting prognosis in AP [42]. The temperature has been reported as a predictor factor for sepsis in AP patients [9]. heart rate has been observed to be associated with severe AP [43]. The interventions also can predict the risk of sepsis in AP. Early vasopressor use was significantly associated with increased in-hospital mortality among critically ill AP patients [44]. We found that the inflammatory markers including RDW and platelets can predict the risk of sepsis in AP patients. As a part of routine blood tests, RDW is a quantitative measurement of the size variability of peripheral blood red blood cells (RBCs), which reflects the heterogeneity of RBCs. Because the changes in the shape and size of circulating red blood cells are often related to the occurrence and development of hematological diseases, RDW is used for the morphological classification of anemia and differential diagnosis of microcytic anemia [45]. RDW is positively associated with AP severity and is likely a useful predictive parameter of AP severity [46]. Platelets are small pieces of cytoplasm shed by mature megakaryocytes, which participate in the hemostasis function of the body. When the stress effect secondary to acute and critical diseases occurs in the body, the number of platelets will change, and the degree of platelet change will affect sepsis [47]. A study by Feng et al. found that a low platelet count increases the risk of sepsis in patients with AP [9]. Simple, routine, and widespread individual laboratory parameter, BUN has been proposed as marker of disease severity [48]. In this study, BUN could be used to predict the risk of sepsis in AP patients. A study by Hong et al. demonstrated that BUN could predict severe AP [49]. Farrell et al. found that persistent elevation of BUN is associated with the development of severe AP [50]. Previous studies have also suggested that BUN is strongly associated with sepsis [51, 52]. GCS was originally used as an assessment tool for patients with head injuries to assess the coma of patients, which has become an important part of the system to determine the severity of an injury [53]. In this study, GCS could predict the risk of sepsis in patients with AP. A retrospective analysis also demonstrated that GCS was among the predictive factor of sepsis among patients with AP [9].

Our study has several strengths. To the best of our knowledge, we first report the application of machine learning models to predict the risk of sepsis in AP patients using the MIMIC database. The optimal model was screened using a variety of machine learning methods and showed significantly better predictive value than LR and scoring systems, providing a basis for the accurate prediction of sepsis risk in AP patients. The sample size in this study is very sufficient for the construction and validation of prediction models. A larger sample size is valuable for developing a more robust prediction model, which has good generalization ability and good statistical efficacy for a wider population. However, the study was still subject to some limitations. First, the retrospective nature of the study may have introduced unavoidable selection bias, which limits the interpretation of the results. Second, the MIMIC data were obtained from a single center in the United States, which may affect the generalizability of the prediction model to other populations. The results may not be representative of the entire population of AP patients, although we attempted to provide detailed information in our study. Third, the study included AP patients in MIMIC-III and IV, which included hospitalized patients from 2001 to 2019. The population studied here is not consecutive and therefore different biases may have been introduced. As treatment regimens are developed and optimized, consistency of treatment regimens cannot be guaranteed, which may introduce some bias into the results. Fourth, radiological results in AP, specific chemoradiotherapy information, and medication dosage in vasopressors and mechanical ventilation may have an impact on our results, but the lack of radiological data in the database prevented us from performing further analyses. Fifth, the study lacked external validation. External validation is crucial to assess the generalizability and reliability of the model, especially when using data from a single center. Therefore, it would be important to perform further validation on an independent dataset in future studies to examine the robustness and generalization ability of the proposed model, which might greatly increase the impact of the current finding. Future research will need to explore other machine learning algorithms for predicting sepsis in AP patients.

Conclusions

This study constructed and validated machine learning models to predict sepsis in patients with AP. The GBDT model, based on 13 predictive factors, showed promising performance in predicting sepsis in AP patients. A prediction model is a useful tool for the early identification of high-risk patients and timely clinical intervention.

Data Availability

The datasets generated and/or analyzed during the current study are available in the MIMIC III database (https://mimic.mit.edu/docs/iii/) and MIMIC IV database (https://mimic.mit.edu/docs/iv/).

Abbreviations

AP:

Acute pancreatitis

SIRS:

systemic inflammatory response syndrome

SOFA:

sequential organ failure assessment

qSOFA:

quick-SOFA

SAPS II:

simplified acute physiology score II

LR:

logistic regression

AUC:

area under the receiver

ROC:

AUC of the operating characteristic curve

AKI:

acute kidney injury

MIMIC III:

Medical Information Mart for Intensive Care III

SBP:

systolic blood pressure

DBP:

diastolic blood pressure

GCS:

Glasgow Coma Scale

CCI:

charlson comorbidity index

INR:

International Normalized Ratio

WBC:

white blood cell

PT:

prothrombin time

PTT:

partial thromboplastin time

BUN:

blood urea nitrogen

ICD:

International Classification of Diseases

SVM:

support vector machine

KNN:

K-nearest neighbor

MLP:

multi-layer perceptron

GBDT:

gradient boosting decision tree

AdaBoost:

adaptive enhancement algorithm

PPV:

positive prediction value

NPV:

negative prediction value

SD:

standard deviations

References

  1. Kiriyama M, Jitsuiki K, Muramatsu KI, Furusawa H, Moriya S, Yanagawa Y. Posttraumatic Pancreatitis Four Days after Renal Injury with Massive Retroperitoneal Hematoma. Case Rep Emerg Med 2021, 2021:6693259.

  2. Peery AF, Crockett SD, Murphy CC, Jensen ET, Kim HP, Egberg MD, Lund JL, Moon AM, Pate V, Barnes EL, et al. Burden and cost of gastrointestinal, liver, and pancreatic Diseases in the United States: Update 2021. Gastroenterology. 2022;162(2):621–44.

    PubMed  Google Scholar 

  3. Lee PJ, Papachristou GI. New insights into acute pancreatitis. Nat Rev Gastroenterol Hepatol. 2019;16(8):479–96.

    CAS  PubMed  Google Scholar 

  4. van den Berg FF, van Dalen D, Hyoju SK, van Santvoort HC, Besselink MG, Wiersinga WJ, Zaborina O, Boermeester MA, Alverdy J. Western-type diet influences mortality from necrotising pancreatitis and demonstrates a central role for butyrate. Gut. 2021;70(5):915–27.

    PubMed  Google Scholar 

  5. Párniczky A, Lantos T, Tóth EM, Szakács Z, Gódi S, Hágendorn R, Illés D, Koncz B, Márta K, Mikó A, et al. Antibiotic therapy in acute pancreatitis: from global overuse to evidence based recommendations. Pancreatology: official journal of the International Association of Pancreatology (IAP) [et al]. 2019;19(4):488–99.

    PubMed  Google Scholar 

  6. Rudd KE, Johnson SC, Agesa KM, Shackelford KA, Tsoi D, Kievlan DR, Colombara DV, Ikuta KS, Kissoon N, Finfer S, et al. Global, regional, and national sepsis incidence and mortality, 1990–2017: analysis for the global burden of Disease Study. Lancet. 2020;395(10219):200–11.

    PubMed  PubMed Central  Google Scholar 

  7. Wang C, Zhang J, Liu L, Qin W, Luo N. Early predictive value of presepsin for secondary sepsis and mortality in intensive care unit patients with severe acute pancreatitis. Shock; 2023.

  8. Susak YM, Dirda OO, Fedorchuk OG, Tkachenko OA, Skivka LM. Infectious complications of Acute Pancreatitis is Associated with Peripheral Blood Phagocyte functional exhaustion. Dig Dis Sci. 2021;66(1):121–30.

    CAS  PubMed  Google Scholar 

  9. Feng A, Ao X, Zhou N, Huang T, Li L, Zeng M, Lyu J. A Novel Risk-Prediction Scoring System for Sepsis among Patients with Acute Pancreatitis: A Retrospective Analysis of a Large Clinical Database. Int J Clin Pract 2022, 2022:5435656.

  10. Teng TZJ, Tan JKT, Baey S, Gunasekaran SK, Junnarkar SP, Low JK, Huey CWT, Shelat VG. Sequential organ failure assessment score is superior to other prognostic indices in acute pancreatitis. World J Crit Care Med. 2021;10(6):355–68.

    PubMed  PubMed Central  Google Scholar 

  11. van der Woude SW, van Doormaal FF, Hutten BA, Holleman FJN. Classifying sepsis patients in the emergency department using SIRS, qSOFA or MEWS. Neth J Med. 2018;76(4):158–66.

    PubMed  Google Scholar 

  12. Kaplan M, Ates I, Oztas E, Yuksel M, Akpinar MY, Coskun O, Kayacetin E. A new marker to Determine Prognosis of Acute Pancreatitis: PLR and NLR Combination. J Med Biochem. 2018;37(1):21–30.

    CAS  PubMed  PubMed Central  Google Scholar 

  13. Askim Ã, Moser F, Gustad LT, Stene H, Gundersen M, Åsvold BO, Dale J, Bjørnsen LP, Damås JK, Solligård E. Poor performance of quick-SOFA (qSOFA) score in predicting severe sepsis and mortality - a prospective study of patients admitted with infection to the emergency department. Scand J Trauma Resusc Emerg Med. 2017;25(1):56.

    PubMed  PubMed Central  Google Scholar 

  14. Luo XQ, Kang YX, Duan SB, Yan P, Song GB, Zhang NY, Yang SK, Li JX, Zhang H. Machine learning-based prediction of Acute kidney Injury following Pediatric Cardiac surgery: Model Development and Validation Study. J Med Internet Res. 2023;25:e41142.

    PubMed  PubMed Central  Google Scholar 

  15. Hou N, Li M, He L, Xie B, Wang L, Zhang R, Yu Y, Sun X, Pan Z, Wang K. Predicting 30-days mortality for MIMIC-III patients with sepsis-3: a machine learning approach using XGboost. J Transl Med. 2020;18(1):462.

    CAS  PubMed  PubMed Central  Google Scholar 

  16. Wang D, Li J, Sun Y, Ding X, Zhang X, Liu S, Han B, Wang H, Duan X, Sun T. A machine learning model for Accurate Prediction of Sepsis in ICU patients. Front Public Health. 2021;9:754348.

    PubMed  PubMed Central  Google Scholar 

  17. Yin M, Zhang R, Zhou Z, Liu L, Gao J, Xu W, Yu C, Lin J, Liu X, Xu C, et al. Automated machine learning for the early prediction of the severity of Acute Pancreatitis in Hospitals. Front Cell Infect Microbiol. 2022;12:886935.

    CAS  PubMed  PubMed Central  Google Scholar 

  18. Yang Y, Xiao W, Liu X, Zhang Y, Jin X, Li X. Machine learning-assisted ensemble analysis for the prediction of Acute Pancreatitis with Acute kidney Injury. Int J Gen Med. 2022;15:5061–72.

    PubMed  PubMed Central  Google Scholar 

  19. Yang J, Li Y, Liu Q, Li L, Feng A, Wang T, Zheng S, Xu A, Lyu J. Brief introduction of medical database and data mining technology in big data era. J Evid Based Med. 2020;13(1):57–69.

    PubMed  PubMed Central  Google Scholar 

  20. Zhang L, Wang Z, Xu F, Han D, Li S, Yin H, Lyu J. Effects of stress hyperglycemia on short-term prognosis of patients without diabetes Mellitus in Coronary Care Unit. Front Cardiovasc Med. 2021;8:683932.

    CAS  PubMed  PubMed Central  Google Scholar 

  21. Wu WT, Li YJ, Feng AZ, Li L, Huang T, Xu AD, Lyu J. Data mining in clinical big data: the frequently used databases, steps, and methodological models. Mil Med Res. 2021;8(1):44.

    PubMed  PubMed Central  Google Scholar 

  22. Singer M, Deutschman CS, Seymour CW, Shankar-Hari M, Annane D, Bauer M, Bellomo R, Bernard GR, Chiche JD, Coopersmith CM, et al. The Third International Consensus Definitions for Sepsis and septic shock (Sepsis-3). JAMA. 2016;315(8):801–10.

    CAS  PubMed  PubMed Central  Google Scholar 

  23. Khwannimit B, Bhurayanontachai R, Vattanavanit V. Comparison of the performance of SOFA, qSOFA and SIRS for predicting mortality and organ failure among sepsis patients admitted to the intensive care unit in a middle-income country. J Crit Care. 2018;44:156–60.

    PubMed  Google Scholar 

  24. Le Gall JR, Lemeshow S, Saulnier F. A new simplified Acute Physiology score (SAPS II) based on a European/North american multicenter study. JAMA. 1993;270(24):2957–63.

    PubMed  Google Scholar 

  25. Hagjer S, Kumar N. Evaluation of the BISAP scoring system in prognostication of acute pancreatitis - A prospective observational study. Int J Surg. 2018;54(Pt A):76–81.

    PubMed  Google Scholar 

  26. Chakraborty RK, Burns B. Systemic inflammatory response syndrome. In: StatPearls, editor. Treasure Island (FL): StatPearls Publishing Copyright © 2022. StatPearls Publishing LLC.; 2022.

  27. Riley RD, Ensor J, Snell KIE, Harrell FE Jr, Martin GP, Reitsma JB, Moons KGM, Collins G, van Smeden M. Calculating the sample size required for developing a clinical prediction model. BMJ. 2020;368:m441.

    PubMed  Google Scholar 

  28. Shah AD, Bartlett JW, Carpenter J, Nicholas O, Hemingway H. Comparison of random forest and parametric imputation models for imputing missing data using MICE: a CALIBER study. Am J Epidemiol. 2014;179(6):764–74.

    PubMed  PubMed Central  Google Scholar 

  29. Luo XQ, Yan P, Zhang NY, Luo B, Wang M, Deng YH, Wu T, Wu X, Liu Q, Wang HS, et al. Machine learning for early discrimination between transient and persistent acute kidney injury in critically ill patients with sepsis. Sci Rep. 2021;11(1):20269.

    CAS  PubMed  PubMed Central  Google Scholar 

  30. Zhang Z, Ho KM, Hong Y. Machine learning for the prediction of volume responsiveness in patients with oliguric acute kidney injury in critical care. Crit Care. 2019;23(1):112.

    PubMed  PubMed Central  Google Scholar 

  31. Li S, Dou R, Song X, Lui KY, Xu J, Guo Z, Hu X, Guan X, Cai C. Developing an interpretable machine learning model to Predict in-hospital mortality in Sepsis Patients: a retrospective temporal validation study. J Clin Med 2023, 12(3).

  32. Yuan L, Ji M, Wang S, Wen X, Huang P, Shen L, Xu J. Machine learning model identifies aggressive acute pancreatitis within 48 h of admission: a large retrospective study. BMC Med Inf Decis Mak. 2022;22(1):312.

    Google Scholar 

  33. Qiu Q, Nian YJ, Guo Y, Tang L, Lu N, Wen LZ, Wang B, Chen DF, Liu KJ. Development and validation of three machine-learning models for predicting multiple organ failure in moderately severe and severe acute pancreatitis. BMC Gastroenterol. 2019;19(1):118.

    PubMed  PubMed Central  Google Scholar 

  34. Zhou Y, Ge YT, Shi XL, Wu KY, Chen WW, Ding YB, Xiao WM, Wang D, Lu GT, Hu LH. Machine learning predictive models for acute pancreatitis: a systematic review. Int J Med Informatics. 2022;157:104641.

    Google Scholar 

  35. İnce AT, Silahtaroğlu G, Seven G, Koçhan K, Yıldız K, Şentürk H. Early prediction of the severe course, survival, and ICU requirements in acute pancreatitis by artificial intelligence. Pancreatology: official journal of the International Association of Pancreatology (IAP) [et al]. 2023;23(2):176–86.

    PubMed  Google Scholar 

  36. Islam MM, Nasrin T, Walther BA, Wu CC, Yang HC, Li YC. Prediction of sepsis patients using machine learning approach: a meta-analysis. Comput Methods Programs Biomed. 2019;170:1–9.

    PubMed  Google Scholar 

  37. Fleuren LM, Klausch TLT, Zwager CL, Schoonmade LJ, Guo T, Roggeveen LF, Swart EL, Girbes ARJ, Thoral P, Ercole A, et al. Machine learning for the prediction of sepsis: a systematic review and meta-analysis of diagnostic test accuracy. Intensive Care Med. 2020;46(3):383–400.

    PubMed  PubMed Central  Google Scholar 

  38. Lin S, Zou Y, Hu J, Xiang L, Guo L, Lin X, Zou D, Gao X, Liang H, Zou J, et al. Development and assessment of machine learning models for predicting recurrence risk after endovascular treatment in patients with intracranial aneurysms. Neurosurg Rev. 2022;45(2):1521–31.

    PubMed  Google Scholar 

  39. Lee KH, Chu YC, Tsai MT, Tseng WC, Lin YP, Ou SM, Tarng DC. Artificial Intelligence for Risk Prediction of End-Stage Renal Disease in Sepsis Survivors with Chronic Kidney Disease. Biomedicines 2022, 10(3).

  40. Qu C, Gao L, Yu XQ, Wei M, Fang GQ, He J, Cao LX, Ke L, Tong ZH, Li WQ. Machine Learning Models of Acute Kidney Injury Prediction in Acute Pancreatitis Patients. Gastroenterol Res Pract 2020, 2020:3431290.

  41. Hong WD, Chen XR, Jin SQ, Huang QK, Zhu QH, Pan JY. Use of an artificial neural network to predict persistent organ failure in patients with acute pancreatitis. Clin (Sao Paulo). 2013;68(1):27–31.

    Google Scholar 

  42. Miller J, Wu Y, Safa R, Marusca G, Bhatti S, Ahluwalia G, Dandashi J, Acevedo HG, Farook N, Scott A, et al. Derivation and validation of the ED-SAS score for very early prediction of mortality and morbidity with acute pancreatitis: a retrospective observational study. BMC Emerg Med. 2021;21(1):16.

    PubMed  PubMed Central  Google Scholar 

  43. Park HS, In SG, Yoon HJ, Lee WJ, Woo SH, Kim D. Predictive values of neutrophil-lymphocyte ratio as an early indicator for severe acute pancreatitis in the emergency department patients. J Lab Physicians. 2019;11(3):259–64.

    CAS  PubMed  PubMed Central  Google Scholar 

  44. Shi H, Sun SY, He YS, Peng Q. Association between early vasopressor administration and in-hospital mortality in critically ill patients with acute pancreatitis: a cohort study from the MIMIC-IV database. Eur Rev Med Pharmacol Sci. 2023;27(2):787–98.

    CAS  PubMed  Google Scholar 

  45. Fornal M, Wizner B, Cwynar M, Królczyk J, Kwater A, Korbut RA, Grodzicki T. Association of red blood cell distribution width, inflammation markers and morphological as well as rheological erythrocyte parameters with target organ damage in hypertension. Clin Hemorheol Microcirc. 2014;56(4):325–35.

    CAS  PubMed  Google Scholar 

  46. Zhang T, Liu H, Wang D, Zong P, Guo C, Wang F, Wu D, Tang M, Zhou J, Zhao Y. Predicting the Severity of Acute Pancreatitis with Red cell distribution width at early admission stage. Shock. 2018;49(5):551–5.

    PubMed  Google Scholar 

  47. Lei WNG, Guifang C, Huaxing C. Values of platelet, PT and TAT in diagnosis and prognostic evaluation of patients with severe acute pancreatitis complicated by sepsis and diffuse intravascular coagulation. Clin Misdiagnosis Mistreatment 2021, 34(9).

  48. Koutroumpakis E, Wu BU, Bakker OJ, Dudekula A, Singh VK, Besselink MG, Yadav D, Mounzer R, van Santvoort HC, Whitcomb DC, et al. Admission hematocrit and rise in blood Urea Nitrogen at 24 h outperform other laboratory markers in Predicting Persistent Organ failure and pancreatic necrosis in Acute Pancreatitis: a Post Hoc Analysis of three large prospective databases. Am J Gastroenterol. 2015;110(12):1707–16.

    PubMed  Google Scholar 

  49. Hong W, Lin S, Zippi M, Geng W, Stock S, Zimmer V, Xu C, Zhou M. High-density lipoprotein cholesterol, blood Urea Nitrogen, and serum creatinine can predict severe Acute Pancreatitis. Biomed Res Int. 2017;2017:1648385.

    PubMed  PubMed Central  Google Scholar 

  50. Farrell PR, Hornung L, Farmer P, DesPain AW, Kim E, Pearman R, Neway B, Serrette A, Sehgal S, Heubi JE, et al. Who’s at risk? A prognostic model for Severity Prediction in Pediatric Acute Pancreatitis. J Pediatr Gastroenterol Nutr. 2020;71(4):536–42.

    CAS  PubMed  PubMed Central  Google Scholar 

  51. Njim T, Dondorp A, Mukaka M, Ohuma EO. Identifying risk factors for the development of sepsis during adult severe malaria. Malar J. 2018;17(1):278.

    PubMed  PubMed Central  Google Scholar 

  52. Li JL, Li G, Jing XZ, Li YF, Ye QY, Jia HH, Liu SH, Li XJ, Li H, Huang R, et al. Assessment of clinical sepsis-associated biomarkers in a septic mouse model. J Int Med Res. 2018;46(6):2410–22.

    CAS  PubMed  PubMed Central  Google Scholar 

  53. Lu HX, Du J, Wen DL, Sun JH, Chen MJ, Zhang AQ, Jiang JX. Development and validation of a novel predictive score for sepsis risk among trauma patients. World J Emerg Surg. 2019;14:11.

    PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

Not applicable.

Funding

Not applicable.

Author information

Authors and Affiliations

  1. Department of Emergency Medicine, Tianjin Medical University General Hospital, 154 Anshan Road, Heping District, Tianjin, 300052, P.R. China

    Fei Liu & Songtao Shou

  2. Department of Anesthesiology, The First Affiliated Hospital of Hebei North University, Zhangjiakou, Hebei, 075000, P.R. China

    Jie Yao

  3. Department of Intensive Care Unit, The First Affiliated Hospital of Hebei North University, Zhangjiakou, Hebei, 075000, P.R. China

    Chunyan Liu

Authors
  1. Fei Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jie Yao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chunyan Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Songtao Shou
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

FL and SS designed the study. FL wrote the manuscript. JY and CL collected, analyzed, and interpreted the data. SS critically reviewed, edited, and approved the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Songtao Shou.

Ethics declarations

Ethics approval and consent to participate

The requirement of ethical approval for this was waived by the Institutional Review Board of Tianjin Medical University General Hospital, because the data was accessed from MIMIC III database and MIMIC IV database (publicly available database). The need for written informed consent was waived by the Institutional Review Board of Tianjin Medical University General Hospital due to retrospective nature of the study. All methods were performed in accordance with the relevant guidelines and regulations.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Table 1

Variables with missing values below 20%

Supplementary Table 2

comparations of the data before and after imputation

Supplementary Figure 1

. The practical applicability of the GBDT model using SHAP

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, F., Yao, J., Liu, C. et al. Construction and validation of machine learning models for sepsis prediction in patients with acute pancreatitis. BMC Surg 23, 267 (2023). https://doi.org/10.1186/s12893-023-02151-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12893-023-02151-y

Keywords

BMC Surgery

ISSN: 1471-2482

Contact us