Evaluating the Performance of Agreement Metrics in a Delphi Study on Chemical, Biological, Radiological and Nuclear Major Incidents Preparedness Using Classical and Machine Learning Approaches

Farhat, Hassan, Batt, Alan M., Helou, Mariana, Shin, Heejun, Laughton, James, Dumbeck, Carolyn, Dehghani, Arezoo, Rezaei, Fatemeh, Bajow, Nidaa, Mortelmans, Luc, Abougalala, Walid, Mugavero, Roberto, Ciottone, Gregory, Alinier, Guillaume and Dhiab, Mohamed Ben (2025) Evaluating the Performance of Agreement Metrics in a Delphi Study on Chemical, Biological, Radiological and Nuclear Major Incidents Preparedness Using Classical and Machine Learning Approaches. Journal of Contingencies and Crisis Management, 33 (2): e70044. pp. 1-20. ISSN 0966-0879
Copy

Delphi studies in disaster medicine lack consensus on expert agreement metrics. This study examined various metrics using a Delphi study on chemical, biological, radiological, and nuclear (CBRN) preparedness in the Middle East and North Africa region. Forty international disaster medicine experts evaluated 133 items across ten CBRN Preparedness Assessment Tool themes using a 5‐point Likert scale. Agreement was measured using Kendall's W, Intraclass Correlation Coefficient, and Cohen's Kappa. Statistical and machine learning techniques compared metric performance. The overall agreement mean score was 4.91 ± 0.71, with 89.21% average agreement. Kappa emerged as the most sensitive metric in statistical and machine learning analyses, with a feature importance score of 168.32. The Kappa coefficient showed variations across CBRN PAT themes, including medical protocols, logistics, and infrastructure. The integrated statistical and machine learning approach provides a promising method for understanding expert consensus in disaster preparedness, with potential for future refinement by incorporating additional contextual factors.


picture_as_pdf
1468-5973.70044.pdf
subject
Published Version
Available under Creative Commons: BY 4.0

View Download

Atom BibTeX OpenURL ContextObject in Span OpenURL ContextObject Dublin Core MPEG-21 DIDL Data Cite XML EndNote HTML Citation METS MODS RIOXX2 XML Reference Manager Refer ASCII Citation
Export

Downloads