Background/Objectives: Clinical record abstraction is widely used in patient safety and healthcare quality research. Nevertheless, the reliability of data from these studies is frequently underreported. The SAFEST study (Improving Quality and Patient SAFEty in Surgical Care through STandardization and Harmonization of Perioperative Care in Europe) evaluates surgical complications and adherence to perioperative patient safety practices. It is a learning collaborative of ten hospitals across five countries. Hospital-based health professionals perform electronic health record (EHR) abstraction. This study describes inter-rater agreement among raters from different SAFEST hospitals.

Methods: All raters underwent a structured two-step calibration exercise involving abstraction from fictional cases and intra-hospital assessment of real EHR. Initially, raters participated in a webinar detailing all variables necessary for abstraction. Subsequently, raters independently abstracted data from three fictional clinical cases. A second webinar followed, addressing variables with lower inter-rater agreement. Thereafter, raters abstracted data from three additional fictional cases. A panel comprising one surgeon, two anaesthesiologists, and two researchers independently established the gold standards. In the second step, raters abstracted data independently from the EHR of four surgical cases. Inter-rater reliability was assessed by calculating the percent agreement among all raters for the fictional clinical cases alongside Fleiss’ kappa statistics for intra-hospital analysis.

Results: Forty raters completed calibration exercises, ranging from one to eight raters per hospital. All raters successfully abstracted the fictional clinical cases; however, one hospital with only one rater didn't perform the intra-hospital analysis. Analysis of fictional cases revealed overall percent agreement of 85.12% (83.56-87.34%) for the first three cases. Percent agreement improved for the subsequent three cases, averaging 88.74% (85.38-89.22%). Intra-hospital analysis resulted in a median Fleiss’ kappa of 0.81 (0.67-0.97) and a median percent agreement of 79.87% (51.62-95.45%).

Conclusions/Recommendations: Results indicate variability in abstraction quality among SAFEST hospitals. Assessing inter-rater reliability allowed for personalised feedback to hospitals with lower data reliability.

Funding: European Union: No. 101057825.