Choosing a simulation assessement tool: A comparative analysis in Bihar, India

Abstract

Background: To explore whether clinical, teamwork, and interpersonal skills of nurses could be monitored inexpensively at scale, we compared two simulation measurement tools at 160 primary health care facilities in Bihar, India. The aim was to determine if the less expensive, less resource-intensive real-time assessment tool (RTAT) could be used in place of the more expensive, more resource-intensive video coding tool.

Methods: Embedded in a large-scale ongoing nurse mentoring program “AMANAT”, we conducted a cross-sectional study comparing the measurement  tools to assess clinical performance in normal and emergency obstetrical and neonatal simulations. The RTAT consisted of a series of seven global  questions related to clinical identification, management, use of evidence-based practices, communication and teamwork. We also analysed the  simulations using a video coding tool, which included 80 clinical, teamwork and communication indicators. We calculated the means and standard errors  for each indicator using both tools. We calculated the Cohen’s kappa coefficient to determine the level of agreement between the two tools.

Results: We  analysed paired data from the two tools for 222 simulations in three types of scenarios: normal spontaneous vaginal deliveries, neonatal resuscitations  and post-partum haemorrhages. The mean scores on the RTAT were generally higher than those on the video coding tool. The kappa coefficients for all  indicators indicated no agreement between the two tools.

Conclusion: RTAT performed poorly against the video coding tool to measure the clinical skills  of nurses in simulated normal and complicated births at primary health care facilities in Bihar.