Real-world testing of artificial intelligence system for surgical safety management

Abstract

Purpose : Here we report the reality of the feedback process for the social implementation of artificial intelligence (AI) systems of surgical safety management. The purpose of this report is to share a process that can help overcome barriers to AI applications in the medical field and improve the efficiency of social security costs.

Methods : This study is an analysis report of our original surgical safety management AI system (Safety AI) demonstration experiment conducted from April to October 2019 at the Department of Ophthalmology at Tsukazaki Hospital in Himeji City,Hyogo,Japan. During the period, tests were conducted in three phases. Safety AI consisted of three AI applications, patient face recognition AI (Face AI), left and right eye identification AI (L/R AI), and intraocular lens identification AI (IOL AI). In addition to the usual safety checks (Call confirmation, barcode wrapped around wrist, etc) during cataract surgery, a demonstration experiment was conducted. The identification results by AI were evaluated and recorded by multiple medical staff, and the identification success rate was calculated. A failure was recorded if the identification itself was not possible due to some reason, such as the slow response of AI and the progress of cataract surgery. Based on the results of the first and second experiments, AI models were rebuilt and on-site operational procedures was improved.

Results : The experiment was conducted with 126 face recognitions, 379 intraocular lens certifications, and 441 left-right certifications for the 1st, 2nd and 3rd periods. Our models were reconstructed after the 1st and 2nd phases. The success rate of dstinguishment was 87% → 95% → 99% for Face AI, 93% → 97% → 100% for IOL AI, and 81% → 94% → 99% for IOL AI. We received various kinds of feedback from many medical professionals to improve the Satey AI. A lot of reorganization of the operation was carried out especially about the photography acquisition method in face recognition.

Conclusions : Many challenges are found in the social implementation experiment of AI in the medical field, and the identification ability is improved and the operation method is constructed.

This is a 2020 ARVO Annual Meeting abstract.

 

Face recognition is quite difficult. Wearing a funny hat, no makeup, and sleeping on the bed, the face changes with gravity. The most difficult thing in the medical real-world is that taking a facial photo is psychologically resistant to the patient.