Supporting the classifcation of patients in public hospitals in Chile by designing, deploying and validating a system based on natural language processing

Abstract

Background: In Chile, a patient needing a specialty consultation or surgery has to frst be referred by a general practitioner, then placed on a waiting list. The Explicit Health Guarantees (GES in Spanish) ensures, by law, the maximum time to solve 85 health problems. Usually, a health professional manually verifes if each referral, written in natural language, corresponds or not to a GES-covered disease. An error in this classifcation is catastrophic for patients, as it puts them on a non-prioritized waiting list, characterized by prolonged waiting times. Methods: To support the manual process, we developed and deployed a system that automatically classifes referrals as GES-covered or not using historical data. Our system is based on word embeddings specially trained for clinical text produced in Chile. We used a vector representation of the reason for referral and patient’s age as features for training machine learning models using human-labeled historical data. We constructed a ground truth dataset combining classifcations made by three healthcare experts, which was used to validate our results. Results: The best performing model over ground truth reached an AUC score of 0.94, with a weighted F1-score of 0.85 (0.87 in precision and 0.86 in recall). During seven months of continuous and voluntary use, the system has amended 87 patient misclassifcations. Conclusion: This system is a result of a collaboration between technical and clinical experts, and the design of the classifer was custom-tailored for a hospital’s clinical workfow, which encouraged the voluntary use of the platform. Our solution can be easily expanded across other hospitals since the registry is uniform in Chile.