Machine Learning Infrastructure for Vision Screening

GoCheck Kids (GCK), is a clinically validated, easy-to-use, comprehensive photoscreening, and visual acuity app that helps to prevent vision impairments, blindness, and vision-related learning challenges in children 1-18. GoCheck Kids enables healthcare practitioners to screen for amblyopia (aka “lazy eye”) risk factors and other vision risks by providing an integrated, affordable solution in a simple smartphone app. The company currently serves over 6,500 pediatric teams in the US and Europe. The app is FDA-registered and CE-certified.

What was the challenge?

GoCheck was looking to enhance the image classification component of its pediatric photoscreening application through machine learning. Since GCK is a cloud-based system, it is a data-rich environment where it was hypothesized that the application of machine learning could improve the usability of the app by supplementing the image analysis and improving user actions to help capture the best image possible. GoCheck needed a robust and resilient ML infrastructure to run experiments on a dataset of more than one million images faster and more cost-efficiently, to train a highly accurate image classification model for the application.

What was the solution?

GoCheck, in collaboration with Amazon Web Services (AWS), identified Provectus as a partner that had the domain experience necessary to execute the experiment. Working closely with GoCheck, Provectus reviewed their image classification software, prior ML methods, and pipelines. As part of the effort to build ML models requested by the Client, Provectus delivered a secure, auditable, and reproducible ML training infrastructure deployed on AWS, including a robust experimentation environment with experiment tracking and model versioning, and active learning pipelines for continuous data re-labeling and re-training.

More efficient data preparation and faster experimentation enabled GCK to increase the recall of ML models by 3X while preserving its precision. The final results of “child not looking” were decreased by asking the user to retake the image.

How it works

In the United States, vision impairment ranks as the most prevalent disabling childhood condition, according to the CDC. Almost always treatable if detected early, eye conditions such as amblyopia and strabismus cause vision impairment and blindness. Currently, only 15% of preschool children receive an eye exam and only 20% receive some type of vision screening. Historically, photoscreening has required significant upfront capital, been difficult to scale and hard to operate, which has limited the access to pediatric photoscreening.

To remedy the situation, in 2013 GoCheck developed a next-generation clinically validated, pediatric photoscreening platform. Consisting of a smartphone app to perform vision screening and a backend hosted in the cloud, it enables healthcare practitioners to screen for amblyopia and other vision disorders in children in a fast, easy and affordable way.

To screen a child for vision disorders, a user of the app snaps a photo of the child’s eyes at a fixed distance in a dim environment. The app evaluates the image in real-time to check if it is acceptable for assessing via image processing. If so, the app analyzes the image, processes the findings through proprietary risk calculation algorithms, and provides a recommendation of “Risk Factors Identified” or “No Risk Factors Identified”. In 1-2% of the screenings, the result may be inconclusive, or “Not Gradable”. Possible reasons for a not gradable result on an image are: child not looking straight at the camera, exposure not adequate, motion blur, extraneous light sources, etc. If the result is inconclusive the pediatrician can choose to rescreen the child or refer if warranted. If the app indicates “Risk Factors Identified,” the child is referred to an eye care professional for further diagnosis and treatment. If “No Risk Factors Identified,” the child should be screened again in one year.

Advanced ML Infrastructure for Data Preparation and ML Experimentation

For GoCheck, faster and more efficient data preparation, including data re-labeling and re-training and ML experimentation means more accurate image classification, timely feedback for test administrators, and happier customers. This naturally translates into reduced rescreening rates, a better user experience for users, and lowered operational costs for GoCheck. Provectus took on the challenge to design, build and implement a machine learning infrastructure to help GoCheck achieve these success criteria.

At the preparatory stage, GCK’s datasets for gaze estimation were reviewed. Existing performance and KPIs were analyzed, and desirable metrics were agreed upon. It was critical to assess the quality of the datasets (in total, they analyzed over 1M images and provided a data set that included 150K+ eye screens) and determine how well they were labeled and “baked” into existing algorithms. ML methods and pipelines used in previous projects were reviewed as well.

Provectus moved forward to deliver a secure and reproducible experimentation and ML training infrastructure, where pipelines are orchestrated and ML models are trained in Kubeflow. Some of the ML models, as well as analytics, were done in Amazon SageMaker.

It was crucial to incorporate a feature store, experiment tracking and ML pipelines. ML model re-evaluation pipelines, as well as active learning pipelines for continuous, efficient data re-labeling and re-training were implemented to support ongoing model retraining, evaluation, tuning and improvements as new labeled data and user feedback arrive.

At the final stage, Provectus integrated the machine learning solution into the mobile application, and into GoCheck’s business workflow.

Faster ML Experimentation Means More Accurate Models, Less Manual Review, and Happier Customers

GoCheck received a secure, auditable, and reproducible machine learning infrastructure deployed on AWS, featuring a robust experimentation environment with experiment tracking and model versioning and active learning pipelines for continuous data re-labeling and re-training.

A new, advanced machine learning infrastructure built by Provectus allowed a team of just three ML Engineers to run more than 100 large-scale experiments in three weeks, which significantly facilitated the training and tuning of machine learning models.

In just three weeks, more than 100 experiments on the entire subset data of 150K images were conducted, making it possible to not only test various improvement hypotheses but to also re-run every quality hypothesis to determine the best hyperparameters. The experiments produced artifacts such as models, metrics, and predictions, to efficiently track and version each of them for further review and re-use.

The machine learning infrastructure helped the Provectus ML team significantly. More efficient data preparation and faster ML experimentation enabled us to train and tune multiple ML models, to select those with the best precision and recall. For instance, GCK’s original classification protocol correctly caught “Child Not Looking” cases with a recall of ~25% (50% precision), but Provectus’ new model increased the recall to 91% while preserving the initial model’s precision.

For GCK, more accurate image analysis in their application means fewer “not gradable” results and less manual messaging, which naturally translates into cost savings and increased user satisfaction.

The robust and resilient infrastructure for machine learning bolsters GoCheck’s quick, sustainable growth in the vision screening market. The GCK application is now equipped with advanced ML algorithms — courtesy of ML infrastructure — that further enable accurate and convenient photoscreening for a wide range of conditions, including amblyopia.

Better access to affordable eye screening with GCK means that millions of children will be diagnosed in time, and will not suffer from vision impairments and blindness as adults.