From Textray to FDA: The journey from basic research to clinical outcomes

By: Yaron Blinder, PhD, Sr. Product Manager, X-ray products,
Zebra Medical Vision

This week we are extremely excited to announce the FDA clearance of our HealthPNX product – the world’s first FDA-cleared product leveraging deep learning to automatically read chest x-ray images to detect and prioritize cases with pneumothorax.

To celebrate this momentous milestone, we wanted to shine a light on some key people involved, and share some insights on the process of taking an idea from basic research to a clinical-grade AI product, designed to provide real-world clinical outcomes.

This product got its start with internal research projects at Zebra Medical Vision. One of these projects was the development of the Radbot-CXR algorithm, led by Chen Brestel, PhD, which was trained to read chest x-ray exams and detect four classes of findings – focal lung opacities, diffuse lung opacity, cardiomegaly, and abnormal hilar prominence. A second project named Textray, led by Jonathan Laserson, PhD, focused on extracting labels from radiology reports. This work laid the foundation for chest x-ray algorithm development at Zebra Medical. It enabled our team to quickly produce proofs-of-concept for detection of 40 different conditions in chest x-rays, and created an essential infrastructural resource for the upcoming chest x-ray product roadmap.

Q: What were the hardest points in developing algorithms for classification of chest x-ray anomalies before Textray?

A: (Chen Brestel, PhD, Machine Learning Researcher)

Machine learning algorithms like classification need data, while more anomalies means more data. Our first challenge was to choose a small subset of findings that has a good coverage on the main abnormalities radiologists are used to look for. The second challenge was to make sure our medical device is trained, validated and tested on a dataset that adequately represents the real world. We had to make sure the various patient demographics, age, gender, co-findings, etc. are well represented. Once we had a dataset we collected thousands of labels from radiologists.

Finally, we trained and chose our best model. It was nice to see that our model reaches a performance which is slightly higher than a team of three expert radiologists. Moreover, we found that the model did a pretty nice job in generalizing. Whilst the training experiments used labels generated by resident radiologists, the algorithm testing was performed using labels generated by a team of three expert radiologists, thus measuring ourselves against the highest standards. We also succeeded to build infrastructure that accelerated our next R&D efforts and achieved a small step on the journey to improve the quality and length of human lives.

Chest x-ray of a patient with pneumothorax

Q: What was the hardest part of developing the Textray project?

A: (Jonathan Laserson, PhD, Lead AI Researcher of the HealthPNX project)

The hardest part was coming up with the ontology of findings to map the reports into. We knew that any mistake there would cost us a lot later on. We reviewed thousands of report sentences manually in order to create the original list of findings “bottom-up”. We consulted a number of expert radiologists in order to turn that list into something that clearly distinguishes between the different findings in a chest x-ray. It took a few versions to get it right, we had to merge and split some of the original categories. As someone that has no clinical background, doing this was the hardest part.

Q: What impact has this had on our approach to chest x-ray product development at Zebra?

A: (Mila Orlovsky, MSc, Clinical Data Scientist)

Using previously well-labeled findings, we were able to apply large-scale inference on our available pool of data, and estimate the prevalence of each finding in the wide population. Additionally, we were able to explore correlations between findings and estimate severity groups of patients. While progressing towards the next product, it leveraged our data curation process from wandering between random samples and prolonged tagging efforts to adopting a more systematic approach. Thus, we managed to more easily and reproducibly prepare tightly distributed representative cohorts of images to tag and use both for training and validating our next products.

Q: How does this work now help develop new products at Zebra?

A: (Eli Goz, PhD, Lead AI Researcher , X-ray products)

There is a significant gap between developing “lab” algorithms and making them applicable to solving real world problems. Especially if these problems lie in the medical realm. Here at Zebra Medical Vision, while being involved in cutting-edge research on a daily basis, we always see in front of us the final goal of bridging this gap and invest huge efforts to translate our results into solid clinical products.

Pneumothorax is just the first and important milestone on our way towards a broad range of AI-based chest x-ray applications, covering the most acute and life threatening thoracic pathologies. To achieve this goal we exploit the experience and numerous lessons learned from these projects in order to achieve an uncompromising combination of quality and quantity by: (1) efficiently managing highly interdisciplinary and complex operations, involving numerous disciplines such as machine learning, software engineering, statistics, data science and medicine;  (2) optimizing development cycles and time to product by engineering scalable and efficient infrastructure applicable to numerous clinical findings, and (3) meeting the highest safety and regulatory standards by providing maximally rigorous clinical evaluation and validation of our products.
Our major aim is to make a real impact on real patients and I believe that now we have all the required knowledge and skills to make it happen.

Q: What clinical factors drove the product development process from “40 most common findings” to triaging pneumothorax?

A: (Michal Cohen-Sfady, PhD, Clinical Information Manager)

Pneumothorax is one of the most critical, time-sensitive findings in radiology. A pneumothorax can be symptomatic, or in some cases the patient may feel only vague chest pain or even worse – they may be sedated and therefore unable to report their symptoms at all. These patients all get chest X-rays but due to the heavy radiologist workload, the diagnosis could be delayed for hours. When we think of acute findings on chest x-ray like pneumonia, or congestive heart failure, we usually see a sick patient who with the right attention and treatment, will improve. The situation with pneumothorax is entirely different. A small pneumothorax can develop to become a life-threatening tension pneumothorax very quickly, therefore timely diagnosis is critical. Triaging such cases to the top of the worklist could therefore save lives. The treatment for pneumothorax requires a chest tube which is often a life saving procedure. Chest X-rays with pneumothorax may be sitting in the radiologist queue undetected all the while delaying this life-saving procedure. Given the critical time-sensitive diagnosis of pneumothorax, we decided to prioritize this diagnosis over other chest X-ray findings.

Q: Where is this product expected to provide the most clinical value?

A: (Eldad Elnekave, Chief Medical Officer)

It’s going to save lives When people least expect it. Here’s what I mean: Most doctors will remember the first time they had to place a central venous catheter (it starts with a needle into the jugular vein just over the right clavicle). It happens every night in every intensive care unit. Sometimes, the needle hits the top of the lung, creating a small pneumothorax. You always get a chest x-ray to make sure that the catheter is in the right position. The problem is, when this happens in the middle of the night, it can be hours before any radiologist looks at the x-ray. Especially for patients who are being ventilated, a small pneumothorax can turn into a life-threatening big one within a few hours. This kind of scenario happens at least once a year in any medium sized hospital. What we hope for is that this product will bring those critical x-rays to attention right away, before harm is done.

As the first of its kind, HealthPNX is a harbinger for a new generation of AI products aimed at reading chest X-rays. We at Zebra Medical Vision are excited to expand our chest x-ray product offering and develop a comprehensive suite of products designed to help radiologists address the ever-growing workflow burdens of medical imaging.

Yaron Blinder, PhD
Sr. Product Manager, X-ray products
Zebra Medical Vision

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Blog at WordPress.com.

Up ↑

%d bloggers like this: