Developing cognitive radiology to change the practice of medicine

In the spring of 1997, Dr. Tanveer Syeda-Mahmood received a call from one of her father’s colleagues — the kind of call that everyone dreads. “There is something wrong with your father,” the man reported. He described symptoms of confusion, slurred speech and severe headache. “So I called my sister [a physician] in India and she said, ‘Take him to the emergency room immediately. He may be having a stroke.’”

It’s just 400 miles from Rochester, NY, to Washington, DC — a quick flight — but Tanveer arrived to find her father in acute distress, wracked by seizures. The nurses rushed him away for a simple contrast MRI, but by the time they wheeled him back into the room, the sudden violence of the seizures had given way to a profound and unnerving stillness.

“His brain is gone,” they told her, after investigating the depth of the coma. It had been damaged beyond repair, and beyond the body’s ability to heal itself. Her father, a healthy, energetic man — himself a Ph.D candidate with a sharp mind and an indomitable will — was now “declared a vegetable.”



Take him to the emergency room immediately. He may be having a stroke.

The wrong kind of doctor

Tanveer had been a star at MIT. As a doctoral candidate she helped invent the robotic vacuum that evolved to become the Roomba. Her active research was in image indexing on computers. She was a doctor of machines, not people, but a problem is a problem and Tanveer is a problem solver by nature, training and trade. It’s who she is, what she does, and exactly how she responded to her father’s condition.

Under mounting pressure from doctors and administrators to terminate life support, Tanveer began splitting time between the hospital and her life back in Rochester. She was already juggling a surging career and a growing family with a husband and two young children. Now, two days at home and two days away, she would make the trip to sit beside her comatose father, hoping for some outward sign of life — and burning through the hospital library in a search of explanations and solutions.

How could this have happened, and how could she fix it?

Doctors looking at scans

The wrong kind of stroke

It turns out that there are two kinds of major stroke. About 88 percent of strokes are ischemic, with a blood clot or some other impediment restricting the blood supply to the brain. Then there’s the near opposite, the much rarer hemorrhagic stroke, which describes bleeding in or near the brain.

Tanveer’s father had been diagnosed with an ischemic stroke and promptly treated with blood thinners — a textbook response. The problem is that he was actually suffering a hemorrhagic stroke, and the thinning agents amplified the damage by rushing even more blood into the cranial vault.

It was a case of misdiagnosis. The type of mistake, Tanveer discovered, that is surprisingly common. She was losing her father to something that could have — that should have — been avoided. As Tanveer studied, looking for anything that might help her father, she began to tug at the threads of a much larger problem.

Image of brain CAT scan layers showing highlight activity


It was a case of misdiagnosis. The type of mistake, Tanveer discovered, that is surprisingly common.


80,000 to 160,000 people are killed or permanently disabled every year as a result of misdiagnosis.

The human problem

Even the best radiologists have human limitations. Eye fatigue — fatigue in general — is a big problem. Repetition dulls the senses and the pressure to go faster erodes the ability of radiologists to make secondary or coincidental diagnoses. They become so focused on scanning for primary issues that other important visual information becomes “hidden” in plain sight.

Tanveer also describes what is, effectively, a data silo. “They are sitting and diagnosing in dark rooms, fairly isolated from the rest of the patient records… Although there is a lot of information about the patient in electronic health records, they simply don’t have the time to go to different systems to get access to the general clinical information about the patient.”

Thinking about how her father’s coma might have been avoided, Tanveer imagined ways to help doctors handle the information overload — a comprehensive system for data-driven clinical decision making. Something that would go beyond rule-based filtering to become a semi-automated assistant.

Image of Radiologist


They become so focused on scanning for primary issues that other important visual information becomes ‘hidden’ in plain sight.

The right kind of doctor

For conventional data we can build analytics systems to do the heavy lifting and number crunching. The extraordinary challenge with visual data, like medical images, is that the evaluation process is qualitative rather than quantitative. This is the kind of challenge for which Tanveer is exactly the right kind of doctor.

“Humans just get it. When they come into a scene they immediately notice things that catch their attention. Or, we can zoom in to scan for something specific… So what is that mechanism in the brain that lets you focus in a top-down fashion when you know what you’re looking for, versus having something in the scene itself attract your attention when you are in an open loop mode?”

This is what is called “the visual attention mechanism,” and is something Tanveer was working to help computers achieve in the Artificial Intelligence Lab at MIT. “I tried to do a computational model for it and build an implementation of an ‘attention machine’ that could actually mimic that behavior in recognizing objects.”

Beyond teaching machines to “see” Tanveer knew she would have to develop the capability for computers to “understand” what they were seeing.

Image of layer scans of brains


Humans just get it. When they come into a scene they immediately notice things that catch their attention

From robots to radiology

Tanveer had nursed a longtime fascination with the relationship between visual recall and declarative memory, and thought that those concepts could be applied to computer image analysis. “I had this theory that [computers] could retrieve scenes using the attention mechanism.”

This tantalizing possibility would allow machines to go beyond an image database with content tagging — beyond simply regurgitating human input. Equipped with their own visual attention mechanism, computers could start seeing connections between images, opening up the possibility of making deductive observations. IBM offered the chance to turn that theory into real research.

Tanveer had become acquainted with Dragutin Petkovic, manager of the IBM QBIC (Query By Image Content) program, at imaging research conferences in the early 1990s. “He encouraged me to apply to IBM and come here, because this is where the critical mass was growing.” Not long after the stroke, Tanveer accepted the offer and moved across the country to start work at the Almaden Research Labs in San Jose, California.

When a competition was held at Almaden to pitch innovative projects, Tanveer used the opportunity to propose a new project called AALIM (Advanced Analytics for Information Management, also meaning “The Knowledgeable One”) to address the problem of building a knowledgeable computer assistant for clinical decision-making.

With the right funding and research latitude, Tanveer began to develop her system for data-driven clinical decision support. When she integrated content-based image retrieval, what started to take shape was a revolution in radiology and an evolutionary leap in the practice of medicine.


What started to take shape was a revolution in radiology and an evolutionary leap in the practice of medicine.

The vision for Medical Sieve

Imagine a learning, reasoning system equipped with vast stores of clinical data, combined with an image analyst that wouldn’t suffer from eye fatigue or lapses of attention, even when sorting through hundreds of thousands of images. In essence, a cognitive assistant with advanced multimodal analytics, clinical knowledge, and reasoning capabilities good enough to assist in clinical decision-making.

Tanveer and IBM are turning this vision into reality: Medical Sieve.

As the name implies, Medical Sieve will give doctors a powerful new way to filter huge volumes of clinical data to find the critical information that will help them make the best decisions. It will serve in much the same way that less-experienced doctors conduct preliminary inquiries to narrow the focus for referral to more seasoned clinicians — but with near-instant insights to help doctors get ahead of emergent situations and avoid mistakes like the misdiagnosis in the case of Tanveer’s father.

Tanveer describes this virtual assistant: “It would be so good that it would be trusted, in a similar way that a good resident is trusted. The way we want to do that is to prove that the machine has entry-level capabilities as a radiologist, and is qualified, to some extent, to make evaluations and summaries so that clinicians can use it for their judgements.”


It would be so good that it would be trusted, in a similar way that a good resident is trusted.

Training the know-it-all intern

Medical Sieve is what IBM calls a “grand challenge project.” In contrast to product development, grand challenges are conceived in the tradition of pure science. They are long-term commitments — research for the sake of research — that have the potential to unlock new ways of improving the human condition.

At this stage, Medical Sieve is being developed with a focus on two high-value areas, radiology and cardiology, but the basic premise and the project development could eventually extend to other fields and functions.

Tanveer describes the challenge: “The comprehensive nature of it is that in the fields of radiology and cardiology there are over 40 modalities — we’re talking about CT, MRI, X-Ray, ultrasound — and then there are the diseases. In the field of cardiology alone there are over 600 diseases to cover. So, for a machine to be an assistant, it needs to have very strong capabilities on the knowledge side, the image interpretation side, and on the reasoning side — and the ability to summarize what is important to the clinician. All of this requires dedicated development of sophisticated algorithms in machine-learning-driven analytics and reasoning.”

It won’t be easy, and it won’t be fast, but the progress is steady and incredibly promising — and the medical community is getting on board. In June 2016, IBM announced the formation of the Watson Health medical imaging collaborative, a global initiative comprising leading health systems, academic medical centers, private radiology practices, ambulatory radiology providers and imaging technology companies, all dedicated to making cognitive imaging an integral part of patient care. In February 2017, IBM debuted its first cognitive imaging offering, Watson Imaging Clinical Review, which is helping to reconcile discrepancies between patients' clinical diagnoses and administrative records.

The story that started it all

While the seeds for Medical Sieve were taking root in Tanveer’s mind, she was still fighting to save her father, navigating the US health care and insurance systems. Here, the numbers were not on her side. For a patient with her father’s profile, the actuarial analysis was grim. High costs and extremely low probability of recovery discouraged further interventions. He was on heavy doses of morphine and an infection in his lungs — a complication from a feeding tube insertion — made the prospects even worse. On life support, he would outlive his insurance cap, and then what?

Tanveer explains, “The hospital was concerned that his money would run out, and then they wouldn’t get reimbursed. They wanted him to be taken off of life support.” But she knew her father’s views on the matter and his disposition: “He was very much a fight-to-the-end kind of person.” It’s a trait they have in common; Tanveer was relentless in her pursuit of ways to keep her father alive.

Image of top-down CAT scan layer


Today, what was lost is found. The man whose ‘brain was gone’ is doing just fine.

Angel in the details

She found one. Buried in the fine print of her father’s insurance agreement was a clause that allowed for repatriation of foreign patients if it could be proven that better care was available in the patient’s home country. Tanveer and her family in India worked together with contacts in the medical community to create a treatment plan that could make a compelling case for medically repatriating her father. “It took a lot of work, but eventually they agreed.”

Within a week of the decision, her father was airlifted to a hospital in the family’s home town of Hyderabad. “The first thing they did was take him off of morphine and clear up his lungs, and he came out of the coma two days later!”

Today, what was lost is found. The man whose “brain was gone” is doing just fine. “There is probably long-term damage that altered his personality. We did see that he became much crankier… but he’s functional.” He has since continued his own research, developing a subject index of the Koran and creating an audio log on the survivors of the 1948 revolution in Hyderabad.

Tanveer redefined what was possible to help save her father’s life. With Medical Sieve, she’s continuing to engineer new possibilities, with the hope of saving many more, “It’s my strong belief that it will ultimately change the practice of medicine.”