Skip to main content

2015 IBM Fellows

John Smith

John Smith

IBM Research
Senior Manager, Intelligent Information Systems

Newly minted IBM Fellow John R. Smith knows that in the world of computer vision, progress is a marathon and not a sprint. It’s a fitting field, then, for the regular NYC Marathon runner and parent.

When he joined IBM in 1997, the company had an impressive technology around retrieval of images in database systems. Query by image content (QBIC) was the industry standard, but John was looking ahead towards a new realm of possibility.

“It was an inspiring thing at that time, but it was limited in that it was a database approach,” he said. “You could search for images that were similar to your query image. If yours had red in it, it could search for and return other images with red in it. It was an incredible feat for that time, but it was hinting at an entirely new direction for computer vision.”

Within a couple of years of beginning his IBM career at the Thomas J. Watson Research Center in New York, John was hard at work creating more complex and intricate search categories, and was among the first to develop an appearance-based approach for content analysis, search and retrieval, and semantic concept extraction using machine learning. “We needed to get beyond the simple physical matching idea. We needed to put words to those pictures, and we needed computers to make sense of them,” he said. “Once we started getting some results on that, we knew we needed to scale it. Five categories wasn’t going to cut it. Thousands or tens of thousands was what we aimed for.”

As the landscape has changed, given advances in machine learning and an explosion of available data, the challenges have become more intricate. “It’s a much harder problem, given the enormous volume of images and data captured by people on their cell phones and devices and uploaded to social media in particular,” he said. “It’s now a question of ‘where’ as much as it is of ‘what’ the image is depicting.”

“There has been an enormous paradigm shift from programming computers to teaching computers what and how to learn,” he said. “This transformation has started happening in the field of computer vision, with capabilities like the simply category matching, but now it’s really accelerating, with things like cognitive computing and Watson developing.”

As the world’s population continues to increase rapidly and urban life in particular grows, the field of computer vision takes on ever greater importance. “Safety and security is a huge concern for governments, from the local and civil up to the federal,” he said. “Transportation too, whether it’s public or private. Governments are looking to computer vision to recognize threats or abnormalities in order to raise alerts.”

“We can’t have every traffic camera or security feed monitored by a human,” he added.

Medicine is another space where the potential in the field to help humanity is immense. In dermatology, for example, the images doctors need to analyze hold crucial subtleties upon which a diagnosis can hinge. It would be incredibly valuable to feed these images through a computer, and have that computer learn to discriminate between what matters and what doesn’t.

My hiring manager told me that the most important thing about IBM is that we respect the individual.

John has been instrumental in the development of both the IBM Multimedia Analysis and Retrieval System (IMARS) and Intelligent Video Analytics (IVA), and he is playing an essential role in establishing deep learning-based modeling and classification services for unstructured image, video and signal data. This means being able to label images with words that describe its semantic contents, such as objects, scenes, people, or activities. He recently lead the development of an integrated demonstration of IMARS and IVA for analysis of video-in-the wild, which included work on Boston Marathon-related image and video analysis.

John says the work he and his team do on improving a computer’s ability to understand and recognize images remains incredibly challenging. “Small children can pick out an animal or a car or a particular object, if they have a little background on what they’re looking for.

“Many of those things are still difficult for a computer,” he said. “But there’s tremendous opportunity there, and our ability to attack these problems improves as computing power increases and the numbers and types of data we can feed into computers increases.”

“We’re providing models for what can be the breakthroughs in this space.”

And what does it mean to him to be named a Fellow? “It’s a milepost, not a finish line,” he said. “You need to stay curious, stay focused, but you need a way to measure your progress and know that you’re heading in the right direction.”

No surprise, then. The marathon continues.

John Smith in his own words...

Who is a key mentor who helped get you to where you are today?

My hiring manager out of graduate school was [IBM Research’s Director of Commercial Systems] Chung-Sheng Li, and he made a great effort to bring me into IBM. From the very beginning, and still to this day, he has had an enormous influence on my work. He told me that the most important thing about IBM is that we respect the individual.

He could have meant any number of things by that, but I took it to mean that there was a place for your ideas, that your colleagues would respect your ideas. Of course, we’re trying to bring value to the client and the company in everything we do, but there is also so much autonomy. You have the right to pursue your own ideas. Chung-Sheng instilled that in me from the very beginning, and it has been hugely impactful for me.

What do you like to do away from work?

I tell my eight-year-old daughter that I used to be a runner, and now I’m a dad who runs. The difference means something to me, but I am not sure she really knows what I mean. I was quite intense about running well into my adult (and IBM professional) life. Fatherhood has tempered some of that intensity and competitiveness, but I still love running. I was "grandfathered" into the NYC Marathon, having run it more than 15 times. It means guaranteed entry for life, which means I can run it for life. It's crazy, but it is something that means a lot to me.

What was your first job, ever?

I had a paper route though most of my young years. It was my first job, and it gave me some nice pocket money, a little something to put away in the bank, and a small amount of independence as a kid.

What is on your iPod right now?

The Beatles were my first favorite band as a kid. I still have them on my phone and listen to them all the time.

How would you like to be remembered?

As a "nice guy" (come see the sign in my office). But, seriously, I am an optimistic person and always try to see the best in people, ideas, collaborations, and projects. I hope people remember me as someone who was generous in interactions and someone they enjoyed and benefited from working with.


Watch the video

Watch how the Fellows program impacts IBM and the world

IBM Fellows by the numbers

This year's class

Year by year