This post is part of a series recognizing unique IBM Research projects and their unexpected ties to pop culture, with “30” or “1986” being the common thread. The series will run once a week, celebrating the 30th anniversary of IBM Research – Almaden in San Jose, CA.
What is IBM’s unlikely connection to the Halley’s Comet sighting in 1986? That year, Halley’s Comet made its orbit from outer space towards Earth, making it the first time since 1910 that it was remotely visible to the naked eye.
IBM’s connection to this interstellar comet
Graham Mackintosh and his Spark@SETI team analyze data at the NASA Hyperwall
This unlikely connection lies in a project between researchers at IBM, along with NASA’s Ames Research Center and the SETI Institute, working together to analyze complex deep space radio signals being collected at a rate of more than 4.5 terabytes per hour – that’s 60GB per second – to hunt for patterns that might identify the presence of intelligent extraterrestrial life.
In this collaboration, IBM and the SETI Institute developed an Apache Spark application to analyze the 168 million radio events detected by the Allen Telescope Array (ATA) over several years. The complex nature of the data requires sophisticated mathematical models to tease out faint signals, and machine learning algorithms to separate terrestrial interference from true signals of interest.
IBM Cloud executive Graham Mackintosh, who oversees the group, details IBM’s recent engagement with Stanford University in which new methodologies and algorithms were developed and are now being assessed by scientists for further use. “By incorporating analytics, machine learning, Spark services and image processing, the partnership promises to greatly improve signal classification techniques for identifying data of interest and nudges us closer to one of mankind’s great questions – are we alone?” Mackintosh said. “The Allen Telescope Array provides a massive – and previously untapped – data set that allows our scientists to analyze, extract and classify our unique findings.”
By analyzing the vast archives of ATA content, the teams can help find signals that have previously been ignored, and also open the way for improvements to the real-time decision-making systems right at the ATA, which will permit better processes for targeting operators to capture data of interest in real time.
Dr. Jill Tarter, Co-Founder of SETI speaks at IBM INSIGHT 2015
Here I describe an approach to efficiently train deep learning models on machine learning cloud platforms (e.g., IBM Watson Machine Learning) when the training dataset consists of a large number of small files (e.g., JPEG format) and is stored in an object store like IBM Cloud Object Storage (COS). As an example, I train a […]