Sharpe Engineering’s clients face business problems that can only be solved by crunching huge volumes of high-velocity data in real time—which requires a powerful stream processing engine.
Sharpe Engineering uses IBM Streams as its platform of choice to build innovative solutions that can analyze data in motion, delivering results that clients previously thought to be impossible.
Identifieda potential “smoking gun” biomarker for post-traumatic epilepsy by analyzing EEG data
Enhanceddetection of underwater activity by analyzing acoustical signals
Minimizedhardware costs by harnessing the computational efficiency of IBM Streams
Business challenge story
Thinking outside the box
Over the course of a distinguished career in the software engineering industry, Jim Sharpe came to recognize a class of problems that clients across all industries were facing—important and commercially valuable problems that seemed almost impossible to solve using traditional approaches.
He begins: “The same fundamental problem has existed for many years across all kinds of businesses and use cases: when you have huge volumes of data streaming into your systems, how can you process it fast enough to enable instant insight and decision support?”
Over the years, Sharpe has seen many software products that claimed to be able to meet this challenge—but most of them quickly fell by the wayside.
“Every year I’d look at the market and see what technologies were available to solve these problems, and nothing really clicked,” recalls Sharpe. “I was even contemplating building a solution myself at one point, although in hindsight, I’m really glad I didn’t! But in 2009, I read an article about a new IBM technology called System S, and the hairs stood up on the back of my neck. I knew it had to be the answer I’d been looking for.”
Enterprise-class stream processing
Sharpe immediately made contact with IBM and built a close working relationship with the development team for System S. Nine years later, System S has evolved into the solution we now know as IBM Streams, and Sharpe has built a successful business on helping his clients use it to solve a wide range of novel and challenging problems.
“I fell in love with Streams from day one, and I’ve been working with it ever since,” says Sharpe. “For years, it was the only credible option in the stream processing space, and even today, it has capabilities that other engines struggle to match.”
Sharpe lists three major reasons why IBM Streams is his platform of choice for enterprise-scale stream processing projects.
First, computational efficiency. According to a major benchmarking project run by Walmart in 2016, IBM Streams performed 20 times better than Apache Storm, and twice as well as Apache Apex. The difference in performance has significant implications for hardware costs: according to the benchmark, choosing IBM Streams over Storm could potentially save almost USD 1.8 million per year.
Benchmarks should always be taken with a pinch of salt, but Sharpe has seen IBM Streams deliver similarly impressive performance in real-world deployments. He comments: “It’s remarkable how resource-efficient the IBM Streams platform is. I genuinely think that even if I hand-coded my algorithms in C++, they wouldn’t run much faster than they do with Streams, and I wouldn’t get all the benefits that the platform provides.”
The second reason for choosing IBM Streams is its maturity and stability: the solution provides a highly reliable platform for applications that need to “just work”.
“It’s not unusual for the uptime of IBM Streams applications to be measured in years, rather than months or weeks,” says Sharpe. “For financial or safety-critical applications, I would choose Streams ahead of any of the alternatives. In my view, it’s the most mature, reliable stream processing engine on the market today.”
The maturity of IBM Streams also contributes to the third advantage that Sharpe identifies: the richness of the tools that have evolved around the platform. For example, in addition to its native Streams Processing Language (SPL), IBM also provides development tools for Java and Python, and a host of sophisticated libraries for machine learning.
“Streams has a great set of libraries around time-series analysis, for example—including algorithms such as discrete wavelet transform [DWT], which are extremely powerful for real-time signal processing,” says Sharpe. “The ability to harness well-designed, well-tested tools is a huge advantage, helping us focus on solving our clients’ business problems instead of going down the rabbit-hole of building our own algorithms from scratch.”
Finding the signal in the noise
Over the past few months, the Sharpe Engineering team has worked on a number of interesting projects that demonstrate how the combination of IBM Streams and sophisticated time-series algorithms can provide eye-opening insight into questions that previously seemed unanswerable.
For example, the team worked with medical researchers at the University of Montana to analyze brainwave data from patients who have suffered traumatic brain injuries. As they recover from these injuries, around 30 percent of patients are likely to develop post-traumatic epilepsy—a debilitating and potentially life-changing condition. However, until now, there has been no reliable way to predict which patients will develop the condition, and which will not.
Sharpe explains: “There are some promising drugs that seem to be able to prevent the onset of post-traumatic epilepsy, but unless you know which patients are at risk, you can’t really tell which treatments are most effective.”
The team used IBM Streams to process a large quantity of brainwave data captured by electroencephalogram, and worked to identify biomarkers that would correlate with cases where the patient would subsequently experience epileptic symptoms.
“We found what seemed to be a smoking gun—there was a clear and definable difference in signals present in the population who went on to develop epilepsy as compared to those in the control group,” says Sharpe. “We need to do more research to rule out the possible contribution of other factors, such as the patient’s existing medications, but we’re optimistic that this project could make a real difference to patients’ lives going forwards.”
If the initial positive findings can be confirmed, Sharpe foresees the possibility of building IBM Streams into a real-time monitoring solution that can be used to assess brain injury patients while they are in the intensive care unit. As soon as Streams detects the warning signs of epilepsy, physicians could move patients onto preventative drugs, and hopefully reduce the risk of the condition developing.
A second project involved working with a government research team on real-time analysis of underwater acoustics, using IBM Streams to process the raw data into a form suitable to feed machine learning algorithms that are capable of discriminating previously undetectable signals, even with no prior examples. This approach could be used in a variety of government, commercial, and research settings.
“One interesting thing is that we were able to use the same algorithms and nearly identical code in both projects,” says Sharpe. “We can use machine learning techniques to detect and classify almost any kind of signal data, and IBM Streams can run the algorithms while the data is in motion. That’s a very powerful combination for real-time insight that could be useful for many industries.”
He concludes: “As one of the earliest adopters of IBM Streams technology, we have seen the product evolve from a revolutionary new concept into a mature, proven, enterprise-class platform—and as IBM continues to add solutions such as Streams Designer to the family, it’s getting easier to use than ever before. As more companies seek to embed machine learning into their business processes, IBM Streams is the ideal platform to help them make real-time cognitive solutions a reality—and gain better knowledge, faster.”
Sharpe Engineering Inc
Sharpe Engineering Inc. is a software engineering company that specializes in the application of advanced technologies to address complex problem spaces—in particular, helping clients understand and utilize data in motion by adopting stream processing and machine learning technologies.
Take the next step
IBM Analytics offers one of the world's deepest and broadest analytics platform, domain and industry solutions that deliver new value to businesses, governments and individuals. For more information about how IBM Analytics helps to transform industries and professions with data, visit ibm.com/analytics. Follow us on Twitter at @IBMAnalytics, on our blog at ibmbigdatahub.com and join the conversation #IBMAnalytics.