Apache Quarks, Watson, and Streaming Analytics: Saving the world, one smart sprinkler at a time

There’s a lot more to IoT than just turning on the lights in your house when you’re out. Find out how a group of us at IBM combined Apache Quarks, Streaming Analytics, the Watson IoT Platform, and other Bluemix services to develop a water conservation solution. Our prototype of a smarter connected sprinkler system enables more efficient water irrigation based on information retrieved about current soil conditions.

Better analytics with Apache Quarks

The proliferation of edge devices has resulted in the widespread need to perform real-time analytics on those devices, and then send only data requiring further analysis to central stream processing systems.

There are a few benefits to performing analytics on the edge devices themselves, or at the gateway:

  • reduced communication costs
  • the ability to react locally
  • offloading processing from the central systems

As a result, IBM recently announced an open source project called Apache Quarks whose purpose is to build a community for accelerating analytics at the edge. Apache Quarks provides the ability to react locally to events while also allowing central streaming analytic systems like IBM Streams to deliver a hierarchy of analytic processing across devices.

Quarks topology

IBM developed this project to provide an SDK and lightweight embeddable runtime for streaming analytics at the edge. Quarks can be integrated with centralized analytic systems like Apache Storm or IBM Streams to provide edge-to-center analytics.

Apache Quarks allows devices to react locally to events and helps reduce communication costs by sending data to central systems only when an event of interest occurs. This allows all relevant data to be delivered the moment an event of interest occurs, rather than just relying on a periodic heartbeat of information, which may not include the signals necessary or may arrive too late to take action on it.

To improve irrigation efficiency and properly enforce water usage restrictions, we implemented a prototype for a smarter, connected sprinkler system.

Device hubs like the Watson IoT Platform provide device management and message brokers for communication. Central Analytic systems like IBM Streams provide full-featured streaming analytics with additional context and state, correlating across devices and accessing data-of-record systems. The integration of Quarks and central systems through a device hub enables automatic application connectivity and central job management and health summaries. The consistent streaming programming model enables applications to control the edge devices based on analytics across devices and the central system.

To learn more about Apache Quarks, watch this recording of our webcast below:

Tech challenge: Water conservation application on Bluemix

Back in April, IBM Canada Labs hosted a tech challenge in which participants were asked to write a web application that would run on Bluemix and would use data from the Weather Company available through the Watson Insights for Weather Service. A few of us at IBM got together and brainstormed a number of ideas for how we could take part. Recognizing that water scarcity is one of the most pressing problems in the world right now, we decided to take on a water conservation project. Our mission became to save the world one smart sprinkler at a time.

According to the World Wildlife Fund (WWF), 70% of our planet is covered by water. However, only 3% of it is fresh water, and only one-third of that is available for consumption. Many water systems are stressed due to the growing human population and over-consumption. Agriculture consumes more water than any other source. Much of the water used in agriculture is wasted due to inefficient irrigation systems. To alleviate this problem, many governments impose restrictions on water usage, especially in drought-stricken states like California. However, these bans are difficult to enforce. For example, some people can afford to violate the restrictions and simply pay the fines.

To improve irrigation efficiency and properly enforce water usage restrictions, we implemented a prototype for a smarter, connected sprinkler system.

Water app architecture

This system consists of two main analytics components:

  • Analytics on edge devices: We used a Raspberry Pi to simulate a smart sprinkler device. On the device, local soil conditions like moisture level are continuously collected and analyzed using Apache Quarks. When the device detects that the soil is too dry and requires watering, it sends a request to the centralized analytics system for permission to turn on the sprinkler. The smart sprinkler system will not turn on the sprinkler unless it is approved by the centralized analytics system.
  • Centralized streaming analytics system: In the centralized analytics system, we used the Streaming Analytics service to analyze the incoming water requests in real time. To approve a water request, the application checks the weather forecast for the next two days using the Watson Insights for Weather API. If there is not enough precipitation in the forecast, it checks to see if a water ban is currently in effect. The Streams application will approve the water request only if there is not enough precipitation in the forecast and if there is no water ban in effect.

For rapid application development, we leveraged other Bluemix services. These services handle the infrastructure and connectivity, allowing us to focus on developing the analytics and business logic of the application. For the device to communicate with the centralized analytics system, we used the Watson Internet of Things Platform. This service handles application connectivity for us, allowing the Apache Quarks application to communicate with the IBM Streams application easily via MQTT.

To visualize the data, we implemented a visualization server using the SDK for Node.js service, and we used React.js to implement a web-based dashboard.

This smart sprinkler system now enables more efficient water irrigation based on current soil conditions. The prototype only takes the moisture level into account, but the system could be extended to monitor and analyze other important factors like moisture evaporation rate, soil mineral content, or irrigation requirements based on the type of vegetation being planted. Controlling the sprinklers in a centralized manner enabled us to implement business rules for managing water resources and properly enforcing water restrictions when necessary.

Complex IoT system delivered quickly

This prototype demonstrates the reference architecture for Apache Quarks, and shows how it can be used in conjunction with a centralized analytics system. By pushing analytics to edge devices, we make it possible to continuously monitor local conditions without having to pay a premium for the bandwidth required to continuously send data to a back-end system for analysis. We only send the relevant data (in this case, the water requests) when there is a need. We can also react to local events in a more timely manner. In other sprinkler systems, the user schedules the sprinkler to be turned on for a duration of time without taking the current soil conditions into account. In our application, the sprinkler is automatically turned off when we detect that there is enough moisture in the soil. This analysis and control was all implemented on the edge device.

Finally, we were able to implement this complex IoT system very quickly by leveraging Apache Quarks, Streaming Analytics, the Watson IoT Platform, and other Bluemix services. Connecting edge devices to Streaming Analytics was very simple, thanks to Watson IoT Platform! These Bluemix services provided the necessary infrastructure and connectivity for this application, allowing us to be up and running within a few days.

Check out this video to see the smart sprinkler system in action:

*Apache Quarks is currently undergoing Incubation at the Apache Software Foundation.
** Special thanks to Mary Komor, Kendrick Wong and Cong Li for working on this prototype with me, and providing their technical expertise.


Share this post:

Share on LinkedIn

Add Comment
No Comments

Leave a Reply

Your email address will not be published.Required fields are marked *

More Community Stories

Code Review is an Architectural Necessity: Quality attributes that code review ensures

This post is the second in a series on code review and architecture. In this installment, I discuss the key attributes that code review guarantees.

IBM DataWorks is renamed IBM Bluemix Data Connect

IBM DataWorks service has been renamed IBM Bluemix Data Connect as part of the strategic announcement to make data simple and accessible to all.

Code Review is an Architectural Necessity: What is it and what problems does it solve?

These days, everything is code. Architecture and design documents can be expressed in code. You set up your systems with configuration management software, right? Infrastructure is code. A typo could be an immediate tenfold cost increase. Code review doesn’t just help build knowledge of application code anymore. It forces us to write down our discussions and make them searchable.