NOTE: The data in this blog was downloaded from the city of Boston’s Open Data portal (https://data.cityofboston.gov/) and independently analyzed using IBM Cognos Analytics and Watson Analytics.
Building on the IBM and Datawatch partnership that was announced earlier this year, the two companies now announce the availability of Datawatch Monarch for IBM Analytics.
Introduced at the IBM Vision conference in Orlando, Florida (May 9 – 12, 2016), Datawatch Monarch for IBM Analytics features self-service data preparation that directly exports your data to IBM Watson Analytics and IBM Cognos Analytics.
Spending too much time preparing data and not enough time analyzing and visualizing it? IBM and Datawatch have a combined solution to help.
With the recent IBM and Datawatch partnership, announced at Gartner Business Intelligence & Analytics Summit, Mar 14-16, you can use Datawatch Monarch to extend the data access and preparation capabilities of IBM Watson Analytics and IBM Cognos Analytics.
Use this collection of self-service data preparation and analytics tools to take your data from acquire, prepare, and merge … to explore, visualize and analyze … a lot quicker.
- Datawatch Monarch – quickly convert and combine semi-structured data from multiple files and formats and then export the structured data directly to Cognos Analytics and Watson Analytics.
- Cognos Analytics – build interactive dashboards from a collection of data sources and visualizations.
- Watson Analytics – explore your data using cognitive technology and guided analytics.
Here’s a summary of an analytics journey using these tools on a collection of data that included open city 311 data, weather data and demographics.
Acquire, prepare, and merge … explore, visualize and analyze
The city of Boston, MA – like many other cities across the country and around the world – logs hundreds of thousands of 311 service requests each year. These requests come from citizens and city workers reporting the need for every-day, but important city services for things like snow plowing, street cleaning and pot hole repair.
For this example, we used Datawatch Monarch, Cognos Analytics and Watson Analytics to import, merge and analyze the 311, weather and demographic data acquired from PDF, xls files and web pages. We started out looking just at the combination of 311 and weather data, but this brought us to the idea of adding demographic data to see what other interesting correlations might exist. This combined data gave us the big picture of the city’s primary customers (the citizens) and their related service requests.
Data wrangling with Datawatch Monarch
Datawatch Monarch made easy work of combining our three disparate data sources – Boston 311 requests, weather data and demographics for Boston’s neighborhood – into one structured data file for analysis. Using Monarch’s core feature of importing and parsing semi-structured data from different file formats, we were able to quickly acquire the data from a range of sources, including PDF and xls files and web pages.
First, we imported almost 200,000 records of Boston 311 requests from 2015 (https://data.cityofboston.gov/City-Services/311-Service-Requests/awu8-dc52). Then we matched that data with several hundred thousand weather data points by zip code and date/time stamp. Finally, we joined all of that data with demographic data from a web page for Boston’s 20+ neighborhoods. For the demographic data, we were able to automatically extract and parse that data directly from an HTML table on the following web page: https://en.wikipedia.org/wiki/Boston.
We made use of the pre-built functions in Monarch, designed specifically for data preparation, to do things like split dates into separate fields and rename column titles to make them more user-friendly and ready for natural language processing (NLP). This last data enhancement made asking questions about the data a lot easier in Watson Analytics.
To join the data, we used Monarch’s intuitive and visual interface to arrange the multiple data tables and configure the data join properties. With a couple clicks, we converted and merged all the different input data into one structured data table for analysis.
The final step was to export the combined output directly to Watson Analytics and Cognos Analytics.
Dashboard creation with Cognos Analytics
Bringing the data directly from Datawatch Monarch into Cognos Analytics allowed us to quickly get to work building dashboards using the easy-to-use and visually interactive features in Cognos Analytics. We were able to create visualizations and dashboards using simple drag and drop actions and animate the data using smart filters and data players.
Using dashboards we were able to visually combine and compare the three different data sources on a single canvas. We could investigate how long it takes to resolve certain requests based on type of request and neighborhood where the request came from. We could also blend in the weather data by looking at request resolution compared to temperature levels and weather conditions. This allowed us to quickly view what happens to requests during the summer months or during heavy snow conditions.
Adding the demographic data into the mix gave another perspective on the interactions and correlations between the different data sources. For example, we could look at population, median income and number of households compared to service requests, response time and weather.
Data discovery and predictive analytics with Watson Analytics
To further explore the data and run predictive analytics on it, we imported the data directly from Datawatch Monarch into Watson Analytics.
We started exploring the data by using the natural language-based question feature in Watson Analytics. The column renaming and data cleansing we did in Datawatch Monarch enabled us to more easily ask questions in plain English in Watson Analytics like: “What is the number of Service Requests by Neighborhood?”
Watson Analytics applied its cognitive smarts to understand our question and suggest relevant visualizations. Choosing the first “very relevant” suggestion displayed a packed bubble visualization. Here, there’s a bubble for each neighborhood where bubble size represents the number of requests from that neighborhood.
Some other questions we asked included “What is the breakdown of Days to Resolution by Neighborhood?” and “What is the breakdown of Days to Resolution by Request Type?”.
Next, we started wondering if there were any insights in the data that might impact how fast a request is resolved. Does it matter what neighborhood you are calling from? What the temperature was when you called? What type of request? Or some other attribute? Too many variables to sift through and analyze on your own.
To do this, we ran a prediction analysis in Watson Analytics with the prediction target set to Resolution Efficiency. Are requests resolved in “less than a week”, “2 to 4 weeks”, “more than 6 months” or other time period?
The results showed a predictive strength of 68% that Resolution Efficiency is impacted by which department the request is assigned to (Department Assigned). So, if your request goes to one department it could be resolved pretty quick, but if your request gets assigned to a different department it could take longer. The spiral diagram in Watson Analytics visually and interactively summed up the results.
Drilling into the prediction results showed us insights that were not intuitive from just looking at the data. For example, the Disability Commission closed out nearly 100% of its requests in less than a week, while other departments take longer to resolve requests.
So there you have it, a trio of tools to help you spend less, but more efficient time wrangling your data and more smarter, productive time analyzing and visualizing it.
- Using Datawatch Monarch, we went from three separate collections of data, all in different formats and domains, to a single unified data set ready for smart data discovery.
- With Cognos Analytics, we quickly built interactive visualizations and dashboards to investigate and visualize trends across 311 requests, weather and demographics.
- With Watson Analytics, we used smart data discovery, asked questions using NLP and ran predictive analytics to find insights that typical data discovery would not have found.
Use this combination to iterate through your data analysis faster, more efficiently and more broadly by merging and visualizing a wider collection of data and information.
- 7 Reasons Why I’m Adding Datawatch Monarch for IBM Analytics to Watson Analytics
- Datawatch Monarch for IBM Watson Analytics Now Available
- IBM and Datawatch announce partnership at 2016 Gartner BI & Analytics Summit
- Datawatch Brings Powerful Self- Service Data Preparation Capabilities to IBM Watson Analytics and IBM Cognos Analytics Users