At the end of the Superbowl, people created 12,233 tweets per second. And it turns out that was less than half the
number of tweets created in Japan
on December 9th, when 25,088 tweets per second were recorded about
the Castle in the Sky anime movie.
Which, according to the Chinese, is nothing compared to the 32,312
messages per second sent on their twitter-like Sina Weibo system during the
beginning of the Chinese new year.
Within the government space, we’re no strangers to our own Big Data. Whether you’re in the DOD or NASA, the IRS or
SSA, you’ve got your own Big Data to deal with.
Last week, Forrester Research released a report that should help those in
government understand the Big Data Market.
It is called “ The Forrester Wave™: Enterprise Hadoop Solutions, Q1 2012,
(February 2, 2012)” report. IBM Technologies evaluated were IBM InfoSphere
BigInsights (IBM’s Hadoop-based offering), and IBM Netezza Analytics. In this
evaluation, IBM was placed in the Leaders category of the Wave and achieved the
highest possible score in both the Strategy and Market Presence segments. In
the third segment, Current Offering, IBM received the second highest score. You
the complete report here.
The report by analyst James
Kobielus states, “IBM has the deepest Hadoop platform and application portfolio.”
The IBM Analytics Solution
Center in Washington, DC
also focused on how to handle Big Data at its January 19th
seminar. The seminar covered various
aspects of Big Data including data-in-motion processing software, Hadoop
software, SONAS (scale out network attached storage), and the Netezza data
1. Big Data in Motion
back to the Tweeting, if you’re a government agency and you need to get
actionable insights into 10s of thousands of tweets per second which might be
about an unfolding crisis, how would you do it?
InfoSphere Streams is unlike anything else in the market in its ability
to ingest, analyze and act on data “in motion” – that is, data is processed and
analyzed at microsecond latencies.
2. Hadoop Big Data
is an open source codebase supported by the Apache software foundation. It is designed to process large volumes of
unstructured data. For example, if a government agency wanted to analyze months
of tweets or documents in non-real time, the Hadoop distributed file system
would be a good choice. The enterprise
class IBM Hadoop-based offering, BigInsights, is designed with system
management, security, and performance features that go beyond what is available
in the open source. It provides the
ability to analyze and extract information from a wide variety of data sources,
and promotes data exploration and discovery.
Attached Storage, or NAS, has become a very popular way to provide storage
within an organization. However NAS has
a number of limitations when dealing with
Big Data including the number of objects (files) it can support, support
for very large files, the i/o bandwidth
it can deliver to applications, and fragmented data management across multiple
systems. The IBM SONAS system is
designed to overcome these limitations and look like a very large virtual
system to the applications.
4. Data Warehouse Appliance
data warehouses when used for large volumes of structured data can be costly to
operate and maintain, and can be very slow when used for sophisticated
analysis. The Netezza appliance is a
dedicated device requiring no tuning or storage administration and with special
hardware chips to accelerate the performance of advanced analytics.
Want to learn more?
- More details on the topics can
be found at the ASC Website under
- On the educational front, we
provide free online training through BigDataUniversity.com. To
date, more than 13,000 students have registered for courses on Hadoop,
cloud computing and more.
We are working with a broad range of clients to help them define
their big data strategies. We look forward to working with you on your Big Data
The Forrester Wave™: Enterprise Hadoop Solutions, Q1 2012,
Forrester Research, Inc., February 2, 2012. The Forrester Wave is copyrighted
by Forrester Research, Inc. Forrester and Forrester Wave are trademarks of
Forrester Research, Inc. The Forrester Wave is a graphical representation of
Forrester's call on a market and is plotted using a detailed spreadsheet with
exposed scores, weightings, and comments. Forrester does not endorse any
vendor, product, or service depicted in the Forrester Wave. Information is
based on best available resources. Opinions reflect judgment at the time and
are subject to change.
The news last week was all about the weak job market.
Fed Chairman Ben Bernanke characterized the
job market as showing “continuing weakness.”
Well, guess what?
The job market
for those with Analytics skills is very hot.
Monster has over 1000 job listings for Business Analytics jobs.
Here at IBM, we have over 100 openings for Business
Analytics and Optimization jobs. Some of
these are associated with our Public Sector Practice, consulting to Federal,
State, and Local Governments or developing data-intensive, analytics solutions
to help them perform their mission.
Why are there so many jobs in this field? Businesses and governments today must figure
out how to do more with less.
Organizations can analyze data coming from their business processes to
develop new approaches to streamlining or even optimizing their business. In the past, many decisions involved in
running an organization were based on “gut instinct.” Today, it is not longer defensible to make
decisions in this way when it is possible to make “fact-based” decisions using
hard data. Data stored in a Business
Intelligence system can be used by every level of an organization to help staff
understand their business better, detect problems, and develop solutions that
will allow them to accomplish their mission better, cheaper and faster. Sophisticated analysis can be done on the
data to predict what will happen if the current trends continue, determine how
to achieve the best outcome, and study the impact of external uncertainties
such as the economy or the weather.
According to the Bureau of Labor Statistics in their
2010-2011 Occupational Outlook Handbook,
the employment of operations
research analysts is expected to grow 22 percent over the 2008-18 period. While not all analytics jobs require an
operations research degree, this gives a good indication of the long term
trend. We know that technology is continuing
to improve both in terms of raw compute power and in the design of efficient
algorithms to analyze and optimize solutions.
This increasing capability will drive the demand to add “smarts” to many
more systems and processes, and will drive the need for analysts who can apply
the technology. So analytics isn’t just
a good short term career choice, but a good target for long-term career
To do these jobs, though, requires in-depth skills and
knowledge. Skills in operations research
(OR) techniques, data mining, optimization, decision theory, and data analysis
are needed, along with some background in IT systems. The ideal candidate will also have some
domain knowledge about government or business functional areas since it is very
hard to apply the mathematical techniques in abstraction.
How to Find Analytics Jobs
Most Analytics jobs aren’t listed under “analytics” and many
won’t even come up under that keyword.
Use search terms like ‘business intelligence,” “performance management,”
“optimization,” and “operations
research.” If you have experience with
actual analytics software such as Cognos, SPSS, Intelligent Miner, or ILOG,
both Monster and IBM’s website return hits on those keywords.
Want to learn more about jobs at IBM in Business Analytics?
Go to www.ibm.com/employment
and click on the “Search for Jobs at IBM” link
You may also write me at ASCdc@us.ibm.com
Analytics Solutions Center of Washingtion, D.C. Director
Do today’s MBAs need Analytical Skills? That was the question that a recent Symposium
tried to answer.
On October 21, George
Institute for Integrating Statistics in Decision Sciences (I2SDS)
and IBM’s Analytics
held a Symposium entitled: Analytics and the 21st Century MBA. The abstract provides a good description of
the thesis of the Symposium:
The 21st century belongs to those who can think and act analytically. No
longer is it good enough to make business decisions, no matter what the field,
based on little more than feelings or gut reactions to events. Consumer
products companies, insurance companies, banks, governments, and even sports
teams are turning to Analytics to improve their bottom line and assure their
survivability in this age of hyper-competition and increasingly severe
This Symposium… will demonstrate how Analytics is, a critical component
of 21st Business careers, whether the practitioner's primary responsibility is
in a functional area (Marketing, Operations, Finance, Strategy, International
Business, HR) or a vertical such as Health Care or Tourism.
The Symposium provided talks by leading users of Analytics in Marketing,
Retail, Finance, and the Public Sector.
More on the Symposium is at: http://business.gwu.edu/decisionsciences/i2sds/pdf/GWU%20ASCOutline.pdf
Do you agree with the thesis? Are you
seeing more need for employees with analytical skills? Do you think those with these skills are
having an easier time getting jobs?
I’d like to hear your thoughts.
Frank Stein, Director, Analytics
Does your government agency monitor the social media for information relevant to your mission? Should it?
IBM's Analytics Solution Center recently held a seminar to explore
how agencies and companies can obtain value and insight using social
Pat Fiorenza discussed how agencies can develop an ROI Model - Return
on Influence Model - for social media. Agencies use social media
analytics to help inform their decision making by gathering
information/research, and learn what other agencies and citizens are
saying. Interesting examples from CDC and Govloop were provided.
Learn more here.
Ed Burek, IBM, talked about how savvy companies are now taping into
customer generated content, how government agencies could do the same to
learn how tax payers feel about government actions and messaging. He
gave examples of how regulatory agencies could received the unvarnished
comments from those impacted by regulations, as well as how they could
stay on top of "negative chatter." IBM has created a framework to
derive business insight from the vast amounts of social media that is
now being transmitted. Called Cognos Consumer Insight it provides real
time information on trends and sentiment.
Rick Lawrence, IBM Manager for Machine Learning at Watson Research
Center next talked about the leading edge of social media analytics. He
provided examples from the research portfolio on discovering Who are
the Key Influencers? , Identifying emerging topics of discussion, and
Mapping the billions of tweet to concepts that we really care about.
All of the presentations are available on the ASC website under Past Events (May 10, 2012)
Does your agency care about what its constituents are saying about it
on social media? Does your agency need to have real time intelligence
on events within its mission space? With 340 million Tweets per Day, 2
million blog posts, and 500 million facebook updates, how can you find
the important information? Social Media Analytics may be an idea
whose time has come.
Analytics Solution Center
P.S. The Center for the Business of Government issued a new report on Tweeting in Government. Pat provided a good overview here.
Apparently, pretty good, according to Nucleus Research. They recently completed 2 ROI Case Studies of 2 government analytics projects. Both showed impressive results:
- Alameda Country Social Service Agency's Social Services Integrated Reporting System (SSIRS) had an ROI of 631% and a payback of 2 months
- Memphis Police Department's Blue CRUSH (Criminal Reduction Utilizing Statistical History) had an ROI of 863% and a payback of 2.7 months
The ROI calculations may even be conservative as Nucleus Research appears to assume that the agency and department will pay taxes on the annual benefits from the solutions.
The SSIRS system helped Alameda County reduce overpayments to non-compliant citizens, improve their win rates when claimants appealed discontinuation of benefits, and improved caseworker productivity. The system is essentially a Business Intelligence solution giving the caseworkers access to information about their clients, with dashboard and drill down capabilities. It also provides the caseworkers and managers with immediate information on "how am I doing?". Providing caseworkers with information on their clients' work participation rate and other performance metrics was key to improving the performance of the social service agency. The solution combined Cognos Business Intelligence, Infosphere Identity Insight, and an Infosphere warehouse to hold all the data. Identity Insight helps the caseworkers track the relationships between the various clients (e.g., parent/child) that may impact services offered. Here is a video where Don Edwards, Assistant Agency Director, talks about the solution: YouTube Video
The Blue CRUSH solution helped the Memphis Police Department (MPD) to identify crime "hot spots" and then target these areas for increased attention. As a result, MPD has reduced violent crime without additional staffing. The solution uses IBM SPSS Predictive Analytics software to analyze crime data pertaining to type of criminal offense, time of day, day of week, location, and the weather. The solution was developed with the assistance of the University of Memphis Department of Criminology and Criminal Justice.
Memphis Police Department received a National award from Nucleus
Research for this solution. They were one of only
ten companies and governmental agencies to receive the Nucleus
Research ROI award. Out of 350 technology projects that were
submitted, the Memphis Police Department was one of only two
governmental agencies to receive an award. The other governmental
agency was the US State Department.
If you'd like more information on these two case studies please contact me at firstname.lastname@example.org.
More information about Analytics, including our Fall Analytics Seminar Series,
can be found at www.ibm.com/ASCdc
Director, Analytics Solution Center
Sam Palmisano, Chairman and
CEO of IBM got together with Michael Dell, Chairman and CEO of Dell to release
an op-ed piece last week that the government can save $1 Trillion through the
use of IT.
for the statement.
Jeffrey Zients, Chief Performance Officer,
penned a blog
shortly thereafter titled, “Seeing Eye to Eye with the Tech CEO Council.”
Many of the
examples cited in the Palmisano/Dell statement relate to the use of analytics:
- Consolidating the government’s myriad supply
chains is likely to save $500 billion.
- Applying advanced analytics to reduce fraud
and error in federal grants, food stamps, Medicare payments, tax refunds
and other programs could save $200 billion.
- Using predictive
technology, New York
State is validating
tax refund requests and saving $889
million by catching phony refunds.
- Identifying suspicious Medicare activity using
analytics has shown North Carolina
how to save $25 million in just three months.
In addition to
helping to uncover fraud, waste, and abuse, I’d like to suggest 3 other ways analytics
can help the government to save money.
- Streamlining Processes: Analytics can help streamline and
optimize programs, reducing the costs of implementation while improving
service to citizens. For example,
IBM worked with Social Security to streamline their processing of
disability claims so that the majority of claims can be expedited with
little risk of allowing through unacceptable claims.
- Managing Performance: Performance management solutions can
help the management and staff of agencies to know their up-to-date
performance, and quickly spot and trouble-shoot performance issues before
they become major problems. Performance management can also help identify
successful approaches that can be replicated throughout and across
- Better decision-making: Analytics can help
agencies decide which programs to fund or the most effective
approach to take for a particular program.
By using modeling, simulation, and other data-driven approaches,
agency staff can make decisions that both save the tax payers’ money and
deliver the best results. For
example, by modeling and optimizing the US Postal Service transportation
network, USPS is able to increase utilization of assets and save hundreds
of millions of dollars.
I’d like to hear
your ideas for how agencies can save money through employing analytics. Write to me at email@example.com.
See our website for
further information on using analytics in government: www.ibm.com/ASCdc
Director, Analytics Solution Center
On November 30, the Partnership for Public
Service (www.ourpublicservice.org) released
their new study, “From Data to Decisions: The Power of Analytics.” [i]
Keynoting the event was Shelley Metzenbaum, Associate Director for
Performance and Personnel Management, OMB.
She told the audience that Performance Management is a core pillar of
the Obama Administration and that Measurement and Analysis was the key tenet to
PM. She encouraged the audience to
identify analytics practices that work and spread the word to others. She exhorted the audience to not just collect
data but to use the data to pinpoint problems – “Ask Why, Why, Why” with
respect to performance problems.
The report studied 7 programs[ii] in 8 federal
agencies to understand how they use analytics and how it helped them achieve
better program results. The study
provides clear examples of how data is being used to understand problems and
improve mission performance. It
documents how CMS is using data to answer the question why isn’t health care
quality better and how can we direct scare resources to improve it? In a similar fashion, VA and HUD are using
data to figure out how to reduce homelessness of Veterans including identifying
bottlenecks that are keeping their voucher program from being more
successful. David Zlowe, Performance
Improvement Officer at VA, emphasized in the study that the power of VA’s
analytics approach isn’t in the numbers but in the discussion that are sparked….having
leadership engage in an appreciative conversation guided by hard data.” The 4th program that the
Partnership reported on in some detail is the FAA’s Safety Management
System. This program helps to identify
risks and to understand what contributes to all levels of hazards.
The Partnership event included a panel discussion with Michelle Snyder,
Deputy COO, CMS; Estelle Richman, COO
and Acting Deputy Secretary, HUD; and David Zlowe, PIO, VA. Ms.
Snyder’s advice to the audience was, “Take data, analyze it, tell the story to
the people so it relates and influences the decision makers.” Ms. Richman’s recommendation was to remember
that the analytics are but a method to accomplish the goal of creating an
outcome that can improve people’s lives.
And Mr. Zlowe summarized by saying, “We don’t lack data, we lack
We’d like to hear your experiences driving decisions based on data
in the government. If you'd like a copy of the report, write to me at: ASCdc@us.ibm.com
[i] The Study was a
collaboration between IBM’s Center for the Business of Government and the
Partnership for Public Service
[ii] HUD and VA Veternans
Affairs Supportive Housing (HUD-VASH) program; Safety Management System (SMS)
in the FAA; HHS CMS nursing homes and transplant programs; Coast Guards’
Business Intelligence system (CGBI); NHTSA “Click it or Ticket” campaign;
Navy’s Naval Aviation Enterprise; SSA’s use of mission analytics n customer
Watson is the only computer on the planet that can answer a Jeopardy!
question in less than three seconds - fast enough to be competitive with the
world’s best human players.
of you that missed the match click here
to see a video clip from the match.)
But can a Watson-like computer help the government?
Watson was optimized to tackle a specific challenge:
competing against the world’s best Jeopardy! contestants. It does this by sifting through large amounts of unstructured information to find potential answers and assigning a confidence measure to each potential answer. When it has high confidence in an answer, it will buzz in and offer the answer. Beyond Jeopardy!,
IBM is working to deploy this technology
to businesses and governments dealing with the information overload
problem. At work, few of us are like
Ken Jennings, able to instantly answer almost every question thrown at us - -
with an 80-90% success rate. There is
simply too much information and more information is coming in all the
time. Whether we’re in finance, HR, IT,
or another area, our success at work depends upon dealing with huge volumes of
information, sifting through it to find
the “good information”, and then using the information to make decisions to do our
job. Technology like that used in Watson can provide for our consideration potential answers as well as the "evidence" it used to come up with potential answers.
In discussions recently with some of our military colleagues,
they came up with numerous ideas for deploying Watson-like technology. They cited the problem of “request overload” - - dealing with all the
requests for Predator and similar UAV missions.
How could they deploy their limited resources to best effect? Another person mentioned the problem of
sifting through all the intelligence information – most of it in the form of
unstructured information formats such as video and text – to find the relevant
information to a mission they were planning.
Another discussed the problem of monitoring their “situational
awareness” and how hard it was to keep track of all the data coming in. “Could Watson help monitor our security
posture and alert us to potential threats?” asked another.
Are you dealing with massive amounts of information? How could a Watson-like system assist you at
work? Do you want to recruit Watson to
work for your agency? We want to hear
your thoughts either in this blog or directly.
Write to me at firstname.lastname@example.org.
We’re hosting 2 free Watson Overview Briefings
on July 26 and 27. More
information at our website: www.ibm.com/ascdc
Frank Stein, Director, Analytics Solution Center
As government leaders do you believe the world is getting
more complex? More volatile? If so, you’re not alone - - Sixty percent of
the CEOs surveyed by IBM in our 2010 CEO Study thought the world was getting
more complex, and even more, 69%, felt the world was getting more
For the first time, we also posed a similar set of questions
to college students. These future
leaders viewed the world as even more complex than the CEOs we surveyed. But
they saw less volatility, and significantly less uncertainty than the CEOs (65%
of the CEOs, but only 48% of the students).
Could it be that the students are more acclimated to economic boom/bust
cycles and feel more comfortable with the uncertainty of today’s world?
Or could it be that in the instrumented, interconnected,
collaborative world that they are used to (most of the students never knew a
world without web browsing and many don’t remember the pre-Facebook era), they
feel more comfortable dealing with this complex world? As a student in France put it, “We will have more
information, so it [the world] should be more predictable.”
We found that students who had the greatest sense of
complexity put much more emphasis on the analytics and predictive capabilities
of information. They were 50% more
likely to expect significant impact from increased information than peers who
did not have the same sense of complexity.
And they were 22% more likely to believe that organizations should focus
on insight and intelligence to enable their strategies. Also,
interestingly, students in China
were significantly more likely to prefer a fact- and research-based style of
decision making than their peers around the world. Does that indicate that the Chinese students
have been trained to feel more comfortable dealing with data than their
With the baby boom heading towards retirement in the coming
years, does this mean the government workers who replace them will be more
comfortable using information and analytical techniques to handle the world’s
problems? Or could it be that complexity
will always rise to be just beyond our ability to manage it with our current
level of technology?
Click here to see the IBM Report: “Inheriting
a complex world”
Click here to see the IBM Report: “2010 Global CEO
More on Analytics for Government here: www.ibm.com/ASCdc
Do you think our future leaders are inheriting a more
complex world? And do you feel they are
more prepared to manage it?
Comment on this blog or write to me at ASCdc@us.ibm.com
Frank Stein, Director of IBM’s Analytics Solution
In these tough fiscal times, all agencies are going to be
focusing on doing more with less. How
does one get more done with less budget and staff? Consider turning to Analytics.
The consulting firm Nucleus Research has been looking at the
Return on Investment (ROI)
for various types of IT projects.
According to David O’Connell, Principal Analyst at Nucleus Research, “projects
involving analytics have some of the highest ROIs of any projects studied.”
Nucleus Research recently studied an analytics project IBM performed at DC
Water, the local water authority for Washington,
DC. In 2008, IBM began a first of a kind project
using advanced analytics to create a smarter water system that analyzes data on
valves, storm drains, service vehicles, truck routes and more to optimize its
infrastructure. With some pipes and other assets that date to the Civil War,
maintaining high levels of service while replacing older infrastructure is an
The project has resulted in the following benefits from a combination of IBM
Asset Management and Analytics technology and services:
Field Services trucks can be automatically
routed to optimize work management. This results in more work orders being
completed each week, as well as up to 20 percent reduction of fuel costs
related to fewer truck rolls and reduced "windshield" time.
Revenue loss from defective or
degrading water meters allowed recapture of $3.8 M because the analytics behind
the advanced metering infrastructure delivers more timely identification and
replacement of those meters. Revenue was
also recaptured because DC Water can now identify and bill locations where
there is unmetered water usage.
DC Water has been able to identify
assets most critically in need of repair using predictive analytics, so aging
infrastructure replacement programs can be more accurately scheduled,
preventing costly incidents that reduce service quality, such as outages and
water main breaks. This reduces both
maintenance labor costs and call center
costs associated with emergency incidents.
Nucleus Research reported in its case
study that the DC Water project resulted in $19.677 M of benefits over 3
years with a cost of $883 K, giving an ROI of 629%.
In 2010, Nucleus Research studied a number of other public
sector analytics projects. The results
from these projects are shown in the chart below. On average, the analytics projects have
resulted in an ROI of almost 600%! This
means that over 3 years, the projects have returned benefits 6 times the
original cost of the projects. The
payback period has been less than a year in all cases. This is important to government agencies because
it means you can see cost savings in the same fiscal year that you invest in an
According to David O’Connell, Principal Analyst at Nucleus
Research, “When government entities adopt
analytics, returns are high for two reasons.
First, waste such as leaky water mains, defective meters, or benefits
overpayments can be identified and eliminated.
Second, by making information more readily available, employees spend
less time looking around for information and more time getting their jobs done.” O’Connell went on to say, “Another improvement is better use of
workers’ time. The more an organization
knows about the public it serves, their needs, and the means of delivering
service, the smarter managers’ decisions are when they hand out workers’
Has your agency implemented any analytics projects? What’s been your experience?
Don't feel comfortable sharing
publicly? I'd be happy to hear your thoughts directly as well (email@example.com).
(net savings year 1 + net savings year 2 + net savings year 3)/3 * 100