With ever-growing demand for near-real time analysis of huge quantities of operational data, how could Nedbank Group expand its data warehouse while maintaining agility and keeping costs in check?
Nedbank Group is building a federated, virtualized analytics environment that integrates IBM® Db2® Warehouse and IBM Fluid Query with its existing IBM PureData® System for Analytics data warehouse.
25-35%reduction in data storage costs as data volumes rise
Providesa single point of control for analyzing data wherever it is stored
Boostsagility with a container architecture that can run on-premise or in the cloud
Business challenge story
Seizing opportunities to build on success
Insight into operational data is essential for banks to develop new products, satisfy regulatory requirements, and protect themselves and their customers against fraud. Because almost all departments make use of analytics to guide their decisions and actions, it’s crucial that banks make sure their analytics capabilities are rapid, reliable and effective.
Paul Morley, Executive, Group Data Services at Nedbank Group explains: “We rely on data analysis to support many of our core operations—and I’m not just talking about generating standard reports and dashboards. Many different areas of the business require an analytics architecture that can support highly demanding applications, such as propensity modeling of customer behavior, fraud detection, and risk analytics.”
To keep its analytics function running effectively and efficiently, Nedbank had already implemented IBM PureData System for Analytics (formerly Netezza) as a data warehouse, centralizing data previously siloed in different systems into one domain.
“We established three core principles to guide our goal of building a centralized data architecture,” continues Paul Morley. “Our new architecture had to be technology-agnostic, agile and offer us the ability to analyze data at near real-time speeds.
“We were attracted to PureData because we were confident that its parallel processing and large onboard memory would provide the power we needed to accelerate enterprise-wide analytics. On top of the PureData platform, we built a Metadata Rules Engine (MRE) which contains all of the business rules and algorithms to help us ensure the consistency of our data sets and load data from all of our various sources into the analytics engine.
“The MRE enabled us to consolidate 13,000 separate load processes into just 1,000, which allows us to load nine terabytes of data each day within the two-hour window we have each night. The solution also shattered all of our previous processing records. For example, one of our propensity modeling jobs went from two days down to just 30 minutes. After seeing how we managed to accelerate our analytics capabilities with this IBM solution, demand within the business for data analysis really began to take off.”
As well as needing rapid data analysis of current data, the business soon realized it also needed an Operational Data Store (ODS) for the analysis of historical data.
Paul Morley adds: “We built the ODS to facilitate analysis of data collected over the past 180 days. This put additional pressure on our data storage resources. And because the ODS was so effective at storing historical data for analysis, parts of the business began to rely on it for data archiving too, which increased capacity needs further.”
Nedbank’s capacity plans became the first casualty of this unprecedented success. The Group Data Services team saw the number of users rise from 88 to 1600, with year-on-year data growth jumping from 5 percent to 18 percent. The original IBM PureData System for Analytics appliance quickly reached capacity, and over the five-year period since the original implementation, Nedbank has upgraded or expanded the environment three times.
“The rapid growth in both user numbers and data volumes makes it much more difficult to conduct effective capacity planning,” says Paul Morley. “We set out to find a cost-effective way of evolving our data warehouse.”
Investing in the future
To meet increased business demand for high-speed analytics, Nedbank worked with IBM Business Partner BITanium to introduce IBM Db2 Warehouse to provide a flexible, scale-out extension to its data storage landscape. Meanwhile, IBM Fluid Query would provide a single point of control for accessing and analyzing data, regardless of whether it was located on the PureData appliance, in Db2 Warehouse, or potentially even in other storage environments such as Hadoop.
Paul Morley explains: “Diversifying our data warehouse architecture with Db2 Warehouse held clear advantages for us. Knowing that IBM is aligning its development roadmaps for PureData and Db2 Warehouse was an important consideration, because we wanted to make sure all the components of the landscape would integrate well and work together as a coherent whole.
“IBM Db2 Warehouse provides us with the agility we need—it is built on Docker, which means it’s essentially infrastructure-agnostic and portable. So, in the future, whether we decide to stick with an on-premise deployment or move some of our data into the cloud, we’ve got that flexibility built in.”
Another aspect of the solution’s versatility is the fact that IBM Db2 Warehouse includes an integrated Apache Spark cluster. This will make it possible for Nedbank to analyze unstructured data as well as traditional SQL database tables.
Paul Morley adds: “To reduce the need to move data between different systems, we’re using IBM Fluid Query. It enables us to conduct analysis in situ on our IBM PureSystem appliance and pull data out from our Db2 environment via a Denodo virtualization layer. The end goal is to build a federated data warehouse with a hierarchy of storage environments, with Fluid Query as a single point of access that can query data regardless of where it resides.”
For data archiving and recovery, Nedbank collects data from the MRE and stores it in a compressed file format on IBM PureData System for Analytics. The sophisticated architecture of this solution helps the bank quickly re-build historical snapshots in its ODS system without reloading data.
Paul Morley adds: “Being an early adopter always has its challenges, but whenever we’ve encountered any issues, IBM has always been eager to help and has fed our experience back into product development.
“Meanwhile, BITanium have been an important partner for many years—they are very much part of our team, and their data warehousing expertise is second to none. The strong relationship we have with both companies has helped drive success at Nedbank, and will continue to play a central role as we develop our systems in the future.”
Technology that pays dividends
With IBM solutions powering its data warehouse, Nedbank has been able to reduce the cost of storing its data, simplify data handling processes, and build a flexible analytics platform that can be easily adapted to meet the changing needs of the bank.
“With IBM Db2 Warehouse we will be able to reduce the cost of data warehouse storage capacity by up to 35 percent,” says Paul Morley. “Combined with IBM PureData System for Analytics and IBM Fluid Query, IBM Db2 Warehouse is helping us build a more versatile and scalable analytics landscape, making growth much more sustainable.
“As a result of increased levels of integration in our architecture, we no longer need to spend significant amount of time performing ETL processes or move large amounts of data between our systems. Business users will receive data faster, helping to reduce time-to-insight for departments handling fraud detection, customer service and risk management to name but a few.”
By containerizing data sets in IBM Db2 Warehouse, Nedbank has also been able to boost agility, enabling it to adapt its infrastructure with ease in response to changing business requirements and technology trends.
Paul Morley says: “Containerization opens up new possibilities for us. We could move into the cloud with minimal effort and almost no disruption to our essential business processes.”
He concludes: “The IBM solutions put us within reaching distance of our goal to create a centralized virtual data warehouse. We can provide business users with the tools they need to provide excellent levels of service and build the next generation of banking products and services for our customers.”
About Nedbank Group
Nedbank Group is one of South Africa's four largest banks, with Nedbank Limited as its principal banking subsidiary. The bank has a presence in 6 countries in the Southern African Development Community (SADC) and the East Africa region: it owns subsidiaries in Namibia, Swaziland, Malawi, Mozambique, Lesotho, Zimbabwe, as well as representative offices in Angola and Kenya.
- Db2 Warehouse Private
- PureData for Analytics N2002-002
Take the next step
IBM Analytics offers one of the world's deepest and broadest analytics platform, domain and industry solutions that deliver new value to businesses, governments and individuals. For more information about how IBM Analytics helps to transform industries and professions with data, visit ibm.com/analytics. Follow us on Twitter at @IBMAnalytics, on our blog at ibmbigdatahub.com and join the conversation #IBMAnalytics.
BITanium delivers specialist professional services in the business intelligence, advanced analytics, and data warehousing spaces. To find out more, visit: bitanium.co.za