First off, all three workflows required data ingestion. We proposed consolidating all data ingestions through Kafka to ingest large volumes of streaming data across different channels. Although their existing platform already used Kafka and other custom ingestion sources, we recommended consolidating multiple ingestions into Kafka and leveraging it as a service on IBM Cloud through IBM Message Hub. Kafka provides guaranteed delivery for streaming messages for real-time application. Since Kafka is an open-source technology with no vendor lock-in, the client would have flexibility in vendor choice.
Second, we proposed IBM Streaming Analytics on IBM Cloud to combine real-time and near real-time application use cases for message processing. IBM Streams is technology proven to scale for up to hundreds of millions of users. Although Streams is not open source, it supports the development of applications in Java, Scala, and Python. IBM Streams also supports Apache Beam, which is a programming model that allows users to create platform-agnostic streaming applications. Operators are the lowest-level Streams programming, which might be an adapter like read/write from Kafka and be able to apply a machine-learning model to make predictions.
In case of real-time/near real-time data ingestion, we recommended IBM Streaming Analytics to process the messages. Consumers of IBM Streams output will go into the downstream component—Druid, elastic stack, Cassandra, or HDFS, depending on the workflow. Additionally, the decision engine in the data pipeline will be triggered through the IBM Streams operator based on the type of event. The events data can be recorded into a Cassandra data store (deployed on a customer cloud account) for historical purposes and for real-time/near real-time decision making. The client was already using Druid as a column data store, and we kept it for OLAP queries, which can provide near real-time dashboarding capability for the interactive user applications needs.
The other important technology decision for the target architecture was centered on Hadoop clusters, which can be either deployed as an IaaS solution or as a cloud-native offering through IBM Analytics Engine. We recommended IBM Analytics engine to provide two key capabilities through a single solution: compute-as-a-service and object storage. A major benefit of IBM Analytics Engine is its independent scalability as opposed to being tightly coupled. This empowers a large number of the organization’s data scientists and their developers to focus on building new algorithms and models to extract insights from massive data sets without worrying about building a permanent cluster. IBM Analytics Engine simplifies the complexity of cluster deployment and provides a straightforward way to spin up Hadoop and Spark clusters on the cloud within minutes.
The role of data governance
Implementing data governance was one of the client’s key requirements in migrating their data platform to the cloud. Our team recommended IBM Data Catalog and Data Refinery, which are now part of the IBM Watson Knowledge Catalog. The Knowledge Catalog provides a broad range of capabilities and tools for data cataloging, discovery, findability, and governance. IBM Watson Knowledge Catalog includes built-in data discovery algorithms that utilize machine learning to auto-classify the contents of each data set and a governance policy manager and engine. When you add the data to the enterprise catalog, for example, sensitive data will automatically become classified. The Catalog also immediately redacts or masks any data that users are not allowed to see, aligning it to an organization’s data governance policies.
IBM Data Refinery is embedded into the IBM Watson Knowledge Catalog and Watson Studio, allowing data scientists to connect to data sources regardless of where the data resides, explore data, and use a wide range of transformations to cleanse and transform data into the format necessary for analysis.
IBM Data Science Experience (DSX), which is also now part of Watson Studio, provides a tooling that data scientists can focus on analyzing data. Creating and running models and spark jobs can be invoked through DSX.
Big data on cloud = no brainer
Implementing a big data platform stack on the cloud can provide flexibility, agility, and innovation for the enterprise. The role of IT infrastructure has changed from a cost center to one that is extremely flexible and innovative. By leveraging the cloud and a big data platform in the cloud, IT can become an innovation team-driving value.
Schedule your complimentary IBM Cloud Garage consultation to get started.
Amit Rai, Big Data Platform Architect, supported this Garage engagement, and he contributed to the article.
