Part 3 of 3: Leveraging Kibana to create custom visualizations - Search, analyze, and visualize Spark application data with IBM Spectrum Conductor
TinaLangridge 270007S9W7 Visits (897)
Kibana is a popular web-based graphical interface to visualize and navigate data within Elasticsearch.
This blog demonstrates how to integrate Kibana into your IBM Spectrum Conductor cluster for data exploration and visualization based on Spark application resource metrics in IBM Spectrum Conductor. This blog also refers to a downloadable sample of how to use Kibana 5.4.2 with IBM Spectrum Conductor 2.3.
This blog is the final part of a three-part blog series:
Before you complete the Kibana integration, ensure that the following prerequisites are met:
Kibana utilizes the Elasticsearch cluster in your IBM Spectrum Conductor cluster. You may need to scale the Elasticsearch clients to meet increased querying demands; do so by increasing the maximum instances of the elk-
Elasticsearch index template
Aggregated metrics, including allocated slots, cores, and memory for both CPU and GPU, are stored in Elasticsearch indices prefixed with ibm-
This index can be used to create custom visualizations in Kibana. Refer to Part
Integrating Kibana with IBM Spectrum Conductor
To use Kibana to access data collected by IBM Spectrum Conductor and stored in an Elasticsearch cluster, you first need to download Kibana to connect to the Elasticsearch cluster configured in IBM Spectrum Conductor. You can get full instructions and sample script files here.
After you have integrated Kibana in IBM Spectrum Conductor, you can easily create custom visualizations or dashboards based on the aggregated Spark resource usage metrics in IBM Spectrum Conductor 2.3.
Follow the instructions in the sample script and launch a Kibana browser. The sample Kibana dashboard and visualizations are included in the sample and detailed below.
Kibana dashboard with Spark resource usage metrics
The sample includes an IBM Spectrum Conductor dashboard, to provide visualizations fed by the Spark resource usage metrics from your IBM Spectrum Conductor cluster. You can modify all visualizations, including the dashboard, to fit your business requirements.
The IBM Spectrum Conductor dashboard consists of the following visualizations (all visualizations refer to a specified time range, which is initially configured to range from two days prior to now):
You can hover over sections in the visualizations to see detailed statistics for the selected section.
You can apply filters by clicking on a section (such as a Spark instance group, application ID, user, or time range), and the filter will be applied to all visualizations in the dashboard to drill down on that specific filter. For example, in the following screen capture, clicking the Spark instance group SIG3 on the dashboard included above; the dashboard below shows the updated metrics for only Spark instance group SIG3, excluding all other Spark instance groups:
Kibana Timelion visualizations with Spark resource usage metrics
Timelion is a Kibana plug-in that allows you to combine data sources driven by a simple expression language to retrieve time series data, perform calculations, and visualize the results.
An IBM Spectrum Conductor Timelion sheet is included in the sample; it provides visualizations fed by the Spark resource usage metrics from your IBM Spectrum Conductor cluster. You can modify any of the visualizations, including the dashboard, to fit your business needs and then save them as Kibana dashboard panels to add to a Kibana dashboard.
To use the IBM Spectrum Conductor Timelion sheet:
The IBM Spectrum Conductor Timelion sheet consists of the following visualizations:
Give it a try!
Now that you learned about Kibana with IBM Spectrum Conductor and Kibana Timelion visualizations, with this final blog in the search, analyze, and visualize Spark application data with IBM Spectrum Conductor series, try out the visualizations included in the downloadable sample. This concludes this three-blog series.
If you have any questions or want to let us know what charts you may be interested in, post them in our forum, or join us on Slack!