How to create a custom Kibana dashboard

13 min read

How to create a custom Kibana dashboard

As of May 23rd IBM Bluemix Container Service now provides a native Kubernetes operations experience while removing the burden of maintaining master nodes. Kubernetes itself is based on the Docker engine for managing software images and instantiating containers. Get the details.

In this blog, I will describe the default Kibana dashboard built in Bluemix that can be used to review logs from IBM Containers, how to modify that default dashboard, and how to create a custom dashboard to display your log data. As an example, I will use the Bluemix Go-router logs to create the custom dashboard. This information is based on Kibana 3.0.

Introduction

Kibana is a tool that creates nice graphs based on the logs that are sent to Elasticsearch by Logstash. Kibana is a purely JavaScript based, so it runs a JSON document as a client side application that is connected to an interface by Elasticsearch. Kibana makes use of the excellent, faceted queries that are provided by Elasticsearch to create tables, histograms, pie charts, and even maps with geo points. Using the interface, you can create lots of different charts to present the data that comes out of Elasticsearch.

The Bluemix logging dashboard comes with a built-in, default Kibana dashboard. You can modify the default dashboard, create your own dashboard, or load additional dashboard files for custom views, which you can import from your computer or from Gist storage on GitHub. The dashboard can be saved in Elasticsearch, saved as the home dashboard, or exported to a local directory as a JSON file.

Kibana supports dashboard templates that are in JSON format or scripted dashboards that are in JavaScript file format. For more information, see Templates and Scripts. There are two mechanisms for creating custom dashboards. The first is to create a custom dashboard from the user interface. The second is to create a custom dashboard from a file. You can create custom dashboards from a template JSON document that is based on a specific schema or create your own template as a JavaScript file.

There are sample dashboards available on the Logstash server that you can access. Here is a list of the sample dashboard files:

dashboard files

To access a sample dashboard, create a URL with the dashboard name in one of the following formats:

  • For a JSON template, add #/dashboard/file/ and the file name of the dashboard to the end of the URL. Here is example to load a blank dashboard template.https://logmet.ng.bluemix.net/kibana/#/dashboard/file/blank.json

  • For a JavaScript file, add #/dashboard/script/ and the name of the JavaScript file to the end of the URL. Example:https://logmet.ng.bluemix.net/kibana/#/dashboard/script/logstash.js

Each dashboard file must be created based on the Kibana dashboard template schema format.

The template schema consists of the following items:

  • Services: Services can be queries or filters, which can reused between different panels. Filters can be created by providing a search query or by clicking elements within the visualization dashboard. Filters can be saved in your dashboard, and then you can disable or enable them. Queries provide a way to search on a specific subset of the transactions. Queries use the Lucene query syntax. You can even create a panel with multiple queries to compare the results of two separate queries. For more information, see the Queries and Filters information in the guide by Elasticsearch.

Services
  • Row: The dashboard is made up of invisible horizontal sections in which panels are added. A row can be viewed in a collapsed or expanded format in the dashboard.

Row
  • Panel: Kibana comes with a number of different section types, which are called panels, that can be added to rows in your dashboard. Panels can be created as tables, histograms, terms, text, maps, and columns.

  • Index: Configure index-specific properties, like the default index and the index pattern. The default log index format is “logstash-<Bluemix space guid>-YYYY.MM.DD”

For more information about the schema and its options, see Dashboard Schema.

Before you start the next section of this document, you might watch these videos:

Bluemix Kibana logging dashboard

Logstash Forwarder assumptions

You must have the Logstash Forwarder configured on each application server to send log files to Logstash server. You must add and configure additional Logstash Forwarder configuration files to send the logs to the Logstash server. The additional configuration can also be added to the /etc/logstash-forwarder.conf file directly. You must define paths for your log files and fields, such as “type”, in your Logstash Forwarder configuration files.

“files”: [
{

“paths”: [
“Your log file with its path”,
 

“Your log file with its path”
],
“fields”: { “type”: “Your log type ” }

}

You must also create your patterns file and filter .conf file based on your message format and your log type.

Start the default dashboard

You must log in by entering the following information and selecting Kibana:

  • User: Bluemix user ID

  • Password: Bluemix password

  • Space: Bluemix namespace

  • Organization: Bluemix organization

Kibana

When you log in, the default dashboard is displayed.

dashboard

I have highlighted with red rectangular boxes the areas worth exploring, which are:

  1. Toolbar panel: Save, open, or share your dashboards. By default, the toolbar has a time picker with predefined relative time options and auto-refresh options.

  2. Query panel: Create queries for the indexed database.

  3. Filter panel: Apply filters from queries to narrow a search and trim the results. By default, time is filtered to the last 24 hours. You can change the time by selecting a new value from time picker in the toolbar.

  4. EVENTS BY TIME: This data histogram panel was added for you inside a Histogram row. By default, the data histogram shows the count for all logs within the time set for the interval (y-axis) over time (x-axis). Click the bars, or click-and-drag, to narrow the time filter.

  5. ALL EVENTS: This table panel was added for you inside an Events row. By default, all logs that match the time filter are displayed.

Dashboard settings

You can click the Settings icon in the toolbar to edit and customize the dashboard settings.

Dashboard settings
  • General tab: You can set the name of the dashboard title and its style.

    General tab

 

    • Title: Enter a name for the title of the dashboard.

    • Style: Define the dashboard style as dark or light themed.

    • Editable: Enable or disable dashboard settings.

    • Hints: Shows hints, such as Add panel, in the empty spaces of the dashboard.

  • Rows tab: You can add, reorder, or remove rows from the dashboard. The default dashboard has 2 rows: Histogram and Events.

    Rows tab

Row settings

Row settings

You can click the Configure Row icon to see the row Settings. The row settings have three tabs:

  1. General tab: Configure the row title and height.

    General tab
  2. Panels tab: Move, hide, or delete the panels that are in the row.

    Panels tab
  3. Add Panel tab: Select a panel type and add that panel to the row.

    Add Panel tab

ALL EVENTS table panel

If you click a log event in the All Events table, you can see the detailed information for each of the fields and their values. A field is a part of the log schema that acts as a label for a single piece of information in the log file.

ALL EVENTS table panel

I have highlighted with red rectangular boxes the areas worth exploring, which are:

  1. Fields list panel: Click the right arrow under the panel name to see the Fields list panel. You can search for specific field names by entering a filter string in the Type to filter box.

    Fields list panel

    If you select a field, that field is added in the table as a separate column. If you clear a  field, then the column for that field is removed from the table.

  2. Fields column list: Shows the list of the fields that are selected in the Fields list panel. You can move a column by clicking the Right arrow or Left arrow by the column name. You can sort a column by clicking the column title and, optionally, the Down arrow or Up arrow to change the sort order.

  3. Fields details view: Shows the values for each field in a Table, JSON or Raw format. You can also select one of the following actions to create a query for a field.

    Fields details view

     

    • If you click the 

      icon

       icon, then a filter is added to match the field value. The dashboard shows all of the log information that matches that filter.

    • If you click the 

      icon

       icon, then a filter is added filter to not match the field value. The  dashboard shows all of the log information that matches that filter.

    • If you click the 

      icon

       icon, then it toggles that filed been shown as a table column.

EVENTS BY TIME histogram panel

The default EVENT BY TIME histogram shows the number of the logs over the interval time. The default interval period is set to 10minutes. To change the interval, click View. A small panel appears just below the views with more options, as shown in the section labeled #1. Change the interval value from 10m to any other value from the interval menu.

EVENTS BY TIME histogram panel

I have highlighted with red rectangular boxes the areas worth exploring, which are:

  1. View panel: You can expand the panel to see the options by clicking the 

     button. The histogram view can be changed by selecting or clearing the different options in the view panel.

  2. Zoom in: If you hover the mouse over the graph, you can zoom in to that part of the data you selected. You return to the full graph clicking the 

    icon

     button.

Filters

Every time you narrow the selection of data, either by using the histogram or by selecting a subset of data for other panels, a filter is created. You can see all of the filters by clicking the Filter section.

Filters

To create, edit, enable, disable, or remove a filter:

  • Add a filter by clicking the 

    icon

     icon.

  • Edit a filter by clicking the 

    icon

     icon.

  • Enable or disable a filter by selecting or clearing the 

    icon

     icon.

  • Remove a filter by clicking the 

    icon

     icon.

Note: Time filters cannot be edited.

Query

The Query section provides a way to query a specific subset of the data from a selected time frame. The query syntax is based on the Lucene query syntax, which allows Boolean operators, wildcards, and field filtering. See the example queries below.

Field based query:

    • job_index: router_1

    • type: router_access_log

  • statusCode: 500

Regexp query:

    • job_index: router*

  • hostName: *ng.bluemix.net

Range query:

    • To search for logs with a response_time from 120 ms to 122 ms

      • response_time: [120 TO 122]

To search for logs with a response_time greater or equal than 120ms:

      • response_time: [120 TO *]

To search for logs with a response_time greater than 120ms:

    • response_time: {120 TO *}

Boolean query:

Boolean operators (AND, OR, NOT) allow the combining of multiple sub-queries through logic operators.

Note: Operators such as AND, OR and NOT must be CAPITALIZED. For more information about the Boolean operators, see Lucene query syntax.

Multiple queries

In some cases, you might want to compare the results of two separate queries. Kibana can handle multiple queries by joining them with OR logic. Then, they are treated as separate methods for influencing visualizations. Click the + icon next to the query input, as shown with red arrow, to add another query.

Click the + icon

To remove a query, click the x icon that appears when you hover over the query input field.

click the x icon

Colors and legends

Kibana automatically determines a color to use for your queries, but you can set your color explicitly. Click the colored dot associated with the query to open the query settings. From there, you can change the color of the query or set a new value for the legend.

Colors and legends

How to customize the default dashboard

Based on previous sections, you have a better understanding of the default Kibana dashboard in Bluemix. In this section, we show how you make customizations that are based on the field information from the Bluemix Go-router access log. Start by logging in to the default Kibana dashboard.

Kibana dashboard

Configuring a custom dashboard

You might want to name your new dashboard and add a row to add your custom panels.

  1. Click the Settings icon in the toolbar. The dashboard settings open.

    Settings
  2. In the General tab, add your dashboard title. For example “bluemix_router_access_log”.

    General tab
  3. In the Row tab, click Create Row and name the row, for example, “Router Access Log Info”.

    Row tab

    You can see the new row has been added.

  4. Rearrange how the rows display in the dashboard by clicking the Up arrow or Down arrow icons.

    clicking the Up arrow or Down arrow icons
  5. Click Save.

    Click Save
  6. Verify that there is a new empty row added to the dashboard

    Verify that there is a new empty row added to the dashboard

Create a new filter from existing log fields

  1. Go to ALL EVENTS table and click a row to expand it. Find the type field.

    Find the type field
  2. Pick a field to create a filter from and then click the 

    icon

     icon. A filter is added to match the value of this field, and then the dashboard shows all of the log information that is based on that filter. For example, you selected the type field with the value router_access_log, and so the filter in the following image is created. The dashboard is also updated based on type=“router_access_log”.

    Pick a field to create a filter

Field-wise analysis by using the ALL EVENTS table panel

Tables allow you to create comparative studies of fields.

  1. Under the panel name, click the Right arrow icon to see the Fields list panel. The Fields list panel shows the all of the fields that are available.

    Right arrow
  2. Select which fields to display in the list. The goal is to analyze the router_access_log with some of its fields, such as response_timejob_indexstatus codesrequest URI, and methods.

    Select which fields to display in the list
  3. You can add fields as table columns. If you clear a field’s check box, then the column for that field is removed from table. If you select a field’s check box, then that field is added to the table. I removed all of the default columns and added job_indexhostnamemethodsstatusCode, and message.

    You can add fields as table columns
  4. To order the columns, click the Left arrow or Right arrow next to a column’s title. Clicking any row element shows the complete data with all of the field information.

  5. Click the column title to sort the table by a column. By default, the table is sorted by timestamp.

  6. You can analyze the values for each field by clicking the field name in the field list panel. For example, by clicking the statusCode field, its values are displayed.

    click Terms and select Pie
  7. Choose how to display terms that are used in the data. You can show the field data as a bar graph, pie chart, or table. For example, if you click Terms and select Pie, then you might see a pie chart like this one below:

    click Terms and select Pie
  8. If you want to keep the field analysis in your dashboard, click the 

    icon

     icon and drag the panel to the “Router Access Log Info” row that you created in previous section. Then, you can close the temporary terms panel.

    keep the field analysis
  9. Open the terms panel that you dragged to the dashboard. In the General tab, you can adjust the size of the panel by changing the panel Span from 12 to 4, for example. You can also clear the Missing or Othercheck boxes to hide empty data. Click Save.

    Click Save
    Click Save
  10. Repeat steps 7-10 to analyze and create terms for the other fields. For example, if we add the analysis for the method and job_index fields, then the updated dashboard might look like this image.

    Repeat steps 7-10
  11. If you click any color in the pie chart, a new filter is created based on that selection and the dashboard shows the log information that meets the filter criteria.

Analysis by using the EVENTS BY TIME histogram panel

The EVENTS BY TIME histogram shows the number of messages that were created over an interval time for type=“router_access_log”. The interval period is set to 10 minutes.  Customize the default histogram by completing the following steps.

  1. Create a query for job_index with a value of router_0. Enter job_index:router_0 in the query panel. The dashboard updates to show information for the router_0.

    Create a query
  2. Create another query for job_index with a value of router_1. Click the + icon next to the query, and another query panel is displayed. Enter job_index:reouter_1 in the query panel. The dashboard updates to show information for both router_0 and router_1.

    Create another query
  3.  Click 

    icon

     to display the histogram options and add some analysis.

  4. Clear the Bars and Stack options and select Lines. You can also change the interval period from 10m to 1m. Both router_0 and router_1 are distributing the same amount of data.

    lear the Bars and Stack
  5. Zoom in to the data by hovering over the graph. Then, a new timestamp filter is created and applied to the dashboard.

    Zoom in to the data
  6. Zoom out to see the full graph by clicking 

    icon

     or clear the new timestamp filter. When you zoom out, the new timestamp filter is deleted automatically.

  7. You can change the histogram data value based on other field values, such as a max, min or mean “response_time” value. Go to histogram’s Settings panel, and change the Chart value to max and the Value field to response_time.

    change the histogram data value
  8. You can turn off either of the 2 queries that we created in step #2 from any dashboard panel. Open the Settings panel, click selected from Queries list, and click one of the selected queries.

    turn off either of the 2 queries

Save the custom dashboard

The structure of a customized dashboard can be saved in Elasticsearch, saved as the home dashboard, or exported to a local directory.

  1. Click the Save icon to save the dashboard in Elasticsearch and use it in dashboard later.

  2. Save as Home replaces the default dashboard with the customized dashboard.

  3. Reset Home returns the home dashboard to its previous dashboard.

  4. Export schema exports the dashboard as a JSON document and saves it to your local directory.

Export schema

 Note: When you save your dashboard in Elasticsearch, Elasticsearch keeps it for a limited amount of time. You might also export the dashboard to save it locally as a method for restoring it, if necessary.

Load a custom dashboard

You can load additional dashboard files for custom views, either from Elasticsearch, local files, or Gist storage on GitHub. To load a custom dashboard from Elasticsearch:

  1. Click the Load icon, then select a dashboard name. The default dashboard that shown is the default dashboard.

To load a custom dashboard from a local file:

  1. Click the Load icon, then hover over Advanced.

  2. Click Choose File and select your dashboard JSON file.

To load a custom dashboard from Gist storage on GitHub:

  1. Click the Load icon, then hover over Advanced.

  2. In Gist number or URL field, enter the URL of your dashboard Gist.

  3. To open the custom dashboard, click Get gist.

  4. To load the dashboard, click the dashboard name below the Gist ID.

load the dashboard

Adding security and disabling access to dashboard configurations

You can disable dashboard settings and prevent other users from changing the dashboard configuration.

Adding security and disabling access to dashboard configurations
    • To remove or disable the save, load, share, query, filter, or timepicker options:

      1. Click the Settings icon.

      2. In the Controls tab, clear the option you want to disable.

        Controls tab
    • To remove the Settings icon:

      1. Before you disable the Settings icon, keep in mind that you cannot enable the Settings icon again by using the user interface in the future. You must export a copy of the dashboard with the enabled Settings icon to restore if you need to update the Settings again. 

      2. Click the Settings icon.

      3. In the General tab, clear the Editable check box, and then click Save. The Settings icon has been removed from the toolbar.

      General tab
      Save

Sign up for Bluemix. It’s free!

References

Be the first to hear about news, product updates, and innovation from IBM Cloud