Knowledge base analysis

When you analyze your knowledge base using a testing set, the system generates test results that you can view using various reports, graphs, and tables. You can use this analysis data to evaluate and improve the performance of your knowledge base.

Some Classification Workbench reports and graphs provide you with a view of the knowledge base's overall performance, while others allow you to see how each individual category performed within the knowledge base.

Important: Classification Workbench reports and graphs indicate how your knowledge base will perform in systems using the latest version of IBM® Content Classification. Knowledge base performance might differ somewhat, depending on your version. Always check the knowledge base in your application before deploying it.

How to view reports and graphs
You can view reports and graphs in different formats.
Summary Reports and Graphs page
Several options are available for viewing reports on the Summary Reports and Graphs page.
Category Graphs and Tables page
Select the Category Graphs and Tables page to access graphs and tables for each selected category.
Working with graphs
Several options are available for working with graphs that are displayed in the Classification Workbench view.
Summary report: Cumulative Success
The Cumulative Success summary report provides an overall view of knowledge base performance, indicating how well the knowledge base suggests the correct categories.
Summary report: Recommended Thresholds
The Recommended Thresholds summary report displays two tables: Recall/Precision and False Positive/False Negative.
Summary report: Knowledge Base Data Sheet
The Knowledge Base Data Sheet summary report provides information that helps you evaluate and troubleshoot your knowledge base.
Summary report: Knowledge Base Summary
The Knowledge Base Summary report provides information about your knowledge base.
Summary report: Content Summary
The Content Summary report report provides information about your content set.
Summary report: Threshold Analysis
The Threshold Analysis report shows the automation accuracy achieved when using a system-wide threshold. This information provides you with precision/recall information for each category in a content set.
Summary graph: Cumulative Success
The Cumulative Success graph provides an overall view of knowledge base performance, indicating how well the knowledge base suggests the correct categories.
Summary graph: Cumulative Success By Date / Over Time
The Cumulative Success by date and Cumulative Success over time graphs show a knowledge base's cumulative success during the lifecycle of a chronological testing set (that is, a content set whose items are in the order in which they were received by the system).
Summary graph: Total Precision vs. Recall
The Total Precision vs. Recall graph displays the Precision vs. Recall for all categories in the knowledge base, based on analysis results. The graph represents a weighted average of the precision for each recall value (that is, categories with more texts receive more weight).
Summary graphs: Total Automation vs. Error
The Total Automation vs. Error graphs show the relationship between the automation rate and error rates for all categories in the knowledge base. Use these graphs to evaluate the performance of your knowledge base and find the right balance between automation rate and error rate.
Category graph: Cumulative Success
The Cumulative Success graph, viewed by category, provides a partial view of the Cumulative Success summary graph.
Category graphs: Precision vs. Recall
The Precision vs. Recall graph displays the same information as the Total Precision vs. Recall graph, but for a single category. The Precision vs. Recall graph provides an immediate, visual sense of the selected category's performance.
Category graph: Precision-Recall vs. Threshold
You can use the Precision-Recall vs. Threshold graph to determine category thresholds, or examine the performance of the selected category.
Category graph: Content Item Scoring
The Content Item Scoring graph shows how each item scored that is, the match value) for the selected category. Note that the match value (a number from 0 to 1) corresponds to the score (a percentage from 0 to 100).
Category graph: Match Value Histogram
The Match Value Histogram provides you with another perspective on category performance within the knowledge base. This graph is related to the Content Item Scoring graph, and provides an immediate, visual sense of how well the category performed.
Category table: Stealing/Stolen
In some cases, poor knowledge base performance occurs when categories "steal" content items from each other (that is, when inappropriate categories receive high scores for items). Ideally, IBM Content Classification should assign high scores to the most applicable categories. When this is not the case, it is important to examine which categories are involved.
Category graph: Cumulative Scoring
The Cumulative Scoring graph shows all items for which the selected category was one of the top-ten scoring categories. This is a detailed representation of the data presented in the Cumulative Success category graph.
Category graph: Automation and Error Rates vs. Threshold
The Automation and Error Rates vs. Threshold graph shows automation rates and error rates at various threshold values for a selected category. The information in this graph can help you fine tune and optimize automation for important categories in your knowledge base.
Comparing reports
When you make changes to a knowledge base, you can compare reports to assess the impact of these changes on knowledge base performance.
Viewing category scores
You can add a field to your content set that displays a specified category's score for each content item.
Viewing significant concepts
Concepts are basic units of linguistic or quantitative information that are derived from the textual content of content items. Classification Workbench identifies concepts that are statistically significant to a content item's relationship to a specified category. You can view significant concepts to understand how IBM Content Classification analyzes and classifies content items.
Analyzing free text
You can test IBM Content Classification's analysis capability and simulate your knowledge base's performance for a single text. Enter text and view a list of top-matching categories and their scores.
Threshold Calculator
Thresholds are used by IBM Content Classification applications to determine when to initiate an automatic action such as auto-classifying a document. The Threshold Calculator shows the empirical effects of changing the threshold of a selected category. You can experiment with different thresholds to see the impact on measures such as precision, recall, and cost ratio.