Thresholds are used by IBM® Content
Classification applications
to determine when to initiate an automatic action such as auto-classifying
a document. The Threshold Calculator shows the empirical effects of
changing the threshold of a selected category. You can experiment
with different thresholds to see the impact on measures such as precision,
recall, and cost ratio.
Threshold Calculator workflow
Important: To use the Threshold Calculator, you must have a
content set and valid analysis data.
Access
the Threshold Calculator from the Tools menu
and select a category. Adjust the value of the threshold in the Threshold
% field to see the effect on various measures. You can
also enter a value for cost ratio, precision, and recall to see the
corresponding threshold value.
Click Save Thresholds to
apply the current threshold value to the category in the knowledge
base or in a Threshold Per Category CSV file. You can choose whether
to save global thresholds to other categories in the knowledge base
or CSV file.
Setting global options
Click
Edit
Global Thresholds to set parameters that affect the data
displayed in the Threshold Calculator and which thresholds are saved:
- Maximum number of matches per item
- Specify the number of top-scoring category matches that are returned
for each content item. Error rates in the Threshold Calculator are
calculated based on this value. If your content must be classified
into one category only, set the Maximum number of matches
per item to 1. If your content can be classified into
more than one relevant category, increase this number accordingly.
- Default threshold for all other categories
- Automation and error rates at various threshold values for the
selected category are calculated based on constant threshold values
for all other categories. The thresholds of all other categories can
be set by a Threshold Per Category file (if one is selected) or a
global value that is specified in the Default threshold
for all other categories box. If you do not have a Threshold
Per Category file, or if the thresholds of some categories in the
Threshold Per Category file are not defined, specify a global threshold
value. The default global threshold value is 75%.
- Threshold Per Category File
- If you have a Threshold Per Category (CSV) file, select the file
in the Threshold Per Category File box. You
can create a Threshold Per Category file by using the Threshold Options
window in the Knowledge Base Editor ().
Data displayed in the Threshold Calculator
The
Threshold Calculator shows the Content Item Scoring category graph
(see Category graph: Content Item Scoring).
Each point on the graph represents a single item in the content set.
The Match Value indicates how items scored
in the current category. The blue and maroon colors are indicators
of belonging to, or not belonging to the category, respectively.
The
threshold level is represented by a red horizontal line that is raised
and lowered on the graph as you change the value in the Threshold
% box.
The following measures are displayed and
change as you vary the threshold:
- Cost ratio
- The ratio of $ saved by automating correctly / $ lost
by automating incorrectly.
- For example, if you save $10 for each correct automated action,
and lose $100 for each incorrect automated action, you should choose
a cost ratio of 1/10.
- For companies that can quantify cost savings from automation,
as well as the penalty for a wrong category, this measure can help
them choose the appropriate level of automation. It can also help
determine how many people will be required to perform a specific task
at a given level of accuracy.
- Precision
- The fraction of items that IBM Content
Classification identifies
as relevant to a category that are actually relevant to the category
(the other items are false positives).
- If you are required to have a specific level of accuracy (that
is, the percentage of items identified in the category that actually
belong in the category) you can set the precision to determine the
corresponding threshold.
- Recall
- The fraction of items that are actually relevant to a category
that are recognized as such by the Content Classification (the other items are false
negatives).
- If you have limited resources, you might need to set your threshold
based on the required number of items that must be automated correctly
for the category. For example, you might be required to place 90%
of the items in a particular category, regardless of precision.
- False Positive
- This refers to the maroon points above the threshold. It is the
percentage of items that will be incorrectly classified in the category
if you automate using the current threshold.
- False Negative
- This refers to the blue points below the current threshold. It
is the percentage of items that will not be automated (by mistake)
for their correct category.
- Automation
- This is the percentage of the content items that belong to the
specified category and were automated, either correctly or incorrectly.
In other words, this automation rate shows the percentage of content
items that belong to the specified category with scores that exceeded
the threshold of one or more categories in the knowledge base, but
not necessarily the correct category.
- Error Rates
- Items incorrectly automated to this category
- Also known as the In Error Rate, this is the percentage
of all items in the content set that exceeded the threshold of the
specified category but belong to another category.
- Items incorrectly automated to a different category
- Also known as the Out Error Rate, this is the percentage
of content items that belong to specified category but exceeded the
threshold of another category.
Threshold Calculator scenarios
- Scenario 1
- You are searching for specific messages to be used as evidence
in a court case. You are willing to read 100 items in order to track
down just one relevant item. In this case, you want to maximize your recall at
the expense of precision. In other words, you are willing to
deal with many false positives to avoid even a single false
negative. Your cost ratio can be said to have a value of
99/1, and your threshold should be 1%.
- Scenario 2
- Your system automatically closes accounts and you need to be very
careful about automation. You want extremely high precision to
ensure that you do not accidentally close the account of a valued
customer. In this case, you are willing to sacrifice recall (or
automation) because you must be confident that any automated actions
are very likely to be correct. Your tolerance of false positives will
be very low, as you will prefer to receive many false negatives instead
of a single false positive, which would result in the automatic
closing of a customer account.