Memory limit

Calculating the correlations between categorical fields takes a lot of memory. There is a memory limit to avoid that the mining run locks up.

If the memory limit is exceeded, the calculation of as many correlations as required to meet this limit is stopped. This might result in slightly worse results, however, the mining process completes successfully.

If the memory limit is too low, you are notified. Repeating the mining run with a higher limit ensures the optimum result.

You can change the memory limit by using the power option '-buf'.

Another factor is the number of different values that are tracked for each field. By default, 100 different values are distinguished and taken into account for correlation calculation. This usually ensures good results. However, you can change this value by using the power option -IDM_MAX_DISCR_COUNT.

If you modify the memory limit, take the following considerations into account:

If you raise the value for the memory limit, the results might improve slightly. However, this can have a dramatic impact on the memory consumption and performance, and this might lead to discarding of correlations.
If you lower the value for the memory limit, the results of the correlation calculation can become unreliable.

Feedback | Information roadmap