Sparsity

Before using feeders, you must understand the typical sparse distribution of multidimensional data and TM1® consolidation logic.

During consolidations, TM1 uses a sparse consolidation algorithm to skip over cells that contain zero or are empty. This algorithm speeds up consolidation calculations in cubes that are highly sparse. A sparse cube is a cube in which the number of populated cells as a percentage of total cells is low.

When consolidating data in cubes that have rules defined, TM1 turns off this sparse consolidation algorithm because one or more empty cells may be calculated by a rule. Skipping rules-calculated cells will cause consolidated totals to be incorrect. When the sparse consolidation algorithm is turned off, every cell is checked for a value during consolidation. This can slow down calculations in cubes that are very large and sparse.

Typically, multidimensional cubes contain many more cells with zeros than with values. For example, consider the Purchase cube, where each type of fish is purchased in just a few markets. A typical view of the Purchase cube looks like this:

Screen showing a multidimensional cube in the Cube Viewer.

Most values here are zeros, an indication that this cube is relatively sparse. Multidimensional cubes are almost always sparse.

Note: There is a semantic distinction between a zero and a value that is missing or non-applicable. Sparsity may apply to any of these possible states, with possibly different results. In TM1 though, values can only be real numbers, and the value zero is used to represent all of these states.

The impact of sparsity on calculations can be tremendous. Consider the consolidated value (8433) at the intersection of Total Markets and Total Fish Types, located in the lower right corner of the view above. If you add up every possible cell required to calculate this value, you must add values from 119 different cells. However, if you add only the cells with non-zero values, the number of components of the calculation drops to 46.

If you change the element of the Date dimension, you see even more sparsity, since not all markets are open on every date, and not all fish are bought on every date. For example, if you change the view and drill down to the single date Jun -16, you see that the number of components of the Total Markets/Total Fish Types consolidation drops to just 20!

On average, the more dimensions a cube has, the greater the degree of sparsity.