Preventing indexing and reindexing failure in Lifecycle Query Engine Jena

Problem

In Lifecycle Query Engine, the indexing or reindexing of data provider with a big data set and TDB in memory-mapped mode failing due to the following exception:

com.hp.hpl.jena.tdb.base.file.FileException: BlockMgrMapped.segmentAllocate.

Solution

To fix the issue, you can configure your operating system and Java VM to make optimal use of MMapDirectory.

The default settings in Linux distributions and Solaris/Windows are fine. But due to some modifications, you might limit the maximum amount of virtual address space that can be allocated by applications. Check that ulimit -v and ulimit -m both report unlimited, otherwise it might happen that MMapDirectory reports mmap failed while opening your index.

If this error still happens on system and you get lots of large indexes, each of them with many segments, you might need to tune your kernel parameters in /etc/sysctl.conf. The default value of vm.max_map_count is 65530, you might need to raise it.

For Windows and Solaris systems, similar settings are available. But it is up to you to find out how to use them. Edit the /etc/sysctl.conf file on your Linux host, and add the following if it is not already present: vm.max_map_count=262144

Reload the file by using the following command: sudo sysctl -p. For example, 71 M resource on Linux server with 800 GB RAM and 150 GB of Heap value of vm.max_map_count is set to 198069493. So, consider the value for your server and try.