Improving the Lifecycle Query Engine performance by compacting indexed data

Edit online

Each time you index the data from your lifecycle management applications, the amount of space required to store that data increases. Over time, as the index files get larger and the amount of available hard disk space decreases, you might experience server performance issues. You can take steps to prevent this situation by monitoring the hard disk usage and, when required, compacting the indexed data.

Because disk space is not reclaimed when you delete data from the index, you must consider running a compaction whenever you delete or reindex the data.

If you deployed Lifecycle Query Engine (LQE) across several nodes, you can select specific nodes for data compaction.

When you start to compact indexed data, the indexing service is turned off. However, you can run queries while the data is being compacted. When the compacting process ends, the older index is replaced with the new compacted index.

During compaction, the indexed data is exported to N-Quads format, which is a line-based, plain text format for encoding a resource description framework (RDF) data set. For more information about the N-Quads format, see RDF N-Quads on the W3C website.

You can modify the compaction heap size that can help curtail these JVM heap space issues. Compactions are run as separate JRE processes (from LQE) and require their own Java heap size to be set. To change the Java heap size in the LQE application, go to Administration > Advanced Properties > Show Internal > Edit . In the Compaction section, change the Maximum Heap (MB) size from 4096 to 8192 MB. Save your changes and retry the compaction.

Before you begin

Ensure that the disk space is at least two times the current size of the indexes under <JTSInstallDir>\server\conf\lqe.

Procedure

  1. On the LQE Administration page, under Configuration, click Compaction.
  2. On the Compaction page, click Compact Now.
    Screen capture of the Compaction page. The Compact Now dialog box is open and the node is selected.
  3. Select the node that you want to compact, and click Compact Now.

What to do next

A compaction task is enabled to run weekly. You can modify the schedule, or configure the task to run on specific nodes. Click the Edit Schedule link, select the node, the time, and the days of the week. Select Yes to enable the schedule, then save your configuration.
Screen capture of the Compaction scheduling page.

Configuring alternative location for compaction

Edit online

About this task

LQE compaction requires available disk space equal to the total size of all indexes on disk for the temporary files that are used during the compaction process. By using an optional setting, LQE administrators can specify an alternative location for the temporary files that are used during the compaction process in LQE. Thus, the requirement of available disk space is imposed upon the alternative location instead of imposed upon the index hosting location.

When you set lqe.compactDataDirectory=/some/location in the lqe.properties file, the compaction process uses the alternative location for the temporary files. The compaction uses less disk space in the location where the indexes are hosted but might take more time to complete.

The indexes are located in the server-install/conf/lqe directory. The LQE compaction normally requires available disk space in the server-install/conf/lqe location.

When lqe.compactDataDirectory=<some_location> is set in the lqe.properties file, the available disk space is required in the alternative location instead.

Procedure

  1. As an LQE administrator, open the lqe.properties file.
  2. Add the lqe.compactDataDirectory=<some_location> to the lqe.properties file.
  3. Save the lqe.properties file.
  4. Restart the LQE application.

Recovering a failed compaction

Edit online

About this task

The tasks that are performed during the automated compaction process, for example, the indexTdb compaction process include:
  1. Exporting conf/lqe/indexTdb to tempDataDirectory/dump.nquads.gz
  2. Forking a JVM process to create a new TDB at tempDataDirectory/default_compacted_<ts>
    Note: <ts> indicates the timestamp.
  3. Renaming conf/server/indexTdb to lqe/conf/default_old_<ts>
  4. Deleting conf/lqe/default_old_<ts>
  5. Copying and renaming tempDataDirectory/default_compacted_<ts> to conf/lqe/indexTdb
  6. Deleting tempDataDirectory/default_compacted_<ts>
  7. Deleting tempDataDirectory/dump.nquads.gz

If the automated compaction fails for an index, then the LQE administrator must complete the following recovery steps:

Procedure

  1. Check the instructions in the lqe.log file.
  2. Shut down LQE.
  3. Open server/conf/lqe.
  4. If a default_old_<ts> directory is present, then proceed to step 5, otherwise skip to step 7.
  5. Delete the indexTdb directory.
  6. Rename the default_old_<ts> directory to indexTdb.
  7. If any other <partition_name>_old_<ts> directories are present, then repeat steps 8 and 9 for each directory.
  8. Delete the <partition_name>Tdb directory that matches the partition name of the <partition_name>_old_<ts> directory.
  9. Rename the <partition_name>_old_<ts> directory to <partition_name>Tdb.
  10. Delete the following files or directories, if they exist:
    • tempDataDirectory/<partition_name>_compacted_<ts>
    • tempDataDirectory/dump.nquads.gz
  11. Restart LQE.