This article introduces you to the indexing mechanism of Lucene, a popular full-text IR library written in the Java language. First, I'll demonstrate how to index your documents with Lucene, then I'll discuss how to improve the indexing performance. Finally, I'll analyze Lucene's index file structure. Keep in mind that Lucene is not a ready-to-use application, but rather an IR Library that lets you add searching and indexing functionality to your application.

Architecture overview

Figure 1 shows the indexing architecture of Lucene. Lucene uses different parsers for different types of documents. Take HTML documents, for example -- an HTML parser does some preprocessing, such as filtering the HTML tags and so on. The HTML parser outputs the text content, and then the Lucene Analyzer extracts tokens and related information, such as token frequency, from the text content. The Lucene Analyzer then writes the tokens and related information into the index files of Lucene.

Indexing your documents with Lucene

I'll show you step by step how to create an index for your documents with Lucene. Lucene can index any data that you can convert into textual format. For example, if you want to index HTML or PDF documents, first you should extract the textual information from the documents and then send the information to Lucene for indexing. The example in this article uses Lucene to index text files with a .txt extension.

1. Prepare the text files

Put some text files with a .txt extension into a directory -- for example, C:\\files_to_index on the Microsoft® Windows® platform.

2. Create the index

Listing 1 shows you how to index the text files you prepared in the first step.

package lucene.index;

import java.io.File;
import java.io.FileReader;
import java.io.Reader;
import java.util.Date;

import org.apache.lucene.analysis.Analyzer;
import org.apache.lucene.analysis.standard.StandardAnalyzer;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.apache.lucene.index.IndexWriter;

/**
 * This class demonstrates the process of creating an index with Lucene 
 * for text files in a directory.
 */
public class TextFileIndexer {
 public static void main(String[] args) throws Exception{
   //fileDir is the directory that contains the text files to be indexed
   File   fileDir  = new File("C:\\files_to_index ");

   //indexDir is the directory that hosts Lucene's index files
   File   indexDir = new File("C:\\luceneIndex");
   Analyzer luceneAnalyzer = new StandardAnalyzer();
   IndexWriter indexWriter = new IndexWriter(indexDir,luceneAnalyzer,true);
   File[] textFiles  = fileDir.listFiles();
   long startTime = new Date().getTime();

   //Add documents to the index
   for(int i = 0; i < textFiles.length; i++){
     if(textFiles[i].isFile() >> textFiles[i].getName().endsWith(".txt")){
       System.out.println("File " + textFiles[i].getCanonicalPath() 
              + " is being indexed");
       Reader textReader = new FileReader(textFiles[i]);
       Document document = new Document();
       document.add(Field.Text("content",textReader));
       document.add(Field.Text("path",textFiles[i].getPath()));
       indexWriter.addDocument(document);
     }
   }

   indexWriter.optimize();
   indexWriter.close();
   long endTime = new Date().getTime();

   System.out.println("It took " + (endTime - startTime) 
              + " milliseconds to create an index for the files in the directory "
              + fileDir.getPath());
  }
}
      

As Listing 1 demonstrates, you can index your text files easily with Lucene. Let's interpret the key statements in Listing 1, beginning with this one:

Analyzer luceneAnalyzer = new StandardAnalyzer();

This statement creates an instance of the StandardAnalyzer class, which is in charge of extracting tokens out of text to be indexed. StandardAnalyzer is just one implementation of the abstract class Analyzer; other implementations, such as SimpleAnalyzer, exist.

Now, take a look at this statement:

IndexWriter indexWriter = new IndexWriter(indexDir,luceneAnalyzer,true);

This statement creates an instance of the IndexWriter class, which is a key component in the indexing process. This class can create a new index or open an existing index and add documents to it. You might notice that its constructor accepts three parameters. The first parameter specifies the directory that stores the index files; the second parameter specifies the analyzer that will be used in the indexing process; the last parameter is a Boolean variable. If true, the class creates a new index; if false, it opens an existing index.

The following code snippet shows the process of adding one document to the index:

Document document = new Document();
document.add(Field.Text("content",textReader));
document.add(Field.Text("path",textFiles[i].getPath()));
indexWriter.addDocument(document);

The first line creates an instance of the Document class, which consists of a collection of fields. You can think of this class as a virtual document, such as an HTML page, a PDF file, or a text file. The fields in a document are often the attributes of a virtual document. Take an HTML page, for example: Its fields can include title, contents, URL, and so on. Different types of Field control which field you should index and which you should store with the index. For more information about Field, you can refer to Lucene's Javadoc. The second and third lines add two fields to the document. Each field contains a field name and the content. This example adds two fields named "content" and "path", which store the content and the path of the text file, respectively. The last line adds the prepared documents to the index.

After you add the documents to the index, don't forget to close the index by calling this method, which guarantees that the index changes are written to the disk:

indexWriter.close();

Using the code in Listing 1, you can add the text documents to the index successfully. Now, let's look at another operation on the index.

3. Remove documents from the index

The IndexReader class in Lucene is responsible for removing documents from the existing index, as demonstrated in Listing 2.

File   indexDir = new File("C:\\luceneIndex");
IndexReader ir = IndexReader.open(indexDir);
ir.delete(1);
ir.delete(new Term("path","C:\\file_to_index\lucene.txt"));
ir.close();

In Listing 2, the second line initializes an instance of the IndexReader class using the static method IndexReader.open(indexDir). The parameter of the method specifies the directory that stores the Lucene index files. IndexReader provides two methods to remove documents, as shown in the third and fourth lines. The third line deletes a document by document ID. Every document has a unique ID in the Lucene index, but the system generates the ID, so it's not convenient to use it to delete the document. The fourth line deletes the documents that contain the string "C:\\file_to_index\lucene.txt" in their field "path." You can easily specify a document to be deleted by its file path. Keep in mind that although the documents aren't searchable, the operations don't physically remove the documents from the index; they just mark the documents that have been deleted by creating a file with a .del extension.

You can easily recover the documents that have been marked as deleted, as shown in Listing 3. First, open the index, then call the ir.undeleteAll() method to complete the recovery process.

File   indexDir = new File("C:\\luceneIndex");
IndexReader ir = IndexReader.open(indexDir);
ir.undeleteAll();
ir.close();

You might want to know how to remove the documents from the index physically. Listing 4 shows the process.

File   indexDir = new File("C:\\luceneIndex");
Analyzer luceneAnalyzer = new StandardAnalyzer();
IndexWriter indexWriter = new IndexWriter(indexDir,luceneAnalyzer,false);
indexWriter.optimize();
indexWriter.close();

The third line in Listing 4 initializes an instance of the IndexWriter class and opens the existing index specified by the first parameter. The fourth line cleans up the index. IndexWriter physically deletes from the disk the documents that have been marked as deleted.

Lucene doesn't provide a method to update the document in the index directly, but if you want to do so, first remove the documents from the index and then add the updated version of this document to the index.

Back to top

Improving the indexing performance

You can make full use of your hardware resources to improve the indexing performance with Lucene. When you need to index a large number of documents, you'll notice that the bottleneck of the indexing is the process of writing the documents into the index files on the disk. To solve this problem, Lucene holds a buffer in the RAM. But how can you control the buffer that Lucene uses? Fortunately, Lucene's IndexWriter class exposes three parameters to let you adjust the size of the buffer and the frequency of the disk writes.

mergeFactor

This parameter determines how many documents you can store in the original segment index and how often you can merge together the segment indexes in the disk. For example, if the value of mergeFactor is 10, all the documents will write to a new segment index on the disk if the number of documents reaches 10 in the memory. Also, if the number of segment indexes on the disk reaches 10, they will merge together. The default value of this parameter is 10, which isn't suitable if you have a large number of documents. The large value of this parameter is better for batch index creation.

minMergeDocs

This parameter also affects the indexing performance. It determines the minimum number of documents that have to be buffered in the RAM before IndexWriter writes them to disk. The default value of this parameter is 10. If you have enough RAM, set the value of this parameter as large as possible to decrease the indexing time dramatically.

maxMergeDocs

This parameter determines the maximum number of documents per segment index. The default value is Integer.MAX_VALUE. Large values are better for batched indexing and speedier searches.

Listing 5 shows the usage of these parameters. Listing 5 is similar to Listing 1 but adds the statements to set the parameters described previously.

/**
 * This class demonstrates how to improve the indexing performance 
 * by adjusting the parameters provided by IndexWriter.
 */
public class AdvancedTextFileIndexer  {
  public static void main(String[] args) throws Exception{
    //fileDir is the directory that contains the text files to be indexed
    File   fileDir  = new File("C:\\files_to_index");

    //indexDir is the directory that hosts Lucene's index files
    File   indexDir = new File("C:\\luceneIndex");
    Analyzer luceneAnalyzer = new StandardAnalyzer();
    File[] textFiles  = fileDir.listFiles();
    long startTime = new Date().getTime();

    int mergeFactor = 10;
    int minMergeDocs = 10;
    int maxMergeDocs = Integer.MAX_VALUE;
    IndexWriter indexWriter = new IndexWriter(indexDir,luceneAnalyzer,true);        
    indexWriter.mergeFactor = mergeFactor;
    indexWriter.minMergeDocs = minMergeDocs;
    indexWriter.maxMergeDocs = maxMergeDocs;

    //Add documents to the index
    for(int i = 0; i < textFiles.length; i++){
      if(textFiles[i].isFile() >> textFiles[i].getName().endsWith(".txt")){
        Reader textReader = new FileReader(textFiles[i]);
        Document document = new Document();
        document.add(Field.Text("content",textReader));
        document.add(Field.Keyword("path",textFiles[i].getPath()));
        indexWriter.addDocument(document);
      }
    }

    indexWriter.optimize();
    indexWriter.close();
    long endTime = new Date().getTime();

    System.out.println("MergeFactor: " + indexWriter.mergeFactor);
    System.out.println("MinMergeDocs: " + indexWriter.minMergeDocs);
    System.out.println("MaxMergeDocs: " + indexWriter.maxMergeDocs);
    System.out.println("Document number: " + textFiles.length);
    System.out.println("Time consumed: " + (endTime - startTime) + " milliseconds");
  }
}

Notice that Lucene gives you enough flexibility to control the size of the buffer pool and the frequency of disk writes. Now, take a look at the key statements in this example. The following statements first create an instance of IndexWriter and then assign the defined values to the parameters of IndexWriter.

int mergeFactor = 10;
int minMergeDocs = 10;
int maxMergeDocs = Integer.MAX_VALUE;
IndexWriter indexWriter = new IndexWriter(indexDir,luceneAnalyzer,true);        
indexWriter.mergeFactor = mergeFactor;
indexWriter.minMergeDocs = minMergeDocs;
indexWriter.maxMergeDocs = maxMergeDocs;

Let's examine these parameters' influence on the indexing time. Notice the values of these parameters and the changes on the indexing time. I prepared 10,000 documents for this test; Table 1 shows the test results.

From Table 1, you can easily see the influence that three parameters have on the indexing time. In practice, you'll often change the value of mergeFactor and minMergeDocs to improve the indexing performance. As long as you have enough RAM, you can assign a big integer value to the mergeFactor and minMergeDocs parameters to decrease the indexing time dramatically.

Back to top

Lucene's index file structure analysis

Before analyzing Lucene's index file structure, you should understand the inverted index concept. An inverted index is an inside-out arrangement of documents in which terms take center stage. Each term points to a list of documents that contain it. On the contrary, in a forwarding index, documents take the center stage, and each document refers to a list of terms it contains. You can use an inverted index to easily find which documents contain certain terms. Lucene uses an inverted index as its index structure.

Logical view of index files

Lucene features segments, which contain some indexed documents. You can search segments independently. Now look at Lucene's logical view of index files in Figure 2. The number of segments is determined by the number of documents to be indexed and the maximum number of documents that one segment can contain.

Key index files in Lucene

The following describes the main index files in Lucene. Some might not include all of the columns, but it won't affect your understanding of the index file.

Segments file

A single file contains the active segments information for each index. This file lists the segments by name, and it contains the size of each segment. Table 2 describes the structure of this file.

Fields information file

As you know, documents in the index are composed of fields, and this file contains the fields information in the segment. Table 3 shows the structure of this file.

Text information file

This core index file stores all of the terms and related information in the index, sorted by term. Table 4 shows the structure of this file.

Frequency file

This file contains the list of documents that contain the terms, along with the term frequency in each document. If Lucene finds a term that matches the search word in the term information file, it will visit the list in the frequency file to find which documents contain the term. Table 5 shows a brief structure of this file. It does not contain all of the fields of this file, but it can help you understand its usage.

Position file

This file contains the list of positions at which the term occurs within each document. You can use this information to rank the search results. Table 6 shows the structure of this file.

I've introduced you to the main index files in Lucene, hopefully allowing you to understand the physical storage structure of Lucene.

Back to top

In conclusion

A number of large, well-known organizations are using Lucene. For example, Lucene provides searching capabilities for the Eclipse help system, MIT's OpenCourseWare, and so on. Upon reading this article, I hope you've gained an understanding of Lucene's indexing system and will find it easy to create an index using Lucene's API.

Resources

Learn

• Parsing, indexing, and searching XML with Digester and Lucene by Otis Gospodnetic (developerWorks, June 2003): Manipulate XML in Lucene and cut your developement time.
  
  
• IBM Search and Index APIs (SIAPI) for WebSphere Information Integrator OmniFind Edition by Srinivas Varma Chitiveli (developerWorks, January 2005): Build your own search solutions based on OmniFind technology, IBM's information retrieval library.
  
  
• Lucene's official Web site: Explore numerous study materials for Lucene, including Javadoc and Lucene's latest release.
  
  
• A lecture on Lucene, presented by Doug Cutting at the University of Pisa on November 24, 2004: Explore this brief introduction to Lucene.
  
  
• Modern Information Retrieval by Ricardo Baeza-Yates and Berthier Ribeiro-Neto: Read about changes in modern information retrieval and how to provide relevant information in this book about IR technology.
  
  
• developerWorks Web Architecture zone: Expand your site development skills with articles and tutorials that specialize in Web technologies.
  
  
• developerWorks technical events and webcasts: Stay current with jam-packed technical sessions that shorten your learning curve, and improve the quality and results of your most difficult software projects.
  
  

Get products and technologies

• Lucene: Download the latest version.
  
  
• Free downloads and learning resources: Improve your work with software downloads from developerWorks.
  
  

Discuss

• Lucene mailing list standards: Ask questions, share knowledge, and discuss issues.
  
  
• developerWorks discussion forums: Join and participate in the developerWorks community.
  
  
• developerWorks blogs:Get involved in the developerWorks community.
  
  

About the author

Deng Peng Zhou is a graduate student from Shanghai Jiaotong University. He works as an intern software engineer in IBM Shanghai Globalization Lab and is interested in Java technology and modern information retrieval.

Rate this article

Comments

Back to top

Table of contents

• Architecture overview
• Indexing your documents with Lucene
• Improving the indexing performance
• Lucene's index file structure analysis
• In conclusion
• Resources
• About the author
• Comments

Dig deeper into Web development on developerWorks

IBM SmartCloud trial. No charge.

Unleash the power of hybrid cloud computing today!

---

Public and private solutions in one trial.

Special offers