Lucene: Add multiple documents to the index

IndexWriter provides 'addDocuments' method to add multiple documents to the index.

 

Signature

public long addDocuments(Iterable<? extends Iterable<? extends IndexableField>> docs) throws IOException

 

Example

indexWriter.addDocuments(documents);

 

DocumentUtil.java

package com.sample.app.util;

import java.util.Arrays;
import java.util.List;

import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.apache.lucene.document.TextField;

public class DocumentUtil {

	private static Document getDocument(String id, String title, String description, String blog) {
		Document doc = new Document();
		doc.add(new TextField("id", id, Field.Store.YES));
		doc.add(new TextField("title", title, Field.Store.YES));
		doc.add(new TextField("description", description, Field.Store.NO));
		doc.add(new TextField("blog", blog, Field.Store.YES));
		
		return doc;

	}

	public static List<Document> getDocuments() {
		Document doc1 = getDocument("1", "JavaWorld",
				"The original independent resource for Java developers, architects, and managers.", " javaworld.com");
		Document doc2 = getDocument("2", "Oracle Blogs | The Java Source",
				" Java powers more than 4.5 billion devices including 800 million computers and 1.5 billion cell phones. If you love Java, this is the blog you must follow.",
				"blogs.oracle.com/java");
		Document doc3 = getDocument("3", "A Java geek",
				"Nicolas Fränkel's blog. IT architect focusing on Java, Java EE, and their surrounding ecosystems. He is a trainer, book writer, speaker & blogger.",
				"blog.frankel.ch");
		Document doc4 = getDocument("4", "Self Learning Java", "Learn Java fundamentals and other java libraries",
				"self-learning-java-tutorial.blogspot.com");

		return Arrays.asList(doc1, doc2, doc3, doc4);

	}
}

 

App.java

package com.sample.app;

import java.io.File;
import java.io.IOException;
import java.util.List;

import org.apache.lucene.analysis.Analyzer;
import org.apache.lucene.analysis.standard.StandardAnalyzer;
import org.apache.lucene.document.Document;
import org.apache.lucene.index.DirectoryReader;
import org.apache.lucene.index.IndexReader;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.search.Query;
import org.apache.lucene.search.ScoreDoc;
import org.apache.lucene.search.TopDocs;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.MMapDirectory;
import org.apache.lucene.store.NoLockFactory;
import org.apache.lucene.util.QueryBuilder;

import com.sample.app.util.DocumentUtil;

public class App {

	public static void main(String args[]) throws IOException {

		Analyzer analyzer = new StandardAnalyzer();
		IndexWriterConfig config = new IndexWriterConfig(analyzer);

		Directory directory = new MMapDirectory(new File("/Users/Shared/lucene").toPath(), NoLockFactory.INSTANCE);

		try (IndexWriter indexWriter = new IndexWriter(directory, config)) {

			List<Document> documents = DocumentUtil.getDocuments();

			indexWriter.addDocuments(documents);

		}

		QueryBuilder queryBuilder = new QueryBuilder(analyzer);
		Query query = queryBuilder.createMinShouldMatchQuery("description", "Professional java developers", 0.7f);
		int maxHitsPerPage = 10;

		try (IndexReader indexReader = DirectoryReader.open(directory)) {
			IndexSearcher indexSearcher = new IndexSearcher(indexReader);

			TopDocs docs = indexSearcher.search(query, maxHitsPerPage);
			ScoreDoc[] hits = docs.scoreDocs;
			System.out.println("Total Hits: " + docs.totalHits);
			System.out.println("Results: ");
			for (int i = 0; i < hits.length; i++) {
				Document document = indexSearcher.doc(hits[i].doc);
				System.out.println("Title: " + document.get("title"));
			}
		}

	}

}

 

Output

Total Hits: 1 hits
Results: 
Title: JavaWorld

 

 

Previous                                                    Next                                                    Home

Lucene: Add Document to the index

IndexWriter class provides ‘addDocument’ method, it is used to add document to an index.

 

Signature

public long addDocument(Iterable<? extends IndexableField> doc) throws IOException

 

Example

indexWriter.addDocument(document);

 

'addDocument' method periodically flushes pending documentscto the Directory, andcalso periodically triggers segment merges in the indexcaccording to the MergePolicy in use.

 

Find the below working application.

 

DocumentUtil.java

package com.sample.app.util;

import java.util.Arrays;
import java.util.List;

import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.apache.lucene.document.TextField;

public class DocumentUtil {

	private static Document getDocument(String id, String title, String description, String blog) {
		Document doc = new Document();
		doc.add(new TextField("id", id, Field.Store.YES));
		doc.add(new TextField("title", title, Field.Store.YES));
		doc.add(new TextField("description", description, Field.Store.NO));
		doc.add(new TextField("blog", blog, Field.Store.YES));
		
		return doc;

	}

	public static List<Document> getDocuments() {
		Document doc1 = getDocument("1", "JavaWorld",
				"The original independent resource for Java developers, architects, and managers.", " javaworld.com");
		Document doc2 = getDocument("2", "Oracle Blogs | The Java Source",
				" Java powers more than 4.5 billion devices including 800 million computers and 1.5 billion cell phones. If you love Java, this is the blog you must follow.",
				"blogs.oracle.com/java");
		Document doc3 = getDocument("3", "A Java geek",
				"Nicolas Fränkel's blog. IT architect focusing on Java, Java EE, and their surrounding ecosystems. He is a trainer, book writer, speaker & blogger.",
				"blog.frankel.ch");
		Document doc4 = getDocument("4", "Self Learning Java", "Learn Java fundamentals and other java libraries",
				"self-learning-java-tutorial.blogspot.com");

		return Arrays.asList(doc1, doc2, doc3, doc4);

	}
}

 

App.java

package com.sample.app;

import java.io.File;
import java.io.IOException;
import java.util.List;

import org.apache.lucene.analysis.Analyzer;
import org.apache.lucene.analysis.standard.StandardAnalyzer;
import org.apache.lucene.document.Document;
import org.apache.lucene.index.DirectoryReader;
import org.apache.lucene.index.IndexReader;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.search.Query;
import org.apache.lucene.search.ScoreDoc;
import org.apache.lucene.search.TopDocs;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.MMapDirectory;
import org.apache.lucene.store.NoLockFactory;
import org.apache.lucene.util.QueryBuilder;

import com.sample.app.util.DocumentUtil;

public class App {

	public static void main(String args[]) throws IOException {

		Analyzer analyzer = new StandardAnalyzer();
		IndexWriterConfig config = new IndexWriterConfig(analyzer);

		Directory directory = new MMapDirectory(new File("/Users/Shared/lucene").toPath(), NoLockFactory.INSTANCE);

		try (IndexWriter indexWriter = new IndexWriter(directory, config)) {

			List<Document> documents = DocumentUtil.getDocuments();

			for (Document document : documents) {
				indexWriter.addDocument(document);
			}

		}

		QueryBuilder queryBuilder = new QueryBuilder(analyzer);
		Query query = queryBuilder.createMinShouldMatchQuery("description", "Professional java developers", 0.7f);
		int maxHitsPerPage = 10;

		try (IndexReader indexReader = DirectoryReader.open(directory)) {
			IndexSearcher indexSearcher = new IndexSearcher(indexReader);

			TopDocs docs = indexSearcher.search(query, maxHitsPerPage);
			ScoreDoc[] hits = docs.scoreDocs;
			System.out.println("Total Hits: " + docs.totalHits);
			System.out.println("Results: ");
			for (int i = 0; i < hits.length; i++) {
				Document document = indexSearcher.doc(hits[i].doc);
				System.out.println("Title: " + document.get("title"));
			}
		}

	}

}

 

Output

Total Hits: 1 hits
Results: 
Title: JavaWorld

 

 

 

 

Previous                                                    Next                                                    Home

Lucene: Building Search Index

Before you perform any search operation, you should add all the documents that you want to search to the index. In coming chapters, you are going to learn different ways to index a document.

 

Lucene do not understand different data formats like tables in RDBMS, PDF files, word documents, spreadsheets etc., It is your responsibility to extract the text from those different sources and map the text to a Lucene document and add to index.

 

Adding the documents to index is not a straightforward process. First the field values of the documents are analyzed, unnecessary data like stopwords (a, an, the etc.,) are deleted, and tokens are generated, stemmed and convert to lower case to support case-insensitive search and finally the tokens are added to index.

 

Let me explain with an example.

 

Document 1

Java is a popular programming language. Lucene is a Java Library.

 

Document 2

Lucene Java Library used to perform search operations.

 

Document 3

Lucene in Action.

 

We have three documents document1, document2 and document 3. Whenever a request issued to add these documents to data store, Lucene creates unique identifier and assign to the documents.

 

For example,

Document	id
Document 1	1
Document 2	2
Document 3	3

 

As you see, above table identifier 1 is assigned to ‘Document 1’, identifier 2 is assigned to ‘Document 2’ and identifier 3 is assigned to ‘Document 3’.

 

Now, lucence tokenizes the words in each document and map the tokens to document ids.

 

Lucene Inverted Index looks like below.

 

Term	Available in Documents
Java	1, 2
is	1
a	1
popular	1
programming	1
language	1
Lucene	1, 2, 3
Library	2
used	2
to	2
perform	2
search	2
operations	2
in	3
Action	3

 

Lucene use this inverted index to serve search queries. For example, if user asks for the term ‘programming’ Lucene can quickly checks that this word is in document 1. If user asks for the term ‘Lucene’, then Lucene can serve the document 1, 2 and 3.

 

This example looks simple, but Lucene has some more capabilities like removing the stop words (words like a, an, the, is, in which has lower importance in search), support case insensitive search by converting the tokens to lowercase while indexing etc.,

 

 

Previous                                                    Next                                                    Home

List: Replace element at specific index

‘List’ interface provide ‘set’ method, it is used to replace the element at the specified position in this list with the specified element.

Signature

E set(int index, E element);

Example

List<Integer> primes = Arrays.asList(2, 3, 5, 7, 11, 13);

primes.set(1, 17);

App.java

package com.sample.app;

import java.util.Arrays;
import java.util.List;

public class App {

 public static void main(String args[]) {
  List<Integer> primes = Arrays.asList(2, 3, 5, 7, 11, 13);

  System.out.println("primes : " + primes);

  System.out.println("Setting element at index 1 to 17");

  primes.set(1, 17);
  
  System.out.println("primes : " + primes);

 }
}

Output

primes : [2, 3, 5, 7, 11, 13]

Setting element at index 1 to 17

primes : [2, 17, 5, 7, 11, 13]

You may like

Miscellaneous

Interview Questions

Programming Questions

How to initialize long in Java?

How to Initialize Multiple Variables to same value in Java?

Extract digits from a string or text

Remove first character of a string

TableSaw: Table: Get column by index

'table.column(indexNumber)' return the column by index.

Example

Column<?> column1 = table.column(0);

App.java

package com.sample.app;

import tech.tablesaw.api.IntColumn;
import tech.tablesaw.api.StringColumn;
import tech.tablesaw.api.Table;
import tech.tablesaw.columns.Column;

public class App {

      public static void main(String args[]) {
            int[] empIds = { 1, 2, 3, 4, 5, 6, 7, 8, 9 };
            String[] firstNames = { "Hari", "Ram", "Sowmya", "Chamu", "Harini", "Lahari", "Rama", "Lakshman", "Sandya" };
            String[] lastNames = { "Krishna", "Gurram", "Maj", "Dev", "Gurram", "Ram", "Sen", "Grandi", "Neelam" };

            Table table = Table.create().addColumns(IntColumn.create("Employee Ids", empIds))
                        .addColumns(StringColumn.create("FirstName", firstNames))
                        .addColumns(StringColumn.create("LastName", lastNames));

            System.out.println(table.print());

            Column<?> column1 = table.column(0);

            System.out.println(column1.print());

      }
}

Output

Employee Ids  |  FirstName  |  LastName  |
-------------------------------------------
            1  |       Hari  |   Krishna  |
            2  |        Ram  |    Gurram  |
            3  |     Sowmya  |       Maj  |
            4  |      Chamu  |       Dev  |
            5  |     Harini  |    Gurram  |
            6  |     Lahari  |       Ram  |
            7  |       Rama  |       Sen  |
            8  |   Lakshman  |    Grandi  |
            9  |     Sandya  |    Neelam  |
Column: Employee Ids
1
2
3
4
5
6
7
8
9

Previous                                                    Next                                                    Home

Why Set do not have get(int index) method?

Set in Java works on the principle of hashing and do not maintain any order while storing. This is the reason ‘why a set do not have get method by index’.

What is Hashing?

Hashing is a principle used to store and retrieve data in constant O(1) time.

Hash Table

Hash table is a data structure used to store collection of items in a way that it is easy to find them in later. Usually you can store and search data of Hash table in constant (O(1)) time.

For example, consider following hash table of size 13 (Technically each position of hash table is called slot, so it has 13 slots). Initially hash table is empty. Let us try to insert some data into hash table based on some hash function.

What is Hash function?

Hash function is used to map an item to a slot (position) in Hash table. Hash function takes an item as input and returns a slot in range between 0 and hash table size (exclusive).

To demonstrate example, I am going to use following hash function.

h(item)=item%13

Item	Hash Value
103	12
65	0
35	9
19	6
23	10
76	11

Based on the Hash value computed, all the items are placed in respective slots. Element 103 is placed in slot 12, Element 65 is placed in slot 0 etc.,

There is a problem in above approach. Suppose you want to insert an element 26 in Hash table, 26 % 13 = 0, 26 has to be placed in slot 0, but slot 0 is already occupied with element 65. Here the concept of collision comes into picture. I will explain more about this later in the post.

Load Factor

Load factor is denoted by λ, λ = (Number Of Items)/(Hash table size). In our example λ = 6/13 = 0.46153846153846156.

How to search for an element in Hash Table?

We can use the same hash function to search for an element in Hash table. Suppose you want to search for an element 23.

a.    Apply hash function on element 23, to find the slot. 23 % 13 = 10

b.    Go to slot 10 and check whether element 23 exist or not.

What is Perfect Hash function?

Perfect Hash function maps each element to a unique slot. Ideally there is no systematic way to construct a perfect hash function. Usually depends on the load factor, hash table is resized to reduce the collisions.

Collision Resolution

Collision occurs when two items mapped to same slot. Collision resolution techniques play an important role in hashing. One way to address the problem of collisions is linear probing.

Linear Probing

In linear probing technique, we try to find next open slot in hash table sequentially. For example, hash table is already occupied with elements 103, 65, 35, 19, 23, 76.

When we try to place element 26 (slot = 26 % 13 = 0) in hash table, collision occurs. We try to find next open slot from the position 0 and place the element 26 in it. Next open position after slot 0 is 1, so we place the element 26 in slot 1.

Suppose you want to insert element 48 into hash table, Find the slot for the element 48 (slot = 48 % 13 = 9). Since slot 9 is already occupied with element 35, we need to find next open slot sequentially.  Slots 10, 11, 12, 0, 1 are occupied with elements; slot 2 is open for element 48. Place the element 48 into the slot 2. Same linear probing technique is used while searching elements in Hash table.

How you search for an element in linear probing?

Suppose you want to search for an element 35, find the slot value using hash function (slot = 35 % 13 = 9). Check the slot 9 for the element 35. Slot 9 has the element 35, return TRUE.

Suppose you want to search for an element 48, calculate the slot value (slot = 48 % 13 = 9). Check the slot 9 for the value 48, slot 9 is occupied with value 35. We cannot simply return FALSE, since there is a possibility of collisions, we must do a sequential search, starting at position 10, looking until either we find the item 48 or we find an empty slot.

Chaining

Another way of solving the Collisions is using chaining. Chaining maintains a linked list kind of data structure at each slot to resolve collisions.

For example, Elements 103, 65, 35, 19, 23, 76 are inserted into hash table of size 13 using the hash function h(item)=item%13.

Suppose you want insert an element 130 into Hash table, calculate the slot (slot = 130 % 13  = 0). Slot 0 is already occupied with element 65, so create a new node with element 130 attach this node the slot 0. If you add elements 130, 91, 71, 50, 89, 206 to the hash table, hash table change like below.

Suppose you want to search for an element 206 in the hash table, you calculate slot for the element 206 (slot = 206 % 13 = 11), you go to the slot 11 and traverse entire linked list at slot 11, if you found the value 206 in the linked list, you return TRUE, else FALSE. One problem with this approach is if linked list size grows, time complexity increases for search and insert operations. To solve this issue Java8 maintains a binary search tree kind of data structure.

I hope with this you got some idea in Hashing, load factor, collision resolution techniques, lets start discuss on HashMap internal working in Java.

HashMap works on the principle of Hashing. HashMap Use two properties, which impact the performance.

a. Capacity

b. Load Factor

Capacity

Capacity is the number of buckets(slots) in the hash table.

Load Factor

The load factor is a measure of how full the hash table is allowed to get before its capacity is automatically increased. By default Load Factor for HashMap is 0.75.

For Example let us assume

Initial capacity is 10 (I.e, Number Of Buckets (or) slots)

Load Factor is 0.75

Each Bucket has the capacity to store 100 data items.

So when all the items reached 0.75 * 100 * 10 = 750, then the internal data structures rehashed and the Capacity (Number Of Buckets (or) slots) increases depends on implementation. The same logic applies in future also. HashMap is not synchronized, So user has to take care while performing put kind of operations.

You may like

Miscellaneous

Interview Questions

Programming Questions

Is Collections.emptyList() singleton?

List Capacity vs Size

ArrayList: add: IndexOutOfBoundsException

How to pick random element from Set?