Java: How to process a large file in chunks?

 

This is continuation to my previous post, where I explained how to read a large file line by line in Java.

 

Problem Statement

Read a large file, where the entire data is in single line.

 

Solution

Since all the data is in single line, reading the file line by line will not help us here. We need to read the chunk of data at a time.

 

How to read chunk of data at  time?

BufferedReader#read method used to read the data one chunk at a time.

 

Signature

public int read(char cbuf[], int off, int len)

Reads characters into a portion of an array. This method return the number of characters read, or -1 if the end of the stream has been reached

 

Below snippet read the file data in chunks.

try (final BufferedReader reader = new BufferedReader(new FileReader(new File(inputFilePath)));
		final BufferedWriter writer = new BufferedWriter(new FileWriter(new File(outputFilePath)))) {
	final char[] ch = new char[65535];

	int noOfCharsRead = -1;
	while ((noOfCharsRead = reader.read(ch, 0, ch.length)) != -1) {

		final String str;
		if (noOfCharsRead != 65535) {
			str = new String(Arrays.copyOf(ch, noOfCharsRead));
		} else {
			str = new String(ch);
		}

		writer.write(transformation.apply(str));

	}

}

 

Above snippet read the file chunk by chubk, apply the transformation on each chunk and write the result to destination.

 

Find the below working application.

 

FileUtil.java

package com.sample.app.util;

import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
import java.util.Arrays;
import java.util.function.Function;

public class FileUtil {

	public static void processFile(String inputFilePath, String outputFilePath, Function<String, String> transformation)
			throws FileNotFoundException, IOException {

		try (final BufferedReader reader = new BufferedReader(new FileReader(new File(inputFilePath)));
				final BufferedWriter writer = new BufferedWriter(new FileWriter(new File(outputFilePath)))) {
			final char[] ch = new char[65535];

			int noOfCharsRead = -1;
			while ((noOfCharsRead = reader.read(ch, 0, ch.length)) != -1) {

				final String str;
				if (noOfCharsRead != 65535) {
					str = new String(Arrays.copyOf(ch, noOfCharsRead));
				} else {
					str = new String(ch);
				}

				writer.write(transformation.apply(str));

			}

		}
	}

}

App.java

package com.sample.app;

import java.io.FileNotFoundException;
import java.io.IOException;
import java.util.function.Function;

import com.sample.app.util.FileUtil;

public class App {
	public static void main(String[] args) throws FileNotFoundException, IOException {

		Function<String, String> toUpper = new Function<String, String>() {

			@Override
			public String apply(String t) {
				return t.toUpperCase();
			}

		};

		FileUtil.processFile("/Users/Shared/a.txt", "/Users/Shared/b.txt", toUpper);

	}
}

You may like

File programs in Java

Get the content of resource file as string

Copy the content of file to other location

Write byte array to a file

How to download a binary file in Java?

How to process a huge file line by line in Java?