How to process a huge file in Java?

Problem statement

Assume you have given a file of huge size and process the file line by line and write the data to some destination.

 

Point to consider from problem statement

As per the problem statement, we need to process the file line by line. By considering this, we no need to load all the file content to in-memory. We can read the file line by line and process the line, write the processed result to an external destination.

public static void processFile(String inputFilePath, String outputFilePath,
		Function<String, String> lineTransformation)
		throws FileNotFoundException, IOException {
	try (BufferedReader reader = new BufferedReader(new FileReader(new File(inputFilePath)));
			BufferedWriter writer = new BufferedWriter(new FileWriter(new File(outputFilePath)))) {
		String line;

		while ((line = reader.readLine()) != null) {
			writer.write(lineTransformation.apply(line));
			writer.write("\n");
		}
	}
}

 

Above snippet read the file line by line, apply the lineTransformation on each line and write the result to destination.

 

Find the below working application.

 

FileUtil.java

package com.sample.app.util;

import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
import java.util.function.Function;

public class FileUtil {

	public static void processFile(String inputFilePath, String outputFilePath,
			Function<String, String> lineTransformation)
			throws FileNotFoundException, IOException {
		try (BufferedReader reader = new BufferedReader(new FileReader(new File(inputFilePath)));
				BufferedWriter writer = new BufferedWriter(new FileWriter(new File(outputFilePath)))) {
			String line;

			while ((line = reader.readLine()) != null) {
				writer.write(lineTransformation.apply(line));
				writer.write("\n");
			}
		}
	}

}

 

App.java

package com.sample.app;

import java.io.FileNotFoundException;
import java.io.IOException;
import java.util.function.Function;

import com.sample.app.util.FileUtil;

public class App {
	public static void main(String[] args) throws FileNotFoundException, IOException {

		Function<String, String> toUpper = new Function<String, String>() {

			@Override
			public String apply(String t) {
				return t.toUpperCase();
			}

		};

		FileUtil.processFile("/Users/Shared/a.txt", "/Users/Shared/b.txt", toUpper);
	}
}

 

 

You may like

File programs in Java

Download file from Internet using Java

Get the content of resource file as string

Copy the content of file to other location

Write byte array to a file

How to download a binary file in Java?