Understanding Java Threads and Why Virtual Threads Matter

  

Java has been around for more than 25 years. One of the reasons developers love it is how simple it is to create and use threads. A thread is like a worker inside your program. You can tell this worker like "Hey, go do this task while I do something else".

 

Creating threads in Java is easy, but there’s a problem, when applications grow (like REST APIs, microservices, or big web apps), too many threads can slow things down. That’s where scalability issues begin.

 

In this section, we’ll look at:

 

·      Why traditional threads can become a problem

·      What solutions developers tried in the past

·      Why those solutions had their own challenges

·      And finally, how Java Virtual Threads (Project Loom) bring a big change

 

1. Why Threads Matter in Enterprise Applications?

Most enterprise applications don’t just sit there doing calculations. They talk to other systems.

 

For example:

·      Calling a database (like MySQL or MongoDB)

·      Talking to other microservices

·      Communicating with external services (like payment gateways, weather APIs, or social media APIs)

·      Reading/writing files

 

When a user sends a request to your app (for example, opening their dashboard in a banking app), the app usually:

·      Fetches data from the database.

·      Maybe calls other microservices (like loan service, payment service).

·      Processes the data (calculations, formatting).

·      Sends a final response to the user.

 

These steps fall into two categories:

·      I/O-bound tasks: Waiting for something external (database, network call, file read).

·      CPU-bound tasks: Doing actual work on the CPU (like calculations, algorithms, data transformations).

 

Both are important, but I/O-bound tasks can cause a lot of "waiting around". Imagine a thread that sits idle until a database replies, that’s wasteful of resources.

 

2. How Application Servers Handle Requests?

You don’t have to manage everything yourself. Frameworks like Spring Boot and servers like Tomcat or WebLogic already handle sockets, logging, threads, and database pools for you. But the way they manage threads has a huge impact on performance. Over time, different models were used. Let's understand one-by-one.

 

2.1 Process-per-Request (The Old CGI Days)

CGI stands for Common Gateway Interface. It was one of the first ways web servers (like Apache) could run programs to generate dynamic web pages.

 

For example:

·      A user clicks a button on a website.

·      The web server needs to run some logic (say, fetch data from a database).

·      With CGI, the server starts a new process to handle that request.

 

How it worked?

·      Each HTTP request spawn a brand new process.

·      Process handles the request, then exits.

 

It’s like having a restaurant where, every time a customer orders food, the owner hires a brand-new chef, sets up a new kitchen, and after one meal, shuts it all down.

 

Why This Was a Problem?

·      Processes are heavy: Starting a process means loading the program into memory, setting up resources, etc.

·      Slow startup: Even a small delay per request made websites feel sluggish.

·      Terrible scalability: If 100 users came at the same time, the server had to start 100 processes, the system quickly ran out of CPU and memory.

 

In short, CGI was fine when the web was small, but it collapsed under real world traffic.

 

2.2 The Improvement: FastCGI

To fix CGI’s inefficiency, FastCGI was introduced. Instead of starting a new process for every request, keep a pool of long running processes ready. The web server sends requests to these already running processes.

 

How FastCGI helped?

·      No need to keep creating/destroying processes.

·      Much faster because processes were always warm and ready.

·      Reduced memory and CPU overhead.

 

But Still Not Perfect

·      Processes are still heavier than threads.

·      Communication between web server and FastCGI processes (via sockets/pipes) added extra overhead.

·      Not good enough for massive scalability (thousands or millions of concurrent users).

 

In summary, CGI (process per request) extremely slow, not scalable. FastCGI (process pool) is an improved version over CGI, but still not enough for large scale systems.

 

2.3 Thread-per-Request (Modern Java Servers)

Most popular Java servers today like Tomcat, Jetty, WebLogic, JBoss/WildFly follow this model:

 

·      Every incoming HTTP request gets its own Java thread.

·      The thread stays busy until the response is fully ready.

 

It’s like a restaurant where:

·      Every customer is assigned one waiter.

·      That waiter takes the order, waits while the food is cooked, serves it, and doesn’t serve anyone else until the customer leaves.

 

Why This Was Better Than CGI?

·      Threads are lighter than processes:

·      They share the same memory space (within the JVM).

·      Faster to create than processes.

·      Much better scalability than CGI or FastCGI.

·      Servers could now handle hundreds or thousands of requests without crashing immediately.

 

How It Works?

·      User makes an HTTP request (say, fetch /products).

·      The server assigns a thread from a thread pool.

·      That thread:

o   Parses the request

o   Talks to the database

o   Reads/writes files if needed

o   Prepares the HTTP response

·      Once the response is sent, the thread goes back to the pool to serve another request.

 

The Blocking Model

Threads are blocking. If a thread makes a database call, it waits until the DB responds. While waiting, the thread cannot handle other requests.

 

Example: If you have 1000 threads, and all of them are waiting on slow database queries, your server is stuck, even if the CPU is idle.

 

Advantages of Thread per Request Approach

·      Easy to write & debug: Code looks like "normal Java". Stack traces tell you exactly what went wrong.

·      Simple transaction handling: One thread = one request, transactions (like DB commits/rollbacks) are easy to manage.

·      Resource sharing works well: Thread local variables, DB connection pools, caches all integrate smoothly.

 

Limitations

·      Threads are still expensive: Each Java thread maps to an OS thread, which needs memory (stack space) and kernel scheduling. Java Thread is a thin wrapper around Operating System Thread.

·      Scalability wall: If 10,000 users connect at the same time, server needs ~10,000 threads. Memory usage explodes (hundreds of MB or GB just for thread stacks). Context switching between thousands of threads slows everything down.

 

2.4 Thread Pools

Creating one thread per request works but doesn’t scale (too many OS threads, high memory usage, expensive context switching). The natural improvement that modern Java servers use is the thread pool model.

 

What is a Thread Pool?

·      Instead of creating a new thread for every request, the server:

·      Creates a fixed number of threads in advance (say 200).

·      Keeps them in a "pool".

·      As requests arrive, they are assigned to an available thread.

·      When the thread finishes, it goes back to the pool, ready for the next request.

 

It’s like a restaurant with:

·      A fixed team of waiters (200 waiters).

·      If 201 customers arrive, the extra one must wait in a queue until a waiter is free.

 

Why Thread Pools Help?

·      Limit resource usage: Server will never spawn more than N threads, memory and CPU stay under control.

·      Reuse threads: No overhead of creating/destroying threads repeatedly.

·      Better throughput: Pool threads keep working steadily without overwhelming the OS.

 

Limitations Still Remain

·      If requests block (e.g., slow DB calls), the pool thread still get stuck, new requests pile up in the queue.

·      If too many requests wait too long, users experience timeouts or slow responses.

·      Increasing the pool size helps only up to a point (because of memory + context switching).

 

This is why, for very high concurrency, the industry moved toward non-blocking I/O and later virtual threads.

 

Example: Thread per Request in Action using Regular threads, Thread Pool.

 

RequestHandler.java

package com.sample.app.threads;

import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

public class RequestHandler implements Runnable {
	private final String requestName;

	public RequestHandler(String requestName) {
		this.requestName = requestName;
	}

	@Override
	public void run() {
		System.out.println(Thread.currentThread().getName() + " started processing " + requestName);

		try {
			// Simulate database call (blocking I/O)
			Thread.sleep(2000); // pretend DB took 2 seconds
		} catch (InterruptedException e) {
			Thread.currentThread().interrupt();
		}

		System.out.println(Thread.currentThread().getName() + " finished processing " + requestName);
	}

	public static void main(String[] args) {
		// Thread-per-request demo
		System.out.println("=== Thread-per-Request Demo (Simple) ===");
		for (int i = 1; i <= 5; i++) {
			Thread thread = new Thread(new RequestHandler("Request-" + i));
			thread.start();
		}

		// Using a thread pool (like real servers do)
		System.out.println("\n=== Thread Pool Demo (Realistic Server) ===");
		ExecutorService threadPool = Executors.newFixedThreadPool(3); // only 3 worker threads
		for (int i = 1; i <= 10; i++) {
			threadPool.submit(new RequestHandler("Pooled-Request-" + i));
		}

		threadPool.shutdown(); // allow tasks to finish
	}
}

Sample Output

=== Thread-per-Request Demo (Simple) ===

=== Thread Pool Demo (Realistic Server) ===
Thread-0 started processing Request-1
Thread-2 started processing Request-3
Thread-1 started processing Request-2
Thread-3 started processing Request-4
Thread-4 started processing Request-5
pool-1-thread-3 started processing Pooled-Request-3
pool-1-thread-1 started processing Pooled-Request-1
pool-1-thread-2 started processing Pooled-Request-2
Thread-4 finished processing Request-5
Thread-2 finished processing Request-3
pool-1-thread-3 finished processing Pooled-Request-3
pool-1-thread-2 finished processing Pooled-Request-2
pool-1-thread-1 finished processing Pooled-Request-1
pool-1-thread-1 started processing Pooled-Request-4
Thread-1 finished processing Request-2
pool-1-thread-2 started processing Pooled-Request-5
pool-1-thread-3 started processing Pooled-Request-6
Thread-0 finished processing Request-1
Thread-3 finished processing Request-4
pool-1-thread-2 finished processing Pooled-Request-5
pool-1-thread-3 finished processing Pooled-Request-6
pool-1-thread-2 started processing Pooled-Request-7
pool-1-thread-3 started processing Pooled-Request-8
pool-1-thread-1 finished processing Pooled-Request-4
pool-1-thread-1 started processing Pooled-Request-9
pool-1-thread-3 finished processing Pooled-Request-8
pool-1-thread-3 started processing Pooled-Request-10
pool-1-thread-2 finished processing Pooled-Request-7
pool-1-thread-1 finished processing Pooled-Request-9
pool-1-thread-3 finished processing Pooled-Request-10

   

Output will vary on each run.

 

3. Concurrency, Parallelism, and Sync vs Async in Java

In the previous section, we talked about the thread per user model. Here’s the simple idea how it works:

 

·      When a user sends a request to a server, the server takes a thread from its pool.

·      That thread is now responsible for handling the user’s request.

·      Until the request is fully processed, the thread cannot be reused for other users.

 

So if 100 users send requests at the same time, the server will need 100 threads running in the JVM. Now, here’s the catch, what if your machine only has 3 CPU cores? Clearly, it can’t literally run 50 things at once, it can run 3 tasks at any point in time.

 

This brings us to two important concepts: Concurrency and parallelism.

 

3.1 Concurrency vs Parallelism

Concurrency: The ability of a system to handle multiple tasks at the same time but not necessarily running them at the exact same instant. The CPU switches quickly between tasks, giving the illusion that they’re happening simultaneously. It’s about managing multiple tasks efficiently.

 

For example, imagine you’re working on a laptop. You’re writing code, a download is running, and music is playing. The CPU isn’t doing all three things literally at once, but it switches between them so fast that it feels like everything is happening together.

 

Parallelism: Actually, running multiple tasks at the same exact time, usually on different CPU cores. It’s about executing tasks simultaneously.

 

For example, if your computer has 4 cores, one core can play music, another can handle your download, and two cores can run your code compilation at the same time.

 

Analogy: Restaurant Kitchen

·      Concurrency: One chef (single CPU core) quickly switches between chopping vegetables, stirring soup, and flipping a burger. Each task makes progress, but only one is being done at a time.

·      Parallelism: Three chefs (multiple CPU cores) work at the same time, one chops vegetables, another stirs soup, and the third flips the burger. All tasks truly happen simultaneously.

 

Let’s write an example to understand Concurrency.

 

ConcurrencyDemo.java

package com.sample.app.threads;

import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;

public class ConcurrencyDemo {
	public static void main(String[] args) {
		// Create a thread pool with 3 worker threads
		ExecutorService executor = Executors.newFixedThreadPool(3);

		Runnable downloadTask = () -> {
			for (int i = 1; i <= 5; i++) {
				System.out.println(Thread.currentThread().getName() + " - Downloading file part " + i);
				sleep(200);
			}
			System.out.println(Thread.currentThread().getName() + " - Download complete");
		};

		Runnable uploadTask = () -> {
			for (int i = 1; i <= 5; i++) {
				System.out.println(Thread.currentThread().getName() + " - Uploading chunk " + i);
				sleep(150);
			}
			System.out.println(Thread.currentThread().getName() + " - Upload complete");
		};

		Runnable dataProcessingTask = () -> {
			for (int i = 1; i <= 5; i++) {
				System.out.println(Thread.currentThread().getName() + " - Processing record " + i);
				sleep(100);
			}
			System.out.println(Thread.currentThread().getName() + " - Processing complete ");
		};

		// Submit tasks to executor
		executor.submit(downloadTask);
		executor.submit(uploadTask);
		executor.submit(dataProcessingTask);

		// Shut down executor gracefully
		executor.shutdown();
		try {
			if (!executor.awaitTermination(5, TimeUnit.SECONDS)) {
				executor.shutdownNow();
			}
		} catch (InterruptedException e) {
			executor.shutdownNow();
		}

		System.out.println("All tasks finished!");
	}

	private static void sleep(int ms) {
		try {
			Thread.sleep(ms);
		} catch (InterruptedException ignored) {
		}
	}
}

Output

pool-1-thread-1 - Downloading file part 1
pool-1-thread-3 - Processing record 1
pool-1-thread-2 - Uploading chunk 1
pool-1-thread-3 - Processing record 2
pool-1-thread-2 - Uploading chunk 2
pool-1-thread-1 - Downloading file part 2
pool-1-thread-3 - Processing record 3
pool-1-thread-2 - Uploading chunk 3
pool-1-thread-3 - Processing record 4
pool-1-thread-1 - Downloading file part 3
pool-1-thread-3 - Processing record 5
pool-1-thread-2 - Uploading chunk 4
pool-1-thread-3 - Processing complete 
pool-1-thread-1 - Downloading file part 4
pool-1-thread-2 - Uploading chunk 5
pool-1-thread-2 - Upload complete
pool-1-thread-1 - Downloading file part 5
pool-1-thread-1 - Download complete
All tasks finished!

   

Parallelism (multiple cores):

If the system has multiple cores, those two threads from the previous example may actually run at the same exact time, one thread on each core. You don’t control this directly in Java, the OS and JVM decide how to schedule threads. But with multiple CPU cores, true parallelism can happen.

 

In summary,

·      Concurrency ≠ Parallelism

·      Concurrency = dealing with lots of things at once (interleaving).

·      Parallelism = doing lots of things at the exact same time.

·      Concurrency can happen on a single core.

·      Parallelism requires multiple cores.

 

3.2 Synchronous vs Asynchronous Calls

Synchronous call: You wait for the method to finish before moving on.

 

String result1 = fetch("API 1"); // must finish first
String result2 = fetch("API 2"); // only starts after API 1 finishes

   

Simple to write and debug, but slower because tasks happen one by one.

 

Asynchronous call: You start the work but don’t wait for it to finish immediately. Example using CompletableFuture:

 

AsyncExmple.java

 

package com.sample.app.threads;

import java.util.concurrent.CompletableFuture;

public class AsyncExample {
	public static void main(String[] args) {
		// Start two async API calls
		CompletableFuture<String> api1 = CompletableFuture.supplyAsync(() -> fetch("API 1"));
		CompletableFuture<String> api2 = CompletableFuture.supplyAsync(() -> fetch("API 2"));

		// While APIs are running, main thread can do other work
		System.out.println("Main thread is free to do something else...");

		// Combine results of both APIs once they are ready
		CompletableFuture<String> combined = api1.thenCombine(api2, (res1, res2) -> {
			return res1 + " + " + res2;
		});

		// Add further processing (non-blocking)
		CompletableFuture<Void> pipeline = combined.thenApply(result -> "Processed: " + result)
				.thenAccept(finalResult -> System.out.println("Final Output -> " + finalResult)).exceptionally(ex -> {
					System.out.println("Something went wrong: " + ex.getMessage());
					return null;
				});

		// Wait for all tasks to complete before exiting
		pipeline.join();
		System.out.println("Program finished!");
	}

	static String fetch(String apiName) {
		try {
			System.out.println(Thread.currentThread().getName() + " is fetching " + apiName);
			Thread.sleep(500); // Simulate delay
		} catch (InterruptedException ignored) {
		}
		return "Result from " + apiName;
	}
}

   

Output

 

Main thread is free to do something else...
ForkJoinPool.commonPool-worker-2 is fetching API 2
ForkJoinPool.commonPool-worker-1 is fetching API 1
Final Output -> Processed: Result from API 1 + Result from API 2
Program finished!

   

Analogy: Phone Calls

·      Synchronous: You call a friend, and you don’t do anything else until the conversation ends.

·      Asynchronous: You call a friend, leave a message, and continue doing other things. Later, they call you back with the answer.

 

In summary:

·      Concurrency: Many tasks are in progress at the same time, but not necessarily running simultaneously.

·      Parallelism: Tasks (or subtasks) are actually running at the same time, usually on different cores.

·      Synchronous: You wait for a task to finish before moving on.

·      Asynchronous: You start a task and keep working, the result comes later.

·      In web apps, most tasks are I/O heavy (waiting on databases, APIs), so concurrency, asynchronous mechanisms helps scale better.

 

4. Java Threads and JVM Memory

So far, we’ve talked about how threads help us handle multiple user requests at the same time, and how concepts like concurrency, parallelism, and asynchronous calls play a role. Now, let’s zoom in and understand how Java threads actually work inside the JVM (Java Virtual Machine), and why this is important when building scalable applications.

 

4.1 How a Java Program Runs?

Every Java program begins with a main thread. This is the thread that calls the main() method.

 

·      When you call a method, the JVM creates something called a stack frame to store information about that method (local variables, where you are in the code, etc.).

 

·      If a method calls another method (e.g., processUserDetails() → saveUser()), a new stack frame is added on top of the stack.

 

·      When a method finishes, its stack frame is removed (popped off).

 

Find the below working Application.

 

StackFrameDemo.java

 

package com.sample.app.threads;

public class StackFrameDemo {

    private static void printStackTrace(String context) {
        System.out.println("\n--- Stack Trace (" + context + ") ---");
        StackTraceElement[] stackTrace = Thread.currentThread().getStackTrace();

        // Skip first two elements (getStackTrace + this method itself)
        for (int i = 2; i < stackTrace.length; i++) {
            StackTraceElement element = stackTrace[i];
            System.out.println("  at " + element.getClassName() + "." + element.getMethodName()
                    + "(" + element.getFileName() + ":" + element.getLineNumber() + ")");
        }
        System.out.println("--- End of Stack Trace ---\n");
    }

    public static void main(String[] args) {
        printStackTrace("Start of main");
        System.out.println("Main thread starts");

        processUserDetails("Alice");

        printStackTrace("After processUserDetails");
        System.out.println("Main thread ends");
        printStackTrace("End of main");
    }

    static void processUserDetails(String name) {
        System.out.println("\nEntered processUserDetails() with name = " + name);
        printStackTrace("Inside processUserDetails - before saveUser");

        // Call another method
        saveUser(name);

        printStackTrace("Inside processUserDetails - after saveUser");
        System.out.println("Exiting processUserDetails()");
    }

    static void saveUser(String name) {
        System.out.println("\nEntered saveUser() with name = " + name);
        printStackTrace("Inside saveUser");

        // Local variable (stored in this method's stack frame)
        boolean success = true;

        if (success) {
            System.out.println("User " + name + " saved successfully!");
        }

        printStackTrace("Exiting saveUser");
        System.out.println("Exiting saveUser()");
    }
}

   

Output

 

--- Stack Trace (Start of main) ---
  at com.sample.app.threads.StackFrameDemo.main(StackFrameDemo.java:19)
--- End of Stack Trace ---

Main thread starts

Entered processUserDetails() with name = Alice

--- Stack Trace (Inside processUserDetails - before saveUser) ---
  at com.sample.app.threads.StackFrameDemo.processUserDetails(StackFrameDemo.java:31)
  at com.sample.app.threads.StackFrameDemo.main(StackFrameDemo.java:22)
--- End of Stack Trace ---


Entered saveUser() with name = Alice

--- Stack Trace (Inside saveUser) ---
  at com.sample.app.threads.StackFrameDemo.saveUser(StackFrameDemo.java:42)
  at com.sample.app.threads.StackFrameDemo.processUserDetails(StackFrameDemo.java:34)
  at com.sample.app.threads.StackFrameDemo.main(StackFrameDemo.java:22)
--- End of Stack Trace ---

User Alice saved successfully!

--- Stack Trace (Exiting saveUser) ---
  at com.sample.app.threads.StackFrameDemo.saveUser(StackFrameDemo.java:51)
  at com.sample.app.threads.StackFrameDemo.processUserDetails(StackFrameDemo.java:34)
  at com.sample.app.threads.StackFrameDemo.main(StackFrameDemo.java:22)
--- End of Stack Trace ---

Exiting saveUser()

--- Stack Trace (Inside processUserDetails - after saveUser) ---
  at com.sample.app.threads.StackFrameDemo.processUserDetails(StackFrameDemo.java:36)
  at com.sample.app.threads.StackFrameDemo.main(StackFrameDemo.java:22)
--- End of Stack Trace ---

Exiting processUserDetails()

--- Stack Trace (After processUserDetails) ---
  at com.sample.app.threads.StackFrameDemo.main(StackFrameDemo.java:24)
--- End of Stack Trace ---

Main thread ends

--- Stack Trace (End of main) ---
  at com.sample.app.threads.StackFrameDemo.main(StackFrameDemo.java:26)
--- End of Stack Trace ---

4.2 JVM Memory Model

The JVM uses two key memory areas when running your code:

 

·      Heap (shared memory): This is where all the Java objects live. The heap is shared between all threads. If two threads have a reference to the same object, they can both read or update it.

Refer following posts on Heap Memory.

o   https://self-learning-java-tutorial.blogspot.com/2014/02/heap-memory.html

o   https://self-learning-java-tutorial.blogspot.com/2014/02/how-to-increase-heap-size-in-java.html

 

 

·      Stack (private to each thread) Each thread gets its own stack. This is where local variables and method calls are stored for that thread. Primitive types like int, boolean, and method call details go here. If you create an object, the reference to it is on the stack, but the actual object is on the heap.

Refer following posts on Stack Memory.

o   https://self-learning-java-tutorial.blogspot.com/2014/02/stack-memory.html

o   https://self-learning-java-tutorial.blogspot.com/2014/02/how-to-increase-stack-size-in-java.html

 

4.3 Threads in the JVM: In Java, each thread you create maps directly to an Operating System (OS) thread (1:1 relationship). The OS decides which thread runs on the CPU at any given moment (this is called scheduling). Each Java thread is given about some amount of stack space by default. This space will vary from Operating system to Operating System.

 

4.4 Why Threads Are Expensive?

Threads are powerful, but they’re not free. Here’s why:

 

·      Memory Cost: If each thread gets ~1 MB stack space, creating thousands of threads quickly eats up memory.

·      CPU Cost: The OS has to constantly switch between active threads (called context switching). This slows things down.

·      Blocked Threads Waste Resources: If a thread is waiting (for example, for data from a file or network), the OS thread is also stuck, still holding its resources.

 

4.5 MaxThreadsCheck: Java Thread Capacity Tester

Let’s develop a Java program that helps developers and system administrators understand the maximum number of threads their system can support. Each thread performs a simple task (sleeping for 10 seconds) to simulate a lightweight workload. By running this program, you can:

 

·      Test how many concurrent threads your JVM can handle.

 

·      Observe memory usage and system behavior under heavy thread creation.

 

Tune JVM parameters like -Xmx (maximum heap) or -Xss (thread stack size) to optimize for high concurrency.

 

This is particularly useful for applications that rely on multi-threading, such as servers, parallel processing tasks, or asynchronous systems, where knowing the thread limits helps prevent OutOfMemoryError or performance bottlenecks.

 

MaxThreadsCheck.java  

public class MaxThreadsCheck {

    public static void main(String[] args) {
        if (args.length != 1) {
            System.out.println("Usage: java -Xmx1G MaxThreadsCheck <noOfThreadsToTest>");
            return;
        }

        int noOfThreadsToTest;
        try {
            noOfThreadsToTest = Integer.parseInt(args[0]);
        } catch (NumberFormatException e) {
            System.out.println("Please provide a valid integer for number of threads.");
            return;
        }

        System.out.println("Starting thread test with " + noOfThreadsToTest + " threads...");

        for (int i = 1; i <= noOfThreadsToTest; i++) {
            final int threadNumber = i;

            try {
                Thread t = new Thread(() -> {
                    System.out.println("Thread " + threadNumber + " started");
                    try {
                        Thread.sleep(10_000); // Sleep for 10 seconds
                    } catch (InterruptedException e) {
                        Thread.currentThread().interrupt();
                        System.out.println("Thread " + threadNumber + " interrupted");
                    }
                    System.out.println("Thread " + threadNumber + " finished");
                });

                t.start();

            } catch (OutOfMemoryError | Exception e) {
                System.out.println("Failed to create thread " + threadNumber + ": " + e.getMessage());
                break;
            }
        }

        System.out.println("Thread creation loop finished. Threads will continue to run for 30 seconds.");
    }
}

   

Compile MaxThreadsCheck.java

javac MaxThreadsCheck.java

 

I started the application by executing following command.

java -Xmx1G MaxThreadsCheck 10000

 

I see following output.

Thread 8162 started
[0.795s][warning][os,thread] Failed to start thread "Unknown thread" - pthread_create failed (EAGAIN) for attributes: stacksize: 2048k, guardsize: 16k, detached.
[0.795s][warning][os,thread] Failed to start the native thread for java.lang.Thread "Thread-8162"
Failed to create thread 8163: unable to create native thread: possibly out of memory or process/resource limits reached

This means that the program successfully created 8162 threads. Each thread started running the Runnable and printed this message.

 

When the JVM tried to create the 8163rd thread, but the underlying OS failed to create a new native thread.

 

·      pthread_create is the POSIX system call the JVM uses to create threads on Linux/Unix/macOS.

·      EAGAIN is an OS error code meaning:

o   Resource limits reached (max threads per process or system-wide threads limit), or

o   Insufficient memory (not enough stack memory to allocate for the new thread).

 

 

Feel free to experiment with different options.

a. Reduce thread stack size:

java -Xmx1G -Xss256k MaxThreadsCheck 10000

 

This reduces each thread’s stack from 512KB to 256KB, allowing more threads.

 

b. Increase OS limits:

ulimit -u 65535   # Linux: max processes/threads per user

 

This command sets the maximum number of processes and threads that a single user can run simultaneously on a Linux system.

 

4.6 Thread Limits and Scalability Problems in Java

From the previous example, we learned that there’s always a limit to how many threads you can create in the JVM (Java Virtual Machine).

 

Why? Because:

·      Each thread needs memory (for its stack).

·      The operating system also puts restrictions on how many threads a process can have.

·      So no matter how powerful your machine is, there is always a maximum number of threads you can create.

 

The Problem with Thread Pools

Most application servers (like Tomcat) don’t create a new thread for every request. Instead, they use a thread pool. A thread pool is a fixed group of threads that can be reused. This prevents the server from creating unlimited threads and crashing.

 

For example:

·      Suppose Tomcat is configured with a thread pool size of 100 threads.

·      That means it can serve 100 concurrent users at the same time.

·      But what happens if the 101st user makes a request?

·      The request has to wait in a queue until a thread becomes available.

·      If no thread is free quickly enough, the request may time out or be rejected, and the user sees an error.

 

Let’s say you increase the pool size to 500 threads to support more users. On some machines (like a Mac with huge RAM), you might be able to push this even to 5000 threads. But in practice, the application will often crash or slow down before reaching that number.

 

 

Why? Because each thread consumes memory (usually around 512KB to 2 MB per thread stack). With thousands of threads, memory runs out quickly, each thread adds scheduling overhead. The OS has to context switch between them, which slows things down.

 

Threads Waiting on I/O

In a typical web application, threads don’t always use the CPU.

 

On average:

·      20% of the time: CPU work (processing data, calculations, etc.).

·      80% of the time: Waiting (on I/O, like database queries, network calls, or file access).

 

When a thread is waiting, it’s doing nothing useful, simply wasting resources like memory.

 

Threads are expensive resources. If they are stuck waiting on I/O, they cannot serve new users. This leads to request timeouts, rejections, and poor scalability even though your server’s CPU might still have room to handle more work.

 

5. Scaling Applications: Vertical vs Horizontal Scaling

Till now, we learned that you cannot have unlimited threads running in your application at the same time. The number of threads you can create depends on your machine’s memory, operating system, and other factors. Too many threads can cause performance issues or even application crashes.

 

To solve this problem, the industry has developed two main strategies:

 

·      vertical scaling and

·      horizontal scaling.

 

5.1 Vertical Scaling

Vertical scaling means making a single machine more powerful.

 

·      You increase the CPU power.

·      You add more memory (RAM).

·      You upgrade the machine to handle more work.

 

For example, if your application needs to serve 5,000 users at the same time, you might upgrade the server from 84 GB RAM to 16 GB RAM and from 2 CPUs to 8 CPUs. This gives your application more room to handle extra requests.

 

But there’s a limit. Even the most powerful machine has a maximum capacity, and these upgrades can get very expensive. If you need to support millions of users, vertical scaling alone will not be enough.

 

5.2 Horizontal Scaling

Horizontal scaling means adding more machines instead of making one machine more powerful.

 

Think of it like this:

·      Each machine (node) runs the same application.

·      A load balancer sits in front of them, deciding which machine will handle each incoming request.

 

For example, if each machine can handle 1,000 users, then to support 3,000 users, you just need 3 machines. The load balancer spreads the load evenly. This approach is very popular because you can keep adding machines as your user base grows.

 

The downside? It still costs money, especially when scaling to millions of users. The more machines you add, the more you pay your cloud provider or to your own infrastructure.

 

Real systems often use a combination of both strategies. If you only scale vertically (bigger machine), costs skyrocket and you hit physical limits. If you only scale horizontally (more machines), costs also rise as you keep adding servers.

 

That’s why the real solution is a combination:

 

·      Optimize the application itself so it handles requests efficiently.

·      Use vertical scaling when a small boost is enough.

·      Use horizontal scaling when you need massive scalability.

 

Find the below working Application that reflects how a load balancer works.

 

LoadBalancerDemo.java

package com.sample.app.threads;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.atomic.AtomicInteger;

public class LoadBalancerDemo {
    private final List<String> servers;
    private final AtomicInteger currentServer = new AtomicInteger(0);

    public LoadBalancerDemo(List<String> servers) {
        this.servers = new ArrayList<>(servers); // defensive copy
    }

    public String getServerForRequest(String request) {
        int index = currentServer.getAndUpdate(i -> (i + 1) % servers.size());
        return servers.get(index);
    }

    public static void main(String[] args) {
        List<String> serverList = Arrays.asList("Server-1", "Server-2", "Server-3");
        LoadBalancerDemo loadBalancer = new LoadBalancerDemo(serverList);

        // Simulate multiple user requests (multi-threaded)
        ExecutorService executor = Executors.newFixedThreadPool(5);

        for (int i = 1; i <= 10; i++) {
            final int requestId = i;
            executor.submit(() -> {
                String request = "UserRequest-" + requestId;
                String server = loadBalancer.getServerForRequest(request);
                System.out.println(
                        request + " handled by " + server + " (Thread: " + Thread.currentThread().getName() + ")");
            });
        }

        executor.shutdown();
    }
}

   

Sample Output

 

UserRequest-1 handled by Server-1 (Thread: pool-1-thread-1)
UserRequest-4 handled by Server-1 (Thread: pool-1-thread-4)
UserRequest-3 handled by Server-2 (Thread: pool-1-thread-3)
UserRequest-2 handled by Server-3 (Thread: pool-1-thread-2)
UserRequest-5 handled by Server-2 (Thread: pool-1-thread-5)
UserRequest-6 handled by Server-3 (Thread: pool-1-thread-1)
UserRequest-7 handled by Server-1 (Thread: pool-1-thread-4)
UserRequest-8 handled by Server-2 (Thread: pool-1-thread-1)
UserRequest-9 handled by Server-3 (Thread: pool-1-thread-4)
UserRequest-10 handled by Server-1 (Thread: pool-1-thread-1)

   

6. Non-Blocking I/O: Why It Matters and How Virtual Threads Change the Game

Imagine you’re building an app that

a.   downloads a file,

b.   then reads another file from disk,

c.    and finally sends both results back to the user.

 

This is a very common pattern, your app spends a lot of time waiting either for the network (download) or for the disk (read). While waiting, the CPU is not really working, but the thread that handles the request is still holds resources such as stack memory, platform thread etc.,

 

This is where blocking vs non-blocking I/O makes a big difference:

·      Blocking I/O:  the thread sits idle while waiting.

·      Non-blocking I/O: the thread is released to do other work, and later gets notified when results are ready.

·      Virtual threads: let you write blocking-style code but make it scale like non-blocking, because the JVM handles the waiting very efficiently.

 

6.1 Blocking style (Simple but wastes threads)

 

BlockingFileDownload.java

 

package com.sample.app.threads;

import java.io.IOException;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.stream.Collectors;

public class BlockingFileDownload {
	public static void main(String[] args) throws IOException, InterruptedException {
		System.out.println("Starting download...");

		// Download remote file (blocking)
		String data1 = downloadFileBlocking(
				"https://raw.githubusercontent.com/pandas-dev/pandas/e503c13e5de42ac2fc675e564b0958a14221b14a/doc/redirects.csv");
		System.out.println("Downloaded remote file:\n" + data1);

		// Read local file (blocking)
		String data2 = readFileBlocking("localfile.txt");
		System.out.println("Read local file:\n" + data2);

		// Simulate sending response
		sendResponse(data1, data2);
	}

	// Downloads file content using Java HttpClient (blocking)
	static String downloadFileBlocking(String url) throws IOException, InterruptedException {
		HttpClient client = HttpClient.newHttpClient();
		HttpRequest request = HttpRequest.newBuilder().uri(URI.create(url)).build();
		HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());

		return response.body();
	}

	// Reads file from resources (blocking)
	static String readFileBlocking(String resourceName) throws IOException {
		// Get the file from classpath (works in IDE and JAR)
		var resourceUrl = BlockingFileDownload.class.getClassLoader().getResource(resourceName);
		if (resourceUrl == null) {
			throw new IOException("Resource not found: " + resourceName);
		}
		Path path = Paths.get(resourceUrl.getPath());
		return Files.lines(path).collect(Collectors.joining("\n"));
	}

	// Simulates sending response (blocking)
	static void sendResponse(String a, String b) throws InterruptedException {
		Thread.sleep(200); // simulate network delay
		System.out.println("\n=== Final Response ===");
		System.out.println(a + "\n---\n" + b);
	}
}

   

6.2 Non-blocking style with CompletableFuture (Frees threads)

 

CallbackFileDownload.java

 

package com.sample.app.threads;

import java.util.concurrent.CompletableFuture;

public class CallbackFileDownload {
    public static void main(String[] args) {
        log("Starting async tasks...");

        // Simulate remote file download
        CompletableFuture<String> remoteFileFuture = CompletableFuture.supplyAsync(() -> {
            log("Downloading remote file...");
            sleep(500); // simulate delay
            return "remote-data";
        });

        // Simulate local file read
        CompletableFuture<String> localFileFuture = CompletableFuture.supplyAsync(() -> {
            log("Reading local file...");
            sleep(300); // simulate delay
            return "local-data";
        });

        // Combine both results (callback style)
        remoteFileFuture.thenCombine(localFileFuture, (remote, local) -> {
            log("Combining results...");
            return remote + " + " + local;
        }).thenAccept(result -> {
            log("Sending response...");
            sleep(200); // simulate delay
            System.out.println("[Result Ignored]");
        }).exceptionally(ex -> {
            log("Error occurred: " + ex.getMessage());
            return null;
        });

        // Keep main thread alive until tasks finish
        sleep(2000);
    }

    // Utility method to log with thread info
    static void log(String message) {
        System.out.printf("[%s] %s%n", Thread.currentThread().getName(), message);
    }

    // Helper to simulate delay
    static void sleep(long millis) {
        try {
            Thread.sleep(millis);
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        }
    }
}

   

Sample Output

[main] Starting async tasks...
[ForkJoinPool.commonPool-worker-1] Downloading remote file...
[ForkJoinPool.commonPool-worker-2] Reading local file...
[ForkJoinPool.commonPool-worker-1] Combining results...
[ForkJoinPool.commonPool-worker-1] Sending response...
[Result Ignored]

   

To make the application simpler, I ignored printing the content from urla and from the file.

 

Here:

·      main starts and schedules two tasks with CompletableFuture.supplyAsync.

·      The ForkJoin common pool picks up tasks:

·      Worker-1 runs “Downloading remote file…”

·      Worker-2 runs “Reading local file…”

·      Once both finish, thenCombine runs, and ForkJoin reuses Worker-1 for that callback.

·      The final thenAccept also happens on Worker-1.

 

In Non-blocking I/O:

·      The thread initiates the I/O and then returns immediately to the pool.

·      The actual I/O (network, disk) happens in the OS, not on your thread.

·      When the OS signals data is ready, the framework schedules a new task onto any available thread in the pool.

·      That callback may run on the same thread or a different one, depending on pool availability.

 

Imagine a restaurant:

 

·      Blocking I/O: A waiter takes an order and waits at the kitchen door until the food is ready. No other tables get served by that waiter in the meantime.

·      Non-blocking I/O: A waiter takes an order, gives it to the kitchen, then immediately goes to serve other tables. When the food is ready, any free waiter can deliver it to the customer.

 

 

7. Introducing Virtual Threads

In Java, every program runs on at least one thread, the main thread. If you want your program to do many things at the same time (for example, downloading files, serving web requests, or handling chat messages), you create more threads.

 

Traditionally, Java has only had platform threads (also called OS threads). These are directly managed by the operating system. They are heavyweight each one takes up a lot of memory and system resources. This is why most programs crash if you try to create more than a few thousand threads.

 

Virtual threads are a new type of thread introduced with Project Loom. They behave like normal threads in your code (they run Runnable tasks, can sleep, block, and be joined), but under the hood they are much lighter. You can create thousands or even millions of them without overwhelming your system.

 

How to Create a Virtual Thread?

Creating a virtual thread is almost as easy as creating a normal thread. Here’s the normal way (platform thread):

 

new Thread(() -> {
    System.out.println("Hello from a normal thread!");
}).start();

   

Here’s the virtual thread way:

 

Thread.startVirtualThread(() -> {
    System.out.println("Hello from a virtual thread!");
});

   

That’s it, just one method call difference.

 

HelloVirtualThread.java

 

package com.sample.app.virtual.threads;

public class HelloVirtualThread {
  
  public static void main(String[] args) throws InterruptedException {
    Thread thread = Thread.startVirtualThread(() -> {
        System.out.println("Hello from a virtual thread!");
    });

    thread.join();
  }

}

   

Output

 

Hello from a virtual thread!

   

Are Virtual Threads Daemon Threads?

Yes, virtual threads are daemon threads by default.

 

What does that mean?

A daemon thread is like a background helper. The JVM does not wait for daemon threads to finish when shutting down. If your program only has daemon threads running, the JVM will exit immediately after the main thread ends.

 

HelloVirtualThread.java

package com.sample.app.virtual.threads;

public class HelloVirtualThread {
  
  public static void main(String[] args) throws InterruptedException {
    Thread thread = Thread.startVirtualThread(() -> {
      System.out.println("Is current thread daemon : " + Thread.currentThread().isDaemon());
        System.out.println("Hello from a virtual thread!");
    });

    thread.join();
  }

}

   

Output

 

Is current thread daemon : true
Hello from a virtual thread!

   

Virtual threads behave like normal threads in your code but are much cheaper to create. By default, virtual threads are daemon threads, so the JVM won’t wait for them unless you join() or otherwise keep the program alive. Virtual threads allow Java programs to handle massive concurrency (e.g., thousands or millions of tasks).

 

7.1 Understanding Virtual Threads Internals

When you create a normal Java thread (often called a platform thread), Java directly connects it to an operating system (OS) thread. This is expensive because OS threads consume a lot of memory and resources. That’s why when we experimented with platform threads, we can able to create a few thousand platform threads before your program slows down or crashes.

 

Let's take an example like this:

void handleUserRequest() {
    System.out.println("Start handling request");
    try {
        readFromApi(); // Assume this API is slow and taking 10 seconds to return response
    } catch (InterruptedException e) {
        Thread.currentThread().interrupt();
    }
    System.out.println("Finished handling request");
}

   

Here:

·      System.out.println() takes a little CPU time.

·      readFromApi() blocks the thread for 10 seconds by calling a REST API.

 

The problem here is, while this thread is "blocked" the OS thread it’s tied to is also stuck doing nothing. It’s like booking a taxi but then making the driver wait for 10 minutes while you’re inside a shop. That driver (the OS thread) can’t help anyone else during that time.

 

If a laksh of requests all made wait 10 seconds each, your program would collapse.

 

How Virtual Threads address this issue?

Instead of tying each Java thread directly to an OS thread, virtual threads are managed by the JVM (Java Virtual Machine). Here’s the idea:

 

·      When your virtual thread is doing CPU work (like processing data), the JVM assigns it to a real OS thread (a platform thread).

·      When your virtual thread is waiting (like during Thread.sleep() or a database call), the JVM detaches it from the OS thread. This frees up the OS thread so it can help other tasks.

·      Later, when the waiting is over, the JVM reschedules the virtual thread on any available OS thread and continues running.

 

That means you can have millions of virtual threads waiting on I/O without wasting OS threads.

 

MaxVirtualThreadsCheck.java

import java.util.ArrayList;
import java.util.List;

public class MaxVirtualThreadsCheck {

    public static void main(String[] args) throws InterruptedException {
        if (args.length != 1) {
            System.out.println("Usage: java -Xmx1G MaxVirtualThreadsCheck <noOfThreadsToTest>");
            return;
        }

        int noOfThreadsToTest;
        try {
            noOfThreadsToTest = Integer.parseInt(args[0]);
        } catch (NumberFormatException e) {
            System.out.println("Please provide a valid integer for number of threads.");
            return;
        }

        System.out.println("Starting virtual thread test with " + noOfThreadsToTest + " threads...");

        // Store references to all virtual threads
        List<Thread> threads = new ArrayList<>();

        for (int i = 1; i <= noOfThreadsToTest; i++) {
            final int threadNumber = i;

            try {
                Thread t = Thread.ofVirtual().start(() -> {
                    System.out.println("Virtual Thread " + threadNumber + " started (ID: "
                            + Thread.currentThread().threadId() + ")");
                    try {
                        Thread.sleep(10_000); // simulate some blocking work
                    } catch (InterruptedException e) {
                        Thread.currentThread().interrupt();
                        System.out.println("Virtual Thread " + threadNumber + " interrupted");
                    }
                    System.out.println("Virtual Thread " + threadNumber + " finished");
                });

                threads.add(t); // keep reference

            } catch (OutOfMemoryError | Exception e) {
                System.out.println("Failed to create virtual thread " + threadNumber + ": " + e.getMessage());
                break;
            }
        }

        System.out.println("Waiting for all virtual threads to finish...");

        // Wait for all threads to finish
        for (Thread t : threads) {
            t.join();
        }

        System.out.println("All virtual threads finished.");
    }
}

   

Compile the Application and run the application with a million threads to check.

 

java -Xmx1G MaxVirtualThreadsCheck 1000000

 

Virtual threads are a game changer in Java’s concurrency model. With traditional platform threads, every Java thread maps directly to an operating system thread, which makes them heavy and limits scalability. Virtual threads break this limitation by letting the JVM manage them more efficiently:

 

·      When the task needs CPU, it’s scheduled on an OS thread.

·      When the task waits (like during I/O), the OS thread is released so it can help other tasks.

 

This means we can run millions of virtual threads without overloading the system, while still writing code that looks and feels just like regular threaded Java code. No need to dive into complex asynchronous callbacks or reactive frameworks, the programming model stays simple and familiar.

 

In real world applications like web servers, databases, or messaging systems, this allows Java programs to handle huge amounts of concurrent work while using fewer resources.

  

Previous                                                    Next                                                    Home