JMH: State objects

State objects encapsulate the state on which benchmark is working on.

 

For example, I want to know how many times a method gets executed while performing the benchmark. To do this, I can maintain a counter that can be shared across all the benchmark execution.

 

State variables are declared in special state classes, and an instance of that state class can then be provided as a parameter to the benchmark method.

 

Step 1: Define a State class.

@State(Scope.Benchmark)
public static class BenchmarkState {
    private AtomicLong totalMethodExecutionCount;

    @Setup(Level.Trial)
    public void doSetup() {
        totalMethodExecutionCount = new AtomicLong(0);
        System.out.println("Do Setup");
    }

    @TearDown(Level.Trial)
    public void doTearDown() {
        System.out.println("Benchmark Method executed " + totalMethodExecutionCount.longValue() + " times");
    }
}

 

Step 2: Pass the BenchmarkState variable as an argument to benchmark method.

@Benchmark
@BenchmarkMode(Mode.AverageTime)
public void measureShared1(BenchmarkState state) throws InterruptedException {
    TimeUnit.MILLISECONDS.sleep(100);
    state.totalMethodExecutionCount.getAndIncrement();
}

 

With this logic, I can check how many times the method ‘measureShared1’ is called by benchmarking.

 

Find the below working application.

 

BenchmarkStateDemo.java

package com.sample.app;

import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicLong;

import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.BenchmarkMode;
import org.openjdk.jmh.annotations.Level;
import org.openjdk.jmh.annotations.Mode;
import org.openjdk.jmh.annotations.Scope;
import org.openjdk.jmh.annotations.Setup;
import org.openjdk.jmh.annotations.State;
import org.openjdk.jmh.annotations.TearDown;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.RunnerException;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;

public class BenchmarkStateDemo {

    @State(Scope.Benchmark)
    public static class BenchmarkState {
        private AtomicLong totalMethodExecutionCount;
        private String methodName = null;

        @Setup(Level.Trial)
        public void doSetup() {
            totalMethodExecutionCount = new AtomicLong(0);
            System.out.println("Do Setup");
        }

        @TearDown(Level.Trial)
        public void doTearDown() {
            System.out.println("Benchmark Method " + methodName + " executed " + totalMethodExecutionCount.longValue() + " times");
        }
    }

    @Benchmark
    @BenchmarkMode(Mode.AverageTime)
    public void measureShared1(BenchmarkState state) throws InterruptedException {
        TimeUnit.MILLISECONDS.sleep(100);
        state.methodName = "measureShared1";
        state.totalMethodExecutionCount.getAndIncrement();
    }
    
    @Benchmark
    @BenchmarkMode(Mode.AverageTime)
    public void measureShared2(BenchmarkState state) throws InterruptedException {
        TimeUnit.MILLISECONDS.sleep(200);
        state.methodName = "measureShared2";
        state.totalMethodExecutionCount.getAndIncrement();
    }

    public static void main(String[] args) throws RunnerException {
        Options opt = new OptionsBuilder()
                .include(BenchmarkStateDemo.class.getSimpleName())
                .forks(1)
                .measurementIterations(2)
                .warmupIterations(2).build();

        Runner runner = new Runner(opt);
        runner.run();

    }
}

 

Output

# JMH version: 1.35
# VM version: JDK 15.0.2, Java HotSpot(TM) 64-Bit Server VM, 15.0.2+7-27
# VM invoker: /Library/Java/JavaVirtualMachines/jdk-15.0.2.jdk/Contents/Home/bin/java
# VM options: -Dfile.encoding=UTF-8 -XX:+ShowCodeDetailsInExceptionMessages
# Blackhole mode: full + dont-inline hint (auto-detected, use -Djmh.blackhole.autoDetect=false to disable)
# Warmup: 2 iterations, 10 s each
# Measurement: 2 iterations, 10 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Average time, time/op
# Benchmark: com.sample.app.BenchmarkStateDemo.measureShared1

# Run progress: 0.00% complete, ETA 00:01:20
# Fork: 1 of 1
# Warmup Iteration   1: Do Setup
0.103 s/op
# Warmup Iteration   2: 0.103 s/op
Iteration   1: 0.103 s/op
Iteration   2: Benchmark Method measureShared1 executed 392 times
0.103 s/op


Result "com.sample.app.BenchmarkStateDemo.measureShared1":
  0.103 s/op


# JMH version: 1.35
# VM version: JDK 15.0.2, Java HotSpot(TM) 64-Bit Server VM, 15.0.2+7-27
# VM invoker: /Library/Java/JavaVirtualMachines/jdk-15.0.2.jdk/Contents/Home/bin/java
# VM options: -Dfile.encoding=UTF-8 -XX:+ShowCodeDetailsInExceptionMessages
# Blackhole mode: full + dont-inline hint (auto-detected, use -Djmh.blackhole.autoDetect=false to disable)
# Warmup: 2 iterations, 10 s each
# Measurement: 2 iterations, 10 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Average time, time/op
# Benchmark: com.sample.app.BenchmarkStateDemo.measureShared2

# Run progress: 50.00% complete, ETA 00:00:45
# Fork: 1 of 1
# Warmup Iteration   1: Do Setup
0.203 s/op
# Warmup Iteration   2: 0.203 s/op
Iteration   1: 0.203 s/op
Iteration   2: Benchmark Method measureShared2 executed 200 times
0.203 s/op


Result "com.sample.app.BenchmarkStateDemo.measureShared2":
  0.203 s/op


# Run complete. Total time: 00:01:31

REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on
why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial
experiments, perform baseline and negative tests that provide experimental control, make sure
the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts.
Do not assume the numbers tell you what you want them to tell.

Benchmark                          Mode  Cnt  Score   Error  Units
BenchmarkStateDemo.measureShared1  avgt    2  0.103           s/op
BenchmarkStateDemo.measureShared2  avgt    2  0.203           s/op

From the output, you can see the below lines.

Iteration   2: Benchmark Method measureShared1 executed 392 times
Iteration   2: Benchmark Method measureShared2 executed 200 times

 

A state object can have following scopes.

 

a.   Benchmark: Benchmark state scope. All threads running the benchmark share the same state object.

b.   Group: Group state scope. Each thread group running the benchmark will create its own instance of the state object.

c.    Thread: Each thread running the benchmark will create its own instance of the state object.

 

Let’s confirm it by below example.

 

StatesDemo.java

package com.sample.app;

import java.util.concurrent.TimeUnit;

import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.BenchmarkMode;
import org.openjdk.jmh.annotations.Mode;
import org.openjdk.jmh.annotations.Scope;
import org.openjdk.jmh.annotations.State;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.RunnerException;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;

public class StatesDemo {
    
    @State(Scope.Benchmark)
    public static class BenchmarkState {
    }

    @State(Scope.Thread)
    public static class ThreadState {
    }

    @Benchmark
    @BenchmarkMode(Mode.AverageTime)
    public void benchmarkState(BenchmarkState benchmarkState) throws InterruptedException {
        TimeUnit.SECONDS.sleep(1);
        String threadName = Thread.currentThread().getName();
        String threadGroupName = Thread.currentThread().getThreadGroup().getName();

        System.out.println("threadName : %s, threadGroupName %s, benchmarkState : %s".formatted(threadName,
                threadGroupName, benchmarkState));
    }

    @Benchmark
    @BenchmarkMode(Mode.AverageTime)
    public void threadState(ThreadState threadState) throws InterruptedException {
        TimeUnit.SECONDS.sleep(1);
        String threadName = Thread.currentThread().getName();
        String threadGroupName = Thread.currentThread().getThreadGroup().getName();

        System.out.println("threadName : %s, threadGroupName %s, threadState : %s".formatted(threadName,
                threadGroupName, threadState));
    }

    public static void main(String[] args) throws RunnerException {
        Options opt = new OptionsBuilder()
                .include(StatesDemo.class.getSimpleName())
                .threads(3)
                .threadGroups(2)
                .forks(1)
                .measurementIterations(1)
                .warmupIterations(1)
                .build();
        

        Runner runner = new Runner(opt);
        runner.run();
     
    }
}

 

Run the above application, you will observe following messages in the console.

threadName : com.sample.app.StatesDemo.benchmarkState-jmh-worker-1, threadGroupName main, benchmarkState : com.sample.app.jmh_generated.StatesDemo_BenchmarkState_jmhType@148ccf8e
threadName : com.sample.app.StatesDemo.benchmarkState-jmh-worker-2, threadGroupName main, benchmarkState : com.sample.app.jmh_generated.StatesDemo_BenchmarkState_jmhType@148ccf8e
threadName : com.sample.app.StatesDemo.benchmarkState-jmh-worker-4, threadGroupName main, benchmarkState : com.sample.app.jmh_generated.StatesDemo_BenchmarkState_jmhType@148ccf8e
threadName : com.sample.app.StatesDemo.benchmarkState-jmh-worker-2, threadGroupName main, benchmarkState : com.sample.app.jmh_generated.StatesDemo_BenchmarkState_jmhType@148ccf8e

 

Same benchmark state object ‘com.sample.app.jmh_generated.StatesDemo_BenchmarkState_jmhType@148ccf8e’ is shared across all the method invocations.

 

threadName : com.sample.app.StatesDemo.threadState-jmh-worker-4, threadGroupName main, threadState : com.sample.app.jmh_generated.StatesDemo_ThreadState_jmhType@148ccf8e
threadName : com.sample.app.StatesDemo.threadState-jmh-worker-3, threadGroupName main, threadState : com.sample.app.jmh_generated.StatesDemo_ThreadState_jmhType@32263bb4
threadName : com.sample.app.StatesDemo.threadState-jmh-worker-2, threadGroupName main, threadState : com.sample.app.jmh_generated.StatesDemo_ThreadState_jmhType@3e078932
threadName : com.sample.app.StatesDemo.threadState-jmh-worker-1, threadGroupName main, threadState : com.sample.app.jmh_generated.StatesDemo_ThreadState_jmhType@261a9ff6
threadName : com.sample.app.StatesDemo.threadState-jmh-worker-4, threadGroupName main, threadState : com.sample.app.jmh_generated.StatesDemo_ThreadState_jmhType@148ccf8e
threadName : com.sample.app.StatesDemo.threadState-jmh-worker-3, threadGroupName main, threadState : com.sample.app.jmh_generated.StatesDemo_ThreadState_jmhType@32263bb4
threadName : com.sample.app.StatesDemo.threadState-jmh-worker-2, threadGroupName main, threadState : com.sample.app.jmh_generated.StatesDemo_ThreadState_jmhType@3e078932
threadName : com.sample.app.StatesDemo.threadState-jmh-worker-1, threadGroupName main, threadState : com.sample.app.jmh_generated.StatesDemo_ThreadState_jmhType@261a9ff6
threadName : com.sample.app.StatesDemo.threadState-jmh-worker-4, threadGroupName main, threadState : com.sample.app.jmh_generated.StatesDemo_ThreadState_jmhType@148ccf8e

 

Each worker thread maintains a separate ThreadState object.

 

Previous                                                 Next                                                 Home