Tuesday, 2 August 2022

JMH: Benchmarking modes

At the time of writing this post, JMH support 4 benchmarking modes.

 

a. Throughput: operations per unit of time. It Measures the raw throughput by continuously calling the benchmark method in a time-bound iteration, and counting how many times we executed the method.

 

b. AverageTime: average time per  operation. It measures the average execution time, and it does it in the way similar to Mode.Throughput.

 

c. SampleTime: samples the time for each operation. With this mode, we are still running the method in a time-bound iteration, but instead of measuring the total time, we measure the time spent in some of the benchmark method calls.

 

d. SingleShotTime: measures the time for a single operation. measures the single method invocation time. We do only the single benchmark method invocation. The iteration time is meaningless in this mode: as soon as benchmark method stops, the iteration is over.

 


How to enable Throughput mode?

Annotate the method with ‘@BenchmarkMode(Mode.Throughput)’.

 

Example

@Benchmark
@BenchmarkMode(Mode.Throughput)
public void measureThroughput() throws InterruptedException {
    TimeUnit.MILLISECONDS.sleep(100);
}

 

How to enable AverageTime mode?

@Benchmark
@BenchmarkMode(Mode.AverageTime)
public void measureAverageTime() {
    sleepNMilliSeconds(100);
}

 

How to enable SampleTime mode?

@Benchmark
@BenchmarkMode(Mode.SampleTime)
public void measureSampleTime() {
    sleepNMilliSeconds(100);
}

How to enable SingleShotTime mode?

@Benchmark
@BenchmarkMode(Mode.SingleShotTime)
public void measureSingleShotTime() {
    sleepNMilliSeconds(100);
}

Find the below working application.

 

BenchmarkingModes.java

package com.sample.app;

import java.util.concurrent.TimeUnit;

import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.BenchmarkMode;
import org.openjdk.jmh.annotations.Mode;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.RunnerException;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;

public class BenchmarkingModes {

    private void sleepNMilliSeconds(int n) {
        try {
            TimeUnit.MILLISECONDS.sleep(100);
        } catch (InterruptedException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
    }

    @Benchmark
    @BenchmarkMode(Mode.Throughput)
    public void measureThroughput() {
        sleepNMilliSeconds(50);
    }

    @Benchmark
    @BenchmarkMode(Mode.AverageTime)
    public void measureAverageTime() {
        sleepNMilliSeconds(50);
    }

    @Benchmark
    @BenchmarkMode(Mode.SampleTime)
    public void measureSampleTime() {
        sleepNMilliSeconds(50);
    }

    @Benchmark
    @BenchmarkMode(Mode.SingleShotTime)
    public void measureSingleShotTime() {
        sleepNMilliSeconds(50);
    }

    public static void main(String[] args) throws RunnerException {
        Options opt = new OptionsBuilder()
                .include(BenchmarkingModes.class.getSimpleName())
                .forks(1)
                .measurementIterations(4)
                .warmupIterations(3)
                .build();

        new Runner(opt).run();
    }
}

Output

# JMH version: 1.35
# VM version: JDK 15.0.2, Java HotSpot(TM) 64-Bit Server VM, 15.0.2+7-27
# VM invoker: /Library/Java/JavaVirtualMachines/jdk-15.0.2.jdk/Contents/Home/bin/java
# VM options: -Dfile.encoding=UTF-8 -XX:+ShowCodeDetailsInExceptionMessages
# Blackhole mode: full + dont-inline hint (auto-detected, use -Djmh.blackhole.autoDetect=false to disable)
# Warmup: 3 iterations, 10 s each
# Measurement: 4 iterations, 10 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: com.sample.app.BenchmarkingModes.measureThroughput

# Run progress: 0.00% complete, ETA 00:03:30
# Fork: 1 of 1
# Warmup Iteration   1: 18.887 ops/s
# Warmup Iteration   2: 19.020 ops/s
# Warmup Iteration   3: 18.874 ops/s
Iteration   1: 18.848 ops/s
Iteration   2: 18.795 ops/s
Iteration   3: 18.930 ops/s
Iteration   4: 18.932 ops/s


Result "com.sample.app.BenchmarkingModes.measureThroughput":
  18.876 ±(99.9%) 0.433 ops/s [Average]
  (min, avg, max) = (18.795, 18.876, 18.932), stdev = 0.067
  CI (99.9%): [18.443, 19.310] (assumes normal distribution)


# JMH version: 1.35
# VM version: JDK 15.0.2, Java HotSpot(TM) 64-Bit Server VM, 15.0.2+7-27
# VM invoker: /Library/Java/JavaVirtualMachines/jdk-15.0.2.jdk/Contents/Home/bin/java
# VM options: -Dfile.encoding=UTF-8 -XX:+ShowCodeDetailsInExceptionMessages
# Blackhole mode: full + dont-inline hint (auto-detected, use -Djmh.blackhole.autoDetect=false to disable)
# Warmup: 3 iterations, 10 s each
# Measurement: 4 iterations, 10 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Average time, time/op
# Benchmark: com.sample.app.BenchmarkingModes.measureAverageTime

# Run progress: 33.33% complete, ETA 00:02:20
# Fork: 1 of 1
# Warmup Iteration   1: 0.053 s/op
# Warmup Iteration   2: 0.053 s/op
# Warmup Iteration   3: 0.053 s/op
Iteration   1: 0.053 s/op
Iteration   2: 0.053 s/op
Iteration   3: 0.052 s/op
Iteration   4: 0.053 s/op


Result "com.sample.app.BenchmarkingModes.measureAverageTime":
  0.053 ±(99.9%) 0.001 s/op [Average]
  (min, avg, max) = (0.052, 0.053, 0.053), stdev = 0.001
  CI (99.9%): [0.051, 0.054] (assumes normal distribution)


# JMH version: 1.35
# VM version: JDK 15.0.2, Java HotSpot(TM) 64-Bit Server VM, 15.0.2+7-27
# VM invoker: /Library/Java/JavaVirtualMachines/jdk-15.0.2.jdk/Contents/Home/bin/java
# VM options: -Dfile.encoding=UTF-8 -XX:+ShowCodeDetailsInExceptionMessages
# Blackhole mode: full + dont-inline hint (auto-detected, use -Djmh.blackhole.autoDetect=false to disable)
# Warmup: 3 iterations, 10 s each
# Measurement: 4 iterations, 10 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Sampling time
# Benchmark: com.sample.app.BenchmarkingModes.measureSampleTime

# Run progress: 66.66% complete, ETA 00:01:10
# Fork: 1 of 1
# Warmup Iteration   1: 0.053 ±(99.9%) 0.001 s/op
# Warmup Iteration   2: 0.053 ±(99.9%) 0.001 s/op
# Warmup Iteration   3: 0.053 ±(99.9%) 0.001 s/op
Iteration   1: 0.053 ±(99.9%) 0.001 s/op
                 measureSampleTime·p0.00:   0.050 s/op
                 measureSampleTime·p0.50:   0.054 s/op
                 measureSampleTime·p0.90:   0.055 s/op
                 measureSampleTime·p0.95:   0.055 s/op
                 measureSampleTime·p0.99:   0.055 s/op
                 measureSampleTime·p0.999:  0.055 s/op
                 measureSampleTime·p0.9999: 0.055 s/op
                 measureSampleTime·p1.00:   0.055 s/op

Iteration   2: 0.053 ±(99.9%) 0.001 s/op
                 measureSampleTime·p0.00:   0.050 s/op
                 measureSampleTime·p0.50:   0.054 s/op
                 measureSampleTime·p0.90:   0.055 s/op
                 measureSampleTime·p0.95:   0.055 s/op
                 measureSampleTime·p0.99:   0.055 s/op
                 measureSampleTime·p0.999:  0.055 s/op
                 measureSampleTime·p0.9999: 0.055 s/op
                 measureSampleTime·p1.00:   0.055 s/op

Iteration   3: 0.053 ±(99.9%) 0.001 s/op
                 measureSampleTime·p0.00:   0.050 s/op
                 measureSampleTime·p0.50:   0.053 s/op
                 measureSampleTime·p0.90:   0.055 s/op
                 measureSampleTime·p0.95:   0.055 s/op
                 measureSampleTime·p0.99:   0.055 s/op
                 measureSampleTime·p0.999:  0.055 s/op
                 measureSampleTime·p0.9999: 0.055 s/op
                 measureSampleTime·p1.00:   0.055 s/op

Iteration   4: 0.053 ±(99.9%) 0.001 s/op
                 measureSampleTime·p0.00:   0.050 s/op
                 measureSampleTime·p0.50:   0.053 s/op
                 measureSampleTime·p0.90:   0.055 s/op
                 measureSampleTime·p0.95:   0.055 s/op
                 measureSampleTime·p0.99:   0.055 s/op
                 measureSampleTime·p0.999:  0.055 s/op
                 measureSampleTime·p0.9999: 0.055 s/op
                 measureSampleTime·p1.00:   0.055 s/op



Result "com.sample.app.BenchmarkingModes.measureSampleTime":
  N = 755
  mean =      0.053 ±(99.9%) 0.001 s/op

  Histogram, s/op:
    [0.050, 0.051) = 72 
    [0.051, 0.051) = 59 
    [0.051, 0.052) = 47 
    [0.052, 0.052) = 51 
    [0.052, 0.053) = 28 
    [0.053, 0.053) = 47 
    [0.053, 0.054) = 67 
    [0.054, 0.054) = 68 
    [0.054, 0.055) = 123 
    [0.055, 0.055) = 90 
    [0.055, 0.056) = 103 

  Percentiles, s/op:
      p(0.0000) =      0.050 s/op
     p(50.0000) =      0.054 s/op
     p(90.0000) =      0.055 s/op
     p(95.0000) =      0.055 s/op
     p(99.0000) =      0.055 s/op
     p(99.9000) =      0.055 s/op
     p(99.9900) =      0.055 s/op
     p(99.9990) =      0.055 s/op
     p(99.9999) =      0.055 s/op
    p(100.0000) =      0.055 s/op


# JMH version: 1.35
# VM version: JDK 15.0.2, Java HotSpot(TM) 64-Bit Server VM, 15.0.2+7-27
# VM invoker: /Library/Java/JavaVirtualMachines/jdk-15.0.2.jdk/Contents/Home/bin/java
# VM options: -Dfile.encoding=UTF-8 -XX:+ShowCodeDetailsInExceptionMessages
# Blackhole mode: full + dont-inline hint (auto-detected, use -Djmh.blackhole.autoDetect=false to disable)
# Warmup: 3 iterations, single-shot each
# Measurement: 4 iterations, single-shot each
# Timeout: 10 min per iteration
# Threads: 1 thread
# Benchmark mode: Single shot invocation time
# Benchmark: com.sample.app.BenchmarkingModes.measureSingleShotTime

# Run progress: 100.00% complete, ETA 00:00:00
# Fork: 1 of 1
# Warmup Iteration   1: 0.054 s/op
# Warmup Iteration   2: 0.050 s/op
# Warmup Iteration   3: 0.055 s/op
Iteration   1: 0.055 s/op
Iteration   2: 0.053 s/op
Iteration   3: 0.054 s/op
Iteration   4: 0.052 s/op


Result "com.sample.app.BenchmarkingModes.measureSingleShotTime":
  N = 4
  mean =      0.053 ±(99.9%) 0.008 s/op

  Histogram, s/op:
    [0.052, 0.052) = 1 
    [0.052, 0.053) = 0 
    [0.053, 0.053) = 0 
    [0.053, 0.053) = 0 
    [0.053, 0.053) = 1 
    [0.053, 0.054) = 0 
    [0.054, 0.054) = 1 
    [0.054, 0.054) = 0 
    [0.054, 0.054) = 0 
    [0.054, 0.055) = 0 
    [0.055, 0.055) = 0 
    [0.055, 0.055) = 0 
    [0.055, 0.055) = 1 
    [0.055, 0.056) = 0 
    [0.056, 0.056) = 0 

  Percentiles, s/op:
      p(0.0000) =      0.052 s/op
     p(50.0000) =      0.053 s/op
     p(90.0000) =      0.055 s/op
     p(95.0000) =      0.055 s/op
     p(99.0000) =      0.055 s/op
     p(99.9000) =      0.055 s/op
     p(99.9900) =      0.055 s/op
     p(99.9990) =      0.055 s/op
     p(99.9999) =      0.055 s/op
    p(100.0000) =      0.055 s/op


# Run complete. Total time: 00:03:31

REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on
why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial
experiments, perform baseline and negative tests that provide experimental control, make sure
the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts.
Do not assume the numbers tell you what you want them to tell.

Benchmark                                                        Mode  Cnt   Score    Error  Units
BenchmarkingModes.measureThroughput                             thrpt    4  18.876 ±  0.433  ops/s
BenchmarkingModes.measureAverageTime                             avgt    4   0.053 ±  0.001   s/op
BenchmarkingModes.measureSampleTime                            sample  755   0.053 ±  0.001   s/op
BenchmarkingModes.measureSampleTime:measureSampleTime·p0.00    sample        0.050            s/op
BenchmarkingModes.measureSampleTime:measureSampleTime·p0.50    sample        0.054            s/op
BenchmarkingModes.measureSampleTime:measureSampleTime·p0.90    sample        0.055            s/op
BenchmarkingModes.measureSampleTime:measureSampleTime·p0.95    sample        0.055            s/op
BenchmarkingModes.measureSampleTime:measureSampleTime·p0.99    sample        0.055            s/op
BenchmarkingModes.measureSampleTime:measureSampleTime·p0.999   sample        0.055            s/op
BenchmarkingModes.measureSampleTime:measureSampleTime·p0.9999  sample        0.055            s/op
BenchmarkingModes.measureSampleTime:measureSampleTime·p1.00    sample        0.055            s/op
BenchmarkingModes.measureSingleShotTime                            ss    4   0.053 ±  0.008   s/op






 

 

Previous                                                 Next                                                 Home

No comments:

Post a Comment