At the time of writing this post, JMH support 4 benchmarking modes.
a. Throughput: operations per unit of time. It Measures the raw throughput by continuously calling the benchmark method in a time-bound iteration, and counting how many times we executed the method.
b. AverageTime: average time per operation. It measures the average execution time, and it does it in the way similar to Mode.Throughput.
c. SampleTime: samples the time for each operation. With this mode, we are still running the method in a time-bound iteration, but instead of measuring the total time, we measure the time spent in some of the benchmark method calls.
d. SingleShotTime: measures the time for a single operation. measures the single method invocation time. We do only the single benchmark method invocation. The iteration time is meaningless in this mode: as soon as benchmark method stops, the iteration is over.
How to enable Throughput mode?
Annotate the method with ‘@BenchmarkMode(Mode.Throughput)’.
Example
@Benchmark
@BenchmarkMode(Mode.Throughput)
public void measureThroughput() throws InterruptedException {
TimeUnit.MILLISECONDS.sleep(100);
}
How to enable AverageTime mode?
@Benchmark
@BenchmarkMode(Mode.AverageTime)
public void measureAverageTime() {
sleepNMilliSeconds(100);
}
How to enable SampleTime mode?
@Benchmark
@BenchmarkMode(Mode.SampleTime)
public void measureSampleTime() {
sleepNMilliSeconds(100);
}
How to enable SingleShotTime mode?
@Benchmark
@BenchmarkMode(Mode.SingleShotTime)
public void measureSingleShotTime() {
sleepNMilliSeconds(100);
}
Find the below working application.
BenchmarkingModes.java
package com.sample.app;
import java.util.concurrent.TimeUnit;
import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.BenchmarkMode;
import org.openjdk.jmh.annotations.Mode;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.RunnerException;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;
public class BenchmarkingModes {
private void sleepNMilliSeconds(int n) {
try {
TimeUnit.MILLISECONDS.sleep(100);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
@Benchmark
@BenchmarkMode(Mode.Throughput)
public void measureThroughput() {
sleepNMilliSeconds(50);
}
@Benchmark
@BenchmarkMode(Mode.AverageTime)
public void measureAverageTime() {
sleepNMilliSeconds(50);
}
@Benchmark
@BenchmarkMode(Mode.SampleTime)
public void measureSampleTime() {
sleepNMilliSeconds(50);
}
@Benchmark
@BenchmarkMode(Mode.SingleShotTime)
public void measureSingleShotTime() {
sleepNMilliSeconds(50);
}
public static void main(String[] args) throws RunnerException {
Options opt = new OptionsBuilder()
.include(BenchmarkingModes.class.getSimpleName())
.forks(1)
.measurementIterations(4)
.warmupIterations(3)
.build();
new Runner(opt).run();
}
}
Output
# JMH version: 1.35
# VM version: JDK 15.0.2, Java HotSpot(TM) 64-Bit Server VM, 15.0.2+7-27
# VM invoker: /Library/Java/JavaVirtualMachines/jdk-15.0.2.jdk/Contents/Home/bin/java
# VM options: -Dfile.encoding=UTF-8 -XX:+ShowCodeDetailsInExceptionMessages
# Blackhole mode: full + dont-inline hint (auto-detected, use -Djmh.blackhole.autoDetect=false to disable)
# Warmup: 3 iterations, 10 s each
# Measurement: 4 iterations, 10 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: com.sample.app.BenchmarkingModes.measureThroughput
# Run progress: 0.00% complete, ETA 00:03:30
# Fork: 1 of 1
# Warmup Iteration 1: 18.887 ops/s
# Warmup Iteration 2: 19.020 ops/s
# Warmup Iteration 3: 18.874 ops/s
Iteration 1: 18.848 ops/s
Iteration 2: 18.795 ops/s
Iteration 3: 18.930 ops/s
Iteration 4: 18.932 ops/s
Result "com.sample.app.BenchmarkingModes.measureThroughput":
18.876 ±(99.9%) 0.433 ops/s [Average]
(min, avg, max) = (18.795, 18.876, 18.932), stdev = 0.067
CI (99.9%): [18.443, 19.310] (assumes normal distribution)
# JMH version: 1.35
# VM version: JDK 15.0.2, Java HotSpot(TM) 64-Bit Server VM, 15.0.2+7-27
# VM invoker: /Library/Java/JavaVirtualMachines/jdk-15.0.2.jdk/Contents/Home/bin/java
# VM options: -Dfile.encoding=UTF-8 -XX:+ShowCodeDetailsInExceptionMessages
# Blackhole mode: full + dont-inline hint (auto-detected, use -Djmh.blackhole.autoDetect=false to disable)
# Warmup: 3 iterations, 10 s each
# Measurement: 4 iterations, 10 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Average time, time/op
# Benchmark: com.sample.app.BenchmarkingModes.measureAverageTime
# Run progress: 33.33% complete, ETA 00:02:20
# Fork: 1 of 1
# Warmup Iteration 1: 0.053 s/op
# Warmup Iteration 2: 0.053 s/op
# Warmup Iteration 3: 0.053 s/op
Iteration 1: 0.053 s/op
Iteration 2: 0.053 s/op
Iteration 3: 0.052 s/op
Iteration 4: 0.053 s/op
Result "com.sample.app.BenchmarkingModes.measureAverageTime":
0.053 ±(99.9%) 0.001 s/op [Average]
(min, avg, max) = (0.052, 0.053, 0.053), stdev = 0.001
CI (99.9%): [0.051, 0.054] (assumes normal distribution)
# JMH version: 1.35
# VM version: JDK 15.0.2, Java HotSpot(TM) 64-Bit Server VM, 15.0.2+7-27
# VM invoker: /Library/Java/JavaVirtualMachines/jdk-15.0.2.jdk/Contents/Home/bin/java
# VM options: -Dfile.encoding=UTF-8 -XX:+ShowCodeDetailsInExceptionMessages
# Blackhole mode: full + dont-inline hint (auto-detected, use -Djmh.blackhole.autoDetect=false to disable)
# Warmup: 3 iterations, 10 s each
# Measurement: 4 iterations, 10 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Sampling time
# Benchmark: com.sample.app.BenchmarkingModes.measureSampleTime
# Run progress: 66.66% complete, ETA 00:01:10
# Fork: 1 of 1
# Warmup Iteration 1: 0.053 ±(99.9%) 0.001 s/op
# Warmup Iteration 2: 0.053 ±(99.9%) 0.001 s/op
# Warmup Iteration 3: 0.053 ±(99.9%) 0.001 s/op
Iteration 1: 0.053 ±(99.9%) 0.001 s/op
measureSampleTime·p0.00: 0.050 s/op
measureSampleTime·p0.50: 0.054 s/op
measureSampleTime·p0.90: 0.055 s/op
measureSampleTime·p0.95: 0.055 s/op
measureSampleTime·p0.99: 0.055 s/op
measureSampleTime·p0.999: 0.055 s/op
measureSampleTime·p0.9999: 0.055 s/op
measureSampleTime·p1.00: 0.055 s/op
Iteration 2: 0.053 ±(99.9%) 0.001 s/op
measureSampleTime·p0.00: 0.050 s/op
measureSampleTime·p0.50: 0.054 s/op
measureSampleTime·p0.90: 0.055 s/op
measureSampleTime·p0.95: 0.055 s/op
measureSampleTime·p0.99: 0.055 s/op
measureSampleTime·p0.999: 0.055 s/op
measureSampleTime·p0.9999: 0.055 s/op
measureSampleTime·p1.00: 0.055 s/op
Iteration 3: 0.053 ±(99.9%) 0.001 s/op
measureSampleTime·p0.00: 0.050 s/op
measureSampleTime·p0.50: 0.053 s/op
measureSampleTime·p0.90: 0.055 s/op
measureSampleTime·p0.95: 0.055 s/op
measureSampleTime·p0.99: 0.055 s/op
measureSampleTime·p0.999: 0.055 s/op
measureSampleTime·p0.9999: 0.055 s/op
measureSampleTime·p1.00: 0.055 s/op
Iteration 4: 0.053 ±(99.9%) 0.001 s/op
measureSampleTime·p0.00: 0.050 s/op
measureSampleTime·p0.50: 0.053 s/op
measureSampleTime·p0.90: 0.055 s/op
measureSampleTime·p0.95: 0.055 s/op
measureSampleTime·p0.99: 0.055 s/op
measureSampleTime·p0.999: 0.055 s/op
measureSampleTime·p0.9999: 0.055 s/op
measureSampleTime·p1.00: 0.055 s/op
Result "com.sample.app.BenchmarkingModes.measureSampleTime":
N = 755
mean = 0.053 ±(99.9%) 0.001 s/op
Histogram, s/op:
[0.050, 0.051) = 72
[0.051, 0.051) = 59
[0.051, 0.052) = 47
[0.052, 0.052) = 51
[0.052, 0.053) = 28
[0.053, 0.053) = 47
[0.053, 0.054) = 67
[0.054, 0.054) = 68
[0.054, 0.055) = 123
[0.055, 0.055) = 90
[0.055, 0.056) = 103
Percentiles, s/op:
p(0.0000) = 0.050 s/op
p(50.0000) = 0.054 s/op
p(90.0000) = 0.055 s/op
p(95.0000) = 0.055 s/op
p(99.0000) = 0.055 s/op
p(99.9000) = 0.055 s/op
p(99.9900) = 0.055 s/op
p(99.9990) = 0.055 s/op
p(99.9999) = 0.055 s/op
p(100.0000) = 0.055 s/op
# JMH version: 1.35
# VM version: JDK 15.0.2, Java HotSpot(TM) 64-Bit Server VM, 15.0.2+7-27
# VM invoker: /Library/Java/JavaVirtualMachines/jdk-15.0.2.jdk/Contents/Home/bin/java
# VM options: -Dfile.encoding=UTF-8 -XX:+ShowCodeDetailsInExceptionMessages
# Blackhole mode: full + dont-inline hint (auto-detected, use -Djmh.blackhole.autoDetect=false to disable)
# Warmup: 3 iterations, single-shot each
# Measurement: 4 iterations, single-shot each
# Timeout: 10 min per iteration
# Threads: 1 thread
# Benchmark mode: Single shot invocation time
# Benchmark: com.sample.app.BenchmarkingModes.measureSingleShotTime
# Run progress: 100.00% complete, ETA 00:00:00
# Fork: 1 of 1
# Warmup Iteration 1: 0.054 s/op
# Warmup Iteration 2: 0.050 s/op
# Warmup Iteration 3: 0.055 s/op
Iteration 1: 0.055 s/op
Iteration 2: 0.053 s/op
Iteration 3: 0.054 s/op
Iteration 4: 0.052 s/op
Result "com.sample.app.BenchmarkingModes.measureSingleShotTime":
N = 4
mean = 0.053 ±(99.9%) 0.008 s/op
Histogram, s/op:
[0.052, 0.052) = 1
[0.052, 0.053) = 0
[0.053, 0.053) = 0
[0.053, 0.053) = 0
[0.053, 0.053) = 1
[0.053, 0.054) = 0
[0.054, 0.054) = 1
[0.054, 0.054) = 0
[0.054, 0.054) = 0
[0.054, 0.055) = 0
[0.055, 0.055) = 0
[0.055, 0.055) = 0
[0.055, 0.055) = 1
[0.055, 0.056) = 0
[0.056, 0.056) = 0
Percentiles, s/op:
p(0.0000) = 0.052 s/op
p(50.0000) = 0.053 s/op
p(90.0000) = 0.055 s/op
p(95.0000) = 0.055 s/op
p(99.0000) = 0.055 s/op
p(99.9000) = 0.055 s/op
p(99.9900) = 0.055 s/op
p(99.9990) = 0.055 s/op
p(99.9999) = 0.055 s/op
p(100.0000) = 0.055 s/op
# Run complete. Total time: 00:03:31
REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on
why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial
experiments, perform baseline and negative tests that provide experimental control, make sure
the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts.
Do not assume the numbers tell you what you want them to tell.
Benchmark Mode Cnt Score Error Units
BenchmarkingModes.measureThroughput thrpt 4 18.876 ± 0.433 ops/s
BenchmarkingModes.measureAverageTime avgt 4 0.053 ± 0.001 s/op
BenchmarkingModes.measureSampleTime sample 755 0.053 ± 0.001 s/op
BenchmarkingModes.measureSampleTime:measureSampleTime·p0.00 sample 0.050 s/op
BenchmarkingModes.measureSampleTime:measureSampleTime·p0.50 sample 0.054 s/op
BenchmarkingModes.measureSampleTime:measureSampleTime·p0.90 sample 0.055 s/op
BenchmarkingModes.measureSampleTime:measureSampleTime·p0.95 sample 0.055 s/op
BenchmarkingModes.measureSampleTime:measureSampleTime·p0.99 sample 0.055 s/op
BenchmarkingModes.measureSampleTime:measureSampleTime·p0.999 sample 0.055 s/op
BenchmarkingModes.measureSampleTime:measureSampleTime·p0.9999 sample 0.055 s/op
BenchmarkingModes.measureSampleTime:measureSampleTime·p1.00 sample 0.055 s/op
BenchmarkingModes.measureSingleShotTime ss 4 0.053 ± 0.008 s/op
No comments:
Post a Comment