Jmh: @Setup, @TearDown fixture methods

@Setup

Setup marks the fixture method to be run before the benchmark.

 

@TearDown

TearDown marks the fixture method to be run after the benchmark

 

Since fixture methods (methods annotated with @Setup, @TearDown) manage the State lifecycles, these can only be declared in State classes.

Fixtures have different Levels to control when they are about to run.

 

Example

@State(Scope.Benchmark)
public static class MyState {

    @Setup(Level.Trial)
    public void setupTrailLevel() {
        System.out.println("setupTrailLevel called");
    }
    
    @Setup(Level.Iteration)
    public void setupIterationLevel() {
        System.out.println("setupIterationLevel called");
    }
    
    @Setup(Level.Invocation)
    public void setupInvocationLevel() {
        System.out.println("setupInvocationLevel called");
    }

    @TearDown(Level.Trial)
    public void tearDownTrailLevel() {
        System.out.println("tearDownTrailLevel called");
    }
    
    @TearDown(Level.Iteration)
    public void tearDownIterationLevel() {
        System.out.println("tearDownIterationLevel called");
    }
    
    @TearDown(Level.Invocation)
    public void tearDownInvocationLevel() {
        System.out.println("tearDownInvocationLevel called");
    }
}

Notice above snippet, @Setup, @TearDown annotations takes a Level attribute. Possible values of Level are documented in below table.

 

Level	Description
Level.Trial	Annotated method is called before/after each run of the benchmark.
Level.Iteration	Annotated method is called before/after each iteration of the benchmark.
Level.Invocation	Annotated method is called once for each call to the benchmark method.

 

 Find the below working application, observe the output to confirm the same on fixture methods with level attribute.

 

SetupTeardownDemo.java

 

package com.sample.app;

import java.util.concurrent.TimeUnit;

import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.BenchmarkMode;
import org.openjdk.jmh.annotations.Level;
import org.openjdk.jmh.annotations.Mode;
import org.openjdk.jmh.annotations.Scope;
import org.openjdk.jmh.annotations.Setup;
import org.openjdk.jmh.annotations.State;
import org.openjdk.jmh.annotations.TearDown;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.RunnerException;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;

public class SetupTeardownDemo {
    
    @State(Scope.Benchmark)
    public static class MyState {

        @Setup(Level.Trial)
        public void setupTrailLevel() {
            System.out.println("\nsetupTrailLevel called\n");
        }
        
        @Setup(Level.Iteration)
        public void setupIterationLevel() {
            System.out.println("\n\tsetupIterationLevel called");
        }
        
        @Setup(Level.Invocation)
        public void setupInvocationLevel() {
            System.out.println("\n\t\tsetupInvocationLevel called");
        }

        @TearDown(Level.Trial)
        public void tearDownTrailLevel() {
            System.out.println("\ntearDownTrailLevel called\n");
        }
        
        @TearDown(Level.Iteration)
        public void tearDownIterationLevel() {
            System.out.println("\ttearDownIterationLevel called");
        }
        
        @TearDown(Level.Invocation)
        public void tearDownInvocationLevel() {
            System.out.println("\t\ttearDownInvocationLevel called");
        }
    }
    
    @Benchmark
    @BenchmarkMode(Mode.AverageTime)
    public void test(MyState myState) throws InterruptedException {
        TimeUnit.SECONDS.sleep(2);
        System.out.println("\t\ttest method executed");
    }
    
    
    public static void main(String[] args) throws RunnerException {
        Options opt = new OptionsBuilder()
                .include(SetupTeardownDemo.class.getSimpleName())
                .forks(2)
                .measurementIterations(1)
                .warmupIterations(1)
                .build();
        

        new Runner(opt).run();
    }

}

 

Output

# JMH version: 1.35
# VM version: JDK 15.0.2, Java HotSpot(TM) 64-Bit Server VM, 15.0.2+7-27
# VM invoker: /Library/Java/JavaVirtualMachines/jdk-15.0.2.jdk/Contents/Home/bin/java
# VM options: -Dfile.encoding=UTF-8 -XX:+ShowCodeDetailsInExceptionMessages
# Blackhole mode: full + dont-inline hint (auto-detected, use -Djmh.blackhole.autoDetect=false to disable)
# Warmup: 1 iterations, 10 s each
# Measurement: 1 iterations, 10 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Average time, time/op
# Benchmark: com.sample.app.SetupTeardownDemo.test

# Run progress: 0.00% complete, ETA 00:00:40
# Fork: 1 of 2
# Warmup Iteration   1: 
setupTrailLevel called


    setupIterationLevel called

        setupInvocationLevel called
        test method executed
        tearDownInvocationLevel called

        setupInvocationLevel called
        test method executed
        tearDownInvocationLevel called

        setupInvocationLevel called
        test method executed
        tearDownInvocationLevel called

        setupInvocationLevel called
        test method executed
        tearDownInvocationLevel called

        setupInvocationLevel called
        test method executed
        tearDownInvocationLevel called
    tearDownIterationLevel called
2.002 s/op
Iteration   1: 
    setupIterationLevel called

        setupInvocationLevel called
        test method executed
        tearDownInvocationLevel called

        setupInvocationLevel called
        test method executed
        tearDownInvocationLevel called

        setupInvocationLevel called
        test method executed
        tearDownInvocationLevel called

        setupInvocationLevel called
        test method executed
        tearDownInvocationLevel called

        setupInvocationLevel called
        test method executed
        tearDownInvocationLevel called
    tearDownIterationLevel called

tearDownTrailLevel called

2.003 s/op

# Run progress: 50.00% complete, ETA 00:00:25
# Fork: 2 of 2
# Warmup Iteration   1: 
setupTrailLevel called


    setupIterationLevel called

        setupInvocationLevel called
        test method executed
        tearDownInvocationLevel called

        setupInvocationLevel called
        test method executed
        tearDownInvocationLevel called

        setupInvocationLevel called
        test method executed
        tearDownInvocationLevel called

        setupInvocationLevel called
        test method executed
        tearDownInvocationLevel called

        setupInvocationLevel called
        test method executed
        tearDownInvocationLevel called
    tearDownIterationLevel called
2.002 s/op
Iteration   1: 
    setupIterationLevel called

        setupInvocationLevel called
        test method executed
        tearDownInvocationLevel called

        setupInvocationLevel called
        test method executed
        tearDownInvocationLevel called

        setupInvocationLevel called
        test method executed
        tearDownInvocationLevel called

        setupInvocationLevel called
        test method executed
        tearDownInvocationLevel called

        setupInvocationLevel called
        test method executed
        tearDownInvocationLevel called
    tearDownIterationLevel called

tearDownTrailLevel called

2.004 s/op


Result "com.sample.app.SetupTeardownDemo.test":
  2.003 s/op


# Run complete. Total time: 00:00:50

REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on
why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial
experiments, perform baseline and negative tests that provide experimental control, make sure
the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts.
Do not assume the numbers tell you what you want them to tell.

Benchmark               Mode  Cnt  Score   Error  Units
SetupTeardownDemo.test  avgt    2  2.003           s/op

 

  

Previous                                                 Next                                                 Home