Integrating a library with JUnit 5

[armandino]

Hi all,

I'm working on a library that populates objects with random data via reflection: https://github.com/instancio/instancio

I would like to integrate the library with JUnit 5 so that if a test fails, I can display the seed value that was used by the random number generator in the failure message. This would allow the user to reproduce the failed test by supplying the seed value back to the library and re-running the test.

Example:

@Test
void verifyPerson() {
    Person person = Instancio.of(Person.class).create();
    // some test code
    // ... an assertion fails
}

I would like to report the following message:

verifyPerson() failed with seed value: 123

Second test run:

@Test
void verifyPerson() {
    Person person = Instancio.of(Person.class)
        .withSeed(123) // re-run the test with failed test data
        .create();
    // ...

I was thinking of creating an extension and putting the value into the ExtensionContext.store,
but I couldn't figure out how to pass the data from the library into the store.

Would this be possible to achieve?

[armandino]

In case it helps someone else, I ended up passing custom seed value using a @Seed annotation on a test method. This was implemented using ThreadLocal:

https://github.com/instancio/instancio/blob/main/instancio-junit/src/main/java/org/instancio/junit/InstancioExtension.java

@Seed(123)
@Test
void verifyPerson() {
    Person person = Instancio.of(Person.class).create();
    // ...

[mpkorstanje]

Instead of using a static method on Instancio you may want to consider an instance method and provide the instance as an argument or annotated field.

The instance could the be pre-seeded.

[Bananeweizen]

Why don't you inject your factory created Person instance into the test method as an argument? Similar to what the ParameterizedTest variants do. That way the code of the test methods would remain unchanged for re-running with a fixed seed, since the injection can provide the non-random value the second time. I would probably use a -Dfoo.bar.seed=123 system property to set a wanted seed at wherever the test runtime is launched. That way a single argument for the seed can be used with 1 or 1000 tests, while your suggestion above doesn't really scale.

[armandino]

@Bananeweizen - thank you for the suggestions. I did actually implement the first one. It is possible to inject arguments into parameterized tests like so:

    @ParameterizedTest
    @InstancioSource({Foo.class, Bar.class, Baz.class})
    void example(Foo foo, Bar bar, Baz baz) {
        // snip...
    }

Currently seed value can be specified by placing @Seed(12345) annotation on a test method. Setting a global seed using a system property is an interesting idea. This would allow running all tests against fixed data. I will run some experiments but I don't see why it couldn't be added.