How to include additional information in OTR report and include it in HTML report using open-test-reporting-cli

[tiefenauer]

I have a plain Java application that runs JUnit tests programmatically, using OpenTestReportGeneratingListener to create an event-based Open Test Report.

I have the following code that that will run JUnit tests programmatically from a Quarkus application (outline only for clarity):

LauncherConfig config = LauncherConfig.builder()... .build();
LauncherDiscoveryRequest request = LauncherDiscoveryRequestBuilder.request().selectClass(QuarkusBridge.class)... .build();

LauncherFactory.create(config).execute(request, new OpenTestReportGeneratingListener());

QuarkusBridge.java provides a test factory that generates the tests dynamically:

// QuarkusBridge.java 
    @TestFactory
    public Collection<DynamicTest> dynamicTestsWithCollection(TestReporter reporter, TestInfo testInfo) {
        return Arc.container().instance(TestCoordinator.class).get().computeTests(reporter, testInfo);
    }

TestCoordinator.java contains a method computeTests that will create instances of DynamicTests to be run as JUnit Tests

    public Collection<DynamicTest> computeTests(TestReporter reporter, TestInfo testInfo) {
       // create a list of DynamicTest objects that contain the test logic
    }

I am using the OTR-CLI to create a HTML report. I have read that it is possible to extend the HTML report and I have checked out the example from JUnit implementation of a Contributor.

What I want now is to extend the HTML report so that for each executed test a link is displayed. The link will be used to download assets that were created during test execution. Is this possible? What steps are needed to add this data to the OTR reports so that it can be included in the HTML report?

I am new to OTR and have used primarily to write "traditional" unit/integration tests so far (surefire reports that are rendered in Jenkins). Unfortunately the OTR docs and example code have not given me enough information and I haven't found any additional documentation.

[marcphilipp]

Extending the HTML report requires information to be present in the OTR XML files that are used as inputs. Since you're using OpenTestReportGeneratingListener to write those XML files, it would have to include that information. Since OpenTestReportGeneratingListener is currently not extensible, you'll have to work with what it supports at the moment. Another use case for making it extensible popped up in cucumber/cucumber-jvm#2947 so we might have to think about that.

In the meantime, I recommend using report entries for this purpose. Assuming you're using JUnit Jupiter, you could do this directly from your test code or a custom extension.

import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestReporter;
import org.junit.jupiter.api.extension.AfterTestExecutionCallback;
import org.junit.jupiter.api.extension.ExtendWith;
import org.junit.jupiter.api.extension.ExtensionContext;

public class ExampleTests {

	@Test
	void testThatReportsLink(TestReporter testReporter) {
		testReporter.publishEntry("my-artifact", "link:https://example.org");
	}

	@Test
	@ExtendWith(ExampleExtension.class)
	void testUsingAnExtensionReportingALink() {
	}

	static class ExampleExtension implements AfterTestExecutionCallback {
		@Override
		public void afterTestExecution(ExtensionContext context) {
			context.publishReportEntry("my-artifact", "link:https://example.org");
		}
	}
}

The link: prefix is an undocumented feature (or maybe a leaked implementation detail, not sure, yet) of OTR's HTML report and will cause it to be rendered like this:

Important

An alternative would be to attach these files to the test report directly using the brand-new feature that I added in #4138. It's not released yet, but feedback would be very welcome!

[tiefenauer]

Thanks, @marcphilipp . I should have mentioned that I am running the tests using @TestFactory, not individual tests annotated with @Test. The test factory will compute the tests dynamically based on the contents of a directory (i.e. creating/executing a test for each file in that directory). I updated my original question above accordingly.

The method containing the logic to create the tests accepts an instance of TestReporter and call TestReporter#publishEntry() for each dynamic test.

For that scenario, your suggestion does indeed work, but not as I expected: The Link is then contained in the OTR, but not in the report for each dynamic test.

Screenshot from the node for the method annotated with @TestFactory:

Screenshot for a node created wor the test results of a dynamic Test 'F1-00001'_

It does not seem to matter at what point I will call TestReporter#publishEntry("my-artifact", "link:https://example.org"), because the entry will always only be contained in the node for the method that is annotated with @TestFactory, never in the nodes for the dynamically generated tests.

I also tried using an Extension implementing AfterEachCallback, but the result is the same.

public class ExampleExtension implements AfterEachCallback {
    @Override
    public void afterEach(ExtensionContext context) throws Exception {
        context.publishReportEntry("my-artifact", "link:https://example.org");
    }
}

I am not sure if I am doing something wrong or if adding an entry on dynamically generated tests is indeed not possible.

However, I played around with the generated OTR-XML and added the entry manually (simply replacing the id of dynamicTestsWithCollection with the ID of a dynamic test):

<e:reported id="4" time="2024-12-09T07:52:53.249890400Z"><attachments><data time="2024-12-09T08:52:53.248871700"><entry key="my-artifact">link:https://example.org</entry></data></attachments></e:reported>

The HTML report then displays the link in the node for the dynamic test, so the OTR-XML still seems to be valid:

Edit: Using TestReporter in conjunction with dynamic tests does not seem to be supported (but a workaround would be helpful): #378

[marcphilipp]

It ain't pretty but it works:

import static org.junit.jupiter.api.DynamicTest.dynamicTest;
import static org.junit.jupiter.api.extension.ExtensionContext.Namespace.GLOBAL;

import java.util.Map;
import java.util.stream.Stream;

import org.junit.jupiter.api.DynamicTest;
import org.junit.jupiter.api.TestFactory;
import org.junit.jupiter.api.TestReporter;
import org.junit.jupiter.api.extension.DynamicTestInvocationContext;
import org.junit.jupiter.api.extension.ExtendWith;
import org.junit.jupiter.api.extension.ExtensionContext;
import org.junit.jupiter.api.extension.InvocationInterceptor;
import org.junit.jupiter.api.extension.ParameterContext;
import org.junit.jupiter.api.extension.ParameterResolutionException;
import org.junit.jupiter.api.extension.ParameterResolver;

public class DynamicReporterDemo {

	@TestFactory
	@ExtendWith(DynamicTestReportingExtension.class)
	Stream<DynamicTest> testFactory(DynamicTestReporter reporter) {
		reporter.publishEntry("factory");
		return Stream.of(dynamicTest("demo", () -> reporter.publishEntry("key", "value")));
	}

	interface DynamicTestReporter extends TestReporter {
	}

	static class DynamicTestReportingExtension implements ParameterResolver, InvocationInterceptor {

		@Override
		public boolean supportsParameter(ParameterContext parameterContext, ExtensionContext extensionContext)
				throws ParameterResolutionException {
			return parameterContext.getParameter().getType().equals(DynamicTestReporter.class);
		}

		@Override
		public Object resolveParameter(ParameterContext parameterContext, ExtensionContext extensionContext)
				throws ParameterResolutionException {
			DynamicTestReporter reporter = new DefaultTestReporter(extensionContext);
			extensionContext.getStore(GLOBAL).put(DynamicTestReporter.class, reporter);
			return reporter;
		}

		@Override
		public void interceptDynamicTest(Invocation<Void> invocation, DynamicTestInvocationContext invocationContext,
				ExtensionContext extensionContext) throws Throwable {
			DefaultTestReporter reporter = extensionContext.getStore(GLOBAL).get(DynamicTestReporter.class, DefaultTestReporter.class);
			if (reporter == null) {
				invocation.proceed();
			} else {
				reporter.currentExtensionContext.set(extensionContext);
				try {
					invocation.proceed();
				}
				finally {
					reporter.currentExtensionContext.remove();
				}
			}
		}

		static class DefaultTestReporter implements DynamicTestReporter {

			final ThreadLocal<ExtensionContext> currentExtensionContext = new ThreadLocal<>();
			private final ExtensionContext fallbackExtensionContext;

			DefaultTestReporter(ExtensionContext fallbackExtensionContext) {
				this.fallbackExtensionContext = fallbackExtensionContext;
			}

			@Override
			public void publishEntry(Map<String, String> map) {
				ExtensionContext resolvedExtensionContext = currentExtensionContext.get();
				if (resolvedExtensionContext == null) {
					resolvedExtensionContext = fallbackExtensionContext;
				}
				resolvedExtensionContext.publishReportEntry(map);
			}
		}
	}
}

[tiefenauer]

confirming this works, although it's not possible to use the default TestReporter and the DynamicTestReporter simultaneously, but I gues that's by design.

Marking this as the answer.

[marcphilipp]

confirming this works, although it's not possible to use the default TestReporter and the DynamicTestReporter simultaneously, but I gues that's by design.

What did you try?

I've updated the example so that DynamicTestReporter also works in the @TestFactory method.

[tiefenauer]

I ran the tests using the DynamicTestReportingExtension you provided an injected both the DynamicTestReporter and the TestReporter as method parameters. This gave me an error that two implementations of TestReporter were found. For some reasons I cannot reproduce the error anymore, so I guess it was a glitch on my side.

I added the changes you made for the DynamicTestReportingExtension and confirm that using the DynamicTestReporter and the default TestReporter it is possible to add a link both on both levels (@TestFactory and generated tests). 👍