Discover tests in suites packaged into jars, e.g. TCKs

[t1]

As discussed in the discussions, we need a way to package a test suite into a jar and run it for several implementations. This is a common requirement when building Technology Compatibility Kits. There are generally three artifacts:

1. api contains some interfaces and a ServiceLoader (or similar IoC mechanism) to load an implementation of the api interfaces.
2. impl1 containing one implementation (sometimes called a reference implementation ri) of the api; includes the services files for the ServiceLoader.
3. tck the test suite to check that an implementation of the api is compliant.

You can run the tests in the tck by adding a runtime dependency on impl1. But the tck should be usable by any other implementation as well: I want the impl1 to load the tck-jar and run it, not the other way around. The tck should be a library of tests that the implementations can include as a test dependency. The dependency should be impl1→tck not tck→impl1.

I assume that scanning all jars for tests would result in unacceptable performance, so we probably need some way to specify the jars to scan, or even better to mark a jar to be a test-suite.

I currently need this for an open source project: https://github.com/t1/smallrye-graphql It's about the TCK in the client. But I've also built a minimal, reproducible example that may be easier to play around with: https://github.com/t1/junit-tck-mre.

Deliverables

• A mechanism to specify the jars to scan for tests
• Update to the User Manual

[mpkorstanje]

I assume that scanning all jars for tests would result in unacceptable performance, so we probably need some way to specify the jars to scan, or even better to mark a jar to be a test-suite.

I would expect that running the ConsoleLauncher with either --select-module, --select-package or --scan-classpath should suffice or alternatively a declarative test suite from #2416 with the equivalent selectors.

[t1]

Is there a plan as to when this will be released?

[mpkorstanje]

#2415 has been released in the latest preview and will presumably be generally available once the 5.8.0 backlog is cleared out.

[t1]

I got it working with 5.8.0-M1. Thanks a lot!

[jbduncan]

@t1 Ooh, nice! Am I right to think that you got it working over at smallrye/smallrye-graphql#796?

I like how the suite-based TCK runs a bunch of other tests, which in turn grab the implementation-under-test from the classpath using ServiceLoader.

I can see now why, for a project of this size, the Test Interface pattern wouldn't work: the TCK already has lots of test classes, so merging them into one to create a Test Interface would be infeasible.

Great work! 👍

[jbduncan]

I think this issue could be re-opened though, as I believe this suite-based TCK pattern is worth documenting in the User Guide.

What do you think, JUnit 5 team?

[mpkorstanje]

I think it is worth documenting this pattern as one particular use case of declarative test suites.

But I'm not part of the JUnit team. I think you have to tag them with @junit-team.

[marcphilipp]

@t1 Could you please outline your approach here?