-
Notifications
You must be signed in to change notification settings - Fork 30
Add TestKit JUnit extension [ECR-3076] #913
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
} | ||
|
||
@Override | ||
public void beforeEach(ExtensionContext extensionContext) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why beforeEach is implemented, creating the Testkit even if it is not requested? I think it must be instantiated when requested (= resolved).
@Override | ||
public void afterEach(ExtensionContext extensionContext) { | ||
TestKit testKit = getStore(extensionContext).get(KEY, TestKit.class); | ||
testKit.disposeInternal(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do we need this if the store will destroy the objects put there, if they implement https://junit.org/junit5/docs/current/api/org/junit/jupiter/api/extension/ExtensionContext.Store.CloseableResource.html?
|
||
private ExtensionContext.Store getStore(ExtensionContext context) { | ||
return context.getStore(ExtensionContext.Namespace.create(getClass(), | ||
context.getRequiredTestMethod())); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it really needed to include the test method as the namespace? Can't we just create a single namespace as a constant? I think stores of, for instance, test methods running in parallel isolated, and TempDirectory
in JUnit 5 suggests they are.
service = testKit.getService(TestService.SERVICE_ID, TestService.class); | ||
checkTestServiceInitialization(testKit, service); | ||
} | ||
void createTestKitForSingleService(TestKit testKit) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is nice if we can inject Testkit directly, but the present implementation makes it impossible to configure the testkit differently for different tests (without @Nested
or separating the tests into multiple files).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, I thought about that, but is it a frequent case for users to test their service with different TestKit configuration? It seems that usually they'd use the same configuration in the same test suite or use @Nested
. Even if they need that, they could just not inject it and create it manually. If not, what do you propose?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
auditor/validator?
If not, what do you propose?
Please see in jira.
* {@link Service#afterCommit(BlockCommittedEvent)} logic will only be called on the main | ||
* TestKit node of this type | ||
*/ | ||
public Builder withNodeType(EmulatedNodeType nodeType) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shall we drop the node type argument from builder method, and just make 'validator' the default?
private static final String KEY = "ResourceKey"; | ||
private static final Set<Class> testKitModificationAnnotations = | ||
ImmutableSet.of(Auditor.class, Validator.class, ValidatorCount.class, | ||
WithoutTimeService.class); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
WithTimeService(true/false)?
// reconfigure it | ||
if (annotationsUsed) { | ||
throw new RuntimeException("TestKit was parameterized with annotations after being" | ||
+ " instantiated in " + extensionContext.getDisplayName()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not sure "after being instantiated in " is correct. It was instantiated sometime earlier (in @BeforeEach
, for instance).
For better reporting/debuggability, I think it is possible to include info about the 'origin' injection (= testkit instantiation) point in the CloseableTestKit.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry for the ambiguity - I intended this message to mean "parameterized in test
", not "instantiated in test
".
The thing is that we can inject either in @BeforeEach
, @Test
, or @AfterEach
. All of them have the same ExtensionContext
, so it's not clear what to put as an origin
.
There are two possible options -
- if a user reconfigures TestKit in
@Test
, then he will get a cleardisplayName
- If he tries to reconfigure in
@AfterEach
then he'll getdisplayNames
of all tests he has.
We could improve the exception message though and make clear what could be incorrect.
// Throw an exception if TestKit was already instantiated in this context, but user tries to | ||
// reconfigure it | ||
if (annotationsUsed) { | ||
throw new RuntimeException("TestKit was parameterized with annotations after being" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A more specific exception is needed: will ParameterResolutionException work (please check its documentation)?
*/ | ||
private void checkExtensionContext(ExtensionContext extensionContext) { | ||
Optional<Method> testMethod = extensionContext.getTestMethod(); | ||
testMethod.orElseThrow(() -> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
😕 I don't see that the actual condition corresponds to the intended one.
Also, unless the extension is in static
field, it won't be able to inject in @BeforeAll
— so we may consider dropping this limitation, just documenting the consequences in the extension documentation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't see that the actual condition corresponds to the intended one.
What do you mean? For @BeforeAll
and @AfterAll
it will be empty, for @BeforeEach
, @Test
, @AfterEach
it will contain a test method.
Also, unless the extension is in static field, it won't be able to inject in @BeforeAll — so we may consider dropping this limitation, just documenting the consequences in the extension documentation.
Unless @TestInstance(Lifecycle.PER_CLASS)
is used, then user is able to declare @BeforeAll
and @AfterAll
as non-static.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I meant that the code said getTestMethod
but the condition referred to BeforeAll
/AfterAll
, and this connection was not obvious from the code, as the fact that getTestMethod
returned non-empty thing in case of BeforeEach
/AfterEach
.
I suggest clarifying that concisely.
|
||
Optional<WithoutTimeService> withTimeServiceAnnotation = | ||
parameterContext.findAnnotation(WithoutTimeService.class); | ||
withTimeServiceAnnotation.ifPresent(withoutTimeService -> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If it needs a time provider which cannot be provided via an annotation, then I think we simply shall not support such configuration.
CloseableTestKit closeableTestKit = | ||
getStore(extensionContext).get(KEY, CloseableTestKit.class); | ||
// Check if any TestKit configuration annotations are used | ||
boolean annotationsUsed = annotationsUsed(parameterContext); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What if we hide this thing inside buildTestkit
and the other branch:
if (closeableTestKit == null) {
testkit = buildTestkit(parameterContext, extensionContext);
getStore(extensionContext).put(KEY, new CloseableTestKit(testkit));
} else {
if (annotationsUsed(parameterContext)) {
fail
}
testkit = closeableTestkit.getTestkit();
}
return testkit;
with buildTestkit
overrideDefaults()/updateTestkitBuilder() — the implementation of that may check if annotations are present itself
return builder.build();
private final TestKit.Builder testKitBuilder; | ||
|
||
public TestKitExtension(TestKit.Builder testKitBuilder) { | ||
this.testKitBuilder = testKitBuilder; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
templateTestkitBuilder/templateBuilder — as it is used as a template to create new testkits; and add documentation explaining you can override defaults with the following annotations: @Foo
, @Bar
, @Baz
public @interface ValidatorCount { | ||
|
||
/** | ||
* Validator count of TestKit network. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Number of validator nodes in the Testkit network. Must be positive.
@see Builder#<apropriate method>
import java.lang.annotation.Target; | ||
|
||
/** | ||
* Changes main TestKit node type to validator for injected TestKit. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@see <appropriate enum/Builder method>
?
private void checkExtensionContext(ExtensionContext extensionContext) { | ||
Optional<Method> testMethod = extensionContext.getTestMethod(); | ||
testMethod.orElseThrow(() -> | ||
new RuntimeException("TestKit can't be injected in @BeforeAll or @AfterAll")); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
On the question of whether we shall or shall not forbid it:
"If you do need the same Testkit instance for all tests in the class, just use the builder directly:
private static final Testkit testkit = Testkit.newBuilder()
.withService(YourServiceModule.class)
.build();
"
… but as I wrote this instruction I remembered that the user will also need to write @AfterAll
that destroys the testkit! So maybe we shall support injection in BeforeAll
after all, shan't we?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, but isn't the idea of @Test
cases is that they are independent? Why would a user want to use the same TestKit? It's usually used to create some blocks and then review the state of the system, so using the same TestKit would lead to errors.
But we could document that and it would be easier for us - no need to throw an exception in @Before/AfterAll
injections.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So the questions are:
- Could anyone (@bullet-tooth) think of a realistic use-case when one needs a shared testkit for all methods?
- Shall we restrict such usage (as error-prone) if we can't think of such use-cases?
I can think of a scenario when setup is performed in @BeforeAll
and all tests perform only read-only operations on testkit — but how realistic that is, and how resistant to future additions of new tests performing write operations? Also, this scenario would also work with @BeforeEach
, somewhat slower, but way more reliable.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also I'd prefer to explicitly prohibit injecting in @AfterAll
, as it doesn't really make sense (it'll just create one more TestKit if injected). Although I agree that users might want to perform only read-only operations in different tests, it seems like a very error-prone approach. I think @BeforeEach
is more reliable indeed even if slow.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it'll just create one more TestKit if injected
It won't — contexts are nested, everything in the parent context (@BeforeAll
) is accessible in children (@Beach
, @Test
).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, let's forbid but let's put something along the lines of
Testkit cannot be injected in @BeforeAll/AfterAll because it is a stateful, mutable, object and sharing it between all tests is error-prone. Consider injecting it in @BeforeEach instead.
If you *do* need the same instance for all tests — just use `Testkit#builder` directly. Don't forget to destroy it in @AfterEach.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It won't — contexts are nested, everything in the parent context (@BeforeAll) is accessible in children (@beach, @test).
I was talking about @AfterAll
. If we inject in @BeforeAll
, we'd have same TestKit in all tests - but if we inject in @AfterAll
, we won't - every Test will have its own instance, including @AfterAll
method.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see, in this case — yes, but it makes no apparent sense to use that :-)
/** | ||
* Changes main TestKit node type to auditor for injected TestKit. | ||
* | ||
* @see TestKit.Builder#withNodeType(EmulatedNodeType) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is second @see
missing? Because the method won't show.
exonum-java-binding/testkit/pom.xml
Outdated
</dependency> | ||
|
||
<dependency> | ||
<groupId>junit</groupId> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is JUnit 4.
<dependency> | ||
<groupId>org.junit.platform</groupId> | ||
<artifactId>junit-platform-testkit</artifactId> | ||
<version>${junit-platform-testkit.version}</version> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
test-scoped?
* | ||
* <pre><code> | ||
* static { | ||
* LibraryLoader.load(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is no longer needed (once the pr lands)
* Allows injecting TestKit into test context: | ||
* | ||
* <pre><code> | ||
* @RegisterExtension |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ must be escaped
import org.junit.jupiter.api.TestInstance.Lifecycle; | ||
import org.junit.jupiter.api.extension.RegisterExtension; | ||
|
||
@TestInstance(Lifecycle.PER_CLASS) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why? It does not seem to be needed.
@RegisterExtension | ||
TestKitExtension testKitExtension = new TestKitExtension( | ||
TestKit.builder(EmulatedNodeType.VALIDATOR) | ||
static TestKitExtension testKitExtension = new TestKitExtension( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shall we use the extension in the test of testkit + its builder?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it's okay to use extension for most of the TestKit tests (not builder tests though).
} | ||
|
||
@Test | ||
void createTestKitWithBuilderForSingleService() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is probably no longer needed, isn't it?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But why? I think it's still a viable test.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry, wrong method — the question is for createTestKitForSingleService^.
short validatorCount = 2; | ||
try (TestKit testKit = TestKit.builder(EmulatedNodeType.AUDITOR) | ||
try (TestKit testKit = TestKit.builder() | ||
.withNodeType(EmulatedNodeType.AUDITOR) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shall we use with
prefix, or maybe drop it (e.g., nodeType
) or add usual set
and add
depending on the operation? On the other hand, if used consistently, it slightly helps with discoverability (if all use with
, you can type wi
and see what it could do).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I like having same prefix for the reason you stated - it's nice to have all methods starting with with
/** | ||
* Changes main TestKit node type to validator for injected TestKit. | ||
* | ||
* @see TestKit.Builder#withNodeType(EmulatedNodeType) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same as above (two @sees
?)
...cy-demo/src/test/java/com/exonum/binding/cryptocurrency/transactions/CreateWalletTxTest.java
Show resolved
Hide resolved
private static final KeyPair ACCOUNT_2 = PredefinedOwnerKeys.SECOND_OWNER_KEY_PAIR; | ||
|
||
@Test | ||
@RequiresNativeLibrary |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The same is here. The class and tests are annotated.
exonum-java-binding/pom.xml
Outdated
<hamcrest.version>2.1</hamcrest.version> | ||
<junit.version>4.12</junit.version> | ||
<junit.jupiter.version>5.4.2</junit.jupiter.version> | ||
<junit-platform-testkit.version>1.5.0-M1</junit-platform-testkit.version> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please move it to the TestKit
pom (or is it needed for other modules?)
* created. | ||
* | ||
* @see <a href="https://exonum.com/doc/version/0.11/get-started/test-service/">TestKit documentation</a> | ||
* <a href="https://exonum.com/doc/version/0.11/advanced/consensus/specification/#pool-of-unconfirmed-transactions">Pool of Unconfirmed Transactions</a> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please fix @linkplain
above ⏫ (redundant }
)
* given builder. Example usage: | ||
* | ||
* <pre><code> | ||
* @RegisterExtension |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see neither examples of creating TestKit
in a declarative way (using @ExtendWith
) or a Javadoc describing why it's not possible/forbidden.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's not possible to use @ExtendWith
because that way we can't pass a TestKit builder as an argument to extension constructor. Should I specify that in Javadoc?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since we provide examples of usage right in the Javaddocs (and, later, on the web), I don't think it is necessary to say that "You can't register it with ExtendWith". If users try and it does not work — we can reasonably assume they will get to the (any) docs.
ZonedDateTime getTime(); | ||
|
||
/** | ||
* Returns a copy of this time provider. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I do not understand how it is supposed to work. It is apparently required when testkit is
static
and it has a time provider set. In this case the time provider needs to be pre-configured
in a static context (= before @BeforeAll
).
Some issues with the current design:
- Does not make sense with FakeTimeProvider unless it is constant (= never changes the current time),
because the client cannot access the actual time provider used by the testkit. We can probably
make testkit return it, but I think it will only complicate things (pass one provider to the
builder, but get another one from the teskit). On the other hand, if we do not copy (in astatic
context),
and the client puts it in astatic
field and then reconfigures in tests —
they will end up with broken initial state in case of single-threaded execution of tests or concurrency
issues (and anything broken) in case of multi-threaded execution. - Cannot use mocks even without the extension because they cannot be cloned (well, can with a wrapper with a throwing copy implementation).
Unless I'm missing something, copying seems restrictive and not that easy to use or fix.
But it is required only for static
extensions. What we could do:
- 'Fix' by adding
Testkit#getTimeProvider
and some obscure documentation. - Ignore the concurrency issue (i.e.,
static
extension with a shared template builder
and a shared time provider will use the same instance in all tests). If a client breaks
it by modifying the provider in any test, it is their problem. - Forbid
static
extensions — we seem to have found a good reason :-) - Anything else?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd forbid (discourage?) static
extensions. It's easier for us, will prevent any other possible issues with having a static extensions (I can't think of any right now though) and won't limit user in any way.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also I suppose it's better to make a copy of templateTestKitBuilder
on every parameter resolve anyway (as we do now), even though we discourage having a static
extension. Even if someone uses static
, they are unlikely to have any issues in most cases.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK, I agree with discouraging it because apparently there is no reasonable way to support static
. Shall we also "forbid" (probably, by supporting class-level callbacks, and, if they get invoked — throwing an Exception)? Are there any downsides in doing that (i.e., is it possible that a user is forced to use static
but it won't cause any problems)?
...nding/integration-tests/src/test/java/com/exonum/binding/test/BlockchainIntegrationTest.java
Outdated
Show resolved
Hide resolved
.../integration-tests/src/test/java/com/exonum/binding/test/TimeSchemaProxyIntegrationTest.java
Outdated
Show resolved
Hide resolved
exonum-java-binding/testkit/src/main/java/com/exonum/binding/testkit/TestKitExtension.java
Outdated
Show resolved
Hide resolved
I see old way usage of TestKit in |
It makes little sense to use the extension when you need to inject it in |
Overview
Add TestKit JUnit extension.
See: https://jira.bf.local/browse/ECR-3076
Definition of Done