Initial native image agent with JVM mode tests integration

[galderz]

Prototype implementation for #36822

Here are the implementation details:

It adds a new profile, called native-with-agent to the generated projects where the JVM mode tests run with the native image agent. This profile sets the native image agent parameter necessary to run the JVM mode tests with. This parameter includes references to filters that are produced before running JVM mode tests.

Before the JVM mode tests run, GenerateCodeTestsMojo has been modified to generated the native image agent filters are produced. This enables filtering out most of the configuration that is quarkus specific, but crucially it doesn't handle resources. One caveat of this early filtering is that we don't have context of the application to do more fine grained filtering. An alternative implementation would skip filtering at this level and instead filter at the build step phase, just like resources are filtered (read on).

During JVM mode tests, native image agent generates config files inside target/generated-native-config.

Then, native image agent specific build steps pick up this configuration, parse it, filtering using regex and produce existing build items, e.g. ReflectiveClassBuildItem, NativeImageResourcePatternsBuildItem...etc.

There is a builder class that produces json but there's no class to read json files, so I've gone ahead and created a small, dependency-free, json parser called JsonReader for this. The objective of this parser was to get something working quickly and without dependencies, so I'm sure there's a lot of ways in which it can be improved :)

As mentioned above, resources cannot be filtered at the native image agent level, so I post-processed the resources configuration file using the JsonReader parser, along with regex, to filter out resources that Quarkus takes care of.

One important note here is that both the native image configuration that Quarkus produces, and the one extracted from the native image agent ends up in the same configuration files inside the generated bytecode. Achieving this is quite simple because we just reuse the same existing build items. But it might make debugging easier if the files were different so that we can distinguish between those Quarkus produces and those that come from the native image agent more easily. Not sure this is doable, @zakkak? Otherwise, you can always run the app with -Dnative and -Dnative-with-agent and compare the json files (see a demonstration below).

I've tested out this in a new project generated from this branch called new-project (attached new-project-0.1.tar.gz). This is a small example that requires reflection to be configured for Alice class lookup in order to respond to the /hello/Alice endpoint. The native image agent integration works when the native integration test for that endpoint is a success. Comparing the output of the following commands with -Dnative and -Dnative-with-agent you can see what impact the native image agent jvm mode test integration has on the configuration:

$ ./mvnw verify -Dnative 
...
$ tar xfO ./target/quarkus-app/quarkus/generated-bytecode.jar META-INF/native-image/reflect-config.json | jq -s '.[] |= sort_by(.name)' > target/reflect-config-no-agent.json
...
$ ./mvnw verify -Dnative-with-agent
...
$ tar xfO ./target/quarkus-app/quarkus/generated-bytecode.jar META-INF/native-image/reflect-config.json | jq -s '.[] |= sort_by(.name)' > target/reflect-config-with-agent.json
...
$ diff target/reflect-config-with-agent.json target/reflect-config-no-agent.json
619,628d618
<     },
<     {
<       "methods": [
<         {
<           "parameterTypes": [],
<           "name": "sayMyName"
<         }
<       ],
<       "allDeclaredConstructors": true,
<       "name": "org.acme.Alice"
632,635d621
<       "name": "org.acme.Bob"
<     },
<     {
<       "allDeclaredConstructors": true,

It has added Alice class and its sayMyName method to the reflection configuration, which is required in order for endpoint to respond.

There are also differences in the resource configuration but they are not relevant here:

$ tar xfO ./new-project/target/quarkus-app/quarkus/generated-bytecode.jar META-INF/native-image/resource-config.json | jq -s '.[].resources.includes |= sort_by(.pattern)' ...
...
$ diff with-agent/resource-config-with-agent.json no-agent/resource-config-no-agent.json
31,78d30
<         },
<         {
<           "pattern": "\\\\QMETA-INF/maven/org\\.apache\\.maven/maven-core/pom\\.properties\\\\E"
<         },
<         {
<           "pattern": "\\\\QMETA-INF/plexus/components\\.xml\\\\E"
<         },
<         {
<           "pattern": "\\\\QMETA-INF/services/java\\.lang\\.System\\$LoggerFinder\\\\E"
<         },
<         {
<           "pattern": "\\\\QMETA-INF/services/java\\.net\\.spi\\.InetAddressResolverProvider\\\\E"
<         },
<         {
<           "pattern": "\\\\QMETA-INF/services/java\\.net\\.spi\\.URLStreamHandlerProvider\\\\E"
<         },
<         {
<           "pattern": "\\\\QMETA-INF/services/java\\.nio\\.channels\\.spi\\.SelectorProvider\\\\E"
<         },
<         {
<           "pattern": "\\\\QMETA-INF/services/java\\.nio\\.file\\.spi\\.FileSystemProvider\\\\E"
<         },
<         {
<           "pattern": "\\\\QMETA-INF/services/java\\.time\\.zone\\.ZoneRulesProvider\\\\E"
<         },
<         {
<           "pattern": "\\\\QMETA-INF/services/javax\\.xml\\.stream\\.XMLInputFactory\\\\E"
<         },
<         {
<           "pattern": "\\\\QMETA-INF/services/org\\.codehaus\\.groovy\\.runtime\\.ExtensionModule\\\\E"
<         },
<         {
<           "pattern": "\\\\QMETA-INF/sisu/javax\\.inject\\.Named\\\\E"
<         },
<         {
<           "pattern": "\\\\Qjava/io/Serializable\\.class\\\\E"
<         },
<         {
<           "pattern": "\\\\Qorg/apache/maven/model/pom-4\\.0\\.0\\.xml\\\\E"
<         },
<         {
<           "pattern": "\\\\Qorg/osgi/annotation/bundle/Requirements\\.class\\\\E"
<         },
<         {
<           "pattern": "java\\.base:\\\\Qjava/io/Serializable\\.class\\\\E"
<         },
<         {
<           "pattern": "java\\.base:\\\\Qjdk/internal/icu/impl/data/icudt72b/nfc\\.nrm\\\\E"

One thing of interest in the resource differences is that it uncovers a bug in the native image integration. It's escaping patterns when it shouldn't, see #36823.

Final note, what to filter for is up for debate. I was quite aggressive in the native image agent filters because there really wasn't need for any of the additional configuration that it generated, but certain filters are debatable. Not sure what is the best approach here. Do we prefer minimal filtering that we now for sure Quarkus takes care of? Or do we prefer aggressive filtering? I think I would probably go for minimal because the aim here is to get something that native image builds and works. The user can trim things later if necessary.

[maxandersen]

what makes something qualified /necessary to have in this list?

[galderz]

Good question.

I created the list based on the output of the native image agent config for the getting started reactive sample JVM mode unit tests, see here.

Something more dynamic would likely be needed here. I guess there could be some general starting point for this list, and then based on the extensions in use, expand it?

[maxandersen]

I just know that this class is going to be misused to add more json stuff....

[galderz]

That can of worms was open when the Json class was added to generate build output metrics, which Foivos' then reused to generate the json config output for native image :)

[maxandersen]

this is just for tests....but how about 'quarkus run' / mvn quarkus:run - should that be configured somehow?

[galderz]

Yeah that should be done too. I'll look into it.

[galderz]

@maxandersen I've looked into add quarkus run integration but I should be deferred to a later stage (we can create a follow up issue). Gradle quarkusRun does not seem to work at all. It starts Quarkus and then finishes the build, see here. Also, there are no integration tests under gradle that verify quarkusRun works at all. Native image agent integration with run targets should be added after that is fixed.

[zakkak]

But it might make debugging easier if the files were different so that we can distinguish between those Quarkus produces and those that come from the native image agent more easily. Not sure this is doable, @zakkak? Otherwise, you can always run the app with -Dnative and -Dnative-with-agent and compare the json files (see a demonstration below).

@galderz that's definitely possible, but we would need to explicitly pass these config files to native-image using -H:ReflectionConfigurationFiles, -H:ResourceConfigurationFiles, -H:JNIConfigurationFiles, and -H:SerializationConfigurationFiles which are considered "experimental".

I wonder if we could add some kind of comment in the build items and output it in the json files, in the future we could augment the proposed reasons field.

Either way would be acceptable I believe.

[zakkak]

Do we prefer minimal filtering that we now for sure Quarkus takes care of? Or do we prefer aggressive filtering? I think I would probably go for minimal because the aim here is to get something that native image builds and works. The user can trim things later if necessary.

Minimal sounds good, as long as the user can include and edit the generated parts in their source code repository. Things might become tricky though when later you do an update and rerun native-with-agent since you will have to do the clean up again.

[zakkak]

Very interesting and less complicated than I expected. Nice work @galderz.

[zakkak]

Ultimately we will need the corresponding code for gradle.

[maxandersen]

don't we control the exeuction of tests and run enough to run with different agents? feel like i recall @stuartwdouglas or @geoand mention we could register agents?

that could avoid need of manipulating pom.xml/build.gradle?

[geoand]

@maxandersen we can fully control the launch of @QuarkusIntegrationTest. The JVM used for @QuarkusTest (and all "unit" tests for that matter) is controlled by the surefire plugin in Maven.

[galderz]

Do we prefer minimal filtering that we now for sure Quarkus takes care of? Or do we prefer aggressive filtering? I think I would probably go for minimal because the aim here is to get something that native image builds and works. The user can trim things later if necessary.

Minimal sounds good, as long as the user can include and edit the generated parts in their source code repository. Things might become tricky though when later you do an update and rerun native-with-agent since you will have to do the clean up again.

This got me thinking.

I think the best way to use this integration would be to run it locally, with GraalVM installed locally, take the generated config and add it to your source tree, as you already suggest above. The main reason for doing this is because we would not be able to execute native-with-agent profile with in-container builds.

If the user then wants to add additional configuration manually via JSON, it would work best if the files were differently named to the ones that were added to the source tree as a result of native-image-agent execution. That would make it easy to re-run native-with-agent and replace those files.

So, going back to your other comment about different files vs reasons, I think different files would work better given the above.

[galderz]

Thanks all for the feedback. I've had a quick chat with @zakkak to clarify a few things. I'll work on the next iteration of the feature and I'll post an update when ready.

[galderz]

I've just pushed a new prototype that works like this:

• The maven project generation still adds a new profile native-with-agent and that makes surefire unit tests run with native image agent.
• The native image agent spits out the configuration into target/native-agent-base-config folder for maven.
• There is no longer filtering done at the agent level and instead all the configuration filtering is delegated to the quarkus build inside NativeImageAgentConfigStep.
• NativeImageAgentConfigStep reads the native image agent json using the new JsonReader class and that is then transformed using the new JsonTransform interface, combined with the existing Json class that emits json (The transformation API is inspired by how the upcoming Class-File API in OpenJDK works). So, there are no longer intermediate build items created. It just takes json in, transforms it and spits it out onto the folder <source-jar-dir>/native-agent-config folder for maven.
• The JsonTransform API only has one function, which is to drop json from the input. That's all that is needed for this use case, but more functions could be added for future use cases.
• The Json API has been enhanced to add a new skipEscape option to avoid escaping strings where they are already escape. That's the case with transformation where the input json is already escaped and hence does not need to be escaped again.
• I've created different regular expression filters for each configuration file. The regular expressions are simple: if there is a reflection class FQN containing the word quarkus, it's filtered out...etc. For each configuration file type, there's a different lambda that figures out at which level the filtering needs to happen.
• The contents of native-agent-config are hooked into native image via -H:ConfigurationFileDirectories= option. @zakkak Looking at the source code this option is stable (iow not experimental), so seems like a better option that individual file configuration parameters.
• integration-tests/maven/src/test/java/io/quarkus/maven/it/NativeAgentIT has been added that verifies that native image integration unit tests work for a project that needs reflection configuration to succeed. See associated project code.
• integration-tests/gradle/src/test/java/io/quarkus/gradle/nativeimage/NativeAgentIntegrationTestIT.java has been added that verifies the same thing but gradle projects.

I have the following questions, doubts or comments:

• The maven project unit test bakes the new profile that would be seen by new projects generated which is not ideal, but I don't see tests that generate maven projects on the file. Any thoughts on this?
• The Gradle project is in integration-tests/gradle/src/main/resources/it-test-native-agent-project and there I manually added the test jvmArgs to contain the native image agent parameters (note that the folder where it spits it out is instead build/native-agent-base-config). I do see gradle project generation is in quarkus, but how would you add a profile (ala -Dnative-with-agent in maven) in Gradle so that those jvmArgs are only added when the given profile/option is active?
• Seems to me the generic type in Json.JsonBuilder adds unnecessary complexity. Its self method, which is the only one that uses the generic is not used at all.
• It probably makes sense to have a single regular expression that works for all configuration file types, because in the end what is filtered is common. This avoids duplication of keywords to match. It is also true that sometimes you're filtering FQN classes and other times resource paths and so the point could be made that finer tuned regular expressions would result in different regex for different configuration files, but seems to me that would makes regex overcomplicated.
• I explored adding quarkus:run integration for maven and that might be doable, but the equivalent feature for Gradle needs further work (see Gradle quarkusRun does not keep the server up #37379), so I think it's better to defer that to a later stage.

[vsevel]

when you are generating the missing declarations, it would be nice to display it somewhere readable. for instance in the build logs.
this would allow the developer to figure out what he missed, and give him the option to add the declaration explicitly in the app.

[zakkak]

Great work, @galderz. I like the new approach even better :)

[galderz]

when you are generating the missing declarations, it would be nice to display it somewhere readable. for instance in the build logs. this would allow the developer to figure out what he missed, and give him the option to add the declaration explicitly in the app.

Fair enough. I could add some log messages that specify the folders where the native image agent generated config files and the transformed ones are located. Does that sound good?

[vsevel]

I could add some log messages that specify the folders where the native image agent generated config files and the transformed ones are located.

ideally I would give more control to the developer. at least in our env, dev teams do not have access to the CI workspace. they have to go through the CI team to get an extract.
I would rather print a log message saying that 26 classes have been added to the reflection json file, and offer to specify an option -DdisplayNativeAgentFileContent=true to show the content. unless you think this could be huge (we could even add a quality gate searching for this pattern).
ideally I would like the dev team to be autonomous. I suspect the CI team is going to be bored pretty quickly. and I suspect they will not want to provide access to the nodes.

[galderz]

I could add some log messages that specify the folders where the native image agent generated config files and the transformed ones are located.

ideally I would give more control to the developer. at least in our env, dev teams do not have access to the CI workspace. they have to go through the CI team to get an extract. I would rather print a log message saying that 26 classes have been added to the reflection json file, and offer to specify an option -DdisplayNativeAgentFileContent=true to show the content. unless you think this could be huge (we could even add a quality gate searching for this pattern). ideally I would like the dev team to be autonomous. I suspect the CI team is going to be bored pretty quickly. and I suspect they will not want to provide access to the nodes.

Ok, I understand the situation now. I think an option to show native image agent configuration files in the console would make sense to cover such scenario, but it would work best if only the post-processed configuration would be shown. So, not what comes out directly out of the native image agent, but after having that filtered out to remove things that Quarkus takes care of.

For the most basic of Quarkus applications that has a just returns a greeting message, these are the sizes of files generated by native image agent:

$ l ./new-project/target/native-agent-base-config
total 872
drwxr-xr-x   9 galder  staff   288B  7 Dec 09:45 .
drwxr-xr-x  18 galder  staff   576B  7 Dec 09:46 ..
-rw-r--r--   1 galder  staff   967B  7 Dec 09:45 jni-config.json
-rw-r--r--   1 galder  staff    86B  7 Dec 09:45 proxy-config.json
-rw-r--r--   1 galder  staff   406K  7 Dec 09:45 reflect-config.json
-rw-r--r--   1 galder  staff   8.6K  7 Dec 09:45 resource-config.json
-rw-r--r--   1 galder  staff   2.1K  7 Dec 09:45 serialization-config.json

The biggest file there is by quite a big marging reflect-config.json with 406K. Once post-processed I see:

$ l ./new-project/target/new-project-1.0.0-SNAPSHOT-native-image-source-jar/native-agent-config
total 152
drwxr-xr-x  7 galder  staff   224B  7 Dec 09:45 .
drwxr-xr-x  6 galder  staff   192B  7 Dec 09:46 ..
-rw-r--r--  1 galder  staff   785B  7 Dec 09:45 jni-config.json
-rw-r--r--  1 galder  staff     2B  7 Dec 09:45 proxy-config.json
-rw-r--r--  1 galder  staff    55K  7 Dec 09:45 reflect-config.json
-rw-r--r--  1 galder  staff   4.8K  7 Dec 09:45 resource-config.json
-rw-r--r--  1 galder  staff   767B  7 Dec 09:45 serialization-config.json

The reflection configuration has dropped to 55K. Technically this could be further brought down but it's not easy for Quarkus to know exactly, because these remaining registrations are all related to java, javax and sun packages, so the application itself could have caused those to be required. So, the post-processing is quite conservative here, but if printed out they can always be added to the source code and do further filtering there if needed.

[vsevel]

So, not what comes out directly out of the native image agent, but after having that filtered out to remove things that Quarkus takes care of.

yes

but if printed out they can always be added to the source code and do further filtering there if needed.

yes exactly
having the agent to detect something means that the developer has forgotten declaring some resources in the app. and we are down to having the tests to identify the missing pieces. if a test is disabled at some point, we could forget declaring some resources, and get a regression on the native build.
it is kind of awkward if you think about it: activating or deactivating a test now has an impact on the built app, whereas tests should only observe and report.

[vsevel]

if/when your reporting feature is available, I would instruct the teams to review the generated content systematically, as it is smell to detect something. I think this should be logged as a WARN

[galderz]

I had a follow up chat with @aloubyansky last week and I'm working on a further iteration of this. I'm also including the feedback that @vsevel provided. I will update once I have something to show once again.

[galderz]

@aloubyansky Pushed commits to fix your comments.

[aloubyansky]

Thanks!

[quarkus-bot]

Status for workflow Quarkus Documentation CI

This is the status report for running Quarkus Documentation CI on commit 1970ed6.

✅ The latest workflow run for the pull request has completed successfully.

It should be safe to merge provided you have a look at the other checks in the summary.

⚠️ There are other workflow runs running, you probably need to wait for their status before merging.

[quarkus-bot]

Status for workflow Quarkus CI

This is the status report for running Quarkus CI on commit 1970ed6.

✅ The latest workflow run for the pull request has completed successfully.

It should be safe to merge provided you have a look at the other checks in the summary.

You can consult the Develocity build scans.

---

Flaky tests - Develocity

⚙️ JVM Tests - JDK 17

📦 extensions/smallrye-reactive-messaging-kafka/deployment

✖ io.quarkus.smallrye.reactivemessaging.kafka.deployment.dev.KafkaDevServicesDevModeTestCase.sseStream - History

• Assertion condition defined as a Lambda expression in io.quarkus.smallrye.reactivemessaging.kafka.deployment.dev.KafkaDevServicesDevModeTestCase Expecting size of: [] to be greater than or equal to 2 but was 0 within 10 seconds. - org.awaitility.core.ConditionTimeoutException

org.awaitility.core.ConditionTimeoutException: 
Assertion condition defined as a Lambda expression in io.quarkus.smallrye.reactivemessaging.kafka.deployment.dev.KafkaDevServicesDevModeTestCase 
Expecting size of:
  []
to be greater than or equal to 2 but was 0 within 10 seconds.
	at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
	at org.awaitility.core.AssertionCondition.await(AssertionCondition.java:119)
	at org.awaitility.core.AssertionCondition.await(AssertionCondition.java:31)

---

⚙️ JVM Tests - JDK 21

📦 integration-tests/reactive-messaging-kafka

✖ io.quarkus.it.kafka.KafkaConnectorTest.testFruits - History

• Assertion condition defined as a Lambda expression in io.quarkus.it.kafka.KafkaConnectorTest expected: <6> but was: <5> within 10 seconds. - org.awaitility.core.ConditionTimeoutException

org.awaitility.core.ConditionTimeoutException: Assertion condition defined as a Lambda expression in io.quarkus.it.kafka.KafkaConnectorTest expected: <6> but was: <5> within 10 seconds.
	at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
	at org.awaitility.core.AssertionCondition.await(AssertionCondition.java:119)
	at org.awaitility.core.AssertionCondition.await(AssertionCondition.java:31)
	at org.awaitility.core.ConditionFactory.until(ConditionFactory.java:1006)
	at org.awaitility.core.ConditionFactory.untilAsserted(ConditionFactory.java:790)
	at io.quarkus.it.kafka.KafkaConnectorTest.testFruits(KafkaConnectorTest.java:63)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)

[vsevel]

awesome @galderz

[vsevel]

just tried out in just released 3.12
works like a charm @galderz
thanks and congrats!