GraalVM Native build needs runtime initialization command line parameters

[lord-fourth0107]

I was trying to authenticate a serviceclient using service principal . On using the azure-identity dependency , it builds a lot of reflections . I tried removing them using lazy initialization by using the --initialize-at-run-time , but even after including the java.security.SecureRandom class in the args , the class gets initialized at build time ,resulting a build failure. Without using azure identityin pom for service principal authentication, i was able to build a native build with a fully functional service client(Service client with Connection String). Please check the reproducer code's application.properties file for further details

Steps to reproduce the issue:
Reproducing the issue:

Clone the project from
(https://github.com/uttamsingh0107/quarkus-reproducable)
2.Run the command "mvn package -Pnative -Dquarkus.native.container-build=true"

Expected Output :
Build should be success

Actual Output:
/project:z --name build-native-dHJoo quay.io/quarkus/ubi-quarkus-native-image:21.3-java11 -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true -J-Dio.netty.leakDetection.level=DISABLED -J-Dio.netty.allocator.maxOrder=3 -J-Duser.language=en -J-Duser.country=US -J-Dfile.encoding=UTF-8 -H:-ParseOnce --initialize-at-run-time=com.microsoft.aad.msal4jextensions.persistence.linux.ISecurityLibrary,com.microsoft.aad.msal4jextensions.persistence.mac.ISecurityLibrary,com.sun.jna.platform.win32.Crypt32,com.sun.jna.platform.win32.Kernel32,java.security.SecureRandom,com.nimbusds.jwt.JWTClaimsSet --trace-class-initialization=java.security.SecureRandom -H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy$BySpaceAndTime -H:+JNI -H:+AllowFoldMethods -H:FallbackThreshold=0 -H:+ReportExceptionStackTraces -H:-AddAllCharsets -H:EnableURLProtocols=http -H:-UseServiceLoaderFeature -H:+StackTrace -H:AdditionalSecurityProviders=com.sun.security.sasl.Provider,org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslClientProvider,org.apache.kafka.common.security.scram.internals.ScramSaslClientProvider 2.0v-runner -jar 2.0v-runner.jar
[2.0v-runner:25] classlist: 8,111.60 ms, 2.68 GB
[2.0v-runner:25] (cap): 682.64 ms, 2.68 GB
[2.0v-runner:25] setup: 2,899.51 ms, 2.68 GB
11:12:21,924 INFO [com.azu.cor.imp.jac.JacksonVersion] Package versions: jackson-annotations=2.12.5, jackson-core=2.12.5, jackson-databind=2.12.5, jackson-dataformat-xml=2.12.5, jackson-datatype-jsr310=2.12.5, azure-core=1.21.0
To see how the classes got initialized, use --trace-class-initialization=java.security.SecureRandom
[nm-iot-egress-2.0v-runner:25] analysis: 9,061.83 ms, 2.64 GB
Error: Classes that should be initialized at run time got initialized during image building:
java.security.SecureRandom the class was requested to be initialized at run time (from the command line with 'java.security.SecureRandom'). To see why java.security.SecureRandom got initialized use --trace-class-initialization=java.security.SecureRandom

com.oracle.svm.core.util.UserError$UserException: Classes that should be initialized at run time got initialized during image building:
java.security.SecureRandom the class was requested to be initialized at run time (from the command line with 'java.security.SecureRandom'). To see why java.security.SecureRandom got initialized use --trace-class-initialization=java.security.SecureRandom

at com.oracle.svm.core.util.UserError.abort(UserError.java:73)
at com.oracle.svm.hosted.classinitialization.ConfigurableClassInitialization.checkDelayedInitialization(ConfigurableClassInitialization.java:555)
at com.oracle.svm.hosted.classinitialization.ClassInitializationFeature.duringAnalysis(ClassInitializationFeature.java:168)
at com.oracle.svm.hosted.NativeImageGenerator.lambda$runPointsToAnalysis$12(NativeImageGenerator.java:727)
at com.oracle.svm.hosted.FeatureHandler.forEachFeature(FeatureHandler.java:73)
at com.oracle.svm.hosted.NativeImageGenerator.runPointsToAnalysis(NativeImageGenerator.java:727)
at com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:529)
at com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:488)
at com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:403)
at com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:569)
at com.oracle.svm.hosted.NativeImageGeneratorRunner.main(NativeImageGeneratorRunner.java:122)
at com.oracle.svm.hosted.NativeImageGeneratorRunner$JDK9Plus.main(NativeImageGeneratorRunner.java:599)
[nm-iot-egress-2.0v-runner:25] [total]: 20,469.69 ms, 2.64 GB

Printing build artifacts to: /project/nm-iot-egress-2.0v-runner.build_artifacts.txt

Error: Image build request failed with exit status 1
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 46.364 s
[INFO] Finished at: 2021-12-02T16:42:25+05:30
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal io.quarkus:quarkus-maven-plugin:2.5.0.Final:build (default) on project nm-iot-egress: Failed to build quarkus application: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
[ERROR] [error]: Build step io.quarkus.deployment.pkg.steps.NativeImageBuildStep#build threw an exception: java.lang.RuntimeException: Failed to build native image
[ERROR] at io.quarkus.deployment.pkg.steps.NativeImageBuildStep.build(NativeImageBuildStep.java:233)
[ERROR] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[ERROR] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[ERROR] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[ERROR] at java.base/java.lang.reflect.Method.invoke(Method.java:566)
[ERROR] at io.quarkus.deployment.ExtensionLoader$2.execute(ExtensionLoader.java:887)
[ERROR] at io.quarkus.builder.BuildContext.run(BuildContext.java:277)
[ERROR] at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
[ERROR] at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2449)
[ERROR] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1478)
[ERROR] at java.base/java.lang.Thread.run(Thread.java:829)
[ERROR] at org.jboss.threads.JBossThread.run(JBossThread.java:501)
[ERROR] Caused by: java.lang.RuntimeException: Image generation failed. Exit code: 1
[ERROR] at io.quarkus.deployment.pkg.steps.NativeImageBuildStep.imageGenerationFailed(NativeImageBuildStep.java:369)
[ERROR] at io.quarkus.deployment.pkg.steps.NativeImageBuildStep.build(NativeImageBuildStep.java:213)
[ERROR] ... 11 more
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1

[timtay-microsoft]

Marking this as "Enhancement" since you are asking for us to support GraalVM

[jamdavi]

This is due to usage of SecureRandom in the DigestProxyChallengeProcessorImpl class up the chain. Change your Graalvm command line specified in your application.properties to this and it will build just fine. Notice I removed SecureRandom from the initialize-at-runtime command line.

If you experience issues with azure identity after the build you'll need to work with that team to understand why it's failing.

quarkus.native.additional-build-args=--initialize-at-run-time=com.microsoft.azure.proton.transport.proxy.impl.DigestProxyChallengeProcessorImpl\\,com.microsoft.aad.msal4jextensions.persistence.linux.ISecurityLibrary\\,com.microsoft.aadmsal4jextensions.persistence.linux.ISecurityLibrary\\,com.microsoft.aad.msal4jextensions.persistence.mac.ISecurityLibrary\\,com.sun.jna.platform.win32.Crypt32\\,com.sun.jna.platform.win32.Kernel32\\,com.nimbusds.jwt.JWTClaimsSet,--allow-incomplete-classpath

[lord-fourth0107]

Hi @jamdavi. Thank You for the reply . I tried to build the native image using the changes you suggested and the build was success. But when deploying the native image i now get some errors from Azure Identity dependency . This is basically due to non-availability of http client . When adding the http client dependency , the build fails. Similar but not exactly same error was found when i tried for jvm build . Let me know in case of any updates

[jamdavi]

@uttamsingh0107 I believe you have an internal ticket open on this and I'll respond with this same information there. I'll also close this issue based on my findings below. But this will be helpful for anyone else who needs this information.

I was able to get this to build with the following changes.

1. I removed the azure-core-http-netty exclusion under the azure-identity dependency in the POM

2. I changed the quarkus.native.additional-build-args parameters to the following:

quarkus.native.additional-build-args=--initialize-at-run-time=reactor.netty.tcp.TcpClientSecure\\,reactor.netty.tcp.SslProvider\\,reactor.netty.http.client.HttpClientSecure\\,reactor.netty.resources.DefaultLoopKQueue\\,reactor.netty.resources.DefaultLoopEpoll\\,reactor.netty.resources.DefaultLoopNativeDetector\\,com.microsoft.azure.proton.transport.proxy.impl.DigestProxyChallengeProcessorImpl\\,com.microsoft.aad.msal4jextensions.persistence.linux.ISecurityLibrary\\,com.microsoft.aadmsal4jextensions.persistence.linux.ISecurityLibrary\\,com.microsoft.aad.msal4jextensions.persistence.mac.ISecurityLibrary\\,com.sun.jna.platform.win32.Crypt32\\,com.sun.jna.platform.win32.Kernel32\\,com.nimbusds.jwt.JWTClaimsSet,--allow-incomplete-classpath,--enable-https

So, the question is, Is the build failing because there's a problem with my app or a dependency? And, when we do get it working it will fail when you run the app? Is this a problem with the IoT SDK? The Azure SDK? Netty?

Short answer

No.

The Long Answer

Since GraalVM (GVM) does ahead-of-time compilation, there are bound to be limitations and concerns about the system it's compiled on and the one it's compiled for. A very obvious example would be if I did a local native compile on Windows it won't work for Linux. Similarly, but more nuanced is, if I built on Linux with a 2.0 kernel it likely won't work on a 5.0 kernel. Just as with platform implementation, the same applies for CPU architectures and endianness.

GVM has a a large set of rules that it looks out for. You've been hit by two of them which are documented:

1. SecureRandom

2. Initialization at build time

SecureRandom

Since SecureRandom is meant to be read at the instant it's needed GVM will search for any instances of it in memory at build time and block on usage. It does this because if it captured state of the RNG entropy at build time it would be locked in forever and you'd have a static seed for all randoms. This is bad.

From the docs:

The SecureRandom implementations open the /dev/random and /dev/urandom files which are used as sources for entropy. These files are usually opened in class initializers. To avoid capturing state from the machine that runs the Native Image builder these classes need to be initialized at run time.

GVM's requirement is that "these classes need to be initialized at run time". So you need to trace back and find what class(s) instantiate(s) SecureRandom. Any library that uses SSL (or any RNG) will have this requirement.

Initialization at build time

If you notice in the build parameters above all but three Microsoft owned classes are specified. For all classes listed they are specified because they initialize SecureRandom in some way, but others are specified because they potentially initialize other classes that are not considered safe, or are unknown to GraalVM.

This is another requirement of ahead-of-time compilation and not due to a specific library implementation.

Conclusion

The exercises I did to get the build working required no changes to your application code, nor the Microsoft SDKs. But instead, required me to follow the GVM instructions and do some iterative changes to the build parameters to satisfy the GraalVM build. The same will be true for runtime exceptions, if libraries (like, azure-core-http-netty) are not included things will fail. If an library has platform dependent native hooks they might also fail.

The amount and depth of errors that arise from dependent classes we use is all due to how GVM handles ahead-of-time compilation and not because our library doesn't officially support GVM.

Officially, we haven't tested with GraalVM. There are likely scenarios and edge cases that won't work without some deep investgation. But, the exercises performed while investigating this ticket state that we likely do handle the majority of cases when GVM is setup correctly. The ground work would need to be done by the user of GVM to get the application built and run. Once we start seeing callstacks that point to our SDK not handling a specific scenario we can start to narrow down the issues.

For now the problems are not specific to our SDK and need GraalVM experts to get the application stood up.

[JayGhiya]

@jamdavi -> Thankyou for detailed answer. Couple of comments- this issue was created to get a graalvm compatible sdk. It is well known that graalvm does build time class initialization as you mentioned which is not a limitation but a trade off that developer decides for very low startup times(8x-10x lower time than jvm) and lower processing times. Thats what high performance and low footprint langs like golang and rust will do. So it is not definitely a limitation but a choice according to app requirements.

Now coming to making secure random work with graalvm requires changes and recommendations from graalvm experts. This was the intent of creating the github issue in the first place. Also -> there have been multiple libs that already use secure random and have been modified to work with graalvm. i would want to reopen the ticket.

[jamdavi]

All instances of SecureRandom in our library are instantiated at runtime. It's the transitive dependencies that are the issue here.

It appears that we can get a build with the existing SDK using well supported and documented methods.

I will move this issue to a discussion as it's a large feature request and might need to be part of our v2 efforts.