Wildfly agent issue

[montyz]

I'm trying to run the disco agent with wildfly 19.0.0. It's acting as if if doesn't know where the disco-java-agent.jar is, but it obviously found it to initialize X-Ray.

Picked up JAVA_TOOL_OPTIONS: -javaagent:/Users/monty/lark/goldfinch/wildfly-config/wildfly-modules/software/amazon/disco/main/disco-java-agent.jar=pluginPath=/Users/monty/lark/goldfinch/wildfly-config/wildfly-modules/software/amazon/disco/main/disco-plugins
Feb 19, 2021 11:52:45 PM com.amazonaws.xray.agent.runtime.config.XRaySDKConfiguration init
INFO: Initializing the X-Ray Agent Recorder
Feb 19, 2021 11:52:49 PM org.jboss.msc.service.ServiceContainerImpl <clinit>
INFO: JBoss MSC version 1.4.11.Final
Feb 19, 2021 11:52:49 PM org.jboss.threads.Version <clinit>
INFO: JBoss Threads version 2.3.3.Final
java.lang.NoClassDefFoundError: software/amazon/disco/agent/concurrent/ExecutorInterceptor$ExecuteAdvice
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceContainerImpl$ContainerExecutor.execute(ServiceContainerImpl.java:945)
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceControllerImpl.doExecute(ServiceControllerImpl.java:795)
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceControllerImpl.commitInstallation(ServiceControllerImpl.java:313)
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceContainerImpl.install(ServiceContainerImpl.java:752)
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceTargetImpl.install(ServiceTargetImpl.java:260)
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceBuilderImpl.install(ServiceBuilderImpl.java:192)
        at org.jboss.as.controller@11.0.0.Final//org.jboss.as.controller.ControlledProcessStateService.addService(ControlledProcessStateService.java:59)
        at org.jboss.as.server@11.0.0.Final//org.jboss.as.server.BootstrapImpl.internalBootstrap(BootstrapImpl.java:118)
        at org.jboss.as.server@11.0.0.Final//org.jboss.as.server.BootstrapImpl.bootstrap(BootstrapImpl.java:80)
        at org.jboss.as.server@11.0.0.Final//org.jboss.as.server.Main.main(Main.java:109)
        at org.jboss.modules.Module.run(Module.java:352)
        at org.jboss.modules.Module.run(Module.java:320)
        at org.jboss.modules.Main.main(Main.java:593)
Feb 19, 2021 11:52:49 PM org.jboss.as.server.SystemExiter$1 logExit
FATAL: WFLYSRV0239: Aborting with exit code 1

Any idea what might be going wrong?

Thanks!

[willarmiros]

Hi @montyz thanks for raising this. This is rather strange that clearly the X-Ray component of premain is being executed but there's a problem with finding one of the disco libraries. Can you try enabling Disco debug logging described here and posting those results? You can also try to use jar -tf disco-java-agent.jar to inspect its contents and see if the class software/amazon/disco/agent/concurrent/ExecutorInterceptor$ExecuteAdvice is actually unavailable in the JAR or just somehow not being found.

[montyz]

➜  .deploy git:(master) jar -tf /Users/monty/lark/goldfinch/wildfly-config/wildfly-modules/software/amazon/disco/main/disco-java-agent.jar | grep ExecutorInterceptor
software/amazon/disco/agent/concurrent/ExecutorInterceptor.class
software/amazon/disco/agent/concurrent/ExecutorInterceptor$ExecuteAdvice.class
software/amazon/disco/agent/concurrent/ScheduledThreadPoolExecutorInterceptor.class
software/amazon/disco/agent/concurrent/ScheduledThreadPoolExecutorInterceptor$DecorateTaskAdvice.class

=========================================================================

  JBoss Bootstrap Environment

  JBOSS_HOME: /Users/monty/lark/wildfly

  JAVA: /Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home/bin/java

  JAVA_OPTS:  -server  -Xms512M -Xmx4096M -Dpolyglot.js.nashorn-compat=true -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/goldfinch.hprof -Djava.net.preferIPv4Stack=true -Dlogging.level.com.amazonaws.xray=DEBUG  --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED

=========================================================================

Picked up JAVA_TOOL_OPTIONS: -javaagent:/Users/monty/lark/goldfinch/wildfly-config/wildfly-modules/software/amazon/disco/main/disco-java-agent.jar=pluginPath=/Users/monty/lark/goldfinch/wildfly-config/wildfly-modules/software/amazon/disco/main/disco-plugins:loggerfactory=software.amazon.disco.agent.reflect.logging.StandardOutputLoggerFactory:verbose
[software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) finished parsing argument list: pluginPath=/Users/monty/lark/goldfinch/wildfly-config/wildfly-modules/software/amazon/disco/main/disco-plugins:loggerfactory=software.amazon.disco.agent.reflect.logging.StandardOutputLoggerFactory:verbose
[software.amazon.disco.agent.plugin.PluginDiscovery] DiSCo(Core) attempting to load JAR file into system classloader: disco-java-agent-aws-plugin.jar
[software.amazon.disco.agent.plugin.PluginDiscovery] DiSCo(Core) attempting to load JAR file into system classloader: disco-java-agent-web-plugin.jar
[software.amazon.disco.agent.plugin.PluginDiscovery] DiSCo(Core) attempting to load JAR file into system classloader: disco-java-agent-sql-plugin.jar
[software.amazon.disco.agent.plugin.PluginDiscovery] DiSCo(Core) attempting to load JAR file into system classloader: aws-xray-agent-plugin.jar
[software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) passing arguments to DataAccessor to process
[software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) passing arguments to HttpServletServiceInterceptor to process
[software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) passing arguments to AWSClientInvokeInterceptor to process
[software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) passing arguments to DataAccessor to process
[software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) passing arguments to JdbcExecuteInterceptor to process
[software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) passing arguments to AWSClientBuilderInterceptor to process
[software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) passing arguments to ExecutorInterceptor to process
[software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) passing arguments to ThreadInterceptor to process
[software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) passing arguments to ApacheHttpClientInterceptor to process
[software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) passing arguments to DataAccessor to process
[software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) passing arguments to DataAccessor to process
[software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) passing arguments to DataAccessor to process
[software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) passing arguments to ForkJoinPoolInterceptor to process
[software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) passing arguments to ForkJoinTaskInterceptor to process
[software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) passing arguments to ScheduledThreadPoolExecutorInterceptor to process
[software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) passing arguments to ThreadSubclassInterceptor to process
[software.amazon.disco.agent.interception.InterceptionInstaller] DiSCo(Core) attempting to install software.amazon.disco.agent.interception.templates.DataAccessor
[software.amazon.disco.agent.interception.InterceptionInstaller] DiSCo(Core) attempting to install software.amazon.disco.agent.web.servlet.HttpServletServiceInterceptor
[software.amazon.disco.agent.interception.InterceptionInstaller] DiSCo(Core) attempting to install software.amazon.disco.agent.awsv1.AWSClientInvokeInterceptor
[software.amazon.disco.agent.interception.InterceptionInstaller] DiSCo(Core) attempting to install software.amazon.disco.agent.interception.templates.DataAccessor
[software.amazon.disco.agent.interception.InterceptionInstaller] DiSCo(Core) attempting to install software.amazon.disco.agent.sql.JdbcExecuteInterceptor
[software.amazon.disco.agent.interception.InterceptionInstaller] DiSCo(Core) attempting to install software.amazon.disco.agent.awsv2.AWSClientBuilderInterceptor
[software.amazon.disco.agent.interception.InterceptionInstaller] DiSCo(Core) attempting to install software.amazon.disco.agent.concurrent.ExecutorInterceptor
[software.amazon.disco.agent.interception.InterceptionInstaller] DiSCo(Core) attempting to install software.amazon.disco.agent.concurrent.ThreadInterceptor
[software.amazon.disco.agent.interception.InterceptionInstaller] DiSCo(Core) attempting to install software.amazon.disco.agent.web.apache.httpclient.ApacheHttpClientInterceptor
[software.amazon.disco.agent.interception.InterceptionInstaller] DiSCo(Core) attempting to install software.amazon.disco.agent.interception.templates.DataAccessor
[software.amazon.disco.agent.interception.InterceptionInstaller] DiSCo(Core) attempting to install software.amazon.disco.agent.interception.templates.DataAccessor
[software.amazon.disco.agent.interception.InterceptionInstaller] DiSCo(Core) attempting to install software.amazon.disco.agent.interception.templates.DataAccessor
[software.amazon.disco.agent.interception.InterceptionInstaller] DiSCo(Core) attempting to install software.amazon.disco.agent.concurrent.ForkJoinPoolInterceptor
[software.amazon.disco.agent.interception.InterceptionInstaller] DiSCo(Core) attempting to install software.amazon.disco.agent.concurrent.ForkJoinTaskInterceptor
[software.amazon.disco.agent.interception.InterceptionInstaller] DiSCo(Core) attempting to install software.amazon.disco.agent.concurrent.ScheduledThreadPoolExecutorInterceptor
[software.amazon.disco.agent.interception.InterceptionInstaller] DiSCo(Core) attempting to install software.amazon.disco.agent.concurrent.ThreadSubclassInterceptor
Feb 22, 2021 10:31:19 PM com.amazonaws.xray.agent.runtime.config.XRaySDKConfiguration init
INFO: Initializing the X-Ray Agent Recorder
[software.amazon.disco.agent.DiscoAgent] DiSCo(Agent) Plugin name: disco-java-agent-web-plugin.jar
        Bootstrap: no
        Installables: software.amazon.disco.agent.web.servlet.HttpServletServiceInterceptor, software.amazon.disco.agent.web.apache.httpclient.ApacheHttpClientInterceptor, software.amazon.disco.agent.interception.templates.DataAccessor, software.amazon.disco.agent.interception.templates.DataAccessor, software.amazon.disco.agent.interception.templates.DataAccessor, software.amazon.disco.agent.interception.templates.DataAccessor, software.amazon.disco.agent.interception.templates.DataAccessor

[software.amazon.disco.agent.DiscoAgent] DiSCo(Agent) Plugin name: disco-java-agent-sql-plugin.jar
        Bootstrap: no
        Installables: software.amazon.disco.agent.sql.JdbcExecuteInterceptor

[software.amazon.disco.agent.DiscoAgent] DiSCo(Agent) Plugin name: disco-java-agent-aws-plugin.jar
        Bootstrap: no
        Installables: software.amazon.disco.agent.awsv1.AWSClientInvokeInterceptor, software.amazon.disco.agent.awsv2.AWSClientBuilderInterceptor

[software.amazon.disco.agent.DiscoAgent] DiSCo(Agent) Plugin name: aws-xray-agent-plugin.jar
        Bootstrap: no
        Init: com.amazonaws.xray.agent.runtime.AgentRuntimeLoader

[software.amazon.disco.agent.DiscoAgent] DiSCo(Agent) agent startup complete
WARNING: Failed to load the specified log manager class org.jboss.logmanager.LogManager
Feb 22, 2021 10:31:24 PM org.jboss.msc.service.ServiceContainerImpl <clinit>
INFO: JBoss MSC version 1.4.11.Final
Feb 22, 2021 10:31:24 PM org.jboss.threads.Version <clinit>
INFO: JBoss Threads version 2.3.3.Final
java.lang.NoClassDefFoundError: software/amazon/disco/agent/concurrent/ExecutorInterceptor$ExecuteAdvice
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceContainerImpl$ContainerExecutor.execute(ServiceContainerImpl.java:945)
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceControllerImpl.doExecute(ServiceControllerImpl.java:795)
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceControllerImpl.commitInstallation(ServiceControllerImpl.java:313)
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceContainerImpl.install(ServiceContainerImpl.java:752)
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceTargetImpl.install(ServiceTargetImpl.java:260)
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceBuilderImpl.install(ServiceBuilderImpl.java:192)
        at org.jboss.as.controller@11.0.0.Final//org.jboss.as.controller.ControlledProcessStateService.addService(ControlledProcessStateService.java:59)
        at org.jboss.as.server@11.0.0.Final//org.jboss.as.server.BootstrapImpl.internalBootstrap(BootstrapImpl.java:118)
        at org.jboss.as.server@11.0.0.Final//org.jboss.as.server.BootstrapImpl.bootstrap(BootstrapImpl.java:80)
        at org.jboss.as.server@11.0.0.Final//org.jboss.as.server.Main.main(Main.java:109)
        at org.jboss.modules.Module.run(Module.java:352)
        at org.jboss.modules.Module.run(Module.java:320)
        at org.jboss.modules.Main.main(Main.java:593)
Feb 22, 2021 10:31:24 PM org.jboss.as.server.SystemExiter$1 logExit
FATAL: WFLYSRV0239: Aborting with exit code 1

Nothing obvious to me.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs in next 7 days. Thank you for your contributions.

[willarmiros]

Sorry for the delayed reply here. It seems like this is a problem with the X-Ray agent being incapable of instrumenting Wildfly server applications. Unfortunately our team is not able to prioritize adding this support at the moment and so we can't provide an ETA. However, X-Ray does have full compatibility with the latest stable release of the OpenTelemetry Java auto-instrumentation agent, which does have wildfly support. You can check out the OpenTelemetry Java for use with X-Ray documentation here.

In the meantime I'll leave this open as a feature request for wildfly support.

[adrian-baker]

I hit the same exception in a standard Spring Boot application. The cause ended up being that I had used the maven distribution (not the github zip release), which gives you a jar with a subtlety different MANIFEST.MF than the one in the release, and this relies on the name of the jar not being changed:

Boot-Class-Path: disco-java-agent-0.11.0.jar

In my case, my gradle script had renamed this to disco-java-agent.jar, which then seemed to trigger the NoClassDefFoundError.

Oddly even with the renamed jar it worked fine using JDK 16, but not on JDK 11.

So maybe just confirm whether the Boot-Class-Path entry inside disco-java-agent.jar/META-INF/MANIFEST.MF matches your actual file name.

Note that although using the git release is recommended, using the maven dist makes it easier to reuse gradle's internal dependency resolution, and the associated caching and parallelisation benefits.