log correlation is broken for at least one configuration

[jackshirazi]

Describe the bug

app with log4j2 logger, instrumented with APM, with log_ecs_reformatting=OVERRIDE and log_sending=true, code using log.error("error with exception", new RuntimeException("My Runtime Exception for error")) fails to write the message ("error with exception") to the log

Also correlation IDs are missing when log_ecs_reformatting=OVERRIDE is set. Looks like it last worked in 1.29.0

Steps to reproduce

• using java agent version 1.30.0 or later (reproduced with 1.51.0 and later)
• create an application with log4j2
• package the application using a "shaded jar" where all the classes are put in a single jar

Analysis

The log correlation instrumentation is not applied when the application is packaged as a "fat jar" where the META-INF/MANIFEST.MF from log4j2 jar is overwritten by the one generated for the application (for example to make it an executable jar).

The log correlation instrumentation has two variants for log4j 2.6 to < 2.7 and for 2.7 and later, due to the lack of version to parse none is applied.

Manually removing the test on manifest version in the agent allows to solve the issue.

There are only a few instrumentations that rely on reading the library version from the manifest by overriding ElasticApmInstrumentation#getProtectionDomainPostFilter, most of the "per version compatibility testing" is done by overriding ElasticApmInstrumentation#getClassLoaderMatcher and test on the availability of known classes (which is also quite likely for major versions).
However if the changes are very minor between versions then this approach might not be always possible, and that will always require an in-depth investigation of multiple jars to ensure this approach is feasible.

The list of instrumentations that rely on getProtectionDomainPostFilter is:

• mongodb 3 and 4
• log4j2 log correlation
• httpclient 4.x

For the libraries that are packaged with maven like log4j2-core, they do contain /META-INF/maven/org.apache.logging.log4j/log4j-core/pom.properties file from which we could also parse the version. Because the path is unique to the library when building a shaded jar application with either maven or gradle the default is to include those so they are likely preserved most of the time.

I think this latter approach could be relevant to implement as a fallback when the version can't be read from the manifest. This fallback would apply for log4j2, but won't for Mongodb 3.x for example.

[SylvainJuge]

Issue description have been updated with my analysis.

In short, the issue is due to the agent not being able to properly detect log4j2 version and thus not applying log correlation instrumentation.