Error sending telemetry items: instrumentationKey must be non-null

[jepperip]

I am trying to send logs from a Apache Camel application running in a Docker container to Application Insights.
I have created an Application Insight resource and provided the application with my connection string.
I can connect to the app via the Live Metrics in AI and see that logs are coming in, but when I try to view historic transaction I see no logs.

When looking in the applicationinsights-logs I can see that the agent is starting without complaints. But, after running a while I get this:

WARN  c.m.a.a.i.telemetry.TelemetryChannel - In the last 5 minutes, the following operation has failed 20 times (out of 20): Sending telemetry to the ingestion service (telemetry will be stored to disk on failure)::
 * Error sending telemetry items: instrumentationKey must be non-null when using ikey redirect policy (20 times)

When looking at the source-code the instrumenation key seems to be missing from the context in com.microsoft.applicationinsights.agent.internal.httpclient.RedirectPolicy

I've tried providing the connection string in the applicationinsights.json-file and via the env-var APPLICATIONINSIGHTS_CONNECTION_STRING without seeing any difference.
I've also tried to provide the instrumentation key explicitly by providing it via APPINSIGHTS_INSTRUMENTATIONKEY without success.

I have tried running a similar setup outside the Docker container, with Camel and the same connection string, and it works fine.

Now, I understand that there may be multiple causes to why I'm not seeing my logs in AI so for this report I'm focusing on the error I get in the logs.

Might be relevant that I get other exceptions before this, so I thought this was related: #1927 but I have upgraded to v3.2.2 as suggested in that issue.

Expected behavior

No error regarding missing instrumentation key when connection string has been provided.

Actual behavior

Getting error "instrumentationKey must be non-null when using ikey redirect policy" when connection string has been provided.

System information

Please provide the following information:

• SDK Version: Java 8
• OS type and version: Linux version 4.19.128-microsoft-standard (oe-user@oe-host) (gcc version 8.2.0 (GCC)) Configured the code to work with Java 6 standards and using gradlew #1 SMP Tue Jun 23 12:58:10 UTC 2020
• Java 8
• Using spring-boot? Yes
• Additional relevant libraries (with version, if applicable): Apache Camel v2.25.1, applicationinsights-agent-3.2.2.

Logs

applicationinsights.log
The logs is from 3 different applications. They all run the same Java and Linux versions. You can see that I have tried different configurations. My last run starts at line 513. I get all kinds of fun exceptions during my runs.

[heyams]

did you notice this error in your log:

2021-10-28 23:43:43.572Z ERROR c.m.a.a.i.telemetry.TelemetryChannel - 
*************************
Unable to find valid certification path to requested target.
*************************

Description:
ApplicationInsights Java Agent failed to send telemetry data.

Action:
Please import the SSL certificate from https://northeurope-2.in.applicationinsights.azure.com/v2.1/track, into the default java key store located at:
/usr/lib/jvm/java-1.8-openjdk/jre/lib/security/cacerts
Learn more about importing the certificate here: https://go.microsoft.com/fwlink/?linkid=2151450

have you fixed the ssl cert issue? please follow the instruction listed under Action and then restart your app.

[trask]

@heyams that error was prior to the latest run:

My last run starts at line 513

It looks like dc.services.visualstudio.com is not resolving, and then the agent is falling back to storing telemetry to disk, and then retrying the telemetry from disk, at which time it doesn't have the ikey since the ikey is embedded in the telemetry data at that point, let's see if we can reproduce locally.

@jepperip in the meantime, can you check the connection string that is displayed in your Application Insights resource in the Azure Portal, and using that? It should have a different IngestionEndpoint in it which will (a) hopefully resolve, and (b) won't need to be further redirected (I think you are getting the redirect because you are using the generic endpoint dc.services.visualstudio.com).

[jepperip]

Thanks for the quick replies.

@heyams yes I have fixed that error. I understand that the logs are confusing because of 1. multiple runs and 2. its from 3 different containers. I'm in the processes of re-running the whole thing to produce a cleaner log file from only one container, if you need it.

@trask I am using the connection string from the portal in the applicationinsights.json config, without having changed it. It looks like this:
InstrumentationKey=XXXX;IngestionEndpoint=https://northeurope-2.in.applicationinsights.azure.com/

[heyams]

thanks @jepperip for clarification. we figured out what caused the failure. I'm working on a fix and will inform you when it becomes available.

[heyams]

fyi: the fix has been released in 3.2.3.