Can not access msk with latest version (2.1.0)

[hoalt14]

Hi friends,

My version of MSK cluster is 3.5.1, and my client is an EC2 instance with IAM role attach exactly policy

I can't work the SASL mechanism = AWS_MSK_IAM with version 2.1.0 (It can not load credentials)

Here is the error logs:
ERROR org.apache.kafka.common.errors.SaslAuthenticationException: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: Failed to find AWS IAM Credentials [Caused by software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[AwsCredentialsProviderChain(credentialsProviders=[EnvironmentVariableCredentialsProvider(), SystemPropertyCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(sections=[])), ContainerCredentialsProvider()])]) ...

But i can work with version 1.1.1

Why? Maybe SDKv2 not working? Help me explain this issue.

Thank you!

[jvdadda]

Hi @hoalt14 , indeed, I miss something during migration, the fix is in PR: #169

[jogilsang]

Hi @hoalt14 , indeed, I miss something during migration, the fix is in PR: #169

still not working 2.1.0
but, It works in 2.0.3

My environment is as follows

• MSK Version (Apache Kafka) : 3.5.1
• Kafka Client : https://downloads.apache.org/kafka/3.7.0/kafka_2.12-3.7.0.tgz

# input
kafka-topics.sh --bootstrap-server $MSK_BOOTSTRAP_ADDRESS --command-config /tmp/client_iam.properties --list

# output
[2024-05-01 16:27:36,380] WARN Exception loading credentials. Retry Attempts: 0 (software.amazon.msk.auth.iam.internals.MSKCredentialProvider)
software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[AwsCredentialsProviderChain(credentialsProviders=[EnvironmentVariableCredentialsProvider(), SystemPropertyCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(sections=[])), ContainerCredentialsProvider()])]) : [AwsCredentialsProviderChain(credentialsProviders=[EnvironmentVariableCredentialsProvider(), SystemPropertyCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(sections=[])), ContainerCredentialsProvider()]): Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[EnvironmentVariableCredentialsProvider(), SystemPropertyCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(sections=[])), ContainerCredentialsProvider()]) : [EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., WebIdentityTokenCredentialsProvider(): Either the environment variable AWS_WEB_IDENTITY_TOKEN_FILE or the javaproperty aws.webIdentityTokenFile must be set., ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(sections=[])): Profile file contained no credentials for profile 'default': ProfileFile(sections=[]), ContainerCredentialsProvider(): Cannot fetch credentials from container - neither AWS_CONTAINER_CREDENTIALS_FULL_URI or AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set.]]
        at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
        at software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain.resolveCredentials(AwsCredentialsProviderChain.java:130)
        at software.amazon.msk.auth.iam.internals.MSKCredentialProvider.loadCredentialsWithRetry(MSKCredentialProvider.java:175)
        at software.amazon.msk.auth.iam.internals.MSKCredentialProvider.resolveCredentials(MSKCredentialProvider.java:162)
        at software.amazon.msk.auth.iam.IAMClientCallbackHandler.handleCallback(IAMClientCallbackHandler.java:99)
        at software.amazon.msk.auth.iam.IAMClientCallbackHandler.handle(IAMClientCallbackHandler.java:77)
        at software.amazon.msk.auth.iam.internals.IAMSaslClient.generateClientMessage(IAMSaslClient.java:139)
        at software.amazon.msk.auth.iam.internals.IAMSaslClient.evaluateChallenge(IAMSaslClient.java:96)
        at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.lambda$createSaslToken$1(SaslClientAuthenticator.java:535)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
        at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.createSaslToken(SaslClientAuthenticator.java:535)
        at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendSaslClientToken(SaslClientAuthenticator.java:434)
        at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendInitialToken(SaslClientAuthenticator.java:333)
        at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.authenticate(SaslClientAuthenticator.java:274)
        at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:181)
        at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
        at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
        at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:585)
        at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1504)
        at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1435)
        at java.base/java.lang.Thread.run(Thread.java:829)

[jvdadda]

@jogilsang The fix is not deployed inside the 2.1.0, so it is "normal" to have it, a maintainer should initiate a release

[kloometropolis]

Will this fix be released soon?

[yermulnik]

We're hitting the same. Can the release be expedited please?

[hhkkxxx133]

Hello, we are targeting on the release by the end of June. Thank you for your continued interest!

[guanlinz]

Hi there,
Facing the same issue, here

[sidyag]

Fix has been released.

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.