Cannot connect to publicly accessible MSK using IAM credentials

[abhishek-parative]

I am following the instructions set here https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html and here https://docs.aws.amazon.com/msk/latest/developerguide/create-topic.html to access my MSK instance.

However, my IAM credentials are not being picked up.

This is what my ~/.aws/config file looks like:

[default]
sso_session = FOOBAR
sso_account_id = 0123456789012
sso_role_name = AdministratorAccess
region = us-west-2
output = json


[profile sample]
sso_session = FOOBAR
sso_account_id = 0123456789012
sso_role_name = AdministratorAccess
sso_region = us-west-2
region = us-west-2
output = json
sso_start_url = foobar_url
sso_registration_scopes = sso:account:access

This is what my client.properties looks like

security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config = software.amazon.msk.auth.iam.IAMLoginModule required awsProfileName="sample";
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler

I also added the aws-msk-iam-auth-1.1.1-all.jar to the kafka/libs folder.

Error Trace:

[2024-05-10 11:23:32,005] ERROR [AdminClient clientId=adminclient-1] Connection to node -1 (REDACTED) failed authentication due to: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: Failed to find AWS IAM Credentials [Caused by aws_msk_iam_auth_shadow.com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [software.amazon.msk.auth.iam.internals.EnhancedProfileCredentialsProvider@1149414c: The SSO session associated with this profile has expired or is otherwise invalid. To refresh this SSO session run aws sso login with the corresponding profile., aws_msk_iam_auth_shadow.com.amazonaws.auth.AWSCredentialsProviderChain@581c9ec8: Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), WebIdentityTokenCredentialsProvider: You must specify a value for roleArn and roleSessionName, software.amazon.msk.auth.iam.internals.EnhancedProfileCredentialsProvider@6a2a7c1c: The SSO session associated with this profile has expired or is otherwise invalid. To refresh this SSO session run aws sso login with the corresponding profile., aws_msk_iam_auth_shadow.com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@60cc5251: Failed to connect to service endpoint: ]]]) occurred when evaluating SASL token received from the Kafka Broker. Kafka Client will go to AUTHENTICATION_FAILED state. (org.apache.kafka.clients.NetworkClient)
...
Caused by: javax.security.sasl.SaslException: Failed to find AWS IAM Credentials [Caused by aws_msk_iam_auth_shadow.com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [software.amazon.msk.auth.iam.internals.EnhancedProfileCredentialsProvider@1149414c: The SSO session associated with this profile has expired or is otherwise invalid. To refresh this SSO session run aws sso login with the corresponding profile., aws_msk_iam_auth_shadow.com.amazonaws.auth.AWSCredentialsProviderChain@581c9ec8: Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), WebIdentityTokenCredentialsProvider: You must specify a value for roleArn and roleSessionName, software.amazon.msk.auth.iam.internals.EnhancedProfileCredentialsProvider@6a2a7c1c: The SSO session associated with this profile has expired or is otherwise invalid. To refresh this SSO session run aws sso login with the corresponding profile., aws_msk_iam_auth_shadow.com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@60cc5251: Failed to connect to service endpoint: ]]]
	at software.amazon.msk.auth.iam.internals.IAMSaslClient.generateClientMessage(IAMSaslClient.java:148)
	at software.amazon.msk.auth.iam.internals.IAMSaslClient.evaluateChallenge(IAMSaslClient.java:95)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.lambda$createSaslToken$1(SaslClientAuthenticator.java:534)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.createSaslToken(SaslClientAuthenticator.java:534)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendSaslClientToken(SaslClientAuthenticator.java:433)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendInitialToken(SaslClientAuthenticator.java:332)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.authenticate(SaslClientAuthenticator.java:273)
	at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:181)
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
	at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:571)
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1413)
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1344)
	at java.base/java.lang.Thread.run(Thread.java:829)

I reauthenticated using aws sso login --profile sample and no dice

[Cyberness]

Same problem with MSF on version 2.1.0. Rollback to version 1.1.9 solved the issue.

[abhishek-parative]

When I use the aws-msk-iam-auth-1.1.9-all.jar, I get the following error:
To use SSO OIDC related properties in the 'sample' profile, the 'ssooidc' service module must be on the class path.

Stack trace:

[2024-05-10 14:38:15,817] WARN Exception loading credentials. Retry Attempts: 0 (software.amazon.msk.auth.iam.internals.MSKCredentialProvider)
aws_msk_iam_auth_shadow.com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [software.amazon.msk.auth.iam.internals.EnhancedProfileCredentialsProvider@713f1d49:


**To use SSO OIDC related properties in the 'sample' profile, the 'ssooidc' service module must be on the class path.**


aws_msk_iam_auth_shadow.com.amazonaws.auth.AWSCredentialsProviderChain@460ddbd6: Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), WebIdentityTokenCredentialsProvider: You must specify a value for roleArn and roleSessionName, software.amazon.msk.auth.iam.internals.EnhancedProfileCredentialsProvider@6c1c84bc: To use SSO OIDC related properties in the 'default' profile, the 'ssooidc' service module must be on the class path., aws_msk_iam_auth_shadow.com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@71f1a25b: Failed to connect to service endpoint: ]]
	at aws_msk_iam_auth_shadow.com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:142)
	at software.amazon.msk.auth.iam.internals.MSKCredentialProvider.loadCredentialsWithRetry(MSKCredentialProvider.java:154)
	at software.amazon.msk.auth.iam.internals.MSKCredentialProvider.getCredentials(MSKCredentialProvider.java:141)
	at software.amazon.msk.auth.iam.IAMClientCallbackHandler.handleCallback(IAMClientCallbackHandler.java:100)
	at software.amazon.msk.auth.iam.IAMClientCallbackHandler.handle(IAMClientCallbackHandler.java:77)
	...
	

[sbernardo]

I had same problem, I found this aws/aws-sdk-java-v2#2961 (comment) , seems need to update some libraries.

This solved for me as well:

Same problem with MSF on version 2.1.0. Rollback to version 1.1.9 solved the issue.

But maybe into version 2.1.1 updating aws libraries solves as well

[sidyag]

We have updated our libraries. Is this still an issue? Please create a new one if you are still facing this.

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.