2.1.1 release includes SaslAuthenticationException regression

[jamielwhite]

In #143, I reported an issue in which the first re-authentication failed for the OAUTHBEARER mechanism. This issue was resolved in release 2.0.3 for the case when awsRoleArn was provided. When I upgraded our apps from 2.0.3 to 2.1.1, I noticed our apps were restarting every hour due to a SaslAuthenticationException: Session too short error. The difference between the behavior now vs in #143 is that the failure occurs after one hour, not after 15 minutes when the role first expires.

I reported this in a comment on a related issue for the case where awsRoleArn is not provided, but since that error also occurs in 2.0.3 I've created this as a separate issue.

security.protocol=SASL_SSL
sasl.mechanism=OAUTHBEARER
sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required awsRoleArn="<role arn>" awsStsRegion="<region>";
sasl.login.callback.handler.class=software.amazon.msk.auth.iam.IAMOAuthBearerLoginCallbackHandler

[james1miller93]

We're also seeing this using version 2.1.0. Looking back through the comments from @jamielwhite, it seems we're running our services in exactly the same way (pods in a kubernetes cluster with iam credentials assigned via irsa roles). As mentioned above, the error is causing our apps to restart.

[sidyag]

I think this PR will fix this. Can you test with it? #182

[jamielwhite]

I'll test it out, but since we aren't using awsProfileName I'm not sure if that branch is going to be hit.

[sidyag]

Interesting. The configuration is provided with awsRoleArn? Let me try and reproduce it in that way.

[jamielwhite]

I tested the jar from the PR and confirmed it still results in the "Session too short" error after 1 hour with awsRoleArn.

[sidyag]

I added the fix for awsRoleArn #183

I think something similar is happening for default creds. Will try to fix that today.

[jamielwhite]

I pulled down the code from #183 and tried out the JAR, and unfortunately it still resulted in the same error after an hour.

[sidyag]

Is that with awsRoleArn or with default? The fix for default is on its way.

[jamielwhite]

It's with awsRoleArn. I inspected the jar and double-checked it includes the new asyncCredentialUpdateEnabled property being set to true.

[sidyag]

Okay. Found the issue. It is a documentation problem.

Try with the following configurations:

security.protocol=SASL_SSL
sasl.mechanism=OAUTHBEARER
sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required awsRoleArn="<role arn>" awsStsRegion="<region>";
sasl.login.callback.handler.class=software.amazon.msk.auth.iam.IAMOAuthBearerLoginCallbackHandler
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMOAuthBearerLoginCallbackHandler

[sidyag]

I have updated docs, and the default chain. Could you try now with mainline?

[jamielwhite]

With awsRoleArn and the addition of sasl.client.callback.handler.class, I'm able to run it longer than 1 hour without any errors 🎉 . I'll try out the default later today, but it will take longer to confirm since that error appeared after 2 hours.

[jamielwhite]

The case without awsRoleArn is also working for longer than 2 hours now. Thanks for the fix and documentation!

[james1miller93]

Thank you for your work on resolving this @sidyag.
Do you know when a release tag is likely to be published with the fix?

[sidyag]

Have published release 2.2.0. It should be available in the next 12-24 hours.

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.