Auth failure when connecting from cross-region to MSK and cross-cloud

[nihalpot]

Currently, we were running into an issue with the MSK Client when it fails due to an authentication error both on cross-region (like let's say ec2 instance is in 'us-east-2' and msk instance is in 'us-west-2'), and cross-account (let's say we are trying to connect to msk from a gcp vm). As I understand this is the reasoning for the issue:

In the AuthenticationRequestParams, the msk client tries to segment the endpoint to find the region metadata using tryGetRegionByEndpointDnsSuffix. In the AWS SDK, if we have defined a regions.xml file the RegionMetadata relies on the LegacyRegionXmlMetadataBuilder to create this. This is built on the InMemoryRegionsProvider which does not have tryGetRegionByEndpointDnsSuffix defined (see here). Thereby, we are running on a node which will have regions.xml defined in the AWS SDK the msk client will default to the same region as that of the ec2 instance it is running on.

This blocks cross-region access since the region configured in AuthenticationRequestParams will be different from the actual region of the MSK broker. Similarly, Regions.getCurrentRegion won't work for non-ec2 instances so this will run into an error on gcp and azure environments.

My proposal to fix this is that the MSK Client should incorporate a awsRegion which essentially overwrites the region field in AuthenticationRequestParams. This is something that is already done for other clients like Kinesis, SNS. What do you think? Should I contribute towards this or can the AWS team build this feature?

[plazma-prizma]

Hello @nihalpot thanks for opening the issue. Please go ahead and send a PR. Could you also provide a setup on how we could re-produce this issue on our end? Also were you using MSK PrivateLInk? https://docs.aws.amazon.com/msk/latest/developerguide/aws-access-mult-vpc.html

[nihalpot]

@plazma-prizma thanks for the reply! I wasn't using MSK privatelink, I was using a public endpoint which, as I understand from the documentation, should support cross-region access. To confirm, is this required for cross-region access?

For instructions to reproduce the issue:

• create a MSK server in a region (let's say us-west-2) and a ec2 instance in different region (let's say us-east-2)

• enable public access for the MSK cluster with required permissions

• add a regions.xml file to aws sdk: "/com/amazonaws/regions/override/regions.xml" or use an installation of that which already has this regions.xml installed

• then use this library to try and connect to MSK instance using the Kafka client, for instance, here's some scala code that I was using to test this:

  import java.util.Properties
  import scala.collection.JavaConverters._
  
  val props = new Properties()
  val kafkatestOptions: Map[String, String] = Map(
      "subscribe" -> "msk-serverless-tutorial",
      "sasl.mechanism" -> "AWS_MSK_IAM",
      "sasl.jaas.config" ->
        "software.amazon.msk.auth.iam.IAMLoginModule required awsDebugCreds=true;",
      "security.protocol" -> "SASL_SSL", 
      "sasl.client.callback.handler.class" ->
        "software.amazon.msk.auth.iam.IAMClientCallbackHandler",
      "bootstrap.servers" -> <your servers>
  )
  for(key <- kafkatestOptions.keys) {
    val value: String = kafkatestOptions.get(key).getOrElse("none")
    props.put(key, value)
  }
  
  <add any IAM creds in env variables or system properties>
  
  val admin = Admin.create(props)
  val topics = Set("<your topic>")
  admin.describeTopics(topics.asJava).all().get().asScala```
  
  

• alternatively if you use a msk cluster in same region, I found that it worked

• I found that this didn't work across different cloud providers (let's say we use a google VM instead to connect to MSK)

In particular, we also just need the following lines of code to test this (pulled from internals/AuthenticationRequestParams.java):

val regionMetadata = RegionMetadataFactory.create()
println("test " + regionMetadata.tryGetRegionByEndpointDnsSuffix("ttt.msk.us-west-2.amazonaws.com"))

with regions.xml installed in aws sdk, this returns null so cross-region access doesn't work

[nihalpot]

@plazma-prizma hi I just added a PR here to fix this issue: #130, I have validated it using the process I described earlier

Could you please review this when you get a chance? thank you!

[GiacomoOrlandi-awin]

Hello, just to double-check, this issue has been fixed in v1.1.9, is that correct?

[plazma-prizma]

v1.1.8...v1.1.9 shows the diff between 1.1.8 and 1.1.9

[GiacomoOrlandi-awin]

@plazma-prizma the diff includes #134, which should fix this issue, so I was wondering why this issue is still open. Did the fix work as expected?

[plazma-prizma]

Hey @GiacomoOrlandi-awin , thanks for the heads up. Issue was just left open. I am closing it now. Please feel free to re-open if you think it wasn't solved.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.