SASL Handshake for AzureEventHub should be V0

[prune998]

Versions

regularly.*
Sarama Version: 0dbba7e
Kafka Version: 1.x
Go Version: 1.12

Configuration

Kafka is from Azure EventHub and is opaque. It is using kafka protobol > 1.0
Sarama is setup as producer using :

	config := sarama.NewConfig()
	config.ClientID = "eventhub-exporter"

	config.Producer.RequiredAcks = sarama.WaitForAll
	config.Producer.Retry.Max = 3
	config.Producer.Return.Successes = true

	config.Version = sarama.V1_1_0_0
	config.Net.TLS.Enable = true
	config.Net.TLS.Config = &tls.Config{
		InsecureSkipVerify: true,
		ClientAuth:         0,
	}

	config.Net.SASL.Enable = true
	config.Net.SASL.Handshake = true
	config.Net.SASL.User = "$ConnectionString"
	config.Net.SASL.Mechanism = "PLAIN"
config.Net.SASL.Password = "Endpoint=sb://xxx.servicebus.windows.net/;SharedAccessKeyName=OnlySend;SharedAccessKey=xxx;EntityPath=eh_test"

Logs

{"level":"info","msg":"starting...","time":"2019-06-20T14:36:19+02:00"}
{"level":"debug","logger":"sarama","msg":"[Initializing new client]","time":"2019-06-20T14:36:19+02:00"}
{"level":"debug","logger":"sarama","msg":"[xxx.servicebus.windows.net:9093]","time":"2019-06-20T14:36:19+02:00"}
{"level":"debug","logger":"sarama","msg":"[xxx.servicebus.windows.net:9093 read tcp 10.213.143.23:58886-\u003exxx:9093: read: connection reset by peer]","time":"2019-06-20T14:36:19+02:00"}
{"level":"debug","logger":"sarama","msg":"[xxx.servicebus.windows.net:9093 write tcp 10.213.143.23:58886-\u003exxx:9093: write: broken pipe]","time":"2019-06-20T14:36:19+02:00"}
{"level":"debug","logger":"sarama","msg":"[-1 read tcp 10.213.143.23:58886-\u003exxx:9093: read: connection reset by peer]","time":"2019-06-20T14:36:19+02:00"}
{"level":"debug","logger":"sarama","msg":"[client/metadata no available broker to send metadata request to]","time":"2019-06-20T14:36:19+02:00"}

Problem Description

Azure Eventhub is using old SASL V0 with Kafka > 1.0.
Sarama, since https://github.com/Shopify/sarama/pull/1354/files is forcing SASL V1 when using Kafka > 1.0, which leads to connexion closed during SASL authentication.

Someone cloned and patch Sarama but I think it was never PRed : evandigby@6509b6f

Maybe the SASL version should be set in a config.Net.SASL.Version parameter ?

I'm sure it's Azure which is not right here, but the fact is that we can't use EventHub with Sarama out of the box. Confluent is using V0 by default, as most java applications do.

I can make a PR if you think it's OK to either make it configurable or change how the decision is made.
Thanks

Reproduce

Create an EventHub in Azure and use the producer at https://github.com/liliankasem/sarama-eventhub-kafka/blob/master/producer/producer.go

[varun06]

Make sense, Please PR and I will have a look and merge. Thanks.

[dnwe]

@prune998 👋 – it was me who made the original change to the default under PR #1354 as I believe the V1 behaviour with a proper authentication response mechanism is highly desirable as the out-of-the-box experience for users

I checked and it does appear that an ApiVersionResponse from an EventHubs Kafka does report that only ApiKey SaslHandshake (17) is available and ApiKey SaslAuthenticate (36) is not returned in the ApiVersionResponse. This is the reason why the Java client continues to work. Normally it would also default to the V1 -> SaslAuthenticate flow, but because the ApiVersionResponse told it that SaslAuthenticate wasn't available, it fell back to the legacy auth request (see SaslClientAuthenticator.java#L213). Rdkafka-based clients all do a similar process.

@varun06 / @bai should we similarly be calling b.ApiVersions(&ApiVersionsRequest) within b.Open() and passing the ApiVersionsResponse struct into b.authenticateViaSASL() to choose the default mechanism? Would you be happy to look at a PR that added that behaviour, and @prune998 would you be happy to test such a PR against EventHubs?

[prune998]

that's a great searching work @dnwe. I'll be happy to test this as soon as available.
I have a meeting with Microsoft/Azure guys tomorrow so I'll be happy to report then that. Can I assume they do not respect the protocol, or do no answer what they should ?

[dnwe]

No they're fine. I have no inside knowledge but I'm guessing they're not running a genuine Apache Kafka broker and have instead implemented their own proxy layer that talks the kafka protocol and remuxes it to Event Hubs calls at the backend. Because they've implemented the ApiVersion flow for clients to determine what features are available it will work with clients that query and adhere to that. I'm guessing it implements some subset of
Features that put it somewhere between 0.10 and 1.0 in capabilities

[d1egoaz]

This seems already fixed by #1410.

Feel free to re-open the issue if need additional help.