Connecting to a load balancer puts wrong node IP in ring

[jameshartig]

Please answer these questions before submitting your issue. Thanks!

What version of Cassandra are you using?

Yugabyte 2.8

What version of Gocql are you using?

v1.0.0

What version of Go are you using?

go 1.17

What did you do?

cluster := gocql.NewCluster("my-load-balancer:9042")

What did you expect to see?

yugabyte host up -- addr="192.168.0.2" datacenter="us-east1"
yugabyte host up -- addr="192.168.0.4" datacenter="us-east1"
yugabyte host up -- addr="192.168.0.3" datacenter="us-east1"

What did you see instead?

yugabyte host up -- addr="192.168.0.2" datacenter="us-east1"
yugabyte host up -- addr="192.168.0.4" datacenter="us-east1"
yugabyte host up -- addr="192.168.0.5" datacenter="us-east1"
...
gocql: events: unable to fetch host info for (192.168.0.3:9042): host not found in peers table

The issue is that the controlConn.setupConn method calls conn.localHostInfo which ends up sending along the connectAddress to hostInfoFromMap and that always prefers the connectAddress when determining how it should connect to the node.

Unfortunately the AddressTranslator interface doesn't give the function access to the HostInfo so there's no way for me to know that 192.168.0.5 should actually be 192.168.0.3.

I had thought to do something like:

gocql.HostFilterFunc(func(host *gocql.HostInfo) bool {
	host.SetConnectAddress(host.RPCAddress())
	return true
}

but the host filter is not called for the control connection. Calling it for the control connection feels like it's the right thing to do but if it returns false we would need to set up a new control connection using another host.

Finally, we could change the control connection setup to not pass connectAddress the first time but there might be people relying on that.

---

If you are having connectivity related issues please share the following additional information

Describe your Cassandra cluster

my-load-balancer resolves to 192.168.0.5

admin@ycqlsh> SELECT peer,rpc_address FROM system.peers;

 peer        | rpc_address
-------------+-------------
 192.168.0.3 | 192.168.0.3
 192.168.0.4 | 192.168.0.4

(2 rows)
admin@ycqlsh> SELECT key, rpc_address FROM system.local;

 key   | rpc_address
-------+-------------
 local | 192.168.0.2

[martin-sucha]

I wonder if something like dialer with host argument would help in this case 🤔 #1487 (comment)

[jameshartig]

Yeah the dialer interface described in that comment would allow you to always dial the correct address and having the host information would be necessary.