JetStream scaler should query the stream consumer leader when clustered

[rayjanoka]

Report

I found that when running a cluster of nats pods, only the stream consumer's leader pod reports the accurate number of messages in the queue.

Expected Behavior

The scaler needs to identify the nats jetstream pod that is the selected stream's consumer leader to get the accurate metric.

Actual Behavior

The scaler is designed to connect to the general k8s svc endpoint so it will connect to a random nats server.

Steps to Reproduce the Problem

1. Add messages to a jetstream queue, consume some of them, then add more
2. Get the leader pod - nats consumer info test-stream durable
3. Query each pod directly via the headless svc to get the number of pending messages

# Consumer Leader (accurate)
➜ curl -s "http://nats-1.nats.nats.svc.cluster.local:8222/jsz?consumers=true" | grep num_pending
              "num_pending": 0,

# Replicas (counting up only)
➜ curl -s "http://nats-0.nats.nats.svc.cluster.local:8222/jsz?consumers=true" | grep num_pending
              "num_pending": 6,
➜ curl -s "http://nats-2.nats.nats.svc.cluster.local:8222/jsz?consumers=true" | grep num_pending
              "num_pending": 6,

Logs from KEDA operator

No response

KEDA Version

2.8.1

Kubernetes Version

1.23

Platform

Any

Scaler Details

JetStream

Anything else?

No response

[rayjanoka]

PR: #3564

[rayjanoka]

Includes a new e2e test to validate a 3-node nats jetstream cluster with stream configurations of 1, 2, and 3 stream replicas.