[BUG] Azure Queue Job Scaler continues to scale up Jobs while the message is being treated

[thomas-lamure]

My problem may look like the issue #525 but the analysis made on this thread does not match mine.
I prefer starting a new one here, if I am mistaken thank you for your help putting this message in the right place.

Expected Behavior

• When I post N messages in the queue, Azure Queue Job Scaler should create only N Jobs to treat the messages.
• When the Jobs read the messages they are made 'invisible' to Peak/Get Messages functions.
• Then, after pollingInterval, when the scaler read the queue there is N 'invisible' messages and 0 'visible' messages.
• The scaler should not create more jobs because there is no 'visible' messages.

Actual Behavior

• When I post N messages in the queue, Azure Queue Job Scaler creates N Jobs to treat the messages.
• When the Jobs read the messages they are made 'invisible' to Peak/Get Messages functions.
• But, after pollingInterval, when the scaler read the queue there is N 'invisible' messages and 0 'visible' messages.
• The scaler creates more jobs because it looks for ApproximateMessagesCount and this returns the global count of messages.

Steps to Reproduce the Problem

1. Deploy a ScaledObject with a scaleType: job deploying a long running task container witch reads one message in the queue and updates its visibility_timeout regularly during the treatment.
2. Put 2 messages in the queue
3. Observe the pods in your cluster and the queue using Azure storage explorer during at least 2 polling intervals
4. You will see no visible messages in your queue but KEDA will scale up the jobs until you reach the queueLength scaler metadata limit (5 by default).

Specifications

• Version: keda.k8s.io/v1alpha1
• Platform: Azure AKS
• Scaler(s): azure-queue

Fix Suggestion

• In /pkg/scalers/azure_queue.go do not read the default Azure Queue Metadata ApproximateMessagesCount.
• As Azure Queue does not give the VisibleMessagesCount Metadata, it is not possible to request directly this information.
• What I think Azure storage explorer is doing:
  • Peak all the messages and calculate the length of the queue based on that.
• Or maybe if someone has its ways into Microsoft here, he can request a new feature on GetProperties to add VisibleMessagesCount

Thank you for your help 😸

[TsuyoshiUshio]

Hi I implement not to refer ApproximateMessagesCount and check the number of the visible queues at least under 32 queues on the PR that is merged.

[zroubalik]

@TsuyoshiUshio so can we close this one then?

[TsuyoshiUshio]

@zroubalik Yes. I implement visible queue count for Azure Storage Queue.

[TsuyoshiUshio]

Also, you can use Scaling strategy as accurate it will perfectly solve this issue. #1227