Removing triggers from ScaledObject doesn't reflect in the HPA it manages

[bishoybassem]

Report

I have a ScaledObject with several rabbitmq triggers, and when deleting one of them from the SO definition, changes are not propagated to the underlying HPA. Adding triggers work fine though.

Example SO:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: consumer
  namespace: keda2
spec:
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown:
          stabilizationWindowSeconds: 60
        scaleUp:
          stabilizationWindowSeconds: 60
  scaleTargetRef:
    name: consumer
  pollingInterval: 5
  minReplicaCount: 1
  maxReplicaCount: 10
  triggers:
  - type: rabbitmq
    metadata:
      protocol: amqp
      queueName: keda-tests
      queueLength: "5"
    authenticationRef:
      name: keda-trigger-auth-rabbitmq-conn
  - type: rabbitmq
    metadata:
      protocol: amqp
      queueName: keda-tests2
      queueLength: "5"
    authenticationRef:
      name: keda-trigger-auth-rabbitmq-conn

which creates the following HPA:

kubectl get hpa keda-hpa-consumer
NAME                REFERENCE             TARGETS                MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-consumer   Deployment/consumer   0/5 (avg), 0/5 (avg)   1         10        1          13s

modify the SO and delete the 2nd trigger:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: consumer
  namespace: keda2
spec:
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown:
          stabilizationWindowSeconds: 60
        scaleUp:
          stabilizationWindowSeconds: 60
  scaleTargetRef:
    name: consumer
  pollingInterval: 5
  minReplicaCount: 1
  maxReplicaCount: 10
  triggers:
  - type: rabbitmq
    metadata:
      protocol: amqp
      queueName: keda-tests
      queueLength: "5"
    authenticationRef:
      name: keda-trigger-auth-rabbitmq-conn

check the HPA:

kubectl get hpa keda-hpa-consumer
NAME                                  REFERENCE                               TARGETS                        MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-consumer                     Deployment/consumer                     0/5 (avg), <unknown>/5 (avg)   1         10        1          15m

notice that KEDA's metrics server is indeed not exposing metrics for the trigger that I deleted, hence the "unknown" current value, and can be confirmed in the events:

Events:
  Type     Reason                   Age                   From                       Message
  ----     ------                   ----                  ----                       -------
  Warning  FailedGetExternalMetric  24s (x26 over 6m47s)  horizontal-pod-autoscaler  unable to get external metric keda2/rabbitmq-keda-tests2/&LabelSelector{MatchLabels:map[string]string{scaledObjectName: consumer,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: No matching metrics found for rabbitmq-keda-tests2

Expected Behavior

Keda operator should propagate all SO modifications to the underlying HPA

Actual Behavior

The HPA stays with old triggers, which could cause unexpected behaviors.

Steps to Reproduce the Problem

1. create a SO with multiple triggers
2. check SO & HPA resources
3. delete one of the SO triggers and apply
4. check SO (doesn't show the deleted trigger) & HPA resources (still contains the deleted metric)

Logs from KEDA operator

2021-03-25T08:44:34.107Z	INFO	controllers.ScaledObject	Creating a new HPA	{"ScaledObject.Namespace": "keda2", "ScaledObject.Name": "consumer", "HPA.Namespace": "keda2", "HPA.Name": "keda-hpa-consumer"}
2021-03-25T08:44:34.225Z	INFO	controllers.ScaledObject	Initializing Scaling logic according to ScaledObject Specification	{"ScaledObject.Namespace": "keda2", "ScaledObject.Name": "consumer"}
2021-03-25T08:44:34.298Z	INFO	controllers.ScaledObject	Reconciling ScaledObject	{"ScaledObject.Namespace": "keda2", "ScaledObject.Name": "consumer"}
2021-03-25T08:44:39.012Z	INFO	controllers.ScaledObject	Reconciling ScaledObject	{"ScaledObject.Namespace": "keda2", "ScaledObject.Name": "consumer"}
2021-03-25T08:54:54.408Z	INFO	controllers.ScaledObject	Reconciling ScaledObject	{"ScaledObject.Namespace": "keda2", "ScaledObject.Name": "consumer"}
2021-03-25T08:54:54.427Z	INFO	controllers.ScaledObject	Initializing Scaling logic according to ScaledObject Specification	{"ScaledObject.Namespace": "keda2", "ScaledObject.Name": "consumer"}
2021-03-25T08:55:06.532Z	INFO	controllers.ScaledObject	Reconciling ScaledObject	{"ScaledObject.Namespace": "keda2", "ScaledObject.Name": "consumer"}

KEDA Version

2.2.0

Kubernetes Version

1.18

Platform

Any

Scaler Details

RabbitMQ