KEDA Autoscale jobs - Not working as expected

[sai25590]

I have installed KEDA 2.0 Beta version. I want it to scale jobs based on a message in Azure service bus queue. For every message in queue I want a job created with a max of 100 parallel jobs. I have trigger auth and managed AAD pod identity to access SB in KEDA.

# https://keda.sh/docs/1.4/concepts/scaling-jobs/ This is for KEDA version 2.0
apiVersion: keda.sh/v1alpha1
kind: ScaledJob
namespace: default # must be same namespace as the Triggerauth
metadata:
  name: autofab-configurator-scaledobject
spec:
  jobTargetRef:
    parallelism: 1 # [max number of desired pods](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#controlling-parallelism)
    completions: 1 # [desired number of successfully finished pods](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#controlling-parallelism)
    activeDeadlineSeconds: 14400 # If a job takes more than 4 hours it is terminated
    backoffLimit: 6 # Specifies the number of retries before marking this job failed. Defaults to 6
    ttlSecondsAfterFinished: 100
    template:
      # describes the [job template](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/)
      metadata:
        labels:
          app: autofab-configurator
          aadpodidbinding: azureresourceaccess
        name: autofab-configurator
      name: autofab-configurator
      spec:   
        containers:
          - name: autofab-configurator
            image: ${acrlogin}/${autofabconfiguratorimage}:${version}
            imagePullPolicy: Always
            resources:
              requests:
                cpu: ${cpurequest} 
            volumeMounts:
              - mountPath: /mnt/azure
                name: volume
            env:
              - name: SERVICEBUS_NAMESPACE
                value: ${sbnamespace}
              - name: CONFIG_QUEUE_NAME
                value: "autofab-configq"
        restartPolicy: Never
        volumes:
          - name: volume
            persistentVolumeClaim:
              claimName: azurefile
      
  pollingInterval: 30 # Optional. Default: 30 seconds
  successfulJobsHistoryLimit: 5
  failedJobsHistoryLimit: 5
  maxReplicaCount: 100 # Max number of pods of this type that can run at the same time.
  triggers:
  # {list of triggers to create jobs}
  - type: azure-servicebus
    metadata:
      # Required: queueName and namespace OR topicName and subscriptionName
      queueName: autofab-configq
      namespace: ${sbnamespace}
      messageCount: "1" # Optional. Default: 5 messages
    authenticationRef:
      name: autoscale-servicebus-triggerauth

When I send two messages in the service bus (both messages have same data) they are processed in serial instead of parallel.

I see this error in KEDA metrics pod Cannot get list of ScaledObjects" "error"="etcdserver: request timed out" "WatchedNamespace"="":

D:\git\Networking-Tycoon\src\AVS2.0-AutoFabric>kubectl logs keda-metrics-apiserver-b559564cb-tw9mh -n keda
I1001 02:11:08.622221       1 main.go:84] keda_metrics_adapter "msg"="KEDA Version: v2"
I1001 02:11:08.622281       1 main.go:85] keda_metrics_adapter "msg"="KEDA Commit: 717aab81817f0d9b6d02b0e6d1499367c751faf2"
I1001 02:11:08.622290       1 main.go:86] keda_metrics_adapter "msg"="Go Version: go1.15.1"
I1001 02:11:08.622297       1 main.go:87] keda_metrics_adapter "msg"="Go OS/Arch: linux/amd64"
I1001 02:11:10.273015       1 request.go:621] Throttling request took 1.047381802s, request: GET:https://100.73.46.1:443/apis/storage.k8s.io/v1beta1?timeout=32s
I1001 02:11:11.125405       1 provider.go:49] keda_metrics_adapter/provider "msg"="starting"
I1001 02:11:11.125461       1 main.go:115] keda_metrics_adapter "msg"="starting adapter..."
2020-10-01 02:11:11.125486 I | Starting metrics server at :9022
I1001 02:11:11.626936       1 serving.go:307] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
I1001 02:11:11.917697       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1001 02:11:11.917731       1 shared_informer.go:223] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1001 02:11:11.917842       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1001 02:11:11.917860       1 shared_informer.go:223] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1001 02:11:11.918003       1 dynamic_serving_content.go:130] Starting serving-cert::apiserver.local.config/certificates/apiserver.crt::apiserver.local.config/certificates/apiserver.key
I1001 02:11:11.918249       1 secure_serving.go:178] Serving securely on [::]:6443
I1001 02:11:11.918276       1 tlsconfig.go:240] Starting DynamicServingCertificateController
I1001 02:11:12.017833       1 shared_informer.go:230] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1001 02:11:12.017894       1 shared_informer.go:230] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
E1001 06:26:29.186441       1 provider.go:135] keda_metrics_adapter/provider "msg"="Cannot get list of ScaledObjects" "error"="etcdserver: request timed out"  "WatchedNamespace"=""

I see no errors in KEDA operator:

D:\git\Networking-Tycoon\src\AVS2.0-AutoFabric>kubectl logs keda-operator-6c8ff569b5-m94h8 -n keda
...
...
2020-10-01T10:32:57.305Z        INFO    scalehandler    Active trigger  {"Scaler": {}, "isTriggerActive": false}
2020-10-01T10:32:57.305Z        INFO    scalehandler    Scaler targetAverageValue       {"Scaler": {}, "targetAverageValue": 1}
2020-10-01T10:32:57.318Z        INFO    scalehandler    QueueLength Metric value        {"Scaler": {}, "queueLength": 0}
2020-10-01T10:32:57.318Z        INFO    scalehandler    Scaler maxValue {"maxValue": 0}
2020-10-01T10:32:57.318Z        INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "autofab-configurator-scaledobject", "scaledJob.Namespace": "default", "Number of running Jobs": 0}

[zroubalik]

@TsuyoshiUshio PTAL

[TsuyoshiUshio]

Hi @sai25590
It looks the same root cause of #1186 however, I'll do more research for the provider.go error message. For the scaling logic, I'll provide fix soon.

[TsuyoshiUshio]

Hi @sai25590
I provide the fix and merge. Please wait until it is released.
#1214