Support profiling for keda components (kedacore#549)

Signed-off-by: yuval weber <yuval199985@gmail.com>
Signed-off-by: unknown <yuval199985@gmail.com>
Co-authored-by: Tom Kerkhove <kerkhove.tom@gmail.com>

keda/README.md

@@ -232,6 +232,17 @@ their default values.
 | `prometheus.webhooks.serviceMonitor.targetLabels` | list | `[]` | TargetLabels transfers labels from the Kubernetes `Service` onto the created metrics |
 | `prometheus.webhooks.serviceMonitor.targetPort` | string | `""` | Name or number of the target port of the Pod behind the Service, the port must be specified with container port property. Mutually exclusive with port |
 
+### Troubleshooting
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `profiling.metricsServer.enabled` | bool | `false` | Enable profiling for KEDA metrics server |
+| `profiling.metricsServer.port` | int | `8083` | Expose profiling on a specific port |
+| `profiling.operator.enabled` | bool | `false` | Enable profiling for KEDA operator |
+| `profiling.operator.port` | int | `8082` | Expose profiling on a specific port |
+| `profiling.webhooks.enabled` | bool | `false` | Enable profiling for KEDA admission webhook |
+| `profiling.webhooks.port` | int | `8084` | Expose profiling on a specific port |
+
 ### Admission Webhooks
 
 | Parameter | Type | Default | Description |
@@ -245,23 +256,6 @@ their default values.
 | `podDisruptionBudget.webhooks` | object | `{}` | Capability to configure [Pod Disruption Budget] |
 | `podLabels.webhooks` | object | `{}` | Pod labels for KEDA Admission webhooks |
 | `podSecurityContext.webhooks` | object | [See below](#KEDA-is-secure-by-default) | [Pod security context] of the KEDA admission webhooks |
-| `prometheus.webhooks.enabled` | bool | `false` | Enable KEDA admission webhooks prometheus metrics expose |
-| `prometheus.webhooks.port` | int | `8080` | Port used for exposing KEDA admission webhooks prometheus metrics |
-| `prometheus.webhooks.prometheusRules.additionalLabels` | object | `{}` | Additional labels to add for KEDA admission webhooks using prometheusRules crd (prometheus operator) |
-| `prometheus.webhooks.prometheusRules.alerts` | list | `[]` | Additional alerts to add for KEDA admission webhooks using prometheusRules crd (prometheus operator) |
-| `prometheus.webhooks.prometheusRules.enabled` | bool | `false` | Enables PrometheusRules creation for the Prometheus Operator |
-| `prometheus.webhooks.prometheusRules.namespace` | string | `""` | Scraping namespace for KEDA admission webhooks using prometheusRules crd (prometheus operator) |
-| `prometheus.webhooks.serviceMonitor.additionalLabels` | object | `{}` | Additional labels to add for metric server using ServiceMonitor crd (prometheus operator) |
-| `prometheus.webhooks.serviceMonitor.enabled` | bool | `false` | Enables ServiceMonitor creation for the Prometheus webhooks |
-| `prometheus.webhooks.serviceMonitor.interval` | string | `""` | Interval at which metrics should be scraped If not specified Prometheus’ global scrape interval is used. |
-| `prometheus.webhooks.serviceMonitor.jobLabel` | string | `""` | jobLabel selects the label from the associated Kubernetes service which will be used as the job label for all metrics. [ServiceMonitor Spec] |
-| `prometheus.webhooks.serviceMonitor.podTargetLabels` | list | `[]` | PodTargetLabels transfers labels on the Kubernetes `Pod` onto the created metrics |
-| `prometheus.webhooks.serviceMonitor.port` | string | `"metrics"` | Name of the service port this endpoint refers to. Mutually exclusive with targetPort |
-| `prometheus.webhooks.serviceMonitor.relabelings` | list | `[]` | List of expressions that define custom relabeling rules for metric server ServiceMonitor crd (prometheus operator). [RelabelConfig Spec] |
-| `prometheus.webhooks.serviceMonitor.relabellings` | list | `[]` | DEPRECATED. List of expressions that define custom relabeling rules for metric server ServiceMonitor crd (prometheus operator). [RelabelConfig Spec] |
-| `prometheus.webhooks.serviceMonitor.scrapeTimeout` | string | `""` | Timeout after which the scrape is ended If not specified, the Prometheus global scrape timeout is used unless it is less than Interval in which the latter is used |
-| `prometheus.webhooks.serviceMonitor.targetLabels` | list | `[]` | TargetLabels transfers labels from the Kubernetes `Service` onto the created metrics |
-| `prometheus.webhooks.serviceMonitor.targetPort` | string | `""` | Name or number of the target port of the Pod behind the Service, the port must be specified with container port property. Mutually exclusive with port |
 | `resources.webhooks` | object | `{"limits":{"cpu":"50m","memory":"100Mi"},"requests":{"cpu":"10m","memory":"10Mi"}}` | Manage [resource request & limits] of KEDA admission webhooks pod |
 | `securityContext.webhooks` | object | [See below](#KEDA-is-secure-by-default) | [Security context] of the admission webhooks container |
 | `topologySpreadConstraints.webhooks` | list | `[]` | [Pod Topology Constraints] of KEDA admission webhooks pod |

keda/README.md.gotmpl

@@ -69,7 +69,7 @@ their default values.
 | Parameter | Type | Default | Description |
 |-----------|------|---------|-------------|
 {{- range .Values }}
-  {{- if and (or (contains "operator" .Key) (contains "keda" .Key)) (not (or (contains "opentelemetry" .Key) (contains "prometheus" .Key))) }}
+  {{- if and (or (contains "operator" .Key) (contains "keda" .Key)) (not (or (contains "opentelemetry" .Key) (contains "prometheus" .Key) (contains "profiling" .Key))) }}
 | `{{ .Key }}` | {{ .Type }} | {{ if .Default }}{{ .Default }}{{ else }}{{ .AutoDefault }}{{ end }} | {{ if .Description }}{{ .Description }}{{ else }}{{ .AutoDescription }}{{ end }} |
   {{- end }}
 {{- end }}
@@ -79,7 +79,7 @@ their default values.
 | Parameter | Type | Default | Description |
 |-----------|------|---------|-------------|
 {{- range .Values }}
-  {{- if and (or (contains "metricServer" .Key) (contains "metricsServer" .Key) (contains "metricsApiServer" .Key) (contains "metricsAdapter" .Key) (hasPrefix "service." .Key)) (not (or (contains "opentelemetry" .Key) (contains "prometheus" .Key)))}}
+  {{- if and (or (contains "metricServer" .Key) (contains "metricsServer" .Key) (contains "metricsApiServer" .Key) (contains "metricsAdapter" .Key) (hasPrefix "service." .Key)) (not (or (contains "opentelemetry" .Key) (contains "prometheus" .Key) (contains "profiling" .Key)))}}
 | `{{ .Key }}` | {{ .Type }} | {{ if .Default }}{{ .Default }}{{ else }}{{ .AutoDefault }}{{ end }} | {{ if .Description }}{{ .Description }}{{ else }}{{ .AutoDescription }}{{ end }} |
   {{- end }}
 {{- end }}
@@ -94,12 +94,22 @@ their default values.
   {{- end }}
 {{- end }}
 
+### Troubleshooting
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+{{- range .Values }}
+  {{- if (contains "profiling" .Key) }}
+| `{{ .Key }}` | {{ .Type }} | {{ if .Default }}{{ .Default }}{{ else }}{{ .AutoDefault }}{{ end }} | {{ if .Description }}{{ .Description }}{{ else }}{{ .AutoDescription }}{{ end }} |
+  {{- end }}
+{{- end }}
+
 ### Admission Webhooks
 
 | Parameter | Type | Default | Description |
 |-----------|------|---------|-------------|
 {{- range .Values }}
-  {{- if contains "webhooks" .Key }}
+  {{- if and (contains "webhooks" .Key) (not (or (contains "opentelemetry" .Key) (contains "prometheus" .Key) (contains "profiling" .Key))) }}
 | `{{ .Key }}` | {{ .Type }} | {{ if .Default }}{{ .Default }}{{ else }}{{ .AutoDefault }}{{ end }} | {{ if .Description }}{{ .Description }}{{ else }}{{ .AutoDescription }}{{ end }} |
   {{- end }}
 {{- end }}

keda/templates/manager/deployment.yaml

@@ -93,6 +93,9 @@ spec:
           {{- if .Values.opentelemetry.operator.enabled }}
           - "--enable-opentelemetry-metrics={{ .Values.opentelemetry.operator.enabled}}"
           {{- end }}
+          {{- if .Values.profiling.operator.enabled }}
+          - "--profiling-bind-address=:{{ .Values.profiling.operator.port }}"
+          {{- end }}
           {{- range $key, $value := .Values.extraArgs.keda }}
           - "--{{ $key }}={{ $value }}"
           {{- end }}

keda/templates/manager/service.yaml

@@ -34,5 +34,10 @@ spec:
     port: {{ .Values.prometheus.operator.port }}
     targetPort: {{ .Values.prometheus.operator.port }}
   {{- end }}
+  {{- if .Values.profiling.operator.enabled }}
+  - name: profiling
+    port: {{ .Values.profiling.operator.port }}
+    targetPort: {{ .Values.profiling.operator.port }}
+  {{- end }}
   selector:
     app: {{ .Values.operator.name }}

keda/templates/metrics-server/deployment.yaml

@@ -129,6 +129,9 @@ spec:
           - --tls-private-key-file={{ .Values.certificates.mountPath }}/tls.key
           - --cert-dir={{ .Values.certificates.mountPath }}
           - --v={{ .Values.logging.metricServer.level }}
+          {{- if .Values.profiling.metricsServer.enabled }}
+          - "--profiling-bind-address=:{{ .Values.profiling.metricsServer.port }}"
+          {{- end }}
           {{- range $key, $value := .Values.extraArgs.metricsAdapter }}
           - --{{ $key }}={{ $value }}
           {{- end }}

keda/templates/metrics-server/service.yaml

@@ -35,5 +35,11 @@ spec:
     port: {{ .Values.prometheus.metricServer.port }}
     targetPort: {{ .Values.prometheus.metricServer.port }}
     protocol: TCP
+  {{- if .Values.profiling.metricsServer.enabled }}
+  - name: profiling
+    port: {{ .Values.profiling.metricsServer.port }}
+    targetPort: {{ .Values.profiling.metricsServer.port }}
+    protocol: TCP
+  {{- end }}
   selector:
     app: {{ .Values.operator.name }}-metrics-apiserver

keda/templates/webhooks/deployment.yaml

@@ -79,6 +79,9 @@ spec:
           - "--port={{ .Values.webhooks.port }}"
           {{- end }}
           - --metrics-bind-address=:{{ .Values.prometheus.webhooks.port }}
+          {{- if .Values.profiling.webhooks.enabled }}
+          - "--profiling-bind-address=:{{ .Values.profiling.webhooks.port }}"
+          {{- end }}
           {{- range $key, $value := .Values.extraArgs.webhooks }}
           - --{{ $key }}={{ $value }}
           {{- end }}

keda/templates/webhooks/service.yaml

@@ -36,6 +36,11 @@ spec:
     port: {{ .Values.prometheus.webhooks.port }}
     targetPort: {{ .Values.prometheus.webhooks.port }}
   {{- end }}
+  {{- if .Values.profiling.webhooks.enabled }}
+  - name: profiling
+    port: {{ .Values.profiling.webhooks.port }}
+    targetPort: {{ .Values.profiling.webhooks.port }}
+  {{- end }}
   selector:
     app: {{ .Values.webhooks.name }}
 {{- end }}

keda/values.yaml

@@ -461,6 +461,27 @@ http:
   # -- The minimum TLS version to use for all scalers that use raw HTTP clients (some scalers use SDKs to access target services. These have built-in HTTP clients, and this value does not necessarily apply to them)
   minTlsVersion: TLS12
 
+## This setting lets you enable profiling for all of the components of KEDA and in the specific port you choose
+## This can be useful when trying to investigate errors like memory leaks or CPU or even look at goroutines to understand better
+## This setting is disabled by default
+profiling:
+  operator:
+    # -- Enable profiling for KEDA operator
+    enabled: false
+    # -- Expose profiling on a specific port
+    port: 8082
+  metricsServer:
+    # -- Enable profiling for KEDA metrics server
+    enabled: false
+    # -- Expose profiling on a specific port
+    port: 8083
+  webhooks:
+    # -- Enable profiling for KEDA admission webhook
+    enabled: false
+    # -- Expose profiling on a specific port
+    port: 8084
+
+
 ## Extra KEDA Operator and Metrics Adapter container arguments
 extraArgs:
   # -- Additional KEDA Operator container arguments