Hashicorp vault auth fails using Pki

[JorTurFer]

Report

KEDA v2.15.0 shows the error error creating the TLS config: error parse X509KeyPair: tls: private key does not match public key"} using pki feature

2024-08-02T13:43:48Z	ERROR	prometheus_scaler	init Prometheus client http transport	{"type": "ScaledObject", "namespace": "hashicorp-vault-test-ns", "name": "hashicorp-vault-test-so", "error": "error creating the TLS config: error parse X509KeyPair: tls: private key does not match public key"}
github.com/kedacore/keda/v2/pkg/scalers.NewPrometheusScaler
	/workspace/pkg/scalers/prometheus_scaler.go:102
github.com/kedacore/keda/v2/pkg/scaling.buildScaler
	/workspace/pkg/scaling/scalers_builder.go:227
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers.func1
	/workspace/pkg/scaling/scalers_builder.go:85
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers
	/workspace/pkg/scaling/scalers_builder.go:90
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).performGetScalersCache
	/workspace/pkg/scaling/scale_handler.go:354
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScalersCache
	/workspace/pkg/scaling/scale_handler.go:282
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).getScaledObjectMetricSpecs
	/workspace/controllers/keda/hpa.go:217
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).newHPAForScaledObject
	/workspace/controllers/keda/hpa.go:72
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).createAndDeployNewHPA
	/workspace/controllers/keda/hpa.go:45
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
	/workspace/controllers/keda/scaledobject_controller.go:447
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
	/workspace/controllers/keda/scaledobject_controller.go:286
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
	/workspace/controllers/keda/scaledobject_controller.go:193
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227

[JorTurFer]

Based on the success executions, the last commit that passed was this: 5ff8f77

doing a checkout of this commit fec0756, the test starts to fail, so that's the commit which breaks it

[JorTurFer]

Just downgrading golang to 1.21 the issue is gone.
I've verified the values passed until here:

keda/pkg/scalers/authentication/authentication_helpers.go

Lines 156 to 163 in ed6dfdd

	func NewTLSConfig(auth *AuthMeta, unsafeSsl bool) (*tls.Config, error) {
	return kedautil.NewTLSConfig(
	auth.Cert,
	auth.Key,
	auth.CA,
	unsafeSsl,
	)
	}

Verifying the values with an external tool, with go 1.21 they match and using go 1.22 they don't match. So they are wrongly gotten at some point

[JorTurFer]

I found the problem, it's here:

keda/pkg/scaling/resolver/hashicorpvault_handler.go

Lines 312 to 352 in ed6dfdd

	func (vh *HashicorpVaultHandler) ResolveSecrets(secrets []kedav1alpha1.VaultSecret) ([]kedav1alpha1.VaultSecret, error) {
	// Group secret by path and type, this allows to fetch a path only once. This is useful for dynamic credentials
	grouped := make(map[SecretGroup][]kedav1alpha1.VaultSecret)
	vaultSecrets := make(map[SecretGroup]*vaultapi.Secret)
	for _, e := range secrets {
	group := SecretGroup{secretType: e.Type, path: e.Path, vaultPkiData: &e.PkiData}
	if _, ok := grouped[group]; !ok {
	grouped[group] = make([]kedav1alpha1.VaultSecret, 0)
	}
	grouped[group] = append(grouped[group], e)
	}
	// For each group fetch the secret from vault
	for group := range grouped {
	vaultSecret, err := vh.fetchSecret(group.secretType, group.path, group.vaultPkiData)
	if err != nil {
	// could not fetch secret, skipping group
	continue
	}
	vaultSecrets[group] = vaultSecret
	}
	// For each secret in each group, fetch the value and add to out
	out := make([]kedav1alpha1.VaultSecret, 0)
	for group, unFetchedSecrets := range grouped {
	vaultSecret := vaultSecrets[group]
	for _, secret := range unFetchedSecrets {
	if vaultSecret == nil {
	// This happens if we were not able to fetch the secret from vault
	secret.Value = ""
	} else {
	value, err := vh.getSecretValue(&secret, vaultSecret)
	if err != nil {
	secret.Value = ""
	} else {
	secret.Value = value
	}
	}
	out = append(out, secret)
	}
	}
	return out, nil
	}

go 1.21 treats the group used for they key as the same group, but go 1.22 doesn't:

keda/pkg/scaling/resolver/hashicorpvault_handler.go

Lines 316 to 322 in ed6dfdd

	for _, e := range secrets {
	group := SecretGroup{secretType: e.Type, path: e.Path, vaultPkiData: &e.PkiData}
	if _, ok := grouped[group]; !ok {
	grouped[group] = make([]kedav1alpha1.VaultSecret, 0)
	}
	grouped[group] = append(grouped[group], e)
	}

It generates 3 certificates instead on just one, and use each one to get one parameter

I'd say that it's because the SecretGroup uses a pointer for vaultPkiData and this reference changes, creating a different key each iteration but I don't see any reference to this change in the go release notes 🤷