Autoscaling using log analytics query not working properly anymore, context deadline exceeded (Client.Timeout exceeded while awaiting headers)

[NicolaeMarius]

Hi everyone,

we seem to be having an issue with our keda autoscaling implementation in our AKS production and testing environments alike, for some reason we keep getting this error from the operator/scaledobject:

"error": "failed to get metrics. Scaled object: scaledObjectName-keda. Namespace:my-namespace. Inner Error: error calling Log Analytics REST api. Inner Error: Post "https://api.loganalytics.io/v1/workspaces//query": context deadline exceeded (Client.Tim
eout exceeded while awaiting headers)"}

2024-01-15T03:10:13ZERRORscale_handlererror getting metric for scaler{"scaledObject.Namespace": "my-namespace", "scaledObject.Name": "scaledObjectName-keda", "scaler": "azureLogAnalyticsScaler", "error": "failed to get metrics. Scaled obj
ect: scaledObjectName-keda. Namespace:my-namespace. Inner Error: error calling Log Analytics REST api. Inner Error: Post \"https://api.loganalytics.io/v1/workspaces/<workspaceID>/query\": context deadline exceeded (Client.Tim
eout exceeded while awaiting headers)"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScaledObjectMetrics
/workspace/pkg/scaling/scale_handler.go:483
github.com/kedacore/keda/v2/pkg/metricsservice.(*GrpcServer).GetMetrics
/workspace/pkg/metricsservice/server.go:47
github.com/kedacore/keda/v2/pkg/metricsservice/api._MetricsService_GetMetrics_Handler
/workspace/pkg/metricsservice/api/metrics_grpc.pb.go:99
google.golang.org/grpc.(*Server).processUnaryRPC
/workspace/vendor/google.golang.org/grpc/server.go:1337
google.golang.org/grpc.(*Server).handleStream
/workspace/vendor/google.golang.org/grpc/server.go:1714
google.golang.org/grpc.(*Server).serveStreams.func1.1
/workspace/vendor/google.golang.org/grpc/server.go:959

It stopped working around 19 days ago according to the events, our keda version is 2.11.0
The default timeout we have on our keda deployments are:

name: KEDA_HTTP_DEFAULT_TIMEOUT
         value: "3000"

We are kind of stuck in our troubleshooting as we don't fully understand what could be causing this issue, I have checked the workspaceID values in the secret of the ta object and it seems to be correct.

If anyone could help point me in the right direction would be very much appreciated, thank you in advance!

[ghantasunil]

I am running into the same issue. @NicolaeMarius did you find any solution for this?

[NicolaeMarius]

Hi @ghantasunil,

actually yes, it was a problem specific with the way the AKS clusters get provisioned on Azure, the load balancer azure will provision for your AKS cluster has a session timeout of 30 minutes, which is a lot, I changed the session timeout to 4 minutes and that fixed my issue, you should check for the snat port exhaustion metric in your load balancer and observe if you have a snat port exhaustion, in my case we did see snat port exhaustion which is what was causing this issue, after changing the session timeout to 4 minutes we saw the snat exhaustion issue went away and keda was able to successfully contact the outside world aka the public log analytics api.
Let me know if you have the same issue or if you need more details on this.

[zroubalik]

Thanks for the insight @NicolaeMarius !