CP-14444: Debug logging for PodStore

[bdrennz]

Description of changes

This PR adds debug logs for PodStore to surface more information about cache operations. This should help with issues where the PodStore fails to scrape metrics from Pods in a cluster and any other potential cache related issues.

Most of the logs are added to PodStore.refreshInternal() which is ultimately called by PodStore.Decorate() any time the cache needs to be refreshed. PodStore also now implements ExposeCache() which lists all entries in the cache along with the remaining time to expiry. Assuming the cache is not empty, the output will look something like:

2023-11-16T14:23:54Z D! [PodStore.ExposeCache] Exposing cache with '5' entries...
2023-11-16T14:23:54Z D! [PodStore.ExposeCache] Entry: [podKey: namespace:default,podName:prometheus-kube-state-metrics-59649d78f-st99c, remainingTime: 89s]
2023-11-16T14:23:54Z D! [PodStore.ExposeCache] Entry: [podKey: namespace:kube-system,podName:aws-node-mbgps, remainingTime: 89s]
2023-11-16T14:23:54Z D! [PodStore.ExposeCache] Entry: [podKey: namespace:kube-system,podName:kube-proxy-wt8bj, remainingTime: 89s]
2023-11-16T14:23:54Z D! [PodStore.ExposeCache] Entry: [podKey: namespace:cloudzero-metrics,podName:cloudzero-cloudwatch-metrics-ggjmv, remainingTime: 89s]
2023-11-16T14:23:54Z D! [PodStore.ExposeCache] Entry: [podKey: namespace:default,podName:prometheus-prometheus-node-exporter-wdf64, remainingTime: 89s]
2023-11-16T14:23:54Z D! [PodStore.ExposeCache] Finished exposing cache

There's some other general information about pods, nodes, containers, etc.

2023-11-16T14:23:54Z D! [PodStore.refreshInternal] Refreshing Pod: 'cloudzero-cloudwatch-metrics-ggjmv':
2023-11-16T14:23:54Z D! [PodStore.refreshInternal] PodKey: 'namespace:cloudzero-metrics,podName:cloudzero-cloudwatch-metrics-ggjmv'
2023-11-16T14:23:54Z D! [PodStore.refreshInternal] Pod Phase: 'Running'
2023-11-16T14:23:54Z D! [PodStore.refreshInternal] Node Name: 'ip-192-168-23-0.ec2.internal'
2023-11-16T14:23:54Z D! [PodStore.refreshInternal] Container Information: [name: 'cloudzero-cloudwatch-metrics', state: '{nil &ContainerStateRunning{StartedAt:2023-11-16 14:18:15 +0000 UTC,} nil}']
2023-11-16T14:23:54Z D! [PodStore.refreshInternal] Refresh for Pod 'cloudzero-cloudwatch-metrics-ggjmv' is complete!

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

Ideally ExposeCache() should have a unit test, but validating logs might be more trouble than it's worth.

Requirements

Before commit the code, please do the following steps.

1. Run make fmt and make fmt-sh
2. Run make linter

[dmepham]

lgtm!