Request for Cache Metrics in Apache Superset

[ankit-kumar-maersk]

Hi @everyone,

I have encountered a challenge regarding the visibility of cache performance and usage metrics.
I need metrics related to cache to monitor effectiveness of caching, trouleshoot performance issue and to optimize the cache configuration.
But It appears that detailed metrics related to cache performance (such as cache hits, misses, and utilization) are not readily available through the existing metrics endpoints.

Is there currently any way to access detailed cache metrics within Superset that I may have overlooked? If such functionality exists, could you please provide guidance or documentation on how to enable and access these metrics?

[dosubot[bot]]

Hi @ankit-kumar-maersk,

I'm here to help you with your Apache Superset issue. Let's find a solution together!

Yes, there is a way to access detailed cache metrics within Apache Superset, such as cache hits, misses, and utilization. The stats_logger object is used to log cache-related metrics.

To enable and access these metrics, you need to configure the STATS_LOGGER in your Superset configuration. The stats_logger is used in the set_and_log_cache function to log cache set operations:

stats_logger: BaseStatsLogger = config["STATS_LOGGER"]

In the set_and_log_cache function, the following line logs the cache set operation:

stats_logger.incr("set_cache_key")

To enable and access these metrics, you need to:

1. Ensure that STATS_LOGGER is properly configured in your Superset configuration file (e.g., superset_config.py).
2. Implement or use an existing BaseStatsLogger that logs or stores the metrics you are interested in (e.g., cache hits, misses, utilization).

Here is an example of how you might configure the STATS_LOGGER:

from superset.stats_logger import BaseStatsLogger

class CustomStatsLogger(BaseStatsLogger):
    def incr(self, stat_name: str, count: int = 1) -> None:
        # Implement your logging logic here
        print(f"Incrementing stat {stat_name} by {count}")

    def decr(self, stat_name: str, count: int = 1) -> None:
        # Implement your logging logic here
        print(f"Decrementing stat {stat_name} by {count}")

    def timing(self, stat_name: str, value: float) -> None:
        # Implement your logging logic here
        print(f"Timing stat {stat_name} with value {value}")

STATS_LOGGER = CustomStatsLogger()

By configuring the STATS_LOGGER in this way, you can log and access detailed cache metrics in Apache Superset ^[1].

To continue talking to Dosu, mention @dosu.

[dinesh-nithyanandam-maersk]

@dosu - you're suggestion was more off a log based metrics which might be really efficient while measuring, we are looking to build prometheus time-series gauge, counter, histogram metrics on determining the Superset Cache performance so in this its Redis metrics distinctively

1. cache_miss_hits_ratio
2. cache_hit_ratio
3. cache_hit_latency
4. cache_exceptions

[dinesh-nithyanandam-maersk]

@dosu

Please find the below snippet code which we planned to implement, but we would like to get the expert view in the group, before we finalize on the implementation

`import time
from prometheus_client import Counter, start_http_server

Define Prometheus counters

CACHE_HIT = Counter('cache_hit', 'Number of cache hits')
CACHE_MISS = Counter('cache_miss', 'Number of cache misses')

def fetch_from_cache(key):
if key in cache:
CACHE_HIT.inc()
return cache[key]
else:
CACHE_MISS.inc()
return fetch_from_db(key)

def fetch_from_db(key):
# Simulated database fetch
# Fetch data from your database and store it in the cache
data = "fetched_data" # Replace with actual database fetch logic
cache[key] = data
return data

if name == "main":
start_http_server(8000)
cache = {}
while True:
fetch_from_cache('example_key')
time.sleep(5)`