Fix caching related issues

[mistercrunch]

Moving to cache the output of the dataframe as opposed to the output of get_data. Caching the output of get_data doesn't work if the basis for the cache_key is the "query object". When changing controls like the moving average configuration, this doesn't affect the query object (same cache key), but it does affect the post processing that append in get_data. If we use the query object as the basis for the cache key, we need to cache the dataframe itself, and re-execute get_data against the cache.

This may not be 100% ready for merging as:

• I'm unclear on whether the serialization of the dataframe (using df.to_dict) is 100% clean, there could be bugs where some things get lost through the serde process
• FilterBox doesn't get cached anymore as it doesn't follow the common pattern of using `get_df

@john-bodley

Also the introduction of a rendered chartState seemed to have broken the cache tag and the row label in the UI.

[mistercrunch]

It seems like we may want to re-think/cleanup the caching logic a bit.

[mistercrunch]

For the record, I'm doing more work on this. Moving to pickling dataframes as other methods of serializations seem incomplete/risky.

[graceguo-supercat]

@mistercrunch Hi Max, do you mind if i create a another PR just to fix JS part? So this PR focus on cache issue?

[mistercrunch]

@john-bodley I think I've got it mostly nailed down now at this point. This turned out to be a much more complex headache then I thought originally. Especially in the area of caching properly multi-queries visualizations.

[mistercrunch]

Unrelated but that fixed the build

[john-bodley]

@mistercrunch overall this LGTM. We'll probably have to test it extensively in our staging environment. Note there's a few flake8, pylint issues.

[michellethomas]

@mistercrunch @john-bodley I'm finding that when a dashboard gets loaded with a filter box viz or a line chart with a time comparison, queries get executed before the dashboard template even gets rendered (resulting in a blank page that is slow to load because it's executing queries). I think this change is causing the issue.

When getting the dashboard data, we get the data for each slice which gets the viz.data and calls run_extra_queries on init and executes the queries. So the dashboard template doesn't get rendered until all of the queries run.

https://github.com/apache/incubator-superset/blob/master/superset/models/core.py#L173

I'm not quite sure how to resolve this because it sounds like this was specifically added here because it needs to run in the init. Any suggestions?

[mistercrunch]

Oh wow right. This should not be here. Let me take a shot at it.

[mistercrunch]

This should do it: #4627 , mind trying it on you side?

[michellethomas]

thanks for the quick fix! I'll test