Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Explore] Streamlined metric definitions for SQLA and Druid #4663

Merged

Conversation

gabe-lyons
Copy link
Contributor

This PR streamlines the definition of AGGREGATE( column )-style metrics in the explore view. It circumvents the fab view entirely for a much improved user experience.

The metric is stored as a simple object of form {column, aggregate, label} in form data. When a query is run, the adhoc metric objects are sent along with saved metrics to the flask backend, where they are manually parsed in SQLA/Druid expressions.

Here are some quick videos demoing what interactions look like:

Adding and editing metrics:
out5

Applying labels to metrics:
out4

Guide to reviewers:

  • These streamlined metrics I refer to as adhocMetrics. Traditional metrics I refer to as savedMetrics
  • The bulk of the frontend logic lives in MetricsControl and AdhocMetricEditPopover. MetricsControl handles data massaging and the dropdown. AdhocMetricEditPopover handles most of the log for editing
  • The bulk of the flask logic is in connectors/druid/models.py and connectors/sqla/models.py. In those files I handle the parsing of adhoc metrics in sqla/druid aggregations.

Open questions:

  • There is custom parsing logic all over viz.py. Since this PR changes the way we post form data, we may not be able to enable this feature on all viz types. I made a fix for line charts, but I fear there may be more such issues lurking. (See viz.py change for details)
  • How should we deal with displaying saved metrics in the dropdown? In this PR, the default order of options in the metrics dropdown is [...columns, ...aggregates, ...saved metrics]. I prefer this ordering because there are many saved metrics that this feature will replace. However I can see it being confusing initially to users to have this dropdown default to unexpected content.
  • Should we prevent aggregate/column combinations based on column type? Right now I just let the users do as they please.
  • How should we think about deprecating saved metrics that are just a column+aggregate?

reviewers:
@michellethomas @john-bodley @williaster @graceguo-supercat @mistercrunch

(after putting up this PR I'll continue testing locally so there may be some bug fixes to come, but this is the done deal!)

@gabe-lyons gabe-lyons force-pushed the gabe_adding_metric_definition_modal branch from 0b0dc07 to b6f2246 Compare March 22, 2018 01:08
@mistercrunch
Copy link
Member

This is super neat and innovative. Can we make the metric (same as metrics but only allows to define one metric) control in-scope for this PR to make sure both metrics and metric are building upon the same building blocks nicely. From what I see it should be pretty trivial, maybe by adding a multiple boolean prop to MetricsControl ?

Copy link
Member

@mistercrunch mistercrunch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Had a few minor comments on the code but overall this looks great. Surprisingly easier to use than I thought and much better than my MetricsControl attempt that I never merged. Glad I held off on it as it just didn't feel right. I think this does.

Few usability ideas:

  • it's not a common or expected flow, maybe provide hints as people type or when the Select element has focus? Maybe like a small yellow info bubble Pick an aggregate function or a column to aggregate?
  • a click to edit the metric popover when hovering a button-option ?

I wonder if it's eventually in-scope to allow free form SUM(CASE WHEN...) or APPROX_DISTINCT_COUNT... in that context. Seemed possible to extend into this if that becomes a priority somehow, but it's probably ok to send people to the old flow for those 10-20% use-cases, at least for now. Maybe by then I'll have rewritten the "Table Editor" into something much more usable.

My MetricsControl had a big of logic around handling aggregation types based on column type, but that doesn't seem super necessary for MVP here. Could be nice to have eventually.

</FormGroup>
<Button
disabled={!stateIsValid}
bsStyle={(hasUnsavedChanges || !stateIsValid) ? 'default' : 'primary'}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NIT: we've veen using bsSize="small" throughout the app

>
Save
</Button>
<Button onClick={this.props.onClose}>Close</Button>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NIT: bsSize="small"

overlay={overlay}
rootClose
>
<Label style={{ cursor: 'pointer' }}>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was told to avoid double curlies whenever possible since they generate new objectids at every render and will trigger PureComponents to rerender when they don't need to, just declare labelStyle in module scope and reference it here.

@@ -0,0 +1,20 @@
import React from 'react';
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's ok to declare more than one component in a module if the component is only used once in that one module. Point being to avoid juggling with lots of small files when possible, the components can be refactored out to its own module when it grows to be used in more than one component.

I've been guilty of the other extreme (very long modules) so I'm ok with this if you prefer.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do prefer more files in general.. I think in this instance it makes sense so all the *Option components are declared in the same way.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

++ small files. it simplifies everything vastly if everything has it's own file.

label = metric.get('label')
if aggregate == 'COUNT_DISTINCT':
sa_metric = sa.func.COUNT(sa.distinct(column(column_name)))
if aggregate == 'COUNT':
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can use a bunch of elif here to save some comparisons

@gabe-lyons
Copy link
Contributor Author

@mistercrunch thanks for the comments. I'll add it to the metric control as well- forgot about that one.

Also will fix python tests that are failing and add a couple more.

@gabe-lyons
Copy link
Contributor Author

Also like the idea of a click to edit the metric popover when hovering a button-option- I think that would be helpful

@codecov-io
Copy link

codecov-io commented Mar 23, 2018

Codecov Report

Merging #4663 into master will increase coverage by 0.31%.
The diff coverage is 83.76%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #4663      +/-   ##
==========================================
+ Coverage   71.56%   71.87%   +0.31%     
==========================================
  Files         191      204      +13     
  Lines       14958    15323     +365     
  Branches     1100     1177      +77     
==========================================
+ Hits        10705    11014     +309     
- Misses       4250     4306      +56     
  Partials        3        3
Impacted Files Coverage Δ
...rset/assets/javascripts/explore/stores/visTypes.js 70.58% <ø> (ø) ⬆️
...et/assets/javascripts/components/OnPasteSelect.jsx 93.18% <0%> (ø) ⬆️
...javascripts/explore/components/AggregateOption.jsx 100% <100%> (ø)
...s/javascripts/explore/components/controls/index.js 100% <100%> (ø) ⬆️
superset/assets/javascripts/dashboard/reducers.js 50% <100%> (+0.43%) ⬆️
...s/javascripts/explore/propTypes/adhocMetricType.js 100% <100%> (ø)
...s/javascripts/explore/propTypes/savedMetricType.js 100% <100%> (ø)
superset/assets/javascripts/explore/AdhocMetric.js 100% <100%> (ø)
superset/connectors/druid/models.py 79.25% <100%> (+0.39%) ⬆️
superset/assets/javascripts/explore/constants.js 100% <100%> (ø)
... and 26 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 485b0c2...7d59385. Read the comment docs.

@@ -0,0 +1,6 @@
import PropTypes from 'prop-types';
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see this being used anywhere, what's this for?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, good catch! I actually don't use this file anywhere at all! I will delete it.

export default PropTypes.shape({
metric_name: PropTypes.string.isRequired,
expression: PropTypes.string.isRequired,
});
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just curious, why have this in a separate file instead of defined as a shape in the file it's used? This isn't being used in multiple places and it doesn't have a complicated structure (with imports). Is it a best practice?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm yeah I could move it into MetricsControl- I agree this doesn't help much.

}

getDefaultLabel() {
return `${this.aggregate || ''}(${(this.column && this.column.column_name) || ''})`;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When would this.aggregate be null?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It wouldn't be often, but in the case that someone clears the aggregate input in the edit popover, and then goes to edit the title, the default title they would see without this logic would be like null(column_name), which seems like a bad user experience. Adding this logic would make the default title appear as (column_name)

column_name = metric.get('column').get('column_name')
aggregate = metric.get('aggregate')
label = metric.get('label')
if aggregate == 'COUNT_DISTINCT':
Copy link
Member

@hughhhh hughhhh Mar 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I would make this a dict() mapping for instance:

aggregate_dict = {
     'SUM' : sa.func.COUNT(column(column_name))
     ...
}
sa_metric = aggregate_dict.get(aggregate)

I think this is much cleaner and would allow others to add more metrics easily later.

PS: This is so dope! i can't wait till this lands so we can had some geo functions!

@gabe-lyons gabe-lyons force-pushed the gabe_adding_metric_definition_modal branch 3 times, most recently from 1aad37e to 0dc1954 Compare March 27, 2018 21:22
@gabe-lyons
Copy link
Contributor Author

gabe-lyons commented Mar 27, 2018

@mistercrunch @michellethomas thanks for the reviews! I addressed all comments.

For Max's more in depth comments:

  • enabled adhoc metrics on the metric controller. Required adding some additional case statements in the MetricsControl component where I assume value will be an array. Also required some small style updates.
  • for making it more clear that the tokens are editable, I had the AdhocMetricEditPopover appear by default when a new metric is created. I also updated the default text in the select component.

I also added another feature that HIDES any auto generated metric that is just an aggregate + column.

Please take another look when you have the chance!

@mistercrunch
Copy link
Member

I'm going to pull the branch personally but would be nice to have gifs in the PR.

@mistercrunch mistercrunch changed the title [Request For Comments][Explore] Streamlined metric definitions for SQLA and Druid [Explore] Streamlined metric definitions for SQLA and Druid Mar 27, 2018
@mistercrunch
Copy link
Member

Removed the [Request for comments] in the title as it's clear we're moving forward with this.

Copy link
Contributor

@williaster williaster left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a few mostly minor comments, great addition!

@@ -38,5 +38,8 @@
"react/no-unescaped-entities": 0,
"react/no-unused-prop-types": 0,
"react/no-string-refs": 0,
"indent": 0,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume these are due to the new eslint version which flagged them?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah exactly

if (type === '' || type === 'expression') {
if (!type) {
stringIcon = '?';
} else if (type === '' || type === 'expression') {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

type === '' is not possible given the first if statement

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch I'll clarify the first statement

this.hasCustomLabel = !!(adhocMetric.hasCustomLabel && adhocMetric.label);
this.fromFormData = !!adhocMetric.optionName;

if (this.hasCustomLabel) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you could consolidate with
this.label = this.hasCustomLabel ? adhocMetric.label : this.getDefaultLabel();

}

this.optionName = adhocMetric.optionName ||
`metric_${Math.random().toString(36).substring(2, 15)}_${Math.random().toString(36).substring(2, 15)}`;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a util func to generate ids that we could use?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is what my internet research suggested i do 🤕

<Popover
id="metrics-edit-popover"
title={popoverTitle}
{...rest}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd call this ...popoverProps instead of rest

return option.column_name && (option.column_name.indexOf(valueAfterAggregate) >= 0);
}

const autoGeneratedMetricRegex = /^(LONG|DOUBLE|FLOAT)?(SUM|AVG|MAX|MIN|COUNT)\([A-Z_][A-Z0-9_]*\)$/i;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice to have tests for this sort of thing esp if people introduce changes in other files in the future.

optionRenderer={VirtualizedRendererWrap(option => (
<MetricDefinitionOption option={option} />
))}
valueRenderer={option => (
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

again lots of render functions

@@ -134,3 +138,7 @@
.datasource-container {
overflow: auto;
}

.edit-adhoc-metric-save-btn {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you use the existing m-r-5 class? (I think this is the name)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh cool! I didn't realize that existed

@@ -49,8 +49,8 @@ export default class OnPasteSelect extends React.Component {
render() {
const SelectComponent = this.props.selectWrap;
const refFunc = (ref) => {
if (this.props.ref) {
this.props.ref(ref);
if (this.props.refFunc) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yay for readability!

@@ -0,0 +1,37 @@
export default class AdhocMetric {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dig the class approach!

@gabe-lyons gabe-lyons force-pushed the gabe_adding_metric_definition_modal branch from 0dc1954 to e316bda Compare March 28, 2018 16:54
@gabe-lyons gabe-lyons force-pushed the gabe_adding_metric_definition_modal branch from e316bda to 3b8c519 Compare March 28, 2018 16:56
@gabe-lyons
Copy link
Contributor Author

@williaster thanks for all the comments- they should be addressed now.

Good callout on the PropTypes.object stragglers- I feel better now that they're cleaned up.

@gabe-lyons gabe-lyons force-pushed the gabe_adding_metric_definition_modal branch from 3b8c519 to 7d59385 Compare March 28, 2018 19:01
@gabe-lyons
Copy link
Contributor Author

@mistercrunch @michellethomas @williaster I believe this PR is ready to merge--- any objectors?

@williaster
Copy link
Contributor

lgtm, will let @mistercrunch pull the trigger tho 📈

@mistercrunch mistercrunch merged commit 68dec24 into apache:master Mar 29, 2018
@mistercrunch
Copy link
Member

napo

@CasperLiu
Copy link

This PR looks brilliant, great work!

Thanks!

Which release/version would this functionilty go into?
No meaning to push, just get the rough schedule in mind.

Jiaxu

@mistercrunch
Copy link
Member

It should be in 0.25.0, no clear timeline but should be a few weeks at most.

@CasperLiu
Copy link

mistercrunch,

Thanks for your explanation!

Regards,
Jiaxu
刘佳旭

michellethomas pushed a commit to michellethomas/panoramix that referenced this pull request May 24, 2018
)

* adding streamlined metric editing

* addressing lint issues on new metrics control

* enabling druid
@congdanh8391
Copy link

@mistercrunch @michellethomas thanks for the reviews! I addressed all comments.

For Max's more in depth comments:

  • enabled adhoc metrics on the metric controller. Required adding some additional case statements in the MetricsControl component where I assume value will be an array. Also required some small style updates.
  • for making it more clear that the tokens are editable, I had the AdhocMetricEditPopover appear by default when a new metric is created. I also updated the default text in the select component.

I also added another feature that HIDES any auto generated metric that is just an aggregate + column.

Please take another look when you have the chance!

Hey,
Adhoc metric is good but in some cases, it's a problem. In tableview, if you define an adhoc metric, it has no format as similar as auto general metric. Please don not hide auto general metric anymore!

Thank

wenchma pushed a commit to wenchma/incubator-superset that referenced this pull request Nov 16, 2018
)

* adding streamlined metric editing

* addressing lint issues on new metrics control

* enabling druid
@mistercrunch mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 0.25.0 labels Feb 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 0.25.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants