Expression datatable formatting hints #43091

flash1293 · 2019-08-12T10:11:23Z

This PR adds an optional property to the column of expression data tables: formatHint. It carries the JSON representation of a field formatter config that can be used by expression renderers to format the values of the column correctly.

This is a design decision with trade-offs: Conceptually formatting belongs to the presentation of the data and shouldn't be part of the "data source"-ey expression functions. But correctly formatting something is very strongly coupled to the source of a given data table - the data source (in this case esaggs) knows about source-specific formatting information (e.g. user defined field formatters on the given index pattern or the interval of a date_histogram aggregation to format dates accordingly). The render functions shouldn't know or care about these concepts and passing in the formatting information as sub-expressions of the renderer function would cause redundant information on the expression that can get out of sync. Providing the data source with a way to pass formatting hints along with the data itself together with a way to overwrite/ignore this formatting offers the benefits of both ways - a clean and robust expression that integrates with existing formatting sources of the system in the standard case, but still full control over the formatting if this is necessary.

The esaggs function is extended with an optional boolean argument includeFormatHints that can be set to add formatting information to the data table returned by esaggs. It uses the same implementation as the build_pipeline helper for the expression loader for Visualize.

Theoretically the Visualize renderers and build_pipeline could be rewritten to not include the formatting information in the arguments of the render function but to use the format hints from esaggs, but that's out of scope for this PR IMHO.

Not 100% sure about the files the various helpers and types should reside, open for suggestions here.

elasticmachine · 2019-08-12T11:12:45Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

lukeelmers

Overall LGTM, just added a few questions & comments.

src/legacy/ui/public/visualize/loader/pipeline_helpers/utilities.ts

lukeelmers · 2019-08-19T22:57:02Z

src/plugins/data/common/expressions/expression_types/kibana_datatable.ts

@@ -21,9 +21,15 @@ import { map } from 'lodash';

 const name = 'kibana_datatable';

+export interface SerializedFieldFormat<TParams = object> {


Since this is public/exported, maybe it would make sense to co-locate it with getFormat & the pipeline utilities, and import it into here instead? Since it doesn't have anything to do with the kibana_datatable type specifically.

To some extent it doesn't matter a lot right now since all of the ui/visualize/loader stuff is going away or moving eventually anyway, but IMO this is the only thing in the PR that feels like maybe it should be moved.

That's how I started out over here (5548579), but somehow the type check fails in completely unrelated places if I do that - something about importing legacy stuff from the new world. Then I thought it's maybe even cleaner that way because the types of the data table are all defined in one place.

Maybe this was a configuration error already fixed, I'll move it over and check.

Ah I didn't think about that @flash1293 -- you are totally right. You can import new platform into legacy, but you cannot import legacy into new platform.

Not sure what makes the most then, since we definitely won't want to keep this in with the datatable type longer term. Maybe instead we should stick it in the types/common.ts file for now? I just don't want it to get lost in expression_types/kibana_datatable.ts

Moved it over to common

src/legacy/ui/public/visualize/loader/pipeline_helpers/build_pipeline.ts

src/legacy/core_plugins/interpreter/public/functions/esaggs.ts

lukeelmers · 2019-08-19T23:19:04Z

src/legacy/core_plugins/interpreter/public/functions/esaggs.ts

-        name: column.name,
-      })),
+      columns: response.columns.map((column: any) => {
+        const cleanedColumn: KibanaDatatable['columns'][0] = {


Do you think enough folks will have a use case for this that it'd be worth exporting Column separately, perhaps as KibanaDatatableColumn, so that you don't have to do stuff like this?

why do we always take the first column ?

Because all of the columns in the array have the same type, this is a hack. I agree with @lukeelmers here, it makes sense to expose KibanaDatatableColumn to be able to cleanly reference this type.

Current solution works like this:
KibanaDatatable is of type { columns: Array<Column> }
KibanaDatatable['columns] is of type Array<Column>

now the question is how do I get to the type of the Column itself? I could do a proper type inference like this: https://www.typescriptlang.org/docs/handbook/release-notes/typescript-2-8.html#type-inference-in-conditional-types or I could do a dirty hack and get the type of the 0 property of KibanaDatatable['columns] which happens to be Column

A bit orthogonal to the discussion here, but in cases where magic like this is necessary, I find the syntax KibanaDatatable['columns'][number] to be slightly easier to grok, mostly because the index number can be distracting as @ppisljar points out.

But agreed that exporting separately feels like the way to go here.

elasticmachine · 2019-08-19T23:27:52Z

Pinging @elastic/kibana-app-arch

ppisljar

code LGTM

ppisljar · 2019-08-20T08:08:52Z

src/legacy/core_plugins/interpreter/public/functions/esaggs.ts

-        name: column.name,
-      })),
+      columns: response.columns.map((column: any) => {
+        const cleanedColumn: KibanaDatatable['columns'][0] = {


why do we always take the first column ?

…ing-hints

flash1293 · 2019-08-20T12:40:24Z

src/legacy/core_plugins/interpreter/public/functions/esaggs.ts

-      timeRange: get(context, 'timeRange', null),
-      query: get(context, 'query', null),
-      filters: get(context, 'filters', null),
+      timeRange: get(context, 'timeRange', undefined),


this went in because the existing RequestHandler types don't allow null here (but they allow undefined)

flash1293 · 2019-08-20T12:42:44Z

src/legacy/ui/public/vis/request_handlers/request_handlers.d.ts

@@ -42,7 +42,7 @@ export interface RequestHandlerParams {
  visParams?: any;
 }

-export type RequestHandler = <T>(params: RequestHandlerParams) => T;
+export type RequestHandler<T = unknown> = (params: RequestHandlerParams) => T;


moving the type parameter to the outer type to be able to specify it in courier-specific type. Inferred types in other usages should work like before.

elasticmachine · 2019-08-20T13:37:58Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

flash1293 · 2019-08-20T13:47:58Z

@lukeelmers :this: is the error I meant - no idea why this happens but if we don't import legacy from within NP world, it works.

This reverts commit 4738f82.

elasticmachine · 2019-08-20T14:48:22Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

…ing-hints

elasticmachine · 2019-08-21T10:45:42Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

flash1293 added 6 commits July 15, 2019 18:15

dont update embeddables on changes in disabled filters

141394c

Merge remote-tracking branch 'upstream/master'

cb43035

Merge remote-tracking branch 'upstream/master'

a96d594

Merge remote-tracking branch 'upstream/master'

dfeb7da

add format hints to kibana datatable

814eafa

remove unrelated changes

8dc7b85

flash1293 added v8.0.0 release_note:skip Skip the PR/issue when compiling release notes v7.4.0 labels Aug 12, 2019

flash1293 requested a review from ppisljar August 12, 2019 10:11

flash1293 requested a review from a team as a code owner August 12, 2019 10:11

flash1293 mentioned this pull request Aug 12, 2019

[Lens] Field formatter support #38874

Merged

lukeelmers reviewed Aug 19, 2019

View reviewed changes

lukeelmers added the Team:AppArch label Aug 19, 2019

lukeelmers added review Feature:ExpressionLanguage Interpreter expression language (aka canvas pipeline) labels Aug 19, 2019

ppisljar approved these changes Aug 20, 2019

View reviewed changes

flash1293 added 3 commits August 20, 2019 14:04

Merge remote-tracking branch 'upstream/master' into datatable-formatt…

7c55f8f

…ing-hints

address feedback

27a8a40

move serializedfieldformat type

4738f82

flash1293 commented Aug 20, 2019

View reviewed changes

Revert "move serializedfieldformat type"

6303a0f

This reverts commit 4738f82.

lukeelmers approved these changes Aug 20, 2019

View reviewed changes

flash1293 added 2 commits August 21, 2019 11:13

Merge remote-tracking branch 'upstream/master' into datatable-formatt…

685b64b

…ing-hints

move type

80e2022

flash1293 merged commit 9dd4369 into elastic:master Aug 21, 2019

flash1293 deleted the datatable-formatting-hints branch August 21, 2019 10:52

flash1293 added a commit to flash1293/kibana that referenced this pull request Aug 21, 2019

Expression datatable formatting hints (elastic#43091)

e0eb659

flash1293 mentioned this pull request Aug 21, 2019

[7.x] Expression datatable formatting hints (#43091) #43656

Merged

flash1293 added a commit that referenced this pull request Aug 21, 2019

Expression datatable formatting hints (#43091) (#43656)

feaa8e8

wylieconlon mentioned this pull request Feb 7, 2020

Agg configs should support formatting options or field formatters #57154

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Expression datatable formatting hints #43091

Expression datatable formatting hints #43091

flash1293 commented Aug 12, 2019

elasticmachine commented Aug 12, 2019

lukeelmers left a comment

lukeelmers Aug 19, 2019

flash1293 Aug 20, 2019

lukeelmers Aug 20, 2019 •

edited

Loading

flash1293 Aug 21, 2019

lukeelmers Aug 19, 2019

ppisljar Aug 20, 2019

flash1293 Aug 20, 2019

lukeelmers Aug 20, 2019

elasticmachine commented Aug 19, 2019

ppisljar left a comment

ppisljar Aug 20, 2019

flash1293 Aug 20, 2019

flash1293 Aug 20, 2019

elasticmachine commented Aug 20, 2019

flash1293 commented Aug 20, 2019

elasticmachine commented Aug 20, 2019

elasticmachine commented Aug 21, 2019

		@@ -21,9 +21,15 @@ import { map } from 'lodash';

		const name = 'kibana_datatable';

		export interface SerializedFieldFormat<TParams = object> {

Expression datatable formatting hints #43091

Expression datatable formatting hints #43091

Conversation

flash1293 commented Aug 12, 2019

elasticmachine commented Aug 12, 2019

💚 Build Succeeded

lukeelmers left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lukeelmers Aug 20, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Aug 19, 2019

ppisljar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Aug 20, 2019

💔 Build Failed

flash1293 commented Aug 20, 2019

elasticmachine commented Aug 20, 2019

💚 Build Succeeded

elasticmachine commented Aug 21, 2019

💚 Build Succeeded

lukeelmers Aug 20, 2019 •

edited

Loading