[VisBuilder] Add Capability to generate dynamic vega #7288

ananzh · 2024-07-17T17:17:30Z

Description

In this PR, we add the capability for Visbuilder to generate dynamic Vega and Vega-Lite specifications based on user settings and aggregation configurations. Some hightlighs:

developed functions buildVegaSpecViaVega and buildVegaSpecViaVegaLite that can create either Vega or Vega-Lite specifications depending on the complexity of the visualization.
added VegaSpec and VegaLiteSpec interfaces to provide better type checking
broken down the specification building into smaller, reusable components (like buildEncoding, buildMark, buildLegend, buildTooltip) to make the code more maintainable and easier to extend.
added flattenDataHandler to prepare and transform data for use in Vega visualizations

Note: In this current implementation, we directly utilize the existing visualization and style configurations from Visbuilder. This approach maintains consistency with the current feature set and minimizes changes in this pull request. For future enhancements, we plan to introduce dedicated Vega-specific properties within both the visualization and style slices. This will allow us to accommodate new Vega-related settings and provide more granular control over Vega visualizations without affecting the existing Visbuilder functionality.

Issues Resolved (Partially)

#7067

Screenshot

2024-07-17_10-27-24.mp4

Testing the changes

Changelog

feat: [VisBuilder] Add Capability to generate dynamic vega

Check List

All tests pass
- yarn test:jest
- yarn test:jest_integration
New functionality includes testing.
New functionality has been documented.
Update CHANGELOG.md
Commits are signed per the DCO using --signoff

github-actions · 2024-07-17T17:18:03Z

❌ Empty Changelog Section

The Changelog section in your PR description is empty. Please add a valid changelog entry or entries. If you did add a changelog entry, check to make sure that it was not accidentally included inside the comment block in the Changelog section.

codecov · 2024-07-17T17:33:31Z

Codecov Report

Attention: Patch coverage is 77.10280% with 49 lines in your changes missing coverage. Please review.

Project coverage is 67.69%. Comparing base (6551af7) to head (9249067).
Report is 2 commits behind head on main.

Files	Patch %	Lines
...lder/public/visualizations/vega/components/mark.ts	72.72%	5 Missing and 4 partials ⚠️
...public/visualizations/common/expression_helpers.ts	0.00%	8 Missing ⚠️
...uilder/public/visualizations/vega/utils/helpers.ts	77.77%	3 Missing and 5 partials ⚠️
...er/public/visualizations/vega/vega_spec_factory.ts	16.66%	5 Missing ⚠️
...r/public/visualizations/vega/components/tooltip.ts	69.23%	1 Missing and 3 partials ⚠️
...blic/visualizations/vega/vega_lite_spec_builder.ts	71.42%	1 Missing and 3 partials ⚠️
...uilder/public/application/components/workspace.tsx	0.00%	2 Missing ⚠️
...ilder/public/embeddable/vis_builder_embeddable.tsx	0.00%	2 Missing ⚠️
.../public/visualizations/vega/components/encoding.ts	85.71%	0 Missing and 2 partials ⚠️
...er/public/visualizations/vega/vega_spec_builder.ts	86.66%	1 Missing and 1 partial ⚠️
... and 3 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7288      +/-   ##
==========================================
+ Coverage   67.62%   67.69%   +0.07%     
==========================================
  Files        3584     3595      +11     
  Lines       72205    72404     +199     
  Branches    11818    11867      +49     
==========================================
+ Hits        48826    49014     +188     
+ Misses      20391    20385       -6     
- Partials     2988     3005      +17

Flag	Coverage Δ
Linux_1	`33.40% <77.10%> (+0.24%)`	⬆️
Linux_2	`55.53% <ø> (ø)`
Linux_3	`44.48% <ø> (?)`
Linux_4	`34.05% <ø> (?)`
Windows_1	`33.42% <77.10%> (+0.24%)`	⬆️
Windows_2	`55.48% <ø> (ø)`
Windows_3	`44.50% <ø> (ø)`
Windows_4	`34.05% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

github-actions · 2024-07-17T17:33:49Z

❌ Empty Changelog Section

The Changelog section in your PR description is empty. Please add a valid changelog entry or entries. If you did add a changelog entry, check to make sure that it was not accidentally included inside the comment block in the Changelog section.

* update file and functions names * fix type errors * fix area chart Signed-off-by: Anan Zhuang <ananzh@amazon.com>

LDrago27 · 2024-07-22T16:54:45Z

src/plugins/vis_builder/public/visualizations/vega/components/encoding.ts

+ * @param {any} formats - The formatting information for axes.
+ * @returns {VegaScale[]} The Vega scale configurations.
+ */
+export const buildVegaScales = (dimensions: any, formats: any): VegaScale[] => {


The function doesn't seem to be using formats parameter. This parameter can be cleaned.

Will update. thx.

Signed-off-by: Anan Zhuang <ananzh@amazon.com>

LDrago27 · 2024-07-22T20:51:50Z

src/plugins/vis_builder/public/visualizations/vega/utils/helpers.ts

+ * @returns {string} The corresponding Vega data type
+ */
+export const mapFieldTypeToVegaType = (fieldType) => {
+  const typeMap = {


nit: We can use an enum to define the typeMap and add it to the constants file

LDrago27 · 2024-07-22T20:56:05Z

src/plugins/vis_builder/public/visualizations/vega/vega_lite_spec_builder.ts

+
+  // Build the base Vega-Lite specification
+  const baseSpec: VegaLiteSpec = {
+    $schema: 'https://vega.github.io/schema/vega-lite/v5.json',


nit: We can store it as a constant so it would be easier to modify it in future if required.

Will polish in a follow up PR.

LDrago27 · 2024-07-22T20:59:09Z

src/plugins/vis_builder/public/visualizations/vega/vega_lite_spec_builder.ts

+    $schema: 'https://vega.github.io/schema/vega-lite/v5.json',
+    data: { values: transformedData },
+    mark: buildMarkForVegaLite(vegaType),
+    encoding: buildVegaLiteEncoding(dimensions, formats),


nit: We are calling buildVegaLiteEncoding(dimensions, formats) three times separately in the function , we can store the value in first call and use it across places.

Will polish in a follow up PR.

Signed-off-by: Anan Zhuang <ananzh@amazon.com>

* [VisBuilder] Add Capability to generate dynamic vega In this PR, we add the capability for Visbuilder to generate dynamic Vega and Vega-Lite specifications based on user settings and aggregation configurations. * developed functions buildVegaSpecViaVega and buildVegaSpecViaVegaLite that can create either Vega or Vega-Lite specifications depending on the complexity of the visualization. * added VegaSpec and VegaLiteSpec interfaces to provide better type checking * broken down the specification building into smaller, reusable components (like buildEncoding, buildMark, buildLegend, buildTooltip) to make the code more maintainable and easier to extend. * added flattenDataHandler to prepare and transform data for use in Vega visualizations Issue Resolve #7067 Signed-off-by: Anan Zhuang <ananzh@amazon.com> * fix PR comments * update file and functions names * fix type errors * fix area chart Signed-off-by: Anan Zhuang <ananzh@amazon.com> * add unit tests Signed-off-by: Anan Zhuang <ananzh@amazon.com> * enable embeddable for useVega Signed-off-by: Anan Zhuang <ananzh@amazon.com> * remove buildVegaScales due to split it to smaller modules Signed-off-by: Anan Zhuang <ananzh@amazon.com> * fix date for vega Signed-off-by: Anan Zhuang <ananzh@amazon.com> * fix test Signed-off-by: Anan Zhuang <ananzh@amazon.com> * Changeset file for PR #7288 created/updated --------- Signed-off-by: Anan Zhuang <ananzh@amazon.com> Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com> (cherry picked from commit faaa45c) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* [VisBuilder] Add Capability to generate dynamic vega In this PR, we add the capability for Visbuilder to generate dynamic Vega and Vega-Lite specifications based on user settings and aggregation configurations. * developed functions buildVegaSpecViaVega and buildVegaSpecViaVegaLite that can create either Vega or Vega-Lite specifications depending on the complexity of the visualization. * added VegaSpec and VegaLiteSpec interfaces to provide better type checking * broken down the specification building into smaller, reusable components (like buildEncoding, buildMark, buildLegend, buildTooltip) to make the code more maintainable and easier to extend. * added flattenDataHandler to prepare and transform data for use in Vega visualizations Issue Resolve #7067 --------- (cherry picked from commit faaa45c) Signed-off-by: Anan Zhuang <ananzh@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com>

github-actions bot added the distinguished-contributor label Jul 17, 2024

github-actions bot added the failed changeset label Jul 17, 2024

ananzh force-pushed the feat/migrate-vega branch 3 times, most recently from f1017e0 to 1083974 Compare July 19, 2024 16:59

opensearch-changeset-bot bot added a commit to ananzh/OpenSearch-Dashboards that referenced this pull request Jul 19, 2024

Changeset file for PR opensearch-project#7288 created/updated

1cf0c1f

github-actions bot removed the failed changeset label Jul 19, 2024

ananzh marked this pull request as ready for review July 19, 2024 17:05

ananzh requested review from kavilla, AMoo-Miki, ashwin-pc, joshuarrrr, abbyhu2000, zengyan-amazon, zhongnansu, manasvinibs, ZilongX, Flyingliuhub, curq, bandinib-amzn, SuZhou-Joe, ruanyl, BionIT, xinruiba, zhyuanqi, mengweieric and LDrago27 as code owners July 19, 2024 17:05

fix PR comments

c37996e

* update file and functions names * fix type errors * fix area chart Signed-off-by: Anan Zhuang <ananzh@amazon.com>

ananzh force-pushed the feat/migrate-vega branch from 3229fac to c37996e Compare July 22, 2024 16:10

LDrago27 reviewed Jul 22, 2024

View reviewed changes

add unit tests

f2ac5cf

Signed-off-by: Anan Zhuang <ananzh@amazon.com>

ashwin-pc mentioned this pull request Jul 22, 2024

[RELEASE] Release version 2.16.0 opensearch-project/opensearch-build#4771

Closed

74 tasks

ananzh added 2 commits July 22, 2024 20:14

enable embeddable for useVega

e695b45

Signed-off-by: Anan Zhuang <ananzh@amazon.com>

remove buildVegaScales due to split it to smaller modules

745baf2

Signed-off-by: Anan Zhuang <ananzh@amazon.com>

LDrago27 reviewed Jul 22, 2024

View reviewed changes

opensearch-changeset-bot bot added a commit to ananzh/OpenSearch-Dashboards that referenced this pull request Jul 22, 2024

Changeset file for PR opensearch-project#7288 created/updated

cbebec5

LDrago27 reviewed Jul 22, 2024

View reviewed changes

fix date for vega

ae9e15b

Signed-off-by: Anan Zhuang <ananzh@amazon.com>

ananzh force-pushed the feat/migrate-vega branch from cbebec5 to ae9e15b Compare July 22, 2024 21:37

opensearch-changeset-bot bot added a commit to ananzh/OpenSearch-Dashboards that referenced this pull request Jul 22, 2024

Changeset file for PR opensearch-project#7288 created/updated

91ff799

fix test

3162063

Signed-off-by: Anan Zhuang <ananzh@amazon.com>

ananzh force-pushed the feat/migrate-vega branch from 91ff799 to 3162063 Compare July 22, 2024 23:07

LDrago27 previously approved these changes Jul 22, 2024

View reviewed changes

Changeset file for PR opensearch-project#7288 created/updated

677c3eb

opensearch-changeset-bot bot dismissed LDrago27’s stale review via 677c3eb July 22, 2024 23:39

Merge branch 'main' into feat/migrate-vega

9249067

ashwin-pc approved these changes Jul 23, 2024

View reviewed changes

LDrago27 approved these changes Jul 23, 2024

View reviewed changes

ashwin-pc merged commit faaa45c into opensearch-project:main Jul 23, 2024
68 checks passed

ashwin-pc added the backport 2.16 label Jul 23, 2024

opensearch-trigger-bot bot mentioned this pull request Jul 23, 2024

[Backport 2.x] [VisBuilder] Add Capability to generate dynamic vega #7409

Merged

opensearch-trigger-bot bot mentioned this pull request Jul 23, 2024

[Backport 2.16] [VisBuilder] Add Capability to generate dynamic vega #7410

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[VisBuilder] Add Capability to generate dynamic vega #7288

[VisBuilder] Add Capability to generate dynamic vega #7288

ananzh commented Jul 17, 2024 •

edited

Loading

github-actions bot commented Jul 17, 2024

codecov bot commented Jul 17, 2024 •

edited

Loading

github-actions bot commented Jul 17, 2024

LDrago27 Jul 22, 2024

ananzh Jul 22, 2024

LDrago27 Jul 22, 2024

LDrago27 Jul 22, 2024

ananzh Jul 22, 2024

LDrago27 Jul 22, 2024

ananzh Jul 22, 2024

[VisBuilder] Add Capability to generate dynamic vega #7288

[VisBuilder] Add Capability to generate dynamic vega #7288

Conversation

ananzh commented Jul 17, 2024 • edited Loading

Description

Issues Resolved (Partially)

Screenshot

Testing the changes

Changelog

Check List

github-actions bot commented Jul 17, 2024

❌ Empty Changelog Section

codecov bot commented Jul 17, 2024 • edited Loading

Codecov Report

github-actions bot commented Jul 17, 2024

❌ Empty Changelog Section

LDrago27 Jul 22, 2024

Choose a reason for hiding this comment

ananzh Jul 22, 2024

Choose a reason for hiding this comment

LDrago27 Jul 22, 2024

Choose a reason for hiding this comment

LDrago27 Jul 22, 2024

Choose a reason for hiding this comment

ananzh Jul 22, 2024

Choose a reason for hiding this comment

LDrago27 Jul 22, 2024

Choose a reason for hiding this comment

ananzh Jul 22, 2024

Choose a reason for hiding this comment

ananzh commented Jul 17, 2024 •

edited

Loading

codecov bot commented Jul 17, 2024 •

edited

Loading