Research on failed visualization types and saved searches in Dashboards

[zhongnansu]

In the poc, we demonstrate how to enable visualizations to retrieve and render data from any user-configured datasource on demand. However there are some visualizations from sample dataset[e-commerce], that failed to render. We’ll dive deep into the reason it fails, and go over all viz types. Finally we either propose a more generic solution, or we decide if we want to exclude certain viz types from phase 1 implementation, but providing a path for future dev.

Failed Visualization Types and saved_search embeddable

[Done]TVSB:

1. E.g [E-commerce]Promotion Tracking.
2. API: api/metrics/vis/data
3. Research:
   1. TVSB register to expression plugin and add viz configs here (https://github.com/opensearch-project/OpenSearch-Dashboards/blob/1.2.0/src/plugins/vis_type_timeseries/public/plugin.ts#L82-L85)
   2. Unlike other viz types, TVSB plugin has it’s own backend
      1. In the backend of TVSB, it builds the query request from index pattern (

         OpenSearch-Dashboards/src/plugins/vis_type_timeseries/server/lib/vis_data/get_table_data.js

         Line 47 in e3b34df

         const { indexPatternObject } = await getIndexPatternObject(req, panelIndexPattern);

         ) and add default search strategy (https://github.com/opensearch-project/opensearch-dashboards/blob/e3b34df1dea59a253884f6da4e49c3e717d362c9/src/plugins/vis_type_timeseries/server/lib/search_strategies/search_strategy_registry.ts#L46) “opensearch” to it. which failed for external datasource
   3. Generated Expression: https://paste.amazon.com/show/szhongna/1654621155
      1. https://quip-amazon.com/sbawAaesYjBI#temp:C:FJD1ab3267588644074846797637
4. Solution:
   1. No need to make change to the generated expression
   2. Adding logic in TVSB backend to identify if datasource field is used in index-pattern, then adding the correct “ext-opensearch” search strategy. TSVB server side will call search api (https://github.com/opensearch-project/opensearch-dashboards/blob/e3b34df1dea59a253884f6da4e49c3e717d362c9/src/plugins/vis_type_timeseries/server/lib/search_strategies/strategies/abstract_search_strategy.ts#L76)of data plugin ** to retrieve data

[Done with issue]Vega:

1. API: use “search/opensearch” api, it should be using search/ext-opensearch
2. Research:
   1. Vega registers to expression plugin and adds viz configs here (https://github.com/opensearch-project/OpenSearch-Dashboards/blob/1.2.0/src/plugins/vis_type_vega/public/plugin.ts#L107-L109)
   2. Generated expression: https://paste.amazon.com/show/szhongna/1654633838
      1. dataSource is able to be added to the pipeline
3. Solution
   1. Vega uses the Elasticsearch search API (https://www.elastic.co/guide/en/elasticsearch/reference/8.2/search-search.html) to get documents and aggregation results from Elasticsearch
   2. Vega builds DSL Query (https://github.com/opensearch-project/opensearch-dashboards/blob/e3b34df1dea59a253884f6da4e49c3e717d362c9/src/plugins/vis_type_vega/public/vega_request_handler.ts#L84). Vega (https://www.elastic.co/guide/en/kibana/current/vega.html#vega) targets both index and index-pattern level. But since Vega has its own “Vega” syntax and it’s an experimental feature, so it will require special handling with respect to UX. We may consider onboarding Vega at a later phase

[Done]Timeline:

1. API: api/timeline/run
2. Research
   1. Timeline register to expression plugin and add viz configs here (https://github.com/opensearch-project/OpenSearch-Dashboards/blob/1.2.0/src/plugins/vis_type_timeline/public/plugin.ts#L108-L110)
   2. Unlike other viz types, TimeLine plugin has it’s own backend, and call low level search api exposed by
   3. Generated Expression:
      1. opensearchDashboards | opensearch_dashboards_context | timeline_vis expression=".es(*)" interval="auto"
3. Solution:
   1. Timeline (https://www.elastic.co/guide/en/kibana/current/timelion.html) directly builds DSL query (https://github.com/opensearch-project/opensearch-dashboards/blob/3fcc6975277194fd48d3e29ec2a2ca04c90a7152/src/plugins/vis_type_timeline/public/helpers/timeline_request_handler.ts#L116) to query OpenSearch. So it’s targeting at index level not index-pattern level. Our poc solution is based on index level to reference datasource. So the same idea can’t be applied in Timeline case.
   2. The above statement is incorrect. Timeline targets index pattern level. We can onboarding Timeline in a similar way as other viz types. First, find the index pattern, then adding logic to specify “ext-searchStrategy” [code ref (https://github.com/opensearch-project/opensearch-dashboards/blob/caaa0efd898c95525c27e25530e894e5c306eafa/src/plugins/vis_type_timeline/server/series_functions/opensearch/index.js#L140-L158)]

[Done] Search embeddable

1. Discover works with datasource, but when adding the saved search to dashboards it fails to render.
2. API usage: internal/search/opensearch
3. Research & Solution:
   1. Related code change should be made in Discover → saved_search_emabedables (https://github.com/opensearch-project/opensearch-dashboards/blob/e3b34df1dea59a253884f6da4e49c3e717d362c9/src/plugins/discover/public/application/embeddable/search_embeddable.ts#L207) → initializeSearchScope()
   2. Need to find the place to set dataSource field of searchSource, similar to Discover.js

if (indexPattern.dataSource) {
        console.log('zhongnan set datasource from search_emabeddable');
        searchSource.dataSource = indexPattern.dataSource;
  }

Summary

• Search embeddable is based on index pattern, with minor change on the browser side logic, they can fit into the current Poc solution.
• Failed visualizations has one thing in common, from the expressions they generated, we can see that they don’t go through opensearchaggs to create the searchSource obj, and call high level search API. Instead, they retrieve the index pattern, build the OpenSearch query from their plugin and directly query OpenSearch by calling the data plugin low level search API. Therefore, we have 2 proposals.
  • 1. We do special handing inside each viz plugins, to send search request with datasourceId as “searchOption“, based on if “datasource” appears in index pattern, and then apply to the search API call.
  • 2. A more ideal solution is, we create searchSource from those viz plugins and replace the data plugin Search API calls with searchSource.search(). And since the logic to use different strategy is already added to SearchSource, this will align with it. But this needs more research and even a poc to prove that any user input from those viz types can be transformed into searchSource.
• Notes: About High level and low level search API [ref (https://github.com/elastic/kibana/blob/main/src/plugins/data/README.mdx#search)]