Cutover aggs to the new platform #59605

lukeelmers · 2020-03-07T00:44:47Z

Summary

Moves agg_types into the new platform under the data plugin's search service. Once this PR is merged, it should unblock anybody waiting on the rest of the search service to move to the new platform.

Some specific items moved include:

agg_types
esaggs
tabify
registration of range action and click action

At this point there is nothing left in src/legacy/core_plugins/data that anybody should need to rely on... anything there is getting left behind for BWC and can be completely deleted when downstream apps complete migration.

For reviewers

If you got pinged for a codeowners review, don't be intimidated by the size of this PR 😬

Most of the significant changes here affect code owned by app arch, but since nearly all of the code has been removed from src/legacy/core_plugins/data at this point, imports needed to be updated across the project. In most cases you should only have a few changed lines of code to review.

For those reviewing large portions of the PR, I recommend going commit-by-commit instead of attempting to peruse the whole thing. I've tried to keep a coherent commit history so it should make things somewhat easier to follow.

For QA

This PR should introduce no functional changes to Kibana; it is 100% internal refactoring as we migrate our search aggregation infrastructure to the new platform.

Additional info

Now that docs extractor has been added to data plugin, a ton of markdown files were created
Updated MIGRATION.md to reflect changes
Exports from ui/agg_types were preserved for BWC and to minimize the changes that need to go into this (already large) PR.
Structure of the contracts that were already in place in core_plugins/data has been preserved
Date-related interval utils have been grouped together under data/common. This includes parseEsInterval, dateHistogramInverval, isValidEsInterval, etc.
Updates were made to ui/new_platform to ensure functionality that was necessary for the legacy platform still works without the legacy data plugin

Dev docs

I am skipping docs on this PR, as the documentation is being added to the 7.7 ui/public cleanup documentation issue.

elasticmachine · 2020-03-07T00:44:49Z

Pinging @elastic/kibana-app-arch (Team:AppArch)

lukeelmers · 2020-03-10T16:37:39Z

src/plugins/data/common/field_formats/converters/source.ts

+ */
+function noWhiteSpace(html: string) {
+  const TAGS_WITH_WS = />\s+</g;
+  return html.replace(TAGS_WITH_WS, '><');


This is a one-line function so I inlined it here rather than importing from legacy kibana plugin as we had done previously.

lukeelmers · 2020-03-10T16:38:48Z

src/legacy/core_plugins/data/public/plugin.ts

-export interface DataSetup {
-  search: SearchSetup;
-}
+export interface DataSetup {} // eslint-disable-line @typescript-eslint/no-empty-interface


Kept these empty interfaces in place because there were a few areas in Kibana still using them. (trying to keep things as small as possible).

lukeelmers · 2020-03-10T16:39:24Z

src/legacy/core_plugins/data/public/plugin.ts

-    setInjectedMetadata(core.injectedMetadata);
-
+export class DataPlugin implements Plugin<DataSetup, DataStart> {
+  public setup(core: CoreSetup) {
    // This is to be deprecated once we switch to the new search service fully
    addSearchStrategy(defaultSearchStrategy);


@lizozom said this will be going away shortly

lukeelmers · 2020-03-10T16:41:52Z

src/plugins/data/common/index.ts

 export * from './index_patterns';
-export * from './es_query';
-export * from './utils';


Removed this export because now the only thing in utils is internal to other items in the common directory.

lukeelmers · 2020-03-10T16:42:33Z

src/plugins/data/kibana.json

@@ -3,6 +3,9 @@
  "version": "kibana",
  "server": true,
  "ui": true,
-  "requiredPlugins": ["uiActions"],
+  "requiredPlugins": [
+    "expressions",


expressions is now required to register esaggs function

lukeelmers · 2020-03-10T16:45:38Z

src/plugins/data/public/plugin.ts

+      actions: {
+        createFiltersFromEvent,
+      },


createFiltersFromEvent is technically stateful as it relies on index patterns behind the scenes. Open to suggestions on where this should live, but in its current form it cannot be exported statically @ppisljar

lukeelmers · 2020-03-10T16:47:01Z

src/plugins/data/public/search/aggs/agg_types.ts

-    geoTileBucketAgg,
-  ],
-};
+export function getAggTypes(deps: { uiSettings: IUiSettingsClient }) {


Now that this has been wrapped with a provider pattern, we can (as a follow up step) get rid of the getters/setters that are currently being used to retrieve stateful services in each agg type, and instead explicitly pass those deps down.

might make sense to pass down something that can be same on client and server (like getConfig function)

lukeelmers · 2020-03-10T16:49:29Z

src/plugins/data/public/search/aggs/test_helpers/mock_data_services.ts

@@ -41,11 +40,10 @@ import {
 export function mockDataServices() {


This still lives as a test helper inside search.aggs, but might be more generically useful elsewhere in the data plugin. Perhaps we consider eventually moving it up to a different directory

lukeelmers · 2020-03-10T16:51:50Z

src/plugins/data/public/search/search_strategy/default_search_strategy.test.ts

          search: newSearchMock,
          __LEGACY: {
            esClient: {
              search: searchMock,
              msearch: msearchMock,
            },
          },
-        },
+        } as unknown) as jest.Mocked<ISearchStart>,


Changed this to casting because there was a circular dependency going back to require the search.aggs mocks, and it felt unnecessary to duplicate the entire search service contract here

lukeelmers · 2020-03-10T16:52:33Z

src/plugins/kibana_utils/public/index.ts

@@ -18,18 +18,19 @@
 */

 export {
+  calculateObjectHash,


calculateObjectHash wasn't exported at the top level so I added it here, and alphabetized

kibanamachine · 2020-03-12T18:43:08Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: fcfa468

History

💔 Build #32628 failed aa00d90
💔 Build #32542 failed 4c02548
💔 Build #32494 failed d9b9dcf
💔 Build #32288 failed 36d078d
💔 Build #32270 failed cc19af4

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

joshdover

Migration guide changes LGTM. Thanks for making the plugins. prefix more clear 😄

ppisljar

code LGTM

sulemanof

LGTM, mostly checked the default editor changes, great work!

flash1293 · 2020-03-13T12:12:52Z

I will review this PR today.

flash1293

Tested and couldn't find any problems, LGTM 👍

lukeelmers added release_note:plugin_api_changes Contains a Plugin API changes section for the breaking plugin API changes section. v8.0.0 Team:AppArch Feature:NP Migration v7.7.0 labels Mar 7, 2020

lukeelmers self-assigned this Mar 7, 2020

lukeelmers force-pushed the aggs/cutover branch 2 times, most recently from 59d7a0d to 79495f9 Compare March 9, 2020 22:24

lukeelmers changed the title ~~WIP: Cutover aggs to the new platform~~ Cutover aggs to the new platform Mar 10, 2020

lukeelmers force-pushed the aggs/cutover branch from ba448d7 to c2e811e Compare March 10, 2020 14:41

lukeelmers mentioned this pull request Mar 10, 2020

Enhancement - EUICodeEditor for Visualize JSON #58679

Merged

6 tasks

lukeelmers force-pushed the aggs/cutover branch from 5b69e10 to c776aba Compare March 10, 2020 16:28

lukeelmers commented Mar 10, 2020

View reviewed changes

This comment has been minimized.

Sign in to view

lukeelmers force-pushed the aggs/cutover branch from c776aba to 399c193 Compare March 10, 2020 20:50

lukeelmers added the Feature:Aggregations Aggregation infrastructure (AggConfig, esaggs, ...) label Mar 10, 2020

lukeelmers force-pushed the aggs/cutover branch from cc19af4 to 3094c26 Compare March 10, 2020 23:27

lukeelmers requested a review from ppisljar March 10, 2020 23:30

lukeelmers added release_note:skip Skip the PR/issue when compiling release notes review and removed release_note:plugin_api_changes Contains a Plugin API changes section for the breaking plugin API changes section. labels Mar 10, 2020

lukeelmers mentioned this pull request Mar 10, 2020

Dev Docs entry: ui/public cleanup 7.7 #55926

Closed

lukeelmers force-pushed the aggs/cutover branch from 3094c26 to 36d078d Compare March 11, 2020 00:03

lukeelmers marked this pull request as ready for review March 11, 2020 03:16

lukeelmers requested a review from a team March 11, 2020 03:16

lukeelmers requested review from a team as code owners March 11, 2020 03:16

lukeelmers added 12 commits March 12, 2020 10:55

Clean up utils & exports!

85efe3d

Migrate mocks!

3c5bf0e

Clean up legacy data plugin!

a292b36

Move tabify!

2d64ad9

Move isValidJson to DefaultEditor!

8807cda

Fix TS!

ff518ca

Move data actions to NP!

55db48c

Move search expressions utils to NP!

848c151

Fix tests, eslint, and conflicts with elastic#58679!

f56bc99

Ensure service setters are still set for legacy apps!

aa5fd4f

Generate docs!

e710b45

Update migration guide!

fcfa468

lukeelmers force-pushed the aggs/cutover branch from aa00d90 to fcfa468 Compare March 12, 2020 17:04

joshdover approved these changes Mar 12, 2020

View reviewed changes

ppisljar approved these changes Mar 13, 2020

View reviewed changes

sulemanof approved these changes Mar 13, 2020

View reviewed changes

flash1293 mentioned this pull request Mar 13, 2020

Meta: Kibana app NP cutover #60097

Closed

69 tasks

flash1293 approved these changes Mar 13, 2020

View reviewed changes

lukeelmers merged commit 5d428b0 into elastic:master Mar 13, 2020

lukeelmers deleted the aggs/cutover branch March 13, 2020 15:12

lukeelmers mentioned this pull request Mar 13, 2020

[7.x] Search Service: cutover agg types to the new platform (#59605) #60112

Merged

lukeelmers added a commit to lukeelmers/kibana that referenced this pull request Mar 13, 2020

Search Service: cutover agg types to the new platform (elastic#59605)

90c76b2

lukeelmers mentioned this pull request Mar 13, 2020

Search 👉 New Platform #42639

Closed

14 tasks

lukeelmers added a commit that referenced this pull request Mar 13, 2020

Search Service: cutover agg types to the new platform (#59605) (#60112)

7cf974c

sulemanof mentioned this pull request Mar 17, 2020

[NP] Use local helper shortenDottedString for discover #60271

Merged

7 tasks

lukeelmers mentioned this pull request Apr 28, 2020

[data.search.aggs] Remove legacy aggs APIs. #64719

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cutover aggs to the new platform #59605

Cutover aggs to the new platform #59605

lukeelmers commented Mar 7, 2020 •

edited

Loading

elasticmachine commented Mar 7, 2020

lukeelmers Mar 10, 2020

lukeelmers Mar 10, 2020

lukeelmers Mar 10, 2020

lukeelmers Mar 10, 2020

lukeelmers Mar 10, 2020

lukeelmers Mar 10, 2020

lukeelmers Mar 10, 2020

ppisljar Mar 11, 2020

lukeelmers Mar 10, 2020

lukeelmers Mar 10, 2020

lukeelmers Mar 10, 2020

This comment has been minimized.

This comment has been minimized.

kibanamachine commented Mar 12, 2020

joshdover left a comment

ppisljar left a comment

sulemanof left a comment

flash1293 commented Mar 13, 2020

flash1293 left a comment

		@@ -41,11 +40,10 @@ import {
		export function mockDataServices() {

Cutover aggs to the new platform #59605

Cutover aggs to the new platform #59605

Conversation

lukeelmers commented Mar 7, 2020 • edited Loading

Summary

For reviewers

For QA

Additional info

Dev docs

elasticmachine commented Mar 7, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

kibanamachine commented Mar 12, 2020

💚 Build Succeeded

History

joshdover left a comment

Choose a reason for hiding this comment

ppisljar left a comment

Choose a reason for hiding this comment

sulemanof left a comment

Choose a reason for hiding this comment

flash1293 commented Mar 13, 2020

flash1293 left a comment

Choose a reason for hiding this comment

lukeelmers commented Mar 7, 2020 •

edited

Loading