Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[meta] Migrate .kibana esArchives to kbnArchiver #102552

Closed
tylersmalley opened this issue Jun 17, 2021 · 10 comments · Fixed by #106896 or #107156
Closed

[meta] Migrate .kibana esArchives to kbnArchiver #102552

tylersmalley opened this issue Jun 17, 2021 · 10 comments · Fixed by #106896 or #107156
Assignees
@wayneseymour
Labels
Meta Team:Operations Team label for Operations Team

Comments

@tylersmalley
Copy link
Contributor

tylersmalley commented Jun 17, 2021 •
edited by wayneseymour
Loading

The esArchiver is replacing the Kibana index, which is not something that Kibana still supports, causing numerous issues. Going forward we will use kbnArchiver, which is a saved object export to handle this data. In cases where isolation is required, Spaces can now be used.

Standard List (Types of Saved Objects that we do migrate)

As alluded to below, we've a finite list of saved object types that we can migrate.
If you run the following curl against a kibana server, you can obtain the list:

$ curl -k "http://elastic:changeme@localhost:5620/api/kibana/management/saved_objects/_allowed_types" | jq '.types | .[].name' | sort

As of July 6, 2022, the list is as follows:

"action"
"alert"
"apm-indices"
"canvas-element"
"canvas-workpad"
"cases"
"config"
"csp-rule-template"
"csp_rule"
"dashboard"
"graph-workspace"
"index-pattern"
"infrastructure-monitoring-log-view"
"infrastructure-ui-source"
"inventory-view"
"lens"
"map"
"metrics-explorer-view"
"osquery-pack"
"osquery-saved-query"
"query"
"search"
"tag"
"uptime-dynamic-settings"
"url"
"visualization"

Note: The list has and can change so I run the curl command periodically.

Helpers

Archives to be Migrated

SSPL

X-Pack

Other:

fgrep -ri '"index": ".kibana' test --include=mappings.json
@botelastic botelastic bot added the needs-team Issues missing a team label label Jun 17, 2021
tylersmalley pushed a commit to tylersmalley/kibana that referenced this issue Jun 17, 2021
[FTR] Stabilize SSLP functional tests
b82d43d 
* Removes spaces check, since spaces should always be available
* Disables Monitoring, SecuritySolutions, and Reporting in SSPL tests
until elastic#102552 is completed

Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
@tylersmalley tylersmalley mentioned this issue Jun 17, 2021
Merged
tylersmalley pushed a commit that referenced this issue Jun 17, 2021
[FTR] Stabilize SSLP functional tests (#102553)
00c1807 
* Removes spaces check, since spaces should always be available
* Disables Monitoring, SecuritySolutions, and Reporting in SSPL tests
until #102552 is completed

Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
tylersmalley pushed a commit to tylersmalley/kibana that referenced this issue Jun 17, 2021
[FTR] Stabilize SSLP functional tests (elastic#102553)
aa5b2e5 
* Removes spaces check, since spaces should always be available
* Disables Monitoring, SecuritySolutions, and Reporting in SSPL tests
until elastic#102552 is completed

Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
tylersmalley pushed a commit to tylersmalley/kibana that referenced this issue Jun 17, 2021
[FTR] Stabilize SSLP functional tests (elastic#102553)
389bdab 
* Removes spaces check, since spaces should always be available
* Disables Monitoring, SecuritySolutions, and Reporting in SSPL tests
until elastic#102552 is completed

Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
tylersmalley pushed a commit that referenced this issue Jun 21, 2021
@kibanamachine
Revert "Revert: [7.x] [FTR][CI] Use default distribution for all tests (
6977be0 
#102019) (#102580)

* Revert "Revert: [7.x] [FTR][CI] Use default distribution for all tests (#102019)"

This reverts commit c272bdb.

* [FTR] Stabilize SSLP functional tests (#102553)

* Removes spaces check, since spaces should always be available
* Disables Monitoring, SecuritySolutions, and Reporting in SSPL tests
until #102552 is completed

Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
@azasypkin azasypkin added the Team:Operations Team label for Operations Team label Jun 21, 2021
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-operations (Team:Operations)

@botelastic botelastic bot removed the needs-team Issues missing a team label label Jun 21, 2021
@bhavyarm bhavyarm mentioned this issue Jun 21, 2021
Merged
This was referenced Jun 21, 2021
Closed
Closed
Merged
This was referenced Jun 22, 2021
Closed
Merged
@rashmivkulkarni rashmivkulkarni mentioned this issue Jun 30, 2021
Merged
@dgieselaar dgieselaar mentioned this issue Jul 27, 2021
Closed
@wayneseymour wayneseymour mentioned this issue Jul 27, 2021
Merged
@wayneseymour
Copy link
Member

Throw an error in esArchiver when a Kibana index is used
@tylersmalley Are we doing this as I like this, perhaps naively :)

@spalger spalger reopened this Jul 29, 2021
This was referenced Aug 2, 2021
Merged
Merged
@wayneseymour wayneseymour reopened this Aug 3, 2021
@tylersmalley
Copy link
Contributor Author

@wayneseymour - just a heads up that you keep closing this issue.

@tylersmalley Are we doing this as I like this, perhaps naively :)

Yes, once we have migrated them all we will do this in the esArchiver.

@wayneseymour
Copy link
Member

just a heads up that you keep closing this issue.

Yeah I noticed. My apologies. No idea why, but I'll pay more attention to it. Maybe my browser is using gestures lol. I'll watch out for it though.

@rashmivkulkarni rashmivkulkarni mentioned this issue Aug 5, 2021
Closed
wayneseymour added a commit to wayneseymour/kibana that referenced this issue Sep 15, 2022
@wayneseymour
[Archive Migrations] example pipelines
485bfcc 
Drop two fields from the orig archive.

Helps with: elastic#102552
@wayneseymour wayneseymour mentioned this issue Sep 15, 2022
Merged
wayneseymour added a commit to wayneseymour/kibana that referenced this issue Sep 15, 2022
@wayneseymour
[Archive Migration] xpack example pipelines
215eb67 
Drop two fields from the orig archive.

Helps with: elastic#102552
wayneseymour added a commit to wayneseymour/kibana that referenced this issue Sep 15, 2022
@wayneseymour
[Archive Migration] xpack example pipelines
c3fb573 
Drop two fields from the orig archive.

Helps with: elastic#102552
wayneseymour added a commit to wayneseymour/kibana that referenced this issue Sep 15, 2022
@wayneseymour
[Archive Migration] xpack example pipelines
50016bd 
Drop two fields from the orig archive.

Helps with: elastic#102552
wayneseymour added a commit to wayneseymour/kibana that referenced this issue Sep 15, 2022
@wayneseymour
[Archive Migration] xpack example pipelines
be6b91c 
Drop two fields from the orig archive.

Helps with: elastic#102552
wayneseymour added a commit to wayneseymour/kibana that referenced this issue Sep 15, 2022
@wayneseymour
[Archive Migrations] timelion-feature-controls
9652663 
Helps with: elastic#102552

Drop archive as it's not used:
"$ find . -type f -exec grep -l "x-pack/test/functional/es_archives/timelion/feature_controls" {} \;"
@wayneseymour wayneseymour mentioned this issue Sep 15, 2022
Merged
wayneseymour added a commit to wayneseymour/kibana that referenced this issue Sep 19, 2022
@wayneseymour
[Archive Migration] xpack example pipelines
fe07cd9 
Drop two fields from the orig archive.

Helps with: elastic#102552
wayneseymour added a commit that referenced this issue Sep 19, 2022
@wayneseymour
[Archive Migrations] timelion-feature-controls (#140803)
6add682 
Helps with: #102552

Drop archive as it's not used:
"$ find . -type f -exec grep -l "x-pack/test/functional/es_archives/timelion/feature_controls" {} \;"
wayneseymour added a commit to wayneseymour/kibana that referenced this issue Sep 19, 2022
@wayneseymour
[Archive Migration] xpack example pipelines
f055263 
Drop two fields from the orig archive.

Helps with: elastic#102552
wayneseymour added a commit that referenced this issue Sep 19, 2022
@wayneseymour
[Archive Migration] xpack example pipelines (#140789)
e66bcf4 
Drop two fields from the orig archive.

Helps with: #102552
wayneseymour added a commit to wayneseymour/kibana that referenced this issue Sep 22, 2022
@wayneseymour
[Archive Migrations] x-pack import_rule_connector
17fd8bd 
Helps with: elastic#102552
wayneseymour added a commit to wayneseymour/kibana that referenced this issue Sep 30, 2022
@wayneseymour
[Archive Migrations] security_solution-timelines
c304957 
Add new kbn archive with an index pattern
taken from the es archive.

Change the test to use both es and kbn archives.
Since all these objects are loaded into a space,
just delete the space afterwards instead of
unloading with es archiver.

Helps with: elastic#102552
@wayneseymour wayneseymour mentioned this issue Sep 30, 2022
Merged
wayneseymour added a commit to wayneseymour/kibana that referenced this issue Sep 30, 2022
@wayneseymour
[Archive Migrations] security_solution-timelines
9ced15f 
Add new kbn archive with an index pattern
taken from the es archive.

Change the test to use both es and kbn archives.
Since all these objects are loaded into a space,
just delete the space afterwards instead of
unloading with es archiver.

Helps with: elastic#102552
wayneseymour added a commit to wayneseymour/kibana that referenced this issue Sep 30, 2022
@wayneseymour
[Archive Migrations] security_solution-timelines
e282a14 
Add new kbn archive with an index pattern
taken from the es archive.

Change the test to use both es and kbn archives.
Since all these objects are loaded into a space,
just delete the space afterwards instead of
unloading with es archiver.

Helps with: elastic#102552
wayneseymour added a commit to wayneseymour/kibana that referenced this issue Oct 3, 2022
@wayneseymour
[Archive Migrations] security_solution-timelines
37694d8 
Add new kbn archive with an index pattern
taken from the es archive.

Change the test to use both es and kbn archives.
Since all these objects are loaded into a space,
just delete the space afterwards instead of
unloading with es archiver.

Helps with: elastic#102552
wayneseymour added a commit to wayneseymour/kibana that referenced this issue Oct 4, 2022
@wayneseymour
[Archive Migrations] x-pack event_log_multiple_indicies
d06def8 
Add a new fn that checks for whether an archive should be
migrated.  This new fn disregards the standard_list.

Helps with: elastic#102552
@wayneseymour wayneseymour mentioned this issue Oct 4, 2022
Closed
wayneseymour added a commit to wayneseymour/kibana that referenced this issue Oct 4, 2022
@wayneseymour
[Archive Migrations] security_solution-timelines
f30dd57 
Add new kbn archive with an index pattern
taken from the es archive.

Change the test to use both es and kbn archives.
Since all these objects are loaded into a space,
just delete the space afterwards instead of
unloading with es archiver.

Helps with: elastic#102552
wayneseymour added a commit to wayneseymour/kibana that referenced this issue Oct 5, 2022
@wayneseymour
[Archive Migrations] security_solution-timelines
084d81e 
Add new kbn archive with an index pattern
taken from the es archive.

Change the test to use both es and kbn archives.
Since all these objects are loaded into a space,
just delete the space afterwards instead of
unloading with es archiver.

Helps with: elastic#102552
wayneseymour added a commit to wayneseymour/kibana that referenced this issue Oct 5, 2022
@wayneseymour
[Archive Migrations] x-pack event_log_multiple_indicies
96e81c8 
Add a new fn that checks for whether an archive should be
migrated.  This new fn disregards the standard_list.

Helps with: elastic#102552
wayneseymour added a commit to wayneseymour/kibana that referenced this issue Oct 6, 2022
@wayneseymour
[Archive Migrations] x-pack event_log_multiple_indicies
f13acfc 
Add a new fn that checks for whether an archive should be
migrated.  This new fn disregards the standard_list.

Helps with: elastic#102552
wayneseymour added a commit to wayneseymour/kibana that referenced this issue Oct 13, 2022
@wayneseymour
[Archive Migrations] security_solution-timelines
b239aca 
Add new kbn archive with an index pattern
taken from the es archive.

Change the test to use both es and kbn archives.
Since all these objects are loaded into a space,
just delete the space afterwards instead of
unloading with es archiver.

Helps with: elastic#102552
wayneseymour added a commit to wayneseymour/kibana that referenced this issue Oct 14, 2022
@wayneseymour
[Archive Migrations] security_solution-timelines
2b6e527 
Add new kbn archive with an index pattern
taken from the es archive.

Change the test to use both es and kbn archives.
Since all these objects are loaded into a space,
just delete the space afterwards instead of
unloading with es archiver.

Helps with: elastic#102552
wayneseymour added a commit that referenced this issue Nov 4, 2022
@wayneseymour @kibanamachine
[Archive Migrations] security_solution-timelines (#142363)
d4064c8 
* [Archive Migrations] security_solution-timelines

Add new kbn archive with an index pattern
taken from the es archive.

Change the test to use both es and kbn archives.
Since all these objects are loaded into a space,
just delete the space afterwards instead of
unloading with es archiver.

Helps with: #102552

* Drop exclusive

* Drop unneeded fn.

* Lint in my pocket.

* Thanks to J. Buttner, "re-drop" non needed objects from the
original archive.

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
@spalger spalger mentioned this issue Nov 8, 2022
Closed
@dmlemeshko dmlemeshko mentioned this issue Jun 19, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@wayneseymour wayneseymour

Labels
Meta Team:Operations Team label for Operations Team
Projects
None yet
Milestone
No milestone
6 participants
@tylersmalley @spalger @azasypkin @wayneseymour @LeeDr @elasticmachine