Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[JEP-226] Improve migration section #292

Merged
merged 4 commits into from
Jul 17, 2020
Merged

[JEP-226] Improve migration section #292

Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
f1c4308
Improve migration section
stellargo Jun 17, 2020
8b39e93
Design changes
stellargo Jun 19, 2020
f0ec1f8
Improve language
stellargo Jul 16, 2020
76834f2
Merge remote-tracking branch 'upstream/master' into migration
stellargo Jul 16, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 22 additions & 27 deletions jep/226/README.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,11 +108,30 @@ However, the API includes functionality to retrieve fingerprints from other inst
This allows for tracing fingerprints across Jenkins instances.
Implementation wise, this is achieved by plugins ensuring that Fingerprints have an associated Jenkins Instance ID.

==== Local to External Migration
==== Migrations

When a plugin is configured and enabled, the fingerprints are transferred from the local storage to the external storage.
We propose to not implement explicit migrations in one go.
The newly configured external storage will gradually take over the fingerprints from disk storage.

Basic idea is this:

* save new fingerprints to the external storage
* when using a fingerprint from disk storage, delete it and save it in the external storage.

Implementation wise, this will involve changes in the methods in the following manner:

* `save`
** remains unaltered

* `load`
** If fingerprint is found in external storage, fetch it and return; just like earlier
** If not found, and `getReady` for file system storage is `true`, try loading the fingerprint from file system storage.
*** If found, load it, save it in external storage, and delete from file system storage.

* `delete`
** Try deleting fingerprint in external storage.
** If `getReady` for file system storage is `true`, then try deleting it in file system storage.

This by followed by clearing the local database of fingerprints from Jenkins HOME ONLY if the writes are successful.

==== Cleaning up Fingerprints

Expand All @@ -129,30 +148,6 @@ The above solution is provided to the user as an option, and can be toggled beca

We do not propose to revamp the `FingerprintCleanupThread` to use the above logic at the moment.

==== External to Local Migration (out of scope for MVP)

In case of external to local migration, a bulk read is issued, and the fingerprints are written to the local XML-based storage.
The following words of caution are included for the user:

* This operation may lead to high traffic
* This operation may take up a significant space of the disk (the user has to ensure such space is available), and
* This operation will not retrieve fingerprints or fingerprint metadata which is associated (created) by other Jenkins instances.

In case the disk may get full due to external factors during the migration or if there is a network failure, the transfer is aborted, all the downloaded fingerprints are deleted, and the user is notified.

==== External to External Migration (out of scope for MVP)

Such a migration can be either intra plugin (same storage system, but different instance), or inter plugin (different plugin).

The proposal offers the following solution:

* A bulk read of the fingerprints is performed. This operation could be memory intensive, which the Jenkins instance may not have. Hence, if this fails, as a fallback we read the fingerprints one at a time.
* Fingerprints are saved to the new external storage (destination).
* Lastly, deletion of the fingerprints from the source is performed.

Note that the above solution only migrates the fingerprints of the Jenkins instance performing the migration and not the other instances.
For a complete migration, the above procedure has to be performed on all the instances.

=== Design

The following new API entities are introduced:
Expand Down