Backfill repo metadata tool #6288

skofman1 · 2018-08-09T00:12:30Z

/// This tool collect repository metadata for all packages in the DB from nuspec files in V3 flat container and updates DB with this data.
/// Usage:
/// 1. To collect repository metadata:
/// a. Configure app.config with DB information and service index url
/// b. Run this tool with: GalleryTools.exe -c
/// This will create a file repositoryMetadata.txt with all collected data. You can stop the job anytime and restart. cursor.txt contains current position.
///
/// 2. To update DB:
/// a. Run GalleryTools.exe -u
/// This will update DB from file repositoryMetadata.txt. You can stop the job anytime and restart.

joelverhagen · 2018-08-09T23:42:35Z

src/GalleryTools/App.config


+    <add key="Gallery.SqlServer" value="Data Source=(localdb)\mssqllocaldb; Initial Catalog=NuGetGallery; Integrated Security=True; MultipleActiveResultSets=True"/>
+    <add key="Gallery.SupportRequestSqlServer" value="Data Source=(localdb)\mssqllocaldb; Initial Catalog=NuGetGallery; Integrated Security=True; MultipleActiveResultSets=True"/>


Why is Gallery.SupportRequestSqlServer added?

DefaultDependenciesModule can't be created without it.

joelverhagen · 2018-08-09T23:43:13Z

Spoke offline about this, but we should have a README.md that is next to the csproj that describes how this command works. Granted, the other commands don't have docs but we can improve this as we go on.

joelverhagen · 2018-08-09T23:45:36Z

src/GalleryTools/Commands/BackfillRepositoryMetadataCommand.cs

+            CommandOption fileName = config.Option("-f | --file", "File to use", CommandOptionType.SingleValue);
+
+            config.HelpOption("-? | -h | --help");
+            config.OnExecute(() =>


OnExecute can accept a Func<Task<int>>. This is means the whole method can be async and you can avoid .GetAwaiter().GetResult().

joelverhagen · 2018-08-09T23:46:57Z

src/GalleryTools/Commands/BackfillRepositoryMetadataCommand.cs

+            {
+                Log.LogMessage("Initializing");
+
+                _context = new EntitiesContext(connectionString, readOnly: true);


👍, readonly. nice.

joelverhagen · 2018-08-09T23:47:42Z

src/GalleryTools/Commands/BackfillRepositoryMetadataCommand.cs

+
+                    Log.LogMessage($"Start time: {startTime.ToString("u")}");
+
+                    var packagesRepository = new EntityRepository<Package>(_context);


You have DI above, why no use it? Makes this more flexible for the future (future backfills...)

joelverhagen · 2018-08-09T23:48:24Z

src/GalleryTools/Commands/BackfillRepositoryMetadataCommand.cs

+
+                    var packagesRepository = new EntityRepository<Package>(_context);
+
+                    var allPackages = packagesRepository.GetAll().Where(p => p.Created < lastCreateTime && p.Created > startTime && p.PackageStatusKey == PackageStatus.Available);


So you can miss validating packages?

Or less importantly failed validation packages that we reflow?

Paging over Edited will mean you never miss a package.

Also, consider adding a 5-30 minute subtract when first starting due to clock skew.

LastEdited is null for ~1M packages. Will stay with Created. Added support for validating packages. Don't see how clock skew is a factor if we compare always to Created that's DB generated.

Those can come in out of order. There is no guarantee that a sequence of Created timestamps are committed in the same order that they are generated.

You can order by LastEdited, PackageKey then. But it's fine if you include validating.

joelverhagen · 2018-08-09T23:56:59Z

src/GalleryTools/Commands/BackfillRepositoryMetadataCommand.cs

+
+            private static async Task<RepositoryMetadata> GetRepositoryMetadata(string id, string normalizedVersion)
+            {
+                string nuspecUri = $"{_flatContainerUri}{id.ToLowerInvariant()}/{normalizedVersion.ToLowerInvariant()}/{id.ToLowerInvariant()}.nuspec";


nit: trim trailing '/' and include '/' in the string format

_flatContainerUri might have trailing / -- or might not. It's an easy config mistake to make.

just a nit

joelverhagen · 2018-08-09T23:58:00Z

src/GalleryTools/Commands/BackfillRepositoryMetadataCommand.cs

+            private static string GetFlatContainerUri(Uri serviceDiscoveryUri)
+            {
+                var serviceDiscoveryClient = new ServiceDiscoveryClient(serviceDiscoveryUri);
+                return serviceDiscoveryClient.GetEndpointsForResourceType("PackageBaseAddress/3.0.0").GetAwaiter().GetResult().First().AbsoluteUri;


These should be async methods. .GetAwaiter().GetResult() should only be in one place per app. (In most cases)

joelverhagen · 2018-08-09T23:58:51Z

src/GalleryTools/Commands/BackfillRepositoryMetadataCommand.cs

+
+            private static async Task WriteMetadata(DateTime creationDate, string packageId, string packageVersion, RepositoryMetadata repositoryMetadata)
+            {
+                await _metadataFileStreamWriter.WriteLineAsync(


Consider one JSON object per line to avoid escaping problems. Or use a CSV library. Repository metadata could have a comma (type in particular).

Using CsvHelper

joelverhagen · 2018-08-10T00:00:07Z

src/GalleryTools/Commands/BackfillRepositoryMetadataCommand.cs

+            }
+
+            private static async Task CommitBatch(DateTime cursorTime)
+            {


You might run into memory problems since the EF context will hold onto all of the old entities.

You can recreate the EF context each batch to mitigate this.

this is indeed an issue for the "collect" part. However, introducing batching there is too much work. Will leave as is. Thanks for the cursor the job can continue after crash.

joelverhagen · 2018-08-10T00:07:58Z

src/GalleryTools/Commands/BackfillRepositoryMetadataCommand.cs

+
+            CommandOption lastCreateTimeOption = config.Option("-l | --lastcreatetime", "The latest creation time of package we should check", CommandOptionType.SingleValue);
+            CommandOption collectData = config.Option("-c | --collect", "Collect Repository metadata and save in file", CommandOptionType.NoValue);
+            CommandOption updateDB = config.Option("-u | --update", "Update DB with Repository metadata", CommandOptionType.NoValue);


Consider a less aggressive starting point for how quickly updateDB updates batches. This could be done with batchSize and sleepSeconds parameters. Then you can do some batches slowly on PROD before kicking it into high gear.

I'm using the same batching config as I did for "hash" Gallery tool. It saved a 100 and had no sleep. There was not DB impact.

joelverhagen · 2018-08-10T00:11:46Z

src/GalleryTools/Commands/BackfillRepositoryMetadataCommand.cs

+                            counter = 0;
+                        }
+
+                        metadata = await TryReadNextMetadata();


nit: do ... while should be able to eliminate the duplicated metadata = await TryReadNextMetadata(); line.

will leave as is...

joelverhagen

⌚️

joelverhagen

skofman1 added 2 commits August 8, 2018 17:06

works

40d874b

improve log

789bdfb

skofman1 requested review from joelverhagen, agr, xavierdecoster, shishirx34, zivkan, ryuyu, chenriksson, scottbommarito and zhhyu August 9, 2018 00:14

joelverhagen reviewed Aug 9, 2018

View reviewed changes

joelverhagen reviewed Aug 10, 2018

View reviewed changes

joelverhagen requested changes Aug 10, 2018

View reviewed changes

skofman1 added 3 commits August 13, 2018 16:51

PR comments

852e381

Fix comments

4deebd5

bug fix

1087f45

joelverhagen approved these changes Aug 14, 2018

View reviewed changes

skofman1 merged commit 1b3707e into dev Aug 15, 2018

skofman1 deleted the repotool branch August 15, 2018 17:28

scottbommarito mentioned this pull request Aug 20, 2018

Deployment 2018-08-23 #6316

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Backfill repo metadata tool #6288

Backfill repo metadata tool #6288

skofman1 commented Aug 9, 2018 •

edited

Loading

joelverhagen Aug 9, 2018

skofman1 Aug 10, 2018

joelverhagen commented Aug 9, 2018

joelverhagen Aug 9, 2018

skofman1 Aug 13, 2018

joelverhagen Aug 9, 2018

joelverhagen Aug 9, 2018

joelverhagen Aug 9, 2018

skofman1 Aug 13, 2018

joelverhagen Aug 9, 2018

joelverhagen Aug 9, 2018 •

edited

Loading

skofman1 Aug 13, 2018

joelverhagen Aug 13, 2018

joelverhagen Aug 13, 2018

joelverhagen Aug 9, 2018

skofman1 Aug 13, 2018

joelverhagen Aug 13, 2018

skofman1 Aug 13, 2018

joelverhagen Aug 9, 2018

joelverhagen Aug 9, 2018

skofman1 Aug 13, 2018

joelverhagen Aug 10, 2018

joelverhagen Aug 10, 2018

skofman1 Aug 13, 2018

joelverhagen Aug 10, 2018

skofman1 Aug 13, 2018

joelverhagen Aug 10, 2018

skofman1 Aug 13, 2018

joelverhagen left a comment

joelverhagen left a comment


		<add key="Gallery.SqlServer" value="Data Source=(localdb)\mssqllocaldb; Initial Catalog=NuGetGallery; Integrated Security=True; MultipleActiveResultSets=True"/>
		<add key="Gallery.SupportRequestSqlServer" value="Data Source=(localdb)\mssqllocaldb; Initial Catalog=NuGetGallery; Integrated Security=True; MultipleActiveResultSets=True"/>


		Log.LogMessage($"Start time: {startTime.ToString("u")}");

		var packagesRepository = new EntityRepository<Package>(_context);


		var packagesRepository = new EntityRepository<Package>(_context);

		var allPackages = packagesRepository.GetAll().Where(p => p.Created < lastCreateTime && p.Created > startTime && p.PackageStatusKey == PackageStatus.Available);

Backfill repo metadata tool #6288

Backfill repo metadata tool #6288

Conversation

skofman1 commented Aug 9, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joelverhagen commented Aug 9, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joelverhagen Aug 9, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joelverhagen left a comment

Choose a reason for hiding this comment

joelverhagen left a comment

Choose a reason for hiding this comment

skofman1 commented Aug 9, 2018 •

edited

Loading

joelverhagen Aug 9, 2018 •

edited

Loading