[Statistics] Introduce Community Statistics Reports #4955

loic-sharma · 2017-11-03T23:18:01Z

Introduce Community Statistics Reports

This change introduces the concept of "community" statistics report that will be displayed by default. These reports filter out packages owned by Microsoft and .NET Framework. Users can choose to "Show all packages" to view reports that include Microsoft's and .NET Framework's packages. With this change, packages like System.Globalization will no longer clutter our default statistic reports! 😄

Note: The JsonStatisticsService had some serious concurrency issue and did not cache reports that it loaded from blob storage. This pull request fixes that by loading reports once an hour.

This change depends on NuGet/NuGet.Jobs#239.
Fixes #3417.

Screenshots

Statistics Homepage

Detailed Report

Clicking on the "Show more..." links at the bottom of the homepage will take you to the following reports:

Most Downloaded Packages

Most Downloaded Package Versions

anangaur · 2017-11-03T23:22:50Z

I dont think we should have the filter on the https://www.nuget.org/stats page. The filter should only be there on the details pages:
https://www.nuget.org/stats/packageversions
https://www.nuget.org/stats/packages

anangaur · 2017-11-03T23:28:09Z

Something like this (ignore the actual list that shows the current data):

loic-sharma · 2017-11-04T00:03:51Z

Added a filter to both index and detailed pages. I will update the screenshots on Monday.

loic-sharma · 2017-11-04T18:58:51Z

src/NuGetGallery/Services/JsonStatisticsService.cs

-        public async Task<StatisticsReportResult> LoadDownloadPackageVersions()
+        private Task<StatisticsReportResult> LoadDownloadPackageVersions()
+        {
+            var statisticsPackagesItemViewModels = (List<StatisticsPackagesItemViewModel>)DownloadPackageVersionsAll;


TODO: Delete this.

loic-sharma · 2017-11-04T19:01:07Z

src/NuGetGallery/Services/StatisticsReportName.cs

+        /// Breakout by version for a community package (drill down from RecentPopularity).
+        /// This excludes Microsoft and ASP.NET packages.
+        /// </summary>
+        RecentCommunityPopularityDetail_,


TODO: This is not needed, remove.

joelverhagen · 2017-11-06T16:43:26Z

src/NuGetGallery/Controllers/ApiController.cs

@@ -704,9 +704,9 @@ protected internal virtual Stream ReadPackageFromRequest()
        [ActionName("StatisticsDownloadsApi")]
        public virtual async Task<ActionResult> GetStatsDownloads(int? count)
        {
-            var result = await StatisticsService.LoadDownloadPackageVersions();
+            await StatisticsService.Refresh();


Why this change?

See the description:

The JsonStatisticsService had some serious concurrency issue and did not cache reports that it loaded from blob storage. This pull request fixes that by loading reports once an hour.

To elaborate a little: all the Load... reports were merged into the Refresh method.

joelverhagen · 2017-11-06T17:07:07Z

src/NuGetGallery/Views/Statistics/Index.cshtml

-
+    <p>
+        Show all packages:
+        <input type="checkbox" data-bind="checked: showAllPackageDownloads" />


The include prerelease checkbox on search and the per-package stats page both are linkable. Meaning clicking on them updates the URL in the address bar so people can link to the results. You should do this either with a GET form or replaceState in JavaScript.

Good suggestion. However, due to feedback this will no longer be linkable.

joelverhagen · 2017-11-06T17:09:38Z

Do users really care about the old report? Is this driven by a specific customer ask? It is unclear from the linked issue if we should provide a way to toggle between them.

joelverhagen

⌚️
@anangaur's points are good. Let's close the loop here.

anangaur · 2017-11-06T18:18:29Z

The filter (check box) should be enabled by default to filter out the Microsoft packages as that is what users want. It serves 2 purposes:

Soft landing for users who are used to seeing the Microsoft packages in the list.
Messaging: We need to tell what is it that we show on the page :)

loic-sharma · 2017-11-07T01:38:25Z

Yes, non-community packages (including Microsoft packages) are filtered out by default. Talked to @anangaur offline, we cannot use "Hide Microsoft packages" as non-community packages include non-Microsoft packages such as .NET Framework System.* packages. We decided to go with "Show all packages" instead.

Addressed all other feedback.

xavierdecoster · 2017-11-08T12:58:59Z

src/NuGetGallery/Controllers/StatisticsController.cs

-                IsDownloadPackageDetailAvailable = availablity[1].Loaded,
-                DownloadPackageVersionsSummary = _statisticsService.DownloadPackageVersionsSummary,
-                IsNuGetClientVersionAvailable = availablity[2].Loaded,
+                IsDownloadPackageAvailable = _statisticsService.DownloadCommunityPackagesResult.Loaded,


Whilst at it, should we rename the *Result properties to be a little more clear? e.g. DownloadCommunityPackagesResult > CommunityPackageDownloadsResult

Same for the Loaded property which currently sounds like an event: IsLoaded?

Will do! :)

loic-sharma self-assigned this Nov 3, 2017

loic-sharma requested review from joelverhagen, xavierdecoster, shishirx34, ryuyu, anangaur, skofman1 and scottbommarito November 3, 2017 23:18

dnfclas added the cla-already-signed label Nov 3, 2017

loic-sharma commented Nov 4, 2017

View reviewed changes

joelverhagen reviewed Nov 6, 2017

View reviewed changes

joelverhagen requested changes Nov 6, 2017

View reviewed changes

xavierdecoster approved these changes Nov 8, 2017

View reviewed changes

joelverhagen approved these changes Nov 8, 2017

View reviewed changes

loic-sharma added 8 commits November 8, 2017 13:55

Refactored statistics service; added new community reports

7df389c

Updated views

909be3f

Tweak title off state

91e1479

Addressed feedback

e13188c

Fixed tests

8078d07

Cleaned up

6f99346

WIP - Refactored; added first test

69e27ec

Added more tests

b6081a3

loic-sharma added 7 commits November 8, 2017 13:55

Fixed issues

46d2e3e

Fixed UI bugs

3f2876d

Addressed feedback

a047544

Fixed caption

9a0891f

Tweaked subheading

5e71d34

Renamed methods; fixed test

8b5cdd0

Tweaked UI

7148615

loic-sharma force-pushed the loshar-stats branch from 4c6186a to 7148615 Compare November 8, 2017 21:55

loic-sharma merged commit 8d911f0 into dev Nov 8, 2017

loic-sharma deleted the loshar-stats branch November 8, 2017 21:56

shishirx34 mentioned this pull request Nov 8, 2017

Deployment 2017-11-30 #4950

Closed

10 tasks

304NotModified mentioned this pull request Jul 2, 2019

stats: Community Packages shows non-Community Packages #7314

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Statistics] Introduce Community Statistics Reports #4955

[Statistics] Introduce Community Statistics Reports #4955

loic-sharma commented Nov 3, 2017 •

edited

Loading

anangaur commented Nov 3, 2017

anangaur commented Nov 3, 2017 •

edited

Loading

loic-sharma commented Nov 4, 2017

loic-sharma Nov 4, 2017

loic-sharma Nov 4, 2017

joelverhagen Nov 6, 2017

loic-sharma Nov 7, 2017 •

edited

Loading

loic-sharma Nov 7, 2017

joelverhagen Nov 6, 2017

loic-sharma Nov 7, 2017

joelverhagen commented Nov 6, 2017

joelverhagen left a comment

anangaur commented Nov 6, 2017

loic-sharma commented Nov 7, 2017 •

edited

Loading

xavierdecoster Nov 8, 2017

xavierdecoster Nov 8, 2017

loic-sharma Nov 8, 2017

[Statistics] Introduce Community Statistics Reports #4955

[Statistics] Introduce Community Statistics Reports #4955

Conversation

loic-sharma commented Nov 3, 2017 • edited Loading

Introduce Community Statistics Reports

Screenshots

Statistics Homepage

Detailed Report

Most Downloaded Packages

Most Downloaded Package Versions

anangaur commented Nov 3, 2017

anangaur commented Nov 3, 2017 • edited Loading

loic-sharma commented Nov 4, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

loic-sharma Nov 7, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joelverhagen commented Nov 6, 2017

joelverhagen left a comment

Choose a reason for hiding this comment

anangaur commented Nov 6, 2017

loic-sharma commented Nov 7, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

loic-sharma commented Nov 3, 2017 •

edited

Loading

anangaur commented Nov 3, 2017 •

edited

Loading

loic-sharma Nov 7, 2017 •

edited

Loading

loic-sharma commented Nov 7, 2017 •

edited

Loading