msbuild /restore support for all NuGet use cases

[infin8x]

Background

As new .NET project types (e.g. .NET Core, among others) are introduced, newer versions of NuGet are required to understand the project type and accompanying target framework. In Visual Studio, this is fine because the VS update that brings the new project types also brings new versions of NuGet libraries that can restore packages for these project types.

VSTS builds are unable to provide this same symmetry because we are unable to use those built-in NuGet libraries, and instead call NuGet.exe or dotnet.exe directly to do restore. This causes issues when the version of NuGet used to restore is too old to work with one or more of the projects in the user's solution. We want to avoid this issue by using the new msbuild /restore functionality, which relies solely on the installed copy of MSBuild, which in turn matches the version of Visual Studio installed on the build agents. This will mean that agents can build new project types as soon as they're updated to a new version of Visual Studio.

Ask

However, msbuild /restore currently only works for .NET Core/PackageRef projects. We'd like to have its functionality extended to support both .NET paradigms (Framework/packages.config is missing). Once that's done, we can simplify the VSTS build templates for .NET project to use this built-in restore capability and avoid hard-to-debug build errors during the restore step.

Related issues

• Created basic CI CD setup and getting strange error microsoft/azure-pipelines-tasks#5997
• Microsoft.NETCore.Targets 1.0.2 is not compatible with UAP,Version=v10.0.14393 microsoft/azure-pipelines-tasks#6039
• Packages failed nuget restore VS 2017 in VSTS projects .net standard microsoft/azure-pipelines-tasks#5236
• Nuget restore fails on net461 project referencing netstandard2.0 project microsoft/azure-pipelines-tasks#5164
• NuGet restore fails when encountering netcoreapp2.0 -> netstandard2.0 references microsoft/azure-pipelines-tasks#5065
• https://developercommunity.visualstudio.com/content/problem/145862/build-nuget-restore-private-feed.html
• https://developercommunity.visualstudio.com/content/problem/158828/nuget-restore-on-private-agent-always-using-v41.html
• https://developercommunity.visualstudio.com/content/problem/100882/nuget-restore-fails-after-aspnet-core-20-migration.html
• https://developercommunity.visualstudio.com/content/problem/32997/hosted-agent-2017-with-nuget-35.html

[emgarten]

The current extension point for the VSTS credential provider is not cross platform and is only used by nuget.exe. (NuGet.Credentials).

I spent some time looking at this recently, it looks like it could be moved down to NuGet.Protocol level so that it can be used for all network calls, regardless of entry point/OS.

We should consider if we want to move the current code, or if we want to move to using @dtivel's plugin protocol to handle external auth.

@anangaur can you help come up with a plan which protocol to use for external auth providers going forward?

//cc @xavierdecoster who has also expressed interest in this area for myget's auth provider.

[infin8x]

@emgarten this is great news, thanks for looking into it! We've had a long-running desire to have better support for Credential Providers (or some similar 2FA-capable auth model) in dotnet.exe, which it sounds like this would enable. If I understand right, this would also work for msbuild /restore?

@anangaur when we talked yesterday, this was lower on the backlog. That still makes sense to me given the other items on your stack and the team's stack. But, if we're going to do the /restore for .NET Framework work (this issue), I wonder if we should have a plan for the auth stuff as well, so we can make a unified recommendation for the restore-in-CI flow ("always use msbuild and a credential provider"). If you'd like, I'd be happy to drive the plan in close partnership with your team.

[anangaur]

There are 2 parts to the issue here:

1. msbuild t:/restore, dotnet restore and msbuild /restore should work for other package format types.
2. Support for auth on the above mentioned CLIs.

If required we should split this issue into these 2 parts so that we do not block 1 because of 2.

@alexmullans @emgarten @xavierdecoster
We should also sync up on the future course of action for auth. I see three options:

1. NuGet (in all the CLIs) having a capability to do the auth. This would work best from user experience perspective. May be we need to do this only to NuGet's cross plat functionality.
2. The current credential provider mechanism. So the additional work for auth is to be able to support the Linux/Mac credential provider that VSTS has.
3. Go the plugin model that @dtivel worked on for the large package support.

[AndreyAkinshin]

Any updates?

[nkolev92]

Duplicate of #8506.

This will ship in 5.5/16.5, @anangaur want to keep it open until it ships or close it?

[aortiz-msft]

Duplicate of #8506 which was released as part of 16.5 Preview 2.