Split project.lock.json into *.lock.json and *.cache.json

[Mpdreamz]

I've read this gist on checking in project.lock files a couple times over and I really feel the current locking design feels optimized for the wrong scenario: finding breaking changes in moving targets. Whereas the more important default scenario should be repeatable builds.

To the best of my understanding the current project.json.lock now serves two purposes, caching the files in play for compilation and optionally locking dependencies using --lock.

This results in a bit of a operational kerfuffle. When do i need to restore using --lock ? Do I need to unignore lock files in release tags, and if so is this not important on dev branch too? if I branch of those tags should i remove the lock file on the first change?

What if locking is simply the norm just as it is in Paket?

Restore writes a very minimal project.lock.json (look at Paket for an example) if not present that you should always commit (eradicating both --lock and lock:false confusion).

When dnx runs the app for the first time it will generate a .project.cache.json which you should never commit.

If folks want to check fast moving targets on a CI server they could do dnu restore --latest on the CI servers explicitly.

• dnu restore --latest updates all * dependencies
• dnu restore --update updates all dependencies to latest
• dnu restore --outdated simply reports on packages that are outdated

To be fair the wiki does not dictate not checking project.lock.json files but does leave it up in the air without a consistent story.

Even if we do not make locking the norm a split will really help those who do choose to commit the lock file to not be faced with merge conflicts in the cached files section of this lock file.

[muratg]

@davidfowl

[analogrelay]

Note: We need to be extremely careful what we do here now that NuGet 3 and MSBuild for UWP apps are using project.json and project.lock.json

[davidfowl]

We spoke about this today and are going to make a split. The current project.lock.json will be the generated file used for runtime/buildtime and there will be another file maybe nuget.lock.json (name is still up in the air) that will just preserve version information.

The guidance will be that nuget.lock.json should be checked in if you want to lock versions. It will not be generated by default, but you can either do dnu restore --lock or dnu lock (there will be equivalent nuget.exe commands).

There is still an open question about putting the current project.lock.json in an output directory (bin\project.lock.json or obj\project.lock.json) but this file is read by the runtime so that seems a little strange.

[forki]

So is this a way to open up for other tools like paket?

[davidfowl]

I don't want to turn this bug into a discussion about packet but the project.lock.json is basically the format that needs to be produced to make dnx apps work.

[Mpdreamz]

I'm over the moon that the split will happen.

Whats the reason for not always generating the file? I see no reason not to always generate the file and losing dnu restore --lock or dnu lock all together? The choice then becomes whether to checkin the nuget.lock.json file or not.

[davidfowl]

Because the behavior of nuget and dnu isn't changing with respect to how dependent versions are resolved. There is really no reason to lock the project in most cases and it breaks version ranges. Meaning you have to keep running commands that constant update the lock file. I agree that it is a valid workflow and there should be a way to work like that but it's not the default required flow. There are still concerns about the project.json file changing and the nuget.lock.json (or whatever we call it) being out of sync.

[Mpdreamz]

Because the behavior of nuget and dnu isn't changing with respect to how dependent versions are
resolved. There is really no reason to lock the project in most cases and it breaks version ranges.

This has absolutely nothing to do with how versions are resolved and that won't need changing either.

When I run dnu restore and commit my working version and come back 2 years later to that commit and run dnu restore it should be repeatable even with wildcard deps in place. This should totally be the default workflow that we promote.

The fact that the project file has dependency version ranges will mean that a dnu update finds updates when I ask for it explicitly. In the current state we are conflating these two responsibilities.

dnu restore

• no lock file in place? resolve all dependencies.
• lock file in place? resolve only dependencies changed in project file.

dnu update|install <id> <version>

• update or dependency to or latest

dnu outdated <id>

• return information on all or package

I'm still happy that I can now choose to atleast --lock and checkin without having to commit all of dnx's cache but I really feel conflating these two responsibilities will bite us in the long run.

[haf]

Wouldn't it be a time-saver if you replaced this with two things?

• a JSON adapter for Paket to read/write your file format
• a --nuget-versions flag in paket that means to avoid using semver and do it the nuget way instead?

An up-side of doing it this way is that @forki would single-handedly carry half of vnext on his back with bugfixes in 7 minutes. ;)

[Mpdreamz]

@haf as much as I love Paket and hope to see it being integrated with DNX in the future, your comment sets the wrong tone for the active discussion in my opinion. This ticket is to discus the merits of locking by default using dnx ootb tooling which I'm still having a hard time understanding why the aspnet team is on the fence about this.

<brainfart>With this split Paket could now detect i'm running from a vnext dir and write a much terser nuget.lock.json file instead of the entire project.lock.json. Command line integration with dnu might be out of the question but its no different then nuget at the moment.

A "restore-command" : "paket install" would be enough in global.json for tooling such as @OmniSharp to attempt paket install instead of dnu restore. </brainfart>

[davidfowl]

With this split Paket could now detect i'm running from a vnext dir and write a much terser nuget.lock.json file instead of the entire project.lock.json. Command line integration with dnu might be out of the question but its no different then nuget at the moment.

That's inaccurate. project.lock.json is always required for the runtime to function. We're splitting responsibilities of the files, the need for those files still exist. But please, not on this issue.

[davidfowl]

This ticket is to discus the merits of locking by default using dnx ootb tooling which I'm still having a hard time understanding why the aspnet team is on the fence about this.

Because there are certain workflows that make cause friction, like the ones we use. I guess the problem is that we don't preserve our nightly builds forever, they get deleted from myget and having versions checked into source control that update every ci build would be very chatty. We have > 50 repositories that consume builds from each other where the version number is bumped on every build. That would be super noisy. Now on projects where you aren't building all of your own dependencies then it makes more sense.

[Mpdreamz]

All it takes for you is to update your CI build to do a dnu update explicitly before building, to make sure whatever is currently committed still works with latest everything. The fact your versions are not stored forever is irrelevant even then in the case of nightlies. For the same reason these versions are probably not tagged there is no need to recommit the changed lock files.

[davidfowl]

All it takes for you is to update your CI build to do a dnu update explicitly before building, to make sure whatever is currently committed still works with latest everything.

eek, and flood source control with commits that say, updating versions? I'd rather not 😄

The fact your versions are not stored forever is irrelevant even then in the case of nightlies. For the same reason these versions are probably not tagged there is no need to recommit the changed lock files.

Sure it is. You made a huge point about repeatable builds. Which lock files would we commit?

[Mpdreamz]

what commits? I do releases on every commit on myget using paket and they don't trigger commits just as they don't get tagged.

[davidfowl]

Who commits the updated lock file so to source control?

[Mpdreamz]

Why would you need to do that? If the CI explicitly updates to latest?

[davidfowl]

Is there ever a lock file in source control?

[Mpdreamz]

In my hypothetical situation yes you commit one.

When you check it out and run dnu restore it will use that *repeatable builds AWESOME.

When you run on CI you call dnu update before dnu restore commit version is build using latest and pushed to myget, who cares about the updated project.lock.json? Its not a commit or a proper release tag. You could even flip the default and add a --no-lock to make sure the local checkout stays unaltered.

If you want CI to only build known states (also VERY common) it can just do dnu restore and build the exact same thing that i just pushed.

[haf]

I'm completely with @Mpdreamz on this one; the lock file should be generated by default just like it is with npm install --save and bundler install.

• auto gen lock file
• if you want rolling versions, do update on CI, don't commit results, log dependency tree in CI
• if you want locked project, which is basically what all end-user projects want, i.e. those which are not infrastructure/library-oriented like the ASPNET team's are, then just don't run update on the CI server

You should really peruse the discussion we had in paket because it's really oriented towards building for production on .net.

[davidfowl]

Developer is working on one of our repos and new versions of packages are on the feed, they update the lock file?

CI builds needs to build against the latest of everything. Run dnu update or whatever but don't commit the lock file.

Who updates source control after the ci passes and produces new packages against the latest dependencies?

[Mpdreamz]

Developer is working on one of our repos and new versions of packages are on the feed, they update
the lock file?

If they update the version in project.json and run dnu restore or upgrade on the command line directly then yes. Never by modifying the lock file directly though.

CI builds needs to build against the latest of everything. Run dnu update or whatever but don't commit > the lock file.
Who updates source control after the ci passes and produces new packages against the latest > dependencies?

Who said this is even necessary? these nightly/daily/continuous CI builds do they get git tagged? If its not tag worthy its not commit worthy.

In situations where you are cutting real tag worthy releases, commiting the nuget.lock.json is a good thing.

If you are doing true continuous deployments to production, never ever would you NOT want to checkin your nuget.project.json.

[davidfowl]

In situations where you are cutting real tag worthy releases, commiting the nuget.lock.json is a good thing.

Totally agree!

If you are doing true continuous deployments to production, never ever would you NOT want to checkin your nuget.project.json.

We're talking about publishing libraries to myget.org (ci) or nuget.org (production). I appreciate that there are other scenarios (like publishing apps to production) but in this scenario I would never check in the lock file until there's a release, like 1.0.0-beta7 for example, which I think is what we do already today.

[Mpdreamz]

Should have been more exact on that last comment when I meant true continuous deployments I mean deploy chains that just deploy directly to production (machines, not production nuget feed) after each successful builds on ci (or you know yolo deploy :)).

In more staged environments or longer deploy chains (using e.g octopus deploy) you generally still cut releases that have to be pushed to a production nuget feed and then make there way across machines.

My point being that there are many ways to setup up integration and production feed publishes combined with many ways to combine those feeds to publish to staged application environments. I can pretty confidently say that the one thing they all have in common is that repeatable builds is of utmost importance. The flows where this is not, such as bleeding edge builds are the exception but our tooling is treating it as the norm.

but in this scenario I would never check in the lock file until there's a release, like 1.0.0-beta7 for example, which I think is what we do already today.

I think here lies our crux, I'm arguing the lock.json file is always a good thing why do you feel its only useful on releases? Is it not useful on branches, or commits from 2 years ago on master?
Again, I agree on bleeding edge CI feeds committing it back is not needed and genuine noise, just as tagging those releases would be. However the presence of the lock file in the repos does not prevent you from doing what you want, you just have to be explicit about your desire to update all dependencies and that is a good thing.

[akoeplinger]

I very strongly agree with @Mpdreamz on this one, for all the reason he mentioned. Running update in CI is the best solution if you want to build against the very latest packages. Reproducibility is preferred in all other cases. I'd be ecstatic if we could get rid of --lock and just always generate the nuget.lock.json (which you then check in or not).

[davidfowl]

Moving to the RC

[glennc]

Conclusions from the last meeting we had on this:

• Resolve confusion around lock.json
  • Rename file resolved.json
  • Move file out of the solution into somewhere project local that can be safely ignored
  • Remove locked:true from the file.

Once the above changes are made then this new resolved.json file is only designed to do the following:

• Give the runtime a consistent view of globally installed packages from build to build (no changing references because a different project pulled a higher version into your user level packages folder)
• Performance gains by caching the results of walking the dependency graph

It shouldn't be required to check the file in and the biggest source of confusion around this will hopefully be resolved.

What we don't have is a file that is designed to provide repeatable builds of a given commit, branch, etc at any point in time (assuming the packages are able to be downloaded still). As we examined this it become obvious that there are many workflows that people use and the more opinionated we are in the runtime the less likely we are going to be able to support them.

We will create a new issue and look at some workflows and how you would achieve something similar to the lock file without actually baking the concept of a lockfile into the runtime. One of our examples was a global command that could pin a project.json (replace * with resolved package version) and unpin (put the project.json back the way it was). We will look at some of these in the context of some workflows and see how they fit. See what sort of composition model or extension points are required in the runtime to make some of the things we have talked about here possible without the locked:true in this file.

[ctaggart]

What command do I run to update all the dependencies to the latest? I was looking for something like dnu update.

[davidfowl]

Yea, we need one of those

[ctaggart]

So it is just a manual process? I'll start with replacing -beta-23225 with -beta-23413 and go from there.
https://github.com/ctaggart/visualfsharp/blob/dnx/src/fsharp/FSharp.Core/project.json
Is the --outdated around in some form anymore?

[davidfowl]

We don't have anything right now. Though if you run outdated on the BCL packages as they are authored right now you can end up with a version that is too new that drops compatibility for platforms you may care about.

[Mpdreamz]

While I am glad the pseudo lock concept is gone, i still feel this statement is going to hurt us in the long run:

What we don't have is a file that is designed to provide repeatable builds of a given commit, branch,
etc at any point in time (assuming the packages are able to be downloaded still).

This thing needs to be designed for repeatable builds from the get go, repeatable builds are of paramount importance. If packages become unavailable then this is the risk of the developers workflow not because the defaults of the framework err on the side of failure.

Besides this decoupling the act of specifying what dependencies I want from their actual resolved version number is pure developer joy.

Consider a greenfield project and i want to use JSON.NET. Now I need to care about the version number. look up what is the lastest and write it down explicitly. If dnx will start to allow * for minor and major versions i bet you a beer or two that most devs will just put "NewtonSoft.Json" : "*" and not care until its too late.

Again the Paket workflow gives me both.

Another issue is that dependencies versions are now strongly tied to projects not as a repos as a whole. How do I make sure all my project.json's reference the same Json.NET ?

I would love to hear which scenario's you feel are not servicable using the Paket workflow? Then hopefully someone other then me can refute that better than I have been doing so far :)

[forki]

@Mpdreamz yes currently paket plays the role of a global dependency manager which helps you understand dependencies in the whole solution. We currently see no way to do the same with vnext projects.

• We could generate the dependencies block from our lock file knowledge in project.json but then you have to commit this to version control - which will give nasty merge conflicts.
• there seems to be no (obvious) way to switch the resolver/restore process that gets invoked when you save project.json
• if you really want to have reproducible builds there needs to be a chance to commit the packages folder (I don't really recommend this, but some users do have good reasons).
• there are now runtime dependencies, which are not in the package meta data. So you need to download the package to do this (which would break resolution perf)

[forki]

I don't want to turn this bug into a discussion about packet but the project.lock.json is basically the format that needs to be produced to make dnx apps work.

is it possible to just create that file and leave dependencies block in project.json empty? we could gitignore *.lock.json then

[augustoproiete]

What we don't have is a file that is designed to provide repeatable builds of a given commit, branch, etc at any point in time

Just adding myself to the group that thinks this is crucial before RTM's - and this file should be created by default (!).

(assuming the packages are able to be downloaded still)

A lot of companies host their own local/private NuGet feed exactly to make sure old versions of packages will be available at any point in the future. Also a number of devs will actually commit the NuGet package to source control, for the same reason (which could make a lot of sense for folks using Git LFS).

I'll just leave this here in the hope that we learn from the Rails devs ;)

[forki]

Closed? What's the resolution?

[davidfowl]

Dnx has been dead for over a year now. Best to open an issue on nuget about creating a true lock file. In csproj what used to be the lock file is now the assets file and in the obj folder.