Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Yarn Upgrade not changing yarn.lock since v1.0 #4476

Closed
bdwain opened this issue Sep 15, 2017 · 39 comments · Fixed by #4636
Closed

Yarn Upgrade not changing yarn.lock since v1.0 #4476

bdwain opened this issue Sep 15, 2017 · 39 comments · Fixed by #4636

Comments

@bdwain
Copy link
Contributor

bdwain commented Sep 15, 2017

Do you want to request a feature or report a bug?
BUG

What is the current behavior?
nothing happens when i run yarn upgrade. No yarn-error.log is generated either. It happens for other people too.

yarn upgrade v1.0.2
success All of your dependencies are up to date.

But they are not all up to date.

in the other project, i see this every time i run yarn upgrade

success Saved lockfile.
success Saved 5 new dependencies.
├─ angular-auto-focus@1.0.4
├─ angular-bind-html-compile@1.4.1
├─ chiri@1.0.0
├─ html2canvas@0.4.1
└─ sass-quatch@1.0.3

(it's worth noting that those 5 packages are the only 5 that come up when i say yarn upgrade-interactive, and they are all direct git repo dependencies, not npm packages. the other project has no such dependencies)

but yarn.lock never changes

yarn upgrade also no longer installs dependencies when node_modules is empty.

If the current behavior is a bug, please provide the steps to reproduce.
I am just running yarn upgrade in my existing project. It's private so I can't link to it. But it happens in multiple projects

What is the expected behavior?
yarn.lock should update along with dependencies

Please mention your node.js, yarn and operating system version.
node: 8.5.0
yarn: 1.0.2
OS: Mac Sierra

Someone else has yarn 0.27.5 with the same project and it updates the dependencies. They are not just all up to date.

One other thing is that I am using an internal yarn registry (with artifactory), but it has never had an issue (and still works as of now) with yarn 0.27.5.

I'll try to post my package.json if that sheds any light on it

{
  "name": "whatever",
  "version": "9.0.1",
  "description": "foo",
  "private": false,
  "module": "es/components.js",
  "publishConfig": {
    "registry": "some internal url"
  },
  "files": [
    "es/"
  ],
  "scripts": {
    "unimportant": null
  },
  "repository": {
    "type": "git",
    "url": "some url"
  },
  "dependencies": {
    "classnames": "^2.2.5",
    "lodash-es": "^4.17.4",
    "numeral": "^2.0.3",
    "react-addons-shallow-compare": "^15.5.2",
    "react-dates": "^12.1.1",
    "react-onclickoutside": "^6.4.0",
    "shallowequal": "^1.0.2"
  },
  "devDependencies": {
    "@storybook/addon-info": "^3.2.0",
    "@storybook/addon-knobs": "^3.2.0",
    "@storybook/addon-options": "^3.2.3",
    "@storybook/react": "^3.2.3",
    "babel-polyfill": "^6.16.0",
    "immutable": "^3.8.1",
    "moment": "^2.18.1",
    "react": "^15.5.4",
    "react-dom": "^15.5.4",
    "react-router-dom": "^4.1.1",
    "svg-sprite-loader": "^3.2.5"
  },
  "peerDependencies": {
    "babel-polyfill": "^6.16.0",
    "moment": "^2.18.1",
    "react": "^15.3.2",
    "react-dom": "^15.3.2"
  },
  "registry": "npm"
}

and i have a .yarnrc file pointing to an internal yarn registry (running on artifactory. it works with 0.27.5)

@bdwain
Copy link
Contributor Author

bdwain commented Sep 18, 2017

any update on this? it's currently hard to update dependencies for my projects.

@tassilogroeper
Copy link

bump. this is critical!!

@tassilogroeper
Copy link

I just downgraded on a mac my local yarn to v0.27.5 node lts-6x - no lock file created/updated.
A mac of my coleguee has has v0.23.2 with node version 8x - no lock file created/updated.

so I sense there is an other problem somewhere?!

@tassilogroeper
Copy link

tassilogroeper commented Sep 19, 2017

FYI for anyone interested in downgrading read this article
http://remarkablemark.org/blog/2017/02/03/install-brew-package-version/

and I used the commit hash to point to the right yarn.rb file like so:

$ brew install https://github.com/Homebrew/homebrew-core/87b41768b63a52d772f7e8bcc8e9360492ea513f/Formula/yarn.rb

@ai
Copy link

ai commented Sep 19, 2017

It is critical blocker for me too

@tassilogroeper
Copy link

quick fix for build environments would be to use
$ yarn install --no-lockfile
but will increase duration of the build step ...

@bdwain
Copy link
Contributor Author

bdwain commented Sep 19, 2017

@pheadeaux did you clear your cache after downgrading?

@bdwain
Copy link
Contributor Author

bdwain commented Sep 19, 2017

downgrading (and yarn cache clean)fixed it for me. thanks @pheadeaux.

I hope this gets fixed soon. We basically can't upgrade to yarn 1 until it's fixed.

@bdwain
Copy link
Contributor Author

bdwain commented Sep 19, 2017

added some details about my pacakge.json to the original post to help give more details.

@publicarray
Copy link

same on yarn 1.0.2 with MacOS Sierra.
yarn install and yarn don't create a yarn.lock file anymore either. I just removed the file from git until this is fixed

@bdwain
Copy link
Contributor Author

bdwain commented Sep 23, 2017

are there any maintainers who can look at this? it's preventing a lot of people from upgrading to version 1.

@bdwain
Copy link
Contributor Author

bdwain commented Sep 26, 2017

@BYK do you know what I can do to get this issue addressed? It's been a while since it was posted and has received no response, but it affects a bunch of people and forces us to stay on yarn 0.27.

@cebor
Copy link

cebor commented Sep 26, 2017

Issue still with yarn 1.1.0

@BYK
Copy link
Member

BYK commented Sep 27, 2017

Ping @rally25rs. I think this is the new expected behavior after 1.0.

@bdwain if it is urgent, you may wanna try digging into the code yourself. The code responsible for this is here: https://github.com/yarnpkg/yarn/blob/master/src/cli/commands/upgrade.js

@bdwain
Copy link
Contributor Author

bdwain commented Sep 27, 2017

@rally25rs @BYK this doesn't seem like expected behavior. The docs explicitly state that yarn upgrade still changes yarn.lock. Is there another way to upgrade the lock file in 1.0 that I'm missing? Yarn upgrade seems to be the normal way of doing that.

@rally25rs
Copy link
Contributor

rally25rs commented Sep 27, 2017

But they are not all up to date.

@bdwain Could you provide an example? (what the package name is, what version is currently installed, what semver range is specified in package.json)


and they are all direct git repo dependencies

Packages that are direct git repos are always considered "out of date" and will be re-downloaded. The SHA hash of the latest commit should be updated in yarn.lock if there is actually a newer commit.

For example:

"my-project@git+ssh://git@github.com/my-org/my-project#v1.2.0":
  version "1.2.0"
  resolved "git+ssh://git@github.com/my-org/my-project#aa05cc498558d27347a3ced52db22b9820b8e7c8"

With a git repo I could push a new commit to the v1.2.0 branch without bumping the version or branch name, so "version" becomes meaningless. The latest SHA hash will change on the end of resolved if there are actually new commits. At the moment Yarn doesn't actually query Github for the list of commit SHAs to see if there is a difference, it just assumes it is always "outdated" (IIRC it worked this way pre v1 also)

So if no new commits are actually pushed to any of those 5 repos, then Yarn will still "upgrade" to them, but the yarn.lock file would appear unchanged because the latest commit SHA is the same.


It is quite reasonable that Yarn could/should query the github repo to check what the latest commit SHA is to determine if it is "outdated", but the existing behavior (uses yarn outdated's logic) already marked all "exotic" resolvers as "always report them as outdated" so I didn't change that between v0 and v1. That is a little more apparent in the output of yarn outdated where is actually says the version is "exotic".

$ yarn outdated
yarn outdated v1.0.2
Package        Current Wanted Latest Package Type    URL
my-project     1.2.0   exotic exotic devDependencies git+ssh://git@github.com/my-org/my-project#v1.2.0

@bdwain
Copy link
Contributor Author

bdwain commented Sep 27, 2017

@rally25rs my package.json is in my original comment. My package is internal to my company so i can't really share it. I changed the name and the publishConfig but otherwise it's the same.

In general, i wasn't so worried about the direct git dependencies. I just pointed them out to see if they were helpful. Really the problem was that I knew some of my semver dependencies were out of date but not getting updated.

The other notable thing in my scenario is that I have a .yarnrc file pointing to an internal yarn registry running on top of artifactory. It forwards stuff to actual yarn and npm and has always worked fine with yarn 0.27, but maybe the registry needs updating for yarn v1?

@rally25rs
Copy link
Contributor

rally25rs commented Sep 27, 2017

@bdwain

Really the problem was that I knew some of my semver dependencies were out of date but not getting updated.

I assume it is one of the dependencies or devDependencies listed above in your package.json? Which one, and what version is currently installed (actual exact version, not semver range)?

(knowing the package name and what version you are currently at, I can try to reproduce the issue)

Edit:

Also keep in mind that there was a change from v0 to v1 that changed what version would be upgraded to. Upgrade now respects the semver range in your package.json file unless you pass the -L/--latest flag. That may or may not be the issue in your case, but was a change in functionality that has caused a few other issues for folks.

@bdwain
Copy link
Contributor Author

bdwain commented Sep 27, 2017

it may also be the dependencies i have in my file. I'm not totally sure because none need an update now. But dependencies of dependencies are definitely affected. The expected upgrades are within the semver range i have though, so that should be the same I think (and I was under the impression that v0 's upgrade respected semver anyway).

so as an example, i have yarn 0.27 installed and i just ran yarn upgrade on my project, and one of my dev dependencies (i didn't list it because it's internal), pulls in webpack-dev-server ^2.6.1

Currently, webpack-dev-server was resolving to 2.8.2 (in my yarn.lock at least). when i ran yarn upgrade on v0, it upgrade to 2.9.0. then i reverted my changes and upgraded yarn to v1 and reran yarn upgrade. Webpack dev server stayed at 2.8.2.

Does yarn look at dependencies' yarn.locks now or something?

EDIT: I added webpack-dev-server ^2.6.1 to my package.json and reran yarn upgrade and it pulled in the new version. So it seems like the issue is with transitive dependencies.

@rally25rs
Copy link
Contributor

Hmm, perhaps that is the issue... I don't think v1 tries to upgrade transient dependencies. I'm not sure if v0 did... never really thought about it. What upgrade does in v1 is basically call the logic for yarn outdated then installs whatever outdated reports is the new version. I think outdated only looks at the dependencies listed in your package.json, not transient dependencies.

It was all confusing back in v0 too because upgrade and upgrade-interactive were actually very different (upgrade-interactive was not at all an "interactive" version of upgrade and they shared no code and would choose different versions to upgrade to) so in v1 we tried to make them actually behave the same. The v1 upgrade really behaves like the v0 upgrade-interactive and outdated did (which only showed upgrades for your direct dependencies).

I suspect in v1 webpack-dev-server will only get upgraded if the package that included it also had an update (otherwise it would be excluded from the upgrade list so it's transient dep on webpack-dev-server would be unchanged).

I'll go re-pull the code for the last v0 build and see what upgrade actually did... It was the oddball of the 3 commands (upgrade, upgrade-interactive, outdated) so saw the most change in v1...

@bdwain
Copy link
Contributor Author

bdwain commented Sep 27, 2017

Cool thanks for looking into this!

My thought was always that an app should always determine the final version of any package it pulls in, whether transitive or not. Libraries should specify valid ranges but the app's yarn.lock was responsible for specifying the actual version (and respecting dependency ranges). So it seemed that ignoring the yarn.lock in dependencies was by design. If not, libraries basically need publish frequent updates with hardcoded dependency versions to let people use the most up to date version of transitive dependencies.

Also, it's worth noting that with v1, if i delete my yarn.lock file and just run yarn, it'll resolve webpack-dev-server to 2.9. So it seems like at least some yarn commands behave in that way.

@rally25rs
Copy link
Contributor

OK did a little hunting...

In v0 all upgrade really did was to take all your deps in package.json and feed them back to yarn add

So if your package.json had:

"dependencies": {
  "foo": "^1.0.0",
  "bar": "~2.0.0"
}

then it was functionally equivalent to yarn add foo@^1.0.0 bar@~2.0.0 which would go and re-install those packages, rechecking the semver range, and also rechecking all the transient dependencies because of it. It really didn't check at all to see if anything was actually out of date.

in v1 I do the same basic thing, feeding packages to yarn add's logic, but I pre-filter the list by packages that are actually outdated (IIRC I did this because a) it seemed faster / more efficient, and b) because it is what upgrade-interactive did already). I hadn't considered the fact that previously it would have the side-effect of also rechecking for upgrades in transient deps. I really was focused on making upgrade and upgrade-interactive actually do the same thing.

@yarnpkg/core The topic "what is the correct thing to do here" is probably worth some additional discussion. We could go back to passing all packages to add whether or not they are out of date, but that would make upgrade work differently from upgrade-interactive again.

@bdwain Although it isn't ideal, I think you could emulate the old v0 upgrade behavior by just deleting your yarn.lock file and running yarn install

@bdwain
Copy link
Contributor Author

bdwain commented Sep 27, 2017

Yup I was aware of that shortcut. For me it was easier to just downgrade to 0.27 temporarily because I just type yarn upgrade without even thinking now lol.

To me, yarn upgrade-interactive seems like a different command from yarn upgrade. I wouldn't expect to be asked about a transitive dependency in an interactive mode, but something needs to upgrade them, and yarn upgrade seems like the place to do it.

I always used upgrade-interactive to see what version upgrades were available to me that yarn upgrade WOULDN'T pull in for me, and just ran yarn upgrade regularly to pull in the latest versions WITHIN my semver range without caring which packages were actually affected. If someone wanted more granular control over their individual package upgrades, I would argue that they should set more strict semver ranges (or not even use ranges at all).

@BYK
Copy link
Member

BYK commented Sep 29, 2017

@yarnpkg/core The topic "what is the correct thing to do here" is probably worth some additional discussion. We could go back to passing all packages to add whether or not they are out of date, but that would make upgrade work differently from upgrade-interactive again.

How about we add a new flag to upgrade such as --all or --force, -f to bring back that old behavior?

@KristofMorva
Copy link

@BYK that's much better than having no option at all, but in my opinion, people usually just do yarn upgrade, and expect all the packages to be upgraded, so I think it should be the default behavior without any flags. Maybe a --use-lockfile or something similar command could bring back the current behavior.

@bdwain
Copy link
Contributor Author

bdwain commented Sep 29, 2017

Agreed. The behavior without any flags should be the intended purpose of the command. Options should be for exceptional cases. I think most people won't care if a package is a transitive dependency or not they'll want the latest version that they or their dependencies specifiy.

That being said, I also think it'd be good to revert upgrade-interactive to its v0 behavior. Like I said earlier, to me it seems fundamentally different from upgrade, so sharing code shouldn't be the goal. The main reason for the interactivity to me is to upgrade outside of your semver range. Otherwise you should just use yarn upgrade or make your semvers more strict. I don't see it being especially useful in its current form without the --latest option.

Thanks again for looking into this! I appreciate the effort!

@rally25rs
Copy link
Contributor

I don't see it being especially useful in its current form without the --latest option.

Counterpoint; to me upgrade-interactive is only useful without --latest.

Example: I have a project that uses phantomjs for unit testing, and it has to be a ~1.0.0 (1.x) version, it can not be a 2.x version, but I want to keep up-to-date on the latest 1.x release. The same can be said of Marionette.js. v3 has breaking changes, so I need the latest ^2.0.0 release only, but not 3.x. I might only want to upgrade a couple packages but not all of them, and using interactive is a nice way to pick what to change.

There were several github issues around this pre-v1 because it was really confusing that each command picked different versions to upgrade to.

That said, I'm fine with upgrade also upgrading transient deps, with or without a flag. I'll see if I can work up a PR to re-introduce that portion of the upgrade...

@bdwain
Copy link
Contributor Author

bdwain commented Sep 29, 2017

@rally25rs It sounds like upgrade-interactive is just masking the fact that your semver ranges could be tweaked to fit your needs better. Why would you ever not want to upgrade a package within the semver range you specify in package.json? It seems every package could be given a range such that you are always comfortable upgrading within that range, and then yarn upgrade always brings you to the latest comfortable versions for all of your packages, while upgrade-interactive could be used to change those comfort ranges.

In your case, you could set the semver for phantom to ^1.0.0 and ^2.0.0 for marionette. then yarn upgrade will keep those packages within their semver ranges and you only need interactive when you're ready to bump phantom to v2 or marionette to v3, and you can pick and choose the packages you want to pull in breaking changes for.

@bdwain
Copy link
Contributor Author

bdwain commented Sep 29, 2017

Actually I'm starting to think the name upgrade-interactive is causing the confusion. The old behavior encouraged people to use it like I was using it, but the name makes it sound like it's just supposed to let you run a subset of the full upgrade.

Maybe the solution is to add a new command that behaves like upgrade-interactive with --latest. If that was done, I think it would satisfy everyone. I think this behavior is common enough that it should be available without adding options to another command.

@rally25rs
Copy link
Contributor

@bdwain yes that is what my semver ranges are set to. However if I selectively only wanted to upgrade say phantomjs to the latest in the semver range, in yarn v0...

  • yarn upgrade would upgrade all packages (not what I wanted)
  • yarn upgrade-interactive would ignore semver and go to v2 (not what I wanted)
  • yarn upgrade phantomjs would ignore semver and go to v2 (because... bug?)

There was no way to make upgrade-interactive respect semver in v0. Now in v1 you can do either with the --latest flag. I'm not trying to make the case for myself really... we had a ton of duplicate issues in v0 of people reporting upgrade and upgrade-interactive not working (or not working the same, or just not understanding why they were upgrading to different versions). We just tried to make the commands make more sense and at least somewhat work the same.

I don't know how many times I replied to issues trying to explain "upgrade-interactive is not an interactive version of upgrade" which really makes no sense because the command naming would lead you to believe that it is (now in v1 is basically is).

Anyway, as soon as I get a free moment I'll see what I can do about getting upgrade to traverse the transient deps again like it used to (so it will function more like the way it did in v0)

@bdwain
Copy link
Contributor Author

bdwain commented Sep 29, 2017

@rally25rs i see. My question is why are you uncomfortable updating other packages at the same time? Is there not a set of semver ranges that you would ALWAYS be comfortable running yarn upgrade with? I always viewed the whole point of semver ranges was to autoupgrade packages without having to pick and choose them individually. Otherwise you could just not set ranges and manually upgrade things when you want to. Though the alternative would be that yarn upgrade PKG upgrades a specific package within its range, which i think it should do.

@rally25rs
Copy link
Contributor

@bdwain well, yes and no... I mean the whole point of Yarn and NPM5 is deterministic builds, right. I've definitely had CI servers fail just because someone pushed a new patch-version of a package and it had a bug, so there are certain packages I look more carefully at before accepting their changes 😆 In a perfect world, yeah upgrade everything 👍

@bdwain
Copy link
Contributor Author

bdwain commented Sep 29, 2017

Yea i get that @rally25rs. What are your thoughts on yarn breaking-changes-interactive (or whatever. name's not important for now) that's just an alias for upgrade-interactive --latest. I do think the name upgrade-interactive makes more sense to behave like it does now, but one of my favorite yarn features was being able to browse the major upgrades I could potentially pull in interactively. I'd hate to see it hidden behind an option rather than be up front as a top level command where people will see it more.

@danez
Copy link
Contributor

danez commented Oct 4, 2017

I already got used to always passing the --latest flag to upgrade-interactive, as for me too it doesn't make sense to run without this flag. In my case this is because (for example the phantomjs update) I only use upgrade-interactive for updates that are not included in the ranges I have in package.json (phantom 1.x -> 2.x, when using ^1.0.0 in pkg.json). If I would want to get the latest versions of phantomjs within my range I would run yarn upgrade, which pre-1.0 upgraded all dependencies.

This is also where the main painpoint currently is for me. I see that the new --latest flag makes sense for upgrade-interactive and also upgrade, but the new behaviour of upgrade does never update subdependencies. In order to get the pre-1.0 behaviour of yarn upgrade (which updates everything that is in range to the latest version and generates a complete new lockfile), I have to remove yarn.lock and node_modules and run yarn install, which I have to do now regularly in all my oss repos. It is not that much work, but it is a pity that yarn does not have a command for yarn upgrade-everything-to-the-possible-latest-in-range-version-and-generate-new-lock anymore.

@rarkins
Copy link
Contributor

rarkins commented Oct 4, 2017

I have to remove yarn.lock and node_modules and run yarn install

@danez what behaviour did you experience if you did simply rm -rf yarn.lock && yarn install vs rm -rf yarn.lock && rm -rf node_modules && yarn install ? I also remove node_modules to be safe but wasn't 100% sure how it can impact lock file resolutions, which I thought were done first/independently of fetching packages.

@danez
Copy link
Contributor

danez commented Oct 4, 2017

Yeah same for me, not sure if it makes a difference I just wanted to be safe.

@rally25rs
Copy link
Contributor

@danez I think I mentioned it above in this discussion, but it wasn't my intent to remove the upgrade of sub-dependencies when I made these changes. Yarn already had a built-in function to find outdated packages, so I just used that instead of blindly feeding all packages to the process, whether they needed upgraded or not. I saw it as an efficiency/speed gain (no sense in "upgrading" a package from v1.2.3 to v1.2.3, it's already there). I didn't think about the side-effect of that packages transient deps potentially being upgraded.

I started on code for this fix yesterday and hope to PR something today...

@bdwain
Copy link
Contributor Author

bdwain commented Oct 4, 2017

@rally25rs seems like yarn upgrade could share code with yarn install right? since it does the same thing assuming there is no yarn.lock

@rally25rs
Copy link
Contributor

For those keeping tabs on this issue... I pushed PR #4636 if anyone wants to try it out and verify that it works as expected.

Sorry for the delay getting this together. It was more complicated than I thought because of the new features that were added as part of v1.

@bdwain to respond to your comment, internally upgrade just passes install a series of packages to install (so they do share code in a way). Determining what version to upgrade to (and preserving ranges and filtering) is done by the upgrade code. There actually does need to be a yarn.lock file though, which causes a big part of the complication, because you can do things like yarn upgrade foo bar which should only upgrade the "foo" and "bar" packages, and no others, so the lockfile entries need to be preserved for any other deps and their transient deps. (so it's not as easy as just nuking the lockfile and re-running install)

Anyway, if someone wants to test out that PR, I'd appreciate it!

arcanis pushed a commit that referenced this issue Oct 11, 2017
* [#4476] Upgrade transient deps during upgrades.

* Rename 'transient' to 'transitive'

* dont upgrade direct deps unless requested, add verbose upgrade logging

* upgrade-interactive reuse lockfile cleaning from upgrade.js
joaolucasl pushed a commit to joaolucasl/yarn that referenced this issue Oct 27, 2017
* [yarnpkg#4476] Upgrade transient deps during upgrades.

* Rename 'transient' to 'transitive'

* dont upgrade direct deps unless requested, add verbose upgrade logging

* upgrade-interactive reuse lockfile cleaning from upgrade.js
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

10 participants