[Feature] Allow opting-out of vendoring berry's files

[aduh95]

• I'd be willing to implement this feature
• This feature can already be implemented through a plugin

Describe the user story

I am a Yarn user, I have run yarn set version berry on my repo. Now I would like not to commit .yarn/releases; it's not my code, I am not confortable having it on my CVS; also, upgrading to a new version produces huge diffs.
One of the reason I am using Yarn, it is to avoid storing vendor files; I feel forcing users to vendor is not the way to go.

Describe the solution you'd like

We could imagine yarn set version xx keeps the actual version number on the .yarnrc.yml; when running yarn, it will first check if the yarnPath file exists, and if not, checks if a yarnVersion is defined and download it from the network. Note that it doesn't change anything for projects that vendor the file, while providing an easy opt-out (adding .yarn/ to .gitignore).

Describe the drawbacks of your solution

Besides the debat vendoring vs using a package manager, I can't think of any.

Describe alternatives you've considered

Put the .yarn folder and the .yarnrc.yml to my .gitignore and put a note on the Getting Started section of the README to run yarn set version berry...

[arcanis]

Although doable, I would prefer to avoid downloading binaries from remote sources on commands that aren't supposed to have anything to do with the network (such as yarn run, for example). Otherwise, projects using Yarn may end up being unusable or vulnerable should it die someday (I don't find that likely to happen anytime soon, but I always try to take the bus factor into account).

I know that a few people from the Node project have toyed with the idea "what if Node didn't ship with any package manager", or "what if Node shipped with a package manager installer" rather than just npm. Maybe it would be the right time to start a discussion to see how we could make it easier for everyone to manage multiple package manager versions?

[aduh95]

projects using Yarn may end up being unusable or vulnerable should it die someday

One day, the npm registry will be gone forever, and then most of my projects using Yarn will become unusable. I guess that's a risk I willing to take, as I don't vendor my npm dependencies and use Yarn to download them for me. But I see your point regarding the security aspect.

Maybe limit the proposed behaviour to yarn install? That command already downloads dependencies from the network, one could argue yarn-berry.js is also a project dependency.

Maybe it would be the right time to start a discussion to see how we could make it easier for everyone to manage multiple package manager versions?

I couldn't agree more, that would be awesome! But Berry would probably keep its "per-project" install policy though, right?

[arcanis]

I couldn't agree more, that would be awesome! But Berry would probably keep its "per-project" install policy though, right?

Yes. But depending on how this package manager installer would work, it could be the right place to store this "install on execution" logic. For example, let's say that (very basic proposal I just came up with, refine it at will):

• Node ships binaries named yarn and pnpm (it could apply to npm as well, but let's ignore that for a minute as there's political issues as well)

• Those binaries would just contain bootstrappers similar to the following

  find closest .pm-version from cwd
  if there is none
      abort and print an error requesting to select one with "node --pm-init <name>"
  
  read the package manager and version from the .pm-version
  if the .pm-version references another package manager
      abort and print an error requesting to use the other package manager
  
  check if the specified version is installed inside $HOME/.node/pms/<name>/<version>.js
  if it isn't
      abort and print an error requesting that the user runs "node --pm-install"
  otherwise
      run the binary we found
  

• And finally, it would require Node to ship a node --pm-init <name> mode which would download and store locally a given version of the package manager and would write the .pm-version file.

By doing something like this, we would solve many problems at once:

• People using zero-install wouldn't have to run npm install -g yarn just to have a Yarn binary into their global PATH (binary that they wouldn't use anyway)

• Runtime performances for people using zero-install would be significantly improved, because we wouldn't have to parse a 6MB file just to shell out to another project.

• We would be able to prevent mistakes when one project is used instead of another, leading to different installs and subpar developer experience (especially when one project uses features unique to the package manager it's been created for).

[andersk]

While we’re waiting for such a .pm-version mechanism to become widely available (which could take years), is it worth trying to see how close we can come with only Yarn changes? This would help users who are interested in adopting Yarn 2 now, and would also provide experience to help build a case for .pm-version and inform its design.

Although doable, I would prefer to avoid downloading binaries from remote sources on commands that aren't supposed to have anything to do with the network (such as yarn run, for example).

Similarly to your above proposal, when there’s a configured Yarn version that hasn’t been downloaded yet, yarn run could print an error telling you to first yarn download-yarn (or yarn install, if we’re comfortable with that downloading Yarn).

Otherwise, projects using Yarn may end up being unusable or vulnerable should it die someday (I don't find that likely to happen anytime soon, but I always try to take the bus factor into account).

It could download Yarn from the registry—and if the registry is hit by a bus, even a vendored Yarn probably isn’t going to be too useful. 🙂 In any case, this would of course be an opt-in mechanism for projects willing to accept the risk of relying on external dependencies.

[arcanis]

Right, but I'm not sure Yarn core is the right layer to implement this (similar to how Node doesn't implement nvm / fnm / nodenv / etc). The system I highlighted can be implemented on top of it.

and if the registry is hit by a bus, even a vendored Yarn probably isn’t going to be too useful

It does. For example, none of my companies will bat an eye when the registry will die (or even regularly goes down, like it did twice these past weeks), because we don't rely on it for our deployments. Something something offline mirrors / check-in your cache 🙂

[andersk]

It might not be technically ideal, but the reason I think this belongs in Yarn core is that, from the perspective of a project considering adopting Yarn 2, it’d be a lot more palatable to ask developers to upgrade their global yarn to 1.23 than to

• wait years for a yarn wrapper to make its way into Node, if that ever even happens, or
• ask developers to replace their global yarn with a weird third-party yarn wrapper, or
• ask developers to upgrade their global yarn to 2.x (breaking compatibility with Yarn 1 projects), or
• commit a vendored yarn into the repository (even though it is likely bigger than the rest of the repository, ruins git grep, and can’t realistically be audited).

I understand that some projects will be comfortable vendoring yarn, and they can keep doing that. But for others, it’s not a good option.

Node users can get by without nvm and friends in the typical case because there’s a reasonably smooth migration path between Node versions—if you install an arbitrary version as the global node and upgrade it occasionally, you’ll probably be happy. The Yarn 1 → 2 upgrade isn’t like that and isn’t intended to be, so another way to bridge that gap is needed.

[arcanis]

I conceptually agree but I personally don't have time to invest into developing this tool at the moment, so that's something that someone else will have to take ownership of.

I'd encourage you to go ahead and build a prototype of the specs I outlined outside of the Yarn repository (something that can be quickly installed with npm i -g @myusername/pm, for example). Then once there's a better idea of the problems and solutions we can decide whether we want to publish it as an official package or not.

[merceyz]

For the CLI you can use corepack https://github.com/nodejs/corepack. Uninstall yarn, run npm install corepack -g, and add "packageManager": "yarn@2.3.3" to your package.json and corepack will handle downloading and running the correct version

[larixer]

I have written the tool that works almost the way the topic starter @aduh95 described, it computes and stores URL of the Yarn 2 bundle and URLs of official plugins in a JSON file .pinyarn.json, it also writes a small (less than 4Kb) wrapper .pinyarn.js which you can easily review. The wrapper is responsible for downloading Yarn bundle and Yarn plugin if they have not been downloaded yet, because the wrapper is very small it does not affect Yarn 2 boot time:
https://github.com/sysgears/pinyarn

[merceyz]

Now that Node.js ships with Corepack and Yarn 4 will prefer it over yarnPath if possible, do you consider this issue resolved @aduh95?

[aduh95]

Yep, I’ve been using Corepack for some time now, and it does fulfill my use case. Hopefully rest of the ecosystem will adopt it swiftly too!

[andersk]

Note that corepack currently lacks the integrity hash protection for downloading Yarn that Yarn itself uses for downloading other packages, so a compromised proxy server could potentially tamper with the download. Until that’s addressed, be aware that you may not be getting the security guarantees you expect.

• Allow checking hashes for exact versions, and check them for default versions nodejs/corepack#37

(Edit: this is marked “closed”, but integrity is still not protected in typical usage; see nodejs/corepack#134 (comment).)

[MatthiasKunnen]

Plugins such as @yarnpkg/plugin-workspace-tools are still required to be checked in. Could not requiring this be considered?

[arcanis]

It's already the case in 4.0, currently RC. All core plugins are distributed by default.

[MatthiasKunnen]

I cannot find any info regarding a v4 release candidate in the releases. Is there a planned release date?

That being said, I would also like to avoid checking in third party plugins.

[merceyz]

It's one of the pinned issues #3591.

That being said, I would also like to avoid checking in third party plugins.

v4 fetches missing plugins automatically #4912.

[MatthiasKunnen]

I'm aware of pinned issue #3591, however I was not able to find an estimated release date. Did I miss it?

[merceyz]

Ah, I misunderstood you comment; no planned release / whenever it's ready.
There are canary builds available if you want to use those

yarn set version canary

[Philzen]

Idea

Instead of committing the raw yarn executable code into my repository, i'd prefer a canonical way, e.g. adding a git submodule and linking to the yarn executable directly.

Advantages

• Avoid unnecessary repository growth (b/c updates are cjs, amount of reusable / compressable code available to git is limited, thus the repository grows faster with every yarn update)
• Refer to a single source of truth, namely https://github.com/yarnpkg/berry instead of copying your project's compiled code into my own
• Clearer Separation Of Concerns (git is for code, not for compiled stuff, just like it is good practice not to check an NPM packages' dist-folder into main)

Now i'd only need a repository that i could link to that contains the .cjs release versions of yarn... Is there something like that out in the wild that anyone would know of or that the yarn team would be up to create?