[Feature] Lock file only updates (skip modules install)

[rarkins]

• I'd be willing to implement this feature (contributing guide)
• This feature is important to have in this repository; a contrib plugin wouldn't do

Describe the user story

As a Yarn user, I want to be able to update the Yarn lock file to match my package.json file(s) without needing to download all dependencies.

Describe the solution you'd like

The ideal solution would be a flag like --lock-file-only where Yarn resolves dependencies, updates yarn.lock accordingly, then exits before downloading modules.

However for Yarn v1 we had a one-line "hack" which is also perfectly sufficient: https://github.com/renovatebot/renovate/blob/22e982503c5d4bff4b0581a524ebf3e340a59648/lib/manager/npm/post-update/yarn.ts#L47

This hack tricked Yarn into exiting early and gracefully, and a similar trick would be fine for Yarn 2+.

Describe the drawbacks of your solution

The main drawback would be maintainability if the solution adds any non-trivial branches of logic.

Describe alternatives you've considered

A plugin solution would be OK but I'm not sure whether the hooks are yet "deep" enough to allow such a change to core functionality. If there were sufficient hooks, then a new command like yarn update-lockfile would be a great alternative.

[arcanis]

A plugin solution would be OK but I'm not sure whether the hooks are yet "deep" enough to allow such a change to core functionality. If there were sufficient hooks, then a new command like yarn update-lockfile would be a great alternative.

You likely would write a new command rather than change the behaviour of install, since they would then semantically do fairly different things. Fortunately it's not too difficult.

The "lockfile" part of the install is available in the Project instance by calling resolveEverything followed by persist (cf how a full install works; a few other things happen at the start, but you don't have to care about most of them). Something like this should be a workable start (using the 3.x command syntax):

module.exports = {
  name: `@yarnpkg/lockfile-only-install`,
  factory: req => {
    return {
      const {Configuration, Cache, Project, StreamReport} = require(`@yarnpkg/core`);
      const {Command} = require(`clipanion`);

      commands: [
        class LockfileOnlyInstall extends Command {
          static paths = [[`install-lockfile-only`]];

          async execute() {
            const configuration = await Configuration.find(this.context.cwd, this.context.plugins);
            const {project, workspace, locator} = await Project.find(configuration, this.context.cwd);
            const cache = await Cache.find(configuration);

            const report = await StreamReport.start({
              configuration,
              stdout: this.context.stdout,
            }, async report => {
              await project.resolveEverything({cache, report});
              await project.persist();
            });

            return report.exitCode();
          }
        },
      ],
    };
  },
};

[rarkins]

Thanks!

I realized that I overlooked one important point. Users who commit the Yarn cache need to do much more than just update the yarn.lock file. Would there still be any optimization worth doing in such a case, or may as well just run the regular yarn install command?

[arcanis]

For those cases (when the cache is checked inside the repo) then you could also call fetchEverything after the resolution. You'd however only skip the linking, and the linking is fairly fast if you do it with PnP (which you should do since you don't care about having an actual node_modules on disk anyway). I'd probably just use install in those cases.

[ylemkimon]

@arcanis I wonder whether it'd be possible to calculate/generate the checksum in the lockfile without fetching the package.

[arcanis]

Hmm - probably not, as the checksums are based on the zip archives, not the tgz the registries return. Unless the npm registry was to support zip packages (unlikely), fetching will always be needed to generate a full lockfile 🤔

(unless you cache it somewhere on the Renovate infra, that is)

[rarkins]

The challenge is:

• If you mean cache the zip files then we'd almost end up "caching the entire registry"
• We have a lot of VMs, so face the choice of each of them caching vs having a reliable shared cache.. which is not easy either
• It's easier today to cache only for the duration of a repository so that we don't need to worry about accidentally caching private/confidential packages or possible leaks

Would there be a deterministic mapping of tgz sha -> zip sha? I know this would be another difficult problem to solve but would it be theoretically possible to keep a DB of zip SHAs, mapped back to the registry's tgz SHAs?

[ylemkimon]

Another way would be to fetch only relevant packages that need to be updated, i.e., of which checksum doesn't exist. I think this can be implemented by skipping fetch if this.storedChecksums.has(pkg.locatorHash) and skipping the link step entirely.

[ylemkimon]

I've created a PR in #2913 and tested it via Renovate in https://github.com/ylemkimon/KaTeX/pulls. It successfully updates the checksum.