Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes settings interpolation with dotenv variables #5544

Merged
merged 1 commit into from
Jun 29, 2023

Conversation

arcanis
Copy link
Member

@arcanis arcanis commented Jun 29, 2023

What's the problem this PR addresses?

I forgot in #4718 to add a test making sure that the yarnrc settings could use the variable set in the dotenv files (the existing tests were mainly around the values set during script execution), and of course I forgot a change 🥲

How did you fix it?

We now pass the updated environment when interpolating values.

Checklist

  • I have set the packages that need to be released for my changes to be effective.
  • I will check that all automated PR checks pass before the PR gets reviewed.

@arcanis arcanis merged commit 603fe11 into master Jun 29, 2023
15 checks passed
@arcanis arcanis deleted the mael/dotenv-interpolate-settings branch June 29, 2023 15:29
@belgattitude
Copy link

belgattitude commented Jun 29, 2023

@arcanis thanks for the new .env support.

Upgrading to rc.47 from rc.46 CI failed in a monorepo. I'm guessing env loaded by yarn leaks into the app/package (process.env) taking precedence in the way nextjs loads my .env file (exposed in global process.env).

I might totally be wrong but preferred to ask you.

PS:

The example here is with ./apps/nextjs-app and the ./apps/nextjs-app/.env.

PR: belgattitude/nextjs-monorepo-example#3971
CI: https://github.com/belgattitude/nextjs-monorepo-example/actions/runs/5411183973/jobs/9833607130?pr=3971

nextjs-app:build: - info Loaded env from /home/runner/work/nextjs-monorepo-example/nextjs-monorepo-example/apps/nextjs-app/.env
nextjs-app:build: error - Invalid server env(s): NEXTAUTH_SECRET
nextjs-app:build: {
nextjs-app:build:   "_errors": [],
nextjs-app:build:   "NEXTAUTH_SECRET": {  // <- IT LOOKS LIKE THAT ENV WAS LOADED into process.env
nextjs-app:build:     "_errors": [
nextjs-app:build:       "String must contain at least 15 character(s)"
nextjs-app:build:     ]
nextjs-app:build:   }
nextjs-app:build: }
nextjs-app:build: ERROR: command finished with error: command (/home/runner/work/nextjs-monorepo-example/nextjs-monorepo-example/apps/nextjs-app) yarn run build exited (1)

I admit the .env contains a incorrect value (-> a comment). But it might show some leakage

NEXTAUTH_URL=http://localhost:3000
NEXTAUTH_SECRET= # Linux: `openssl rand -hex 32` or go to https://generate-secret.now.sh/32

@arcanis
Copy link
Member Author

arcanis commented Jun 29, 2023

That's the expected behaviour - .env (or anything you defined in injectEnvironmentFiles) is both interpolated in the settings, and injected into the scripts run by yarn run.

@belgattitude
Copy link

belgattitude commented Jun 29, 2023

Thanks @arcanis,

In my experience it might cause issues with framework that rely on multiple files convention for loading envs.

Cause global process.env is taking precedence.

I would recommend for safety to limit the env scope to internal yarn, otherwise I feel a lot of users might run into trouble.

@belgattitude belgattitude mentioned this pull request Jun 29, 2023
3 tasks
@arcanis
Copy link
Member Author

arcanis commented Jun 29, 2023

Those docs make me think it should be correct - .env will be read by Yarn and added to the env (ie process.env), so Next.js will use them like it would have before (it will skip the .env, but since it's already in the env it should be fine, right?). You had a problem because in your case you had a value that Next.js then rejected.

One thing I'm not 100% sure is why Next.js didn't reject your .env before, since it was invalid as well, being empty? Perhaps our implementation should skip empty values, rather than set them to an empty string? Or is it a Next.js bug?

@belgattitude
Copy link

belgattitude commented Jun 29, 2023

Yes indeed, there's 2 causes

  • the env reader behaviour isn't the same (the comment was taken as not-empty')
  • But the main point: process.env takes precedence. That short-circuits the reading of the next files. For example in nextjs .env.local will take precedence over .env (used to override the committed .env). If yarn loads .env into process.env that breaks the feature. Same idea for .env.development, env.staging.local, docker... (where you set the environment).

I like the idea of injectEnvironmentFiles: [] as a config... and it's possible to get something that could work with nextjs... (assuming the reading behaviour is the same - dotenv variables....).

Ultimately I don't see how it would work when dealing with multiple frameworks in a monorepo for example. frameworkX would use .env.(staging|production|test) convention, frameworkB another one (.env.docker...).

I'm in favour of making it opt-in, cause I feel there's a real risk in there

@exKAZUu
Copy link

exKAZUu commented Jun 30, 2023

@arcanis

Let's assume we have a .env file including X=1 and a .env.development file including X=2 and NODE_ENV is development.
With previous versions of Berry and Next.js, the .env.development file is loaded before the .env file.
As X is defined in .env.development at first and subsequent definitions are ignored, its eventual value remains as 2.

However, with the current Berry and Next.js, the process is different.
Here, the berry loads the .env before the .env.development.
This means that by the time .env.development is loaded, X is already defined by the .env file, hence its eventual value remains as 1.

This change of load order might conflict with the expectations of the existing ecosystem and potentially lead to unforeseen issues.

@arcanis
Copy link
Member Author

arcanis commented Jun 30, 2023

Yes, I can see that being annoying indeed 🤔 It'd be too bad to release such a nice feature and be met with negativity due to a harder migration path, even if technically a major is allowed to have breaking changes. In this case it's easy enough to make the change that I'll probably do it.

My current thinking is to change the injectEnvironmentFiles default from .env to .env.yarn. I'll make a fix later to this effect (along with a fix for #5545).

@arcanis arcanis mentioned this pull request Jun 30, 2023
3 tasks
arcanis added a commit that referenced this pull request Jun 30, 2023
**What's the problem this PR addresses?**

It was raised in
#5544 (comment) that
automatically injecting the `.env` file in the environment could break
compatibility with some software that already do this in their own way.
It makes sense, and I prefer to avoid doing that if there's a reasonable
compromise.

Also closes #5545.

**How did you fix it?**

Yarn will now default into loading the `.env.yarn` file, rather than
`.env`.

**Checklist**
<!--- Don't worry if you miss something, chores are automatically
tested. -->
<!--- This checklist exists to help you remember doing the chores when
you submit a PR. -->
<!--- Put an `x` in all the boxes that apply. -->
- [x] I have read the [Contributing
Guide](https://yarnpkg.com/advanced/contributing).

<!-- See
https://yarnpkg.com/advanced/contributing#preparing-your-pr-to-be-released
for more details. -->
<!-- Check with `yarn version check` and fix with `yarn version check
-i` -->
- [x] I have set the packages that need to be released for my changes to
be effective.

<!-- The "Testing chores" workflow validates that your PR follows our
guidelines. -->
<!-- If it doesn't pass, click on it to see details as to what your PR
might be missing. -->
- [x] I will check that all automated PR checks pass before the PR gets
reviewed.
@belgattitude
Copy link

Personally i really like the .env.yarn. wonderful idea. Thx @arcanis for the great you're doing.

arcanis added a commit to yarnpkg/example-repo-zipn that referenced this pull request Jul 3, 2023
**What's the problem this PR addresses?**

It was raised in
yarnpkg/berry#5544 (comment) that
automatically injecting the `.env` file in the environment could break
compatibility with some software that already do this in their own way.
It makes sense, and I prefer to avoid doing that if there's a reasonable
compromise.

Also closes yarnpkg/berry#5545.

**How did you fix it?**

Yarn will now default into loading the `.env.yarn` file, rather than
`.env`.

**Checklist**
<!--- Don't worry if you miss something, chores are automatically
tested. -->
<!--- This checklist exists to help you remember doing the chores when
you submit a PR. -->
<!--- Put an `x` in all the boxes that apply. -->
- [x] I have read the [Contributing
Guide](https://yarnpkg.com/advanced/contributing).

<!-- See
https://yarnpkg.com/advanced/contributing#preparing-your-pr-to-be-released
for more details. -->
<!-- Check with `yarn version check` and fix with `yarn version check
-i` -->
- [x] I have set the packages that need to be released for my changes to
be effective.

<!-- The "Testing chores" workflow validates that your PR follows our
guidelines. -->
<!-- If it doesn't pass, click on it to see details as to what your PR
might be missing. -->
- [x] I will check that all automated PR checks pass before the PR gets
reviewed.
@GollyJer
Copy link

Hi @arcanis. This worked great for resolving packages. Thanks! 💪

I'm curious if it's meant to also work for publishing? With yarn npm publish I get...

➤ YN0035:   Response Code: 404 (Not Found)
➤ YN0035:   Request Method: PUT
➤ YN0035:   Request URL: https://registry.yarnpkg.com/@awesomelabs%2feslint-plugin-awesomelabs

Even though I have yarnrc.yaml set up.

npmScopes:
  awesomelabs:
    npmRegistryServer: https://gitlab.com/api/v4/packages/npm/
    npmAuthToken: ${GITLAB_PACKAGE_REGISTRY_TOKEN}

Publishing doesn't look like it's using the yarnrc.yaml npmScopes.
Should it? Or is there a different way? Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants