[BUG] Incompatible .env file behavior

[zlondrej]

Description

Before 2.27.1, .env file was automatically loaded from the project directory. However in 2.27.1, the .env file is loaded only from the working directory and the project directory's .env file is ignore regardless whether there's .env in the working directory.

This affects both --project-directory and --file options and it seems that --env-file is also ignored.

Steps To Reproduce

Behavior can be easily verified using docker compose config command, but affect other commands such as docker compose up.

Let's assume scenario with following project structure:

.   #  current working directory with no .env file
├── sub
│   ├── docker-compose.yaml
│   └── .env
└── test-environment -> sub/.env

# file: sub/.env
COMPOSE_PROFILES=sub

# file: sub/docker-compose.yaml
services:
  dummy:
    image: placeholder/dummy
    profiles:
      - sub

---

2.27.0 and prior at least down to 2.24.5

All 3 of the commands below resulted in the same output and all these commands correctly load the sub profile.

docker compose --project-directory sub config
docker compose --file sub/docker-compose.yaml config
(cd sub && docker compose config)

name: sub
services:
  dummy:
    profiles:
      - sub
    image: placeholder/dummy
    networks:
      default: null
networks:
  default:
    name: sub_default

2.27.1

When docker compose specified either --project-directory sub or --file sub/docker-compose.yaml, the ./.env file was loaded instead of ./sub/.env.

So as expected, command that uses cd still behaves the same way.

(cd sub && docker compose config)

name: sub
services:
  dummy:
    profiles:
      - sub
    image: placeholder/dummy
    networks:
      default: null
networks:
  default:
    name: sub_default

However 2 of the commands below resulted in the different output compared to prior versions as no .env profile is loaded and COMPOSE_PROFILES is not set.

docker compose --project-directory sub config
docker compose --file sub/docker-compose.yaml config

name: sub
services: {}

This then results in behavior such as getting no service selected when using docker compose --project-directory <dir> up. Command which used to work in past suddenly stopped working.

What makes it even worse is that --env-file should have a priority over implicit default, it seems to be ignored regardless of file's location.

docker compose --env-file sub/.env --project-directory sub config
docker compose --env-file test-environment --project-directory sub config

name: sub
services: {}

Compose Version

Docker Compose version 2.27.1

Docker Environment

Client:
 Version:    26.1.3
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  0.14.1
    Path:     /usr/lib/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  2.27.1
    Path:     /usr/lib/docker/cli-plugins/docker-compose

Server:
 Containers: 26
  Running: 1
  Paused: 0
  Stopped: 25
 Images: 233
 Server Version: 26.1.3
 Storage Driver: overlay2
  Backing Filesystem: f2fs
  Supports d_type: true
  Using metacopy: true
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3a4de459a68952ffb703bbe7f2290861a75b6b67.m
 runc version: 
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.10.218-1-MANJARO
 Operating System: Manjaro Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 31.02GiB
 Name: ****************
 ID: COJZ:2U6G:EJAO:3ZCJ:V6SG:FHDV:UQTV:O663:37BW:DH4L:WYU6:ZT5K
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Anything else?

At first glance, this seems to be a result of allow a local .env file to override compose.yaml sibling .env PR.

Note that <project_directory>/.env doesn't have lower priority than $PWD/.env (as that one doesn't exist), but it's entirely ignored instead.

[ndeloof]

This indeed is feature introduced by #11861, which existed (while undocumented) in earlier releases - documentation update will come soon, the whole environment section is being rewritten

I tried to reproduce, but works as expected:

$ cd test
$ cat compose.yaml 
services:
  test:
    image: ${IMAGE}
$ cat .env 
IMAGE=alpine
$ cd ..
$ cat .env
IMAGE=nginx
$ docker compose -f test/compose.yaml config
name: test
services:
  test:
    image: nginx

so I get the $PWD/.env to override values set by /.env

[ndeloof]

ok, got it. The issue is about COMPOSE_PROFILES which is only loaded from $PWD/.env

[zlondrej]

Re-tested and that indeed seems to be the case. All variables are loaded correctly from both files, with $PWD/.env having priority over sub/.env and --env-file working correctly.

I even added variables into the config for debugging:

  ...
  environment:
    - COMPOSE_PROFILES=${COMPOSE_PROFILES}

and COMPOSE_PROFILES has expected value but is not being applied correctly as services with matching profile are filtered out.

[zlondrej]

This issue might be actually a duplicate of #11856, which already has a fix.

[mantiz]

I might have a similar (same?) issue but with COMPOSE_FILE and not COMPOSE_PROFILES.
I use env-files to define the variable COMPOSE_FILE so I can switch compose files using --env-file.
Of course, I could specify multiple files using -f but this gets verbose and besides COMPOSE_FILE I also set other variable like COMPOSE_PROJECT_NAME which might also be affected.

A downgrade to version 2.27.0 works like before, so I think this may be the same issue, maybe all COMPOSE_* variables are effected. However, the referenced bugfix seems to solve the problem for COMPOSE_PROFILES only. 🤔