Server returns empty response

[schklom]

I spun this up with Docker, and cannot do anything with it. curl returns this

$ curl http://localhost:3000
curl: (52) Empty reply from server

I kept the docker-compose.yaml you did (and renamed the containers), set PUID and PGID to 0.
My docker-compose.yaml includes

services:
  kaizoku:
    image: ghcr.io/oae/kaizoku
    container_name: kaizoku
    hostname: kaizoku
    environment:
      DATABASE_URL: "postgresql://kaizoku:kaizoku@kaizoku-db:5432/kaizoku"
      KAIZOKU_PORT: 3000
      REDIS_HOST: kaizoku-redis
      REDIS_PORT: 6379
      PUID: 0
      PGID: 0
      TZ: Europe/Istanbul
    volumes:
      - ${DOCKER}/kaizoku/data:/data
      - ${DOCKER}/kaizoku/config:/config
      - ${DOCKER}/kaizoku/logs:/logs
    depends_on:
      kaizoku-db:
        condition: service_healthy
      kaizoku-redis:
        condition: service_started
    ports:
      - 3000:3000
    networks:
      kaizoku:
  
  kaizoku-redis:
    image: redis:7-alpine
    container_name: kaizoku-redis
    hostname: kaizoku-redis
    volumes:
      - ${DOCKER}/kaizoku-redis:/data
    networks:
      kaizoku:
      
  kaizoku-db:
    image: postgres:alpine
    container_name: kaizoku-db
    hostname: kaizoku-db
    healthcheck:
      test: ['CMD-SHELL', 'pg_isready -U kaizoku']
      interval: 5s
      timeout: 5s
      retries: 5
    environment:
      POSTGRES_USER: kaizoku
      POSTGRES_DB: kaizoku
      POSTGRES_PASSWORD: kaizoku
    volumes:
      - ${DOCKER}/kaizoku-db:/var/lib/postgresql/data
    networks:
      kaizoku:

networks:
  kaizoku:
    ipam:
      config:
        - subnet: 192.168.21.0/24

My logs show nothing wrong

$ docker logs kaizoku
[custom-init] No custom services found, skipping...
[migrations] started
[migrations] no migrations found
cont-init: info: running /etc/cont-init.d/40-config

-------------------------------------
          _         ()
         | |  ___   _    __
         | | / __| | |  /  \
         | | \__ \ | | | () |
         |_| |___/ |_|  \__/


Brought to you by linuxserver.io
-------------------------------------

To support LSIO projects visit:
https://www.linuxserver.io/donate/
-------------------------------------
GID/UID
-------------------------------------

User uid:    0
User gid:    0
-------------------------------------

yarn run v1.22.19
$ /app/node_modules/.bin/prisma migrate deploy
Prisma schema loaded from prisma/schema.prisma
Datasource "db": PostgreSQL database "kaizoku", schema "public" at "kaizoku-db:5432"

1 migration found in prisma/migrations


No pending migrations to apply.
Done in 2.45s

Any idea what I'm doing wrong?

[oae]

I tried your docker-compose file and I can access the web UI without any errors. Can you curl within the container?

[Vodes]

I may or may not have a related issue.

All containers listed here are in the same docker network.
I can ping the others from within the kaizoku one and vice versa too. Just when I try to curl (or open via swag, reverse proxy) kaizoku I get a 502.

Aside from adding the network and changing the container_names everywhere I'm using pretty much the default compose.

Edit:
nevermind. Seems like it was still loading something? I can now access it just fine

[tschuehly]

@Vodes Yes it seems to take a long time at startup without any logs indicating that it is doing something.

[oae]

Can you check what it does with the ps faux command inside the container?

[Vodes]

Issue still present on 1.2.1

[oae]

The container tries to run chown on the mounted directories before startup. Can you check the permissions of the /config, /logs, and /data mounts?

[Vodes]

Actually, after restarting my host system everything seems to work fine now? I really don't get what was going on there, but before that I had to wait quite a bit of time after every restart of the container.

[Vodes]

Seems like it's still ongoing.

Images

I'm not sure if its the cause or anything but is it even necessary to recursively chown the entire data folder?

Also unsure if related or not, but manually downloading something with mangal from inside the container takes forever (like a few minutes to actually start, similarily to the container itself, lol) and I keep getting failed downloads in kaizoku randomly.

Like these

Might be worth noting that these also get delayed for a long time. That chapter was out for almost 3 days by the time it was actually downloaded. (Kaizoku is set to check every 3 hours for this)

docker-compose and host perms

(alex is 1000 and dockusers is the 1111 group)

version: '3'

services:
kaizoku:
  container_name: kaizoku
  image: ghcr.io/oae/kaizoku:latest
  environment:
    - DATABASE_URL=postgresql://kaizoku:kaizoku@kaizoku_db:5432/kaizoku
    - KAIZOKU_PORT=3000
    - REDIS_HOST=kaizoku_redis
    - REDIS_PORT=6379
    - PUID=1000
    - PGID=1111
    - TZ=Europe/Berlin
  volumes:
    - /var/media/Manga/Ongoing:/data
    - ./data/kaizoku:/config
    - ./data/log:/logs
  depends_on:
    - kaizoku_db
    - kaizoku_redis
  networks:
    - default
kaizoku_redis:
  container_name: kaizoku_redis
  image: redis:7-alpine
  networks:
    - default
  volumes:
    - ./data/redis:/data
kaizoku_db:
  container_name: kaizoku_db
  image: postgres:alpine
  restart: unless-stopped
  environment:
    - POSTGRES_USER=kaizoku
    - POSTGRES_DB=kaizoku
    - POSTGRES_PASSWORD=kaizoku
  networks:
    - default
  volumes:
    - ./data/postgres:/var/lib/postgresql/data

networks:
  default:
      name: swagnet
      external: true

[vodkapmp]

Might be worth noting that these also get delayed for a long time. That chapter was out for almost 3 days by the time it was actually downloaded. (Kaizoku is set to check every 3 hours for this)

I've also been seeing this issue a lot lately, seems to have randomly showed up for me over the last week or so

[schklom]

@oae I had some time to test things out. I realized that the database throws errors. I have deleted it and rebuilt it a few times, always the same problem. I have tried on mad64 and arm64, same problem.

Any ideas of what is going on?

$ docker logs kaizoku-db
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... UTC
creating configuration files ... ok
running bootstrap script ... ok
sh: locale: not found
2022-12-10 16:19:37.675 UTC [31] WARNING:  no usable system locales were found
performing post-bootstrap initialization ... ok
syncing data to disk ... ok


Success. You can now start the database server using:

    pg_ctl -D /var/lib/postgresql/data -l logfile start

initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
waiting for server to start....2022-12-10 16:19:38.228 UTC [37] LOG:  starting PostgreSQL 15.1 on aarch64-unknown-linux-musl, compiled by gcc (Alpine 12.2.1_git20220924-r4) 12.2.1 20220924, 64-bit
2022-12-10 16:19:38.231 UTC [37] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-12-10 16:19:38.237 UTC [40] LOG:  database system was shut down at 2022-12-10 16:19:38 UTC
2022-12-10 16:19:38.242 UTC [37] LOG:  database system is ready to accept connections
 done
server started
CREATE DATABASE


/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*

waiting for server to shut down....2022-12-10 16:19:38.381 UTC [37] LOG:  received fast shutdown request
2022-12-10 16:19:38.383 UTC [37] LOG:  aborting any active transactions
2022-12-10 16:19:38.385 UTC [37] LOG:  background worker "logical replication launcher" (PID 43) exited with exit code 1
2022-12-10 16:19:38.385 UTC [38] LOG:  shutting down
2022-12-10 16:19:38.387 UTC [38] LOG:  checkpoint starting: shutdown immediate
2022-12-10 16:19:38.417 UTC [38] LOG:  checkpoint complete: wrote 918 buffers (5.6%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.018 s, sync=0.005 s, total=0.032 s; sync files=250, longest=0.003 s, average=0.001 s; distance=4222 kB, estimate=4222 kB
2022-12-10 16:19:38.424 UTC [37] LOG:  database system is shut down
 done
server stopped

PostgreSQL init process complete; ready for start up.

2022-12-10 16:19:38.501 UTC [1] LOG:  starting PostgreSQL 15.1 on aarch64-unknown-linux-musl, compiled by gcc (Alpine 12.2.1_git20220924-r4) 12.2.1 20220924, 64-bit
2022-12-10 16:19:38.501 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2022-12-10 16:19:38.501 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2022-12-10 16:19:38.505 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-12-10 16:19:38.511 UTC [52] LOG:  database system was shut down at 2022-12-10 16:19:38 UTC
2022-12-10 16:19:38.516 UTC [1] LOG:  database system is ready to accept connections
2022-12-10 16:19:42.277 UTC [62] FATAL:  role "kaizoku" does not exist
2022-12-10 16:19:47.329 UTC [71] FATAL:  role "kaizoku" does not exist
2022-12-10 16:19:52.385 UTC [78] FATAL:  role "kaizoku" does not exist
2022-12-10 16:19:57.462 UTC [85] FATAL:  role "kaizoku" does not exist
2022-12-10 16:20:02.520 UTC [92] FATAL:  role "kaizoku" does not exist
2022-12-10 16:20:07.598 UTC [99] FATAL:  role "kaizoku" does not exist

[oae]

This may happen if you have changed the PostgreSQL user and kept the older container volume. Other than that nothing comes to mind.

[schklom]

@oae To avoid any problems caused by me, I copy-pasted your docker-compose.yaml from README.md and removed+purged all volumes so that any old data would not be used. The containers are healthy, no problems during creation. However, I still can't

docker-compose.yaml

services:
  app:
    container_name: kaizoku
    image: ghcr.io/oae/kaizoku:latest
    environment:
      - DATABASE_URL=postgresql://kaizoku:kaizoku@db:5432/kaizoku
      - KAIZOKU_PORT=3000
      - REDIS_HOST=redis
      - REDIS_PORT=6379
      - PUID=1001
      - PGID=1001
      - TZ=Europe/Istanbul
    depends_on:
      db:
        condition: service_healthy
    ports:
      - '3000:3000'
  redis:
    image: redis:7-alpine
  db:
    image: postgres:alpine
    restart: unless-stopped
    healthcheck:
      test: ['CMD-SHELL', 'pg_isready -U kaizoku']
      interval: 5s
      timeout: 5s
      retries: 5
    environment:
      - POSTGRES_USER=kaizoku
      - POSTGRES_DB=kaizoku
      - POSTGRES_PASSWORD=kaizoku

I start the containers

$ docker compose pull -q app db redis # get the latest images
$ docker volume prune -f # remove old volumes
$ docker compose up -d app redis
[+] Running 3/3
 ⠿ Network docker_default          Created                                                                                                   0.2s
 ⠿ Container docker_db_1           Healthy                                                                                                  13.6s
 ⠿ Container kaizoku               Started                                                                                                  13.6s
 ⠿ Container docker_redis_1  Started                                                                                                         3.2s

and kaizoku log is now

$ docker compose logs app

$ docker compose logs app
kaizoku  | [custom-init] No custom services found, skipping...
kaizoku  | [migrations] started
kaizoku  | [migrations] no migrations found
kaizoku  | cont-init: info: running /etc/cont-init.d/40-config
kaizoku  | 
kaizoku  | -------------------------------------
kaizoku  |           _         ()
kaizoku  |          | |  ___   _    __
kaizoku  |          | | / __| | |  /  \
kaizoku  |          | | \__ \ | | | () |
kaizoku  |          |_| |___/ |_|  \__/
kaizoku  | 
kaizoku  | 
kaizoku  | Brought to you by linuxserver.io
kaizoku  | -------------------------------------
kaizoku  | 
kaizoku  | To support LSIO projects visit:
kaizoku  | https://www.linuxserver.io/donate/
kaizoku  | -------------------------------------
kaizoku  | GID/UID
kaizoku  | -------------------------------------
kaizoku  | 
kaizoku  | User uid:    1001
kaizoku  | User gid:    1001
kaizoku  | -------------------------------------
kaizoku  | 
kaizoku  | Prisma schema loaded from prisma/schema.prisma
kaizoku  | Datasource "db": PostgreSQL database "kaizoku", schema "public" at "db:5432"
kaizoku  | 
kaizoku  | 4 migrations found in prisma/migrations
kaizoku  | 
kaizoku  | Applying migration `20221015234159_init`
kaizoku  | Applying migration `20221027101522_fix_intervals`
kaizoku  | Applying migration `20221103223653_add_settings_schema`
kaizoku  | Applying migration `20221124200902_add_kavita`
kaizoku  | 
kaizoku  | The following migrations have been applied:
kaizoku  | 
kaizoku  | migrations/
kaizoku  |   └─ 20221015234159_init/
kaizoku  |     └─ migration.sql
kaizoku  |   └─ 20221027101522_fix_intervals/
kaizoku  |     └─ migration.sql
kaizoku  |   └─ 20221103223653_add_settings_schema/
kaizoku  |     └─ migration.sql
kaizoku  |   └─ 20221124200902_add_kavita/
kaizoku  |     └─ migration.sql
kaizoku  |       
kaizoku  | All migrations have been successfully applied.

and the db logs

$ docker compose logs db

$ docker compose logs db
db_1  | The files belonging to this database system will be owned by user "postgres".
db_1  | This user must also own the server process.
db_1  | 
db_1  | The database cluster will be initialized with locale "en_US.utf8".
db_1  | The default database encoding has accordingly been set to "UTF8".
db_1  | The default text search configuration will be set to "english".
db_1  | 
db_1  | Data page checksums are disabled.
db_1  | 
db_1  | fixing permissions on existing directory /var/lib/postgresql/data ... ok
db_1  | creating subdirectories ... ok
db_1  | selecting dynamic shared memory implementation ... posix
db_1  | selecting default max_connections ... 100
db_1  | selecting default shared_buffers ... 128MB
db_1  | selecting default time zone ... UTC
db_1  | creating configuration files ... ok
db_1  | running bootstrap script ... ok
db_1  | sh: locale: not found
db_1  | 2022-12-10 17:58:14.021 UTC [30] WARNING:  no usable system locales were found
db_1  | performing post-bootstrap initialization ... ok
db_1  | initdb: warning: enabling "trust" authentication for local connections
db_1  | initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
db_1  | syncing data to disk ... ok
db_1  | 
db_1  | 
db_1  | Success. You can now start the database server using:
db_1  | 
db_1  |     pg_ctl -D /var/lib/postgresql/data -l logfile start
db_1  | 
db_1  | waiting for server to start....2022-12-10 17:58:18.057 UTC [36] LOG:  starting PostgreSQL 15.1 on x86_64-pc-linux-musl, compiled by gcc (Alpine 12.2.1_git20220924-r4) 12.2.1 20220924, 64-bit
db_1  | 2022-12-10 17:58:18.089 UTC [36] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1  | 2022-12-10 17:58:18.191 UTC [45] LOG:  database system was shut down at 2022-12-10 17:58:14 UTC
db_1  | 2022-12-10 17:58:18.191 UTC [46] FATAL:  the database system is starting up
db_1  | 2022-12-10 17:58:18.227 UTC [36] LOG:  database system is ready to accept connections
db_1  |  done
db_1  | server started
db_1  | CREATE DATABASE
db_1  | 
db_1  | 
db_1  | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
db_1  | 
db_1  | waiting for server to shut down....2022-12-10 17:58:18.506 UTC [36] LOG:  received fast shutdown request
db_1  | 2022-12-10 17:58:18.536 UTC [36] LOG:  aborting any active transactions
db_1  | 2022-12-10 17:58:18.538 UTC [36] LOG:  background worker "logical replication launcher" (PID 49) exited with exit code 1
db_1  | 2022-12-10 17:58:18.538 UTC [43] LOG:  shutting down
db_1  | 2022-12-10 17:58:18.570 UTC [43] LOG:  checkpoint starting: shutdown immediate
db_1  | 2022-12-10 17:58:19.024 UTC [43] LOG:  checkpoint complete: wrote 918 buffers (5.6%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.070 s, sync=0.273 s, total=0.486 s; sync files=250, longest=0.199 s, average=0.002 s; distance=4222 kB, estimate=4222 kB
db_1  | 2022-12-10 17:58:19.032 UTC [36] LOG:  database system is shut down
db_1  |  done
db_1  | server stopped
db_1  | 
db_1  | PostgreSQL init process complete; ready for start up.
db_1  | 
db_1  | 2022-12-10 17:58:19.206 UTC [1] LOG:  starting PostgreSQL 15.1 on x86_64-pc-linux-musl, compiled by gcc (Alpine 12.2.1_git20220924-r4) 12.2.1 20220924, 64-bit
db_1  | 2022-12-10 17:58:19.206 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
db_1  | 2022-12-10 17:58:19.206 UTC [1] LOG:  listening on IPv6 address "::", port 5432
db_1  | 2022-12-10 17:58:19.276 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1  | 2022-12-10 17:58:19.378 UTC [58] LOG:  database system was shut down at 2022-12-10 17:58:18 UTC
db_1  | 2022-12-10 17:58:19.413 UTC [1] LOG:  database system is ready to accept connections
db_1  | 2022-12-10 17:58:27.301 UTC [69] LOG:  could not receive data from client: Connection reset by peer

Trying to curl from inside the container

$ docker compose exec -it app bash
root@b544ada76bcb:/app# curl http://localhost:3000
curl: (7) Failed to connect to localhost port 3000 after 3 ms: Connection refused

and from the host

$ curl http://localhost:3000
curl: (52) Empty reply from server

Any idea now? :P

[oae]

From the logs, I see that kaizoku didn't start yet. Can you check the ps faux inside the container

[schklom]

@oae Of course!

$ docker compose exec -it app bash
root@b544ada76bcb:/app# ps faux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         244  0.5  0.0   7632  4008 pts/0    Ss   21:17   0:00 bash
root         251  0.0  0.0  10040  3268 pts/0    R+   21:18   0:00  \_ ps faux
root           1  0.0  0.0    208    64 ?        Ss   20:58   0:00 /package/admin/s6/command/s6-svscan -d4 -- /run/service
root          14  0.0  0.0   2888  1036 ?        Ss   20:58   0:00 /bin/sh -e /run/s6/basedir/scripts/rc.init top
root          41  0.0  0.0    216    72 ?        S    20:58   0:00  \_ s6-rc -v1 -u -t 0 -- change top
root          73  0.0  0.0    196    48 ?        S    20:58   0:00      \_ /package/admin/s6-2.11.1.2/command/s6-sudoc -e -t 30000 -T 0 -- up 17
root          15  0.0  0.0    212    64 ?        S    20:58   0:00 s6-supervise s6-linux-init-shutdownd
root          16  0.0  0.0    200     4 ?        Ss   20:58   0:00  \_ /package/admin/s6-linux-init/command/s6-linux-init-shutdownd -c /run/s6/bas
root          36  0.0  0.0    212    64 ?        S    20:58   0:00 s6-supervise s6rc-fdholder
root          37  0.0  0.0    212    60 ?        S    20:58   0:00 s6-supervise s6rc-oneshot-runner
root          45  0.0  0.0    188     4 ?        Ss   20:58   0:00  \_ /package/admin/s6/command/s6-ipcserverd -1 -- /package/admin/s6/command/s6-
root          74  0.0  0.0    208    60 ?        S    20:58   0:00      \_ /package/admin/s6/command/s6-sudod -t 30000 -- /package/admin/s6-rc/com
root          77  0.0  0.0   2888   956 ?        S    20:58   0:00          \_ /bin/sh -e /package/admin/s6-overlay-3.1.2.1/etc/s6-rc/scripts/cont
root          84  0.0  0.0   7368  3284 ?        S    20:58   0:00              \_ bash /etc/cont-init.d/40-config
root         230  0.6  0.0   6128  2500 ?        D    20:58   0:07                  \_ chown -R abc:abc /app /config /logs /data

I wasn't sure who the user abc was, so I checked and it has id 1001 as specified in the docker-compose.yaml.

[oae]

abc user is using ids from the host. So before starting the server it tries chown to prevent permission issues. After that, the server will start.

kaizoku/docker/root/etc/cont-init.d/40-config

Lines 12 to 16 in d4786d2

	chown -R abc:abc \
	/app \
	/config \
	/logs \
	/data

[schklom]

Thanks for the info.
Does ps faux help identify the problem?

[oae]

This seems to be a docker bug. docker/for-linux#388

[schklom]

Do you think it would it change anything to run chown in the Dockerfile instead of in a script?
https://docs.docker.com/engine/reference/builder/#copy

For example,

COPY --from=builder --chown=${PUID:-1001}:${PGID:-1001} /app/next.config.mjs ./
COPY --from=builder --chown=${PUID:-1001}:${PGID:-1001} /app/public ./public
COPY --from=builder --chown=${PUID:-1001}:${PGID:-1001} /app/package.json ./package.json
COPY --from=builder --chown=${PUID:-1001}:${PGID:-1001} /app/.next ./.next
COPY --from=builder --chown=${PUID:-1001}:${PGID:-1001} /app/prisma ./prisma
COPY --from=builder --chown=${PUID:-1001}:${PGID:-1001} /app/dist ./dist
COPY --from=builder --chown=${PUID:-1001}:${PGID:-1001} /app/node_modules ./node_modules

COPY --chown=${PUID:-1001}:${PGID:-1001} docker/mangal /mangal
COPY --chown=${PUID:-1001}:${PGID:-1001} docker/root/ /

instead of

COPY --from=builder /app/next.config.mjs ./
COPY --from=builder /app/public ./public
COPY --from=builder /app/package.json ./package.json
COPY --from=builder /app/.next ./.next
COPY --from=builder /app/prisma ./prisma
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules

COPY docker/mangal /mangal
COPY docker/root/ /

and then

RUN chown --recursive ${PUID:-1001}:${PGID:-1001} \ 
 	/config \ 
 	/logs \ 
 	/data 

for the rest.

My thinking is that it would take not take much time on the user machine, instead it would be already specified in the image during compilation.

[oae]

This doesn't work because PUID and PGID are only relevant to runtime. Building with hosts' PUID, and PGID will make the image unusable for different people.

[schklom]

Thank you so much, I will test it when the image is released! :)

[oae]

Feel free to open this issue again if the problem persists.