Role "skynet-quay-database" does not exist

[djjudas21]

I'm new to Quay but I've followed the "batteries-included, zero-config" instructions in the README on my OKD cluster. It's unable to start up because of a problem with Postgres that causes it to crashloop:

Unhealthy deployment:

[jgazeley@hephaestos quay-operator]$ oc get po
NAME                                         READY     STATUS                  RESTARTS   AGE
quay-operator-tng-68fd4fcbc8-r46cd           1/1       Running                 0          19m
skynet-quay-app-upgrade-6944487677-7tcbw     0/1       CrashLoopBackOff        8          18m
skynet-quay-config-editor-596bf4dbdc-rmdp6   1/1       Running                 0          18m
skynet-quay-database-6cfd8cb54-sshfr         0/1       Running                 2          4m23s
skynet-quay-mirror-58567f9b6-59h4v           0/1       Init:CrashLoopBackOff   5          17m
skynet-quay-postgres-init-jjbpl              0/1       Completed               0          18m
skynet-quay-postgres-init-q2p8w              0/1       Completed               0          18m
skynet-quay-redis-bfcb7995b-lqz5d            1/1       Running                 0          18m

Postgres having problems:

[jgazeley@hephaestos quay-operator]$ oc logs skynet-quay-database-6cfd8cb54-sshfr
pg_ctl: another server might be running; trying to start server anyway
waiting for server to start....2021-02-19 13:33:25.395 UTC [25] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2021-02-19 13:33:25.435 UTC [25] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2021-02-19 13:33:26.086 UTC [25] LOG:  redirecting log output to logging collector process
2021-02-19 13:33:26.086 UTC [25] HINT:  Future log output will appear in directory "log".
........................................................................ done
server started
/var/run/postgresql:5432 - accepting connections
=> sourcing /usr/share/container-scripts/postgresql/start/set_passwords.sh ...
ERROR:  role "skynet-quay-database" does not exist

And of course, stuff that depends on Postgres is broken too:

[jgazeley@hephaestos quay-operator]$ oc logs skynet-quay-app-upgrade-6944487677-7tcbw
   __   __
  /  \ /  \     ______   _    _     __   __   __
 / /\ / /\ \   /  __  \ | |  | |   /  \  \ \ / /
/ /  / /  \ \  | |  | | | |  | |  / /\ \  \   /
\ \  \ \  / /  | |__| | | |__| | / ____ \  | |
 \ \/ \ \/ /   \_  ___/  \____/ /_/    \_\ |_|
  \__/ \__/      \ \__
                  \___\ by Red Hat
 Build, Store, and Distribute your Containers
Running all default registry services
Running init script '/quay-registry/conf/init/certs_create.sh'
Generating a RSA private key
..........................................................................................................++++
...........................................................................................................................................................................++++
writing new private key to 'mitm-key.pem'
-----
Running init script '/quay-registry/conf/init/certs_install.sh'
Installing extra certificates found in /quay-registry/conf/stack/extra_ca_certs directory
Running init script '/quay-registry/conf/init/copy_config_files.sh'
Running init script '/quay-registry/conf/init/d_validate_config_bundle.sh'
Validating Configuration
+------------------------+--------------------------------------------------------------------------------+--------+
|      Field Group       |                                     Error                                      | Status |
+------------------------+--------------------------------------------------------------------------------+--------+
| AccessSettings         | -                                                                              | 🟢     |
+------------------------+--------------------------------------------------------------------------------+--------+
| ActionLogArchiving     | -                                                                              | 🟢     |
+------------------------+--------------------------------------------------------------------------------+--------+
| AppTokenAuthentication | -                                                                              | 🟢     |
+------------------------+--------------------------------------------------------------------------------+--------+
| BitbucketBuildTrigger  | -                                                                              | 🟢     |
+------------------------+--------------------------------------------------------------------------------+--------+
| BuildManager           | -                                                                              | 🟢     |
+------------------------+--------------------------------------------------------------------------------+--------+
| Database               | Could not connect to database. Error: dial tcp 172.30.26.212:5432: i/o timeout | 🔴     |
+------------------------+--------------------------------------------------------------------------------+--------+
| DistributedStorage     | -                                                                              | 🟢     |
+------------------------+--------------------------------------------------------------------------------+--------+
| ElasticSearch          | -                                                                              | 🟢     |
+------------------------+--------------------------------------------------------------------------------+--------+
| Email                  | -                                                                              | 🟢     |
+------------------------+--------------------------------------------------------------------------------+--------+
| GitHubBuildTrigger     | -                                                                              | 🟢     |
+------------------------+--------------------------------------------------------------------------------+--------+
| GitHubLogin            | -                                                                              | 🟢     |
+------------------------+--------------------------------------------------------------------------------+--------+
| GitLabBuildTrigger     | -                                                                              | 🟢     |
+------------------------+--------------------------------------------------------------------------------+--------+
| GoogleLogin            | -                                                                              | 🟢     |
+------------------------+--------------------------------------------------------------------------------+--------+
| HostSettings           | -                                                                              | 🟢     |
+------------------------+--------------------------------------------------------------------------------+--------+
| JWTAuthentication      | -                                                                              | 🟢     |
+------------------------+--------------------------------------------------------------------------------+--------+
| LDAP                   | -                                                                              | 🟢     |
+------------------------+--------------------------------------------------------------------------------+--------+
| OIDC                   | -                                                                              | 🟢     |
+------------------------+--------------------------------------------------------------------------------+--------+
| QuayDocumentation      | -                                                                              | 🟢     |
+------------------------+--------------------------------------------------------------------------------+--------+
| Redis                  | -                                                                              | 🟢     |
+------------------------+--------------------------------------------------------------------------------+--------+
| RepoMirror             | -                                                                              | 🟢     |
+------------------------+--------------------------------------------------------------------------------+--------+
| SecurityScanner        | -                                                                              | 🟢     |
+------------------------+--------------------------------------------------------------------------------+--------+
| SigningEngine          | -                                                                              | 🟢     |
+------------------------+--------------------------------------------------------------------------------+--------+
| TeamSyncing            | -                                                                              | 🟢     |
+------------------------+--------------------------------------------------------------------------------+--------+
| TimeMachine            | -                                                                              | 🟢     |
+------------------------+--------------------------------------------------------------------------------+--------+
| UserVisibleSettings    | -                                                                              | 🟢     |
+------------------------+--------------------------------------------------------------------------------+--------+

I'm running this operator from master at commit e2cdb6d. Happy to gather any other diagnostics that would be helpful. Thanks.

[djjudas21]

Also logged as https://issues.redhat.com/browse/PROJQUAY-1664

[alecmerdler]

Thanks for finding this, I'm looking into it now.

[alecmerdler]

@djjudas21 We've merged #427, please let us know if you can still reproduce the issue.

[flavianmissi]

I have tested a similar setup on latest master and could not reproduce the problem, so I'm closing this issue.