Project can not be accessed or deleted via Portal (WebUI) or API

[srvsstieber]

Hi,

i want to report the following issue: one particular habor project can not be accessed or deleted via Portal or API

• when you click in browser on the project, you'll receive the following message in the browsers dev-tools

  {"errors":[{"code":"UNKNOWN","message":"internal server error"}]}
  

• Harbor Version is Version v2.5.4-a39bd2bc
• the requested URL is like: https://<redacted-company-url>/api/v2.0/projects/1101
• the project contains images, OCI artifacts and helm charts, which are stored in chartmuseum
• when we try to get the project infos via API with this method: GET /projects the result is:

  {
    "chart_count": 0,
    "creation_time": "2022-05-09T12:58:07.885Z",
    "current_user_role_id": 1,
    "current_user_role_ids": [
      1
    ],
    "cve_allowlist": {
      "creation_time": "0001-01-01T00:00:00.000Z",
      "id": 74,
      "items": [],
      "project_id": 1101,
      "update_time": "0001-01-01T00:00:00.000Z"
    },
    "metadata": {
      "auto_scan": "true",
      "enable_content_trust": "false",
      "prevent_vul": "false",
      "public": "false",
      "reuse_sys_cve_allowlist": "true",
      "severity": "low"
    },
    "name": "<redacted-project-name>",
    "owner_id": 1,
    "owner_name": "admin",
    "project_id": 1101,
    "repo_count": 2,
    "update_time": "2022-05-09T12:58:07.885Z"
  }

• the chart count is 0 but there is one helm chart with ~300 version in this project
• the repo-count is correct
• at the same time we try to access the project the following log entries are in Harbor Core

2023-02-08T13:14:34Z [ERROR] [/server/v2.0/handler/project.go:505][requestID="bd3b2187-6d8e-49ef-be78-5b422a0a8a1e"]: failed to populate properties for project <redacted>, error: get chart count of project 2564 failed: http error: code 500, message
2023-02-08T13:15:40Z [ERROR] [/server/v2.0/handler/project.go:505][requestID="06efb455-74b9-4af2-90ac-fd6f5efef83a"]: failed to populate properties for project <redacted>, error: get chart count of project 2564 failed: http error: code 500, message
2023-02-08T13:15:45Z [ERROR] [/server/v2.0/handler/project.go:505][requestID="d44fe523-ebc9-40a8-ac54-aac5cca23685"]: failed to populate properties for project <redacted>, error: get chart count of project 2564 failed: http error: code 500, message
2023-02-08T13:36:03Z [ERROR] [/server/v2.0/handler/project.go:505][requestID="abf6b3c6-51ae-40e8-a776-e7870abc5456"]: failed to populate properties for project <redacted>, error: get chart count of project 2564 failed: http error: code 500, message
2023-02-08T13:43:24Z [ERROR] [/server/middleware/security/oidc_cli.go:62][requestID="5c854f41-4ad3-4795-ae97-6dcdeb4f5f38"]: failed to verify secret, username: css_thcr_robot_<redacted>+Trivy-8df50045-a7b6-11ed-ba6d-ae3038d0e59f, error: failed to get oidc user info, error: <QuerySeter> no row found
2023-02-08T13:43:24Z [INFO] [/pkg/notifier/notifier.go:205]: Handle notification with Handler 'InternalArtifact' on topic 'PULL_ARTIFACT': ID-18850, Repository-<redacted>/<redacted> Tags-[] Digest-sha256:2734c9006d744296255604eca94e310165f05cde027d8cc75dd75f45b1650e57 Operator-css_thcr_robot_<redacted>+Trivy-8df50045-a7b6-11ed-ba6d-ae3038d0e59f OccurAt-2023-02-08 13:43:24
2023-02-08T13:43:24Z [INFO] [/pkg/notifier/notifier.go:205]: Handle notification with Handler 'AuditLog' on topic 'PULL_ARTIFACT': ID-18850, Repository-<redacted>/<redacted> Tags-[] Digest-sha256:2734c9006d744296255604eca94e310165f05cde027d8cc75dd75f45b1650e57 Operator-css_thcr_robot_<redacted>+Trivy-8df50045-a7b6-11ed-ba6d-ae3038d0e59f OccurAt-2023-02-08 13:43:24
2023-02-08T13:43:24Z [INFO] [/pkg/notifier/notifier.go:205]: Handle notification with Handler 'ArtifactWebhook' on topic 'PULL_ARTIFACT': ID-18850, Repository-<redacted>/<redacted> Tags-[] Digest-sha256:2734c9006d744296255604eca94e310165f05cde027d8cc75dd75f45b1650e57 Operator-css_thcr_robot_<redacted>+Trivy-8df50045-a7b6-11ed-ba6d-ae3038d0e59f OccurAt-2023-02-08 13:43:24
2023-02-08T13:43:35Z [INFO] [/pkg/notifier/notifier.go:205]: Handle notification with Handler 'ScanWebhook' on topic 'SCANNING_COMPLETED': Artifact-&{NamespaceID:2564 Repository:<redacted>/<redacted> Tag:1.8.5-2ca53808-production-master Digest:sha256:2734c9006d744296255604eca94e310165f05cde027d8cc75dd75f45b1650e57 MimeType:application/vnd.docker.distribution.manifest.v2+json} Operator-auto OccurAt-2023-02-08 13:43:35
2023-02-08T13:43:35Z [INFO] [/pkg/notifier/notifier.go:205]: Handle notification with Handler 'P2PPreheat' on topic 'SCANNING_COMPLETED': Artifact-&{NamespaceID:2564 Repository:<redacted>/<redacted> Tag:1.8.5-2ca53808-production-master Digest:sha256:2734c9006d744296255604eca94e310165f05cde027d8cc75dd75f45b1650e57 MimeType:application/vnd.docker.distribution.manifest.v2+json} Operator-auto OccurAt-2023-02-08 13:43:35
2023-02-08T13:44:01Z [ERROR] [/server/v2.0/handler/project.go:505][requestID="7ba1a837-6401-4163-9c38-573188fcadc4"]: failed to populate properties for project <redacted>, error: get chart count of project 2564 failed: http error: code 500, message

• the log in chartmuseum shows this error:

2023/02/08 14:48:36 [Recovery] 2023/02/08 - 14:48:36 panic recovered:
runtime error: invalid memory address or nil pointer dereference
/usr/local/go/src/runtime/panic.go:221 (0xf16795)
/usr/local/go/src/runtime/signal_unix.go:735 (0xf16765)
/go/src_code/pkg/chartmuseum/server/multitenant/index.go:43 (0x26f7fe1)
/go/src_code/pkg/chartmuseum/server/multitenant/handlers.go:100 (0x26f4357)
/go/src_code/pkg/chartmuseum/router/router.go:249 (0x1c606fa)
/go/pkg/mod/github.com/gin-gonic/gin@v1.7.7/context.go:168 (0x1c5cf14)
/go/pkg/mod/github.com/zsais/go-gin-prometheus@v0.1.0/middleware.go:364 (0x1c5cef5)
/go/pkg/mod/github.com/gin-gonic/gin@v1.7.7/context.go:168 (0x1c11d29)
/go/pkg/mod/github.com/gin-contrib/size@v0.0.0-20220102055520-f75bacbc2df3/size.go:88 (0x1c11d11)
/go/pkg/mod/github.com/gin-gonic/gin@v1.7.7/context.go:168 (0x1c5eac9)
/go/src_code/pkg/chartmuseum/router/middleware.go:48 (0x1c5eaa6)
/go/pkg/mod/github.com/gin-gonic/gin@v1.7.7/context.go:168 (0x1bf2181)
/go/pkg/mod/github.com/gin-gonic/gin@v1.7.7/recovery.go:99 (0x1bf216c)
/go/pkg/mod/github.com/gin-gonic/gin@v1.7.7/context.go:168 (0x1bf082c)
/go/pkg/mod/github.com/gin-gonic/gin@v1.7.7/gin.go:591 (0x1bf0815)
/go/pkg/mod/github.com/gin-gonic/gin@v1.7.7/gin.go:584 (0x1bf0564)
/go/pkg/mod/github.com/gin-gonic/gin@v1.7.7/gin.go:511 (0x1befef1)
/usr/local/go/src/net/http/server.go:2879 (0x11bdd5a)
/usr/local/go/src/net/http/server.go:1930 (0x11b9387)
/usr/local/go/src/runtime/asm_amd64.s:1581 (0xf331e0)

• when we try to delete this project via API with the method: DELETE: /projects/{projectID}

{
  "errors": [
    {
      "code": "UNKNOWN",
      "message": "internal server error"
    }
  ]
}

• next step would be to clean the redis cache, but with this we are not so familiar, if we search through the redis cli keys *chart* we get this result:

1) "helm_chart_cache:a1e87e77ca94253063f61d9827e8f6d2b8769bb03488c2fa96686a344891af83"
2) "helm_chart_cache:5c5ffcd4a19377f9e2eeb40fd8c0a2a7a9f851db74453247c357c83e1775c330"
3) "helm_chart_cache:ad3a8c414f0e5af8f06bcf4030c1a3a58f7073f63627a6dbda6c47e5a77705c0"
4) "helm_chart_cache:e60a35b9a616eccb2fbb7c7e3a6c5b0e8178971ee9129841d1b0b92d8c8287d4"
5) "helm_chart_cache:891a16d9172e885e38af85821ad227ba55a43977cf0640fdbb0c03cfd4d60380"
6) "helm_chart_cache:e2b8f9ec991809a55e28a872b2cb5a6df4c9616d1a6650fe7564706614b28e1d"
7) "helm_chart_cache:b19a8af7f34e7bd9eeaea9c425cf1272efe2edac0f073e1cd42690be6dd463ad"

• and when we search for keys *chart*, we get:

1177) "cache:trimmedmanifestlist:sha256:1bc7e3426c72dabdfbaf6e8d63f774ca5c26640d7c1921302cb43a60e20e94d4"1178) "cache:trimmedmanifestlist:sha256:769422529210357f359e7e5620246028837f84fabe92c838ca2aef75fe2e0741"1179) "cache:trimmedmanifestlist:sha256:088b19c260d977637a5ab9501f8112cad5d558ca07b06f028e25dc7496d60fb2"1180) "cache:manifestlist:ghcr/kedacore/keda:sha256:b75a859a594cea7248784da534aee6f7e3bd13a0fd90d09ceaf130e90c629772"1181) "cache:trimmedmanifestlist:sha256:f4cd32e7a418d9c9ba043e7d561243388202b654c740bcc85ca40b41d9fb4f1e"1182) "cache:trimmedmanifestlist:sha256:bedef0f1d248508fe0a16d2cacea1d2e68e899b2220e2258f1b604e1f327d475"1183) "cache:trimmedmanifestlist:sha256:c5ad98f1fae0b6dfd23a13b17a7b0d97beac8944a0f9f1d676b9ac362351cea5"1184) "cache:trimmedmanifestlist:sha256:e35696a0a313a1797c05dc741a8fb22be5ab833dab50d07aede68c184811b225"1185) "cache:manifestlist:dockerhub/istio/proxyv2:sha256:f6f97fa4fb77a3cbe1e3eca0fa46bd462ad6b284c129cf57bf91575c4fb50cf9:contenttype"1186) "cache:trimmedmanifestlist:sha256:c345c783d8fbbee4edeca2ebc47e7c454fcc2e26686820b8037de97369280ba0"1187) "cache:trimmedmanifestlist:sha256:d6f5c8d70e750117f38ce84253f7677373514c355ca6cbae0a4278e4461497ef"1188) "cache:trimmedmanifestlist:sha256:9b9bad11be62d5acfc5f816bba854423a58b209e55eb2127b9827df71dfb9de7"1189) "cache:trimmedmanifestlist:sha256:a861ee2ce3693ef85bbf0f96e715dde6f3fbd1546333d348993cc123a00a0290"1190) "cache:trimmedmanifestlist:sha256:78c0331e57f0d9fb2c0ba108d89334cc0b616843b675eaccafc85719b37e225b"1191) "cache:trimmedmanifestlist:sha256:c48cf8c493930d6b5fbada793144b177113fefeda5397e99173938c59933285d"1192) "cache:manifestlist:dockerhub/library/node:sha256:ada99ae6ef09a78bd1782e3c3b58567d5a41e695c72d44b8d31994fabbe55072:contenttype"1193) "cache:trimmedmanifestlist:sha256:35ab2bf57814e9ff49e365efd5a5935b6915eede5c7f8581e9e1b85e0eecbe16"1194) "cache:trimmedmanifestlist:sha256:09a36e2be1dbda6009641235d90be8627c9616d42a0b887b770fc92c1753b74a"1195) "cache:trimmedmanifestlist:sha256:1214aee00e17b2f62228071c1f74c5ebc7a081e4aff1a314e13e225304f0afbf"1196) "cache:trimmedmanifestlist:sha256:cb9817249c346d6cfadebe383ed3b3cd4c540f623db40c4ca00da2ada45259bb"1197) "cache:trimmedmanifestlist:sha256:fd4d9a3dd1fd0125108417be21be917f19cc76262347086509a0d43f29b80e98"1198) "cache:trimmedmanifestlist:sha256:dba1787bc2bdf835a2831dc6b7ae397d3fe9d4619170040f5570b2a19205e1a9"1199) "cache:trimmedmanifestlist:sha256:f06a1c7aece9f6fd82f9f5ff0c6e5d395da93a3a54ed46dd0399510f42c9bbee"1200) "cache:trimmedmanifestlist:sha256:cef3a11de97d6aeb94e2d8efdbeecffc3565a6de7ffaf234ee1a2acad559cbcb"1201) "cache:manifestlist:dockerhub/moby/buildkit:sha256:46241014ed9b1a09ec05ed01116621ac01af97d9950777ca281f42ea058934fe:contenttype"

Question: which one of them is to clean up? Both? Thank you for your support.

[stonezdj]

Because chartmuseum is deprecated in Harbor, suggest to reconfig Harbor without the --with-chartmuseum option, for example:

./prepare --with-trivy --with-notary

Restart Harbor and try again.

[srvsstieber]

Because chartmuseum is deprecated in Harbor, suggest to reconfig Harbor without the --with-chartmuseum option, for example:

./prepare --with-trivy --with-notary

Restart Harbor and try again.

Thanks for the reply. Since many of our internal teams still using this component (especially for production deployments) , we can not easily shut it down. Bu we are preparing a migration plan. Until then, it would be great if you can help us to delete the "broken" project. There was one suggestion, to empty the redis cache. Can you say, which one one the db entries we can / should delete, in oder to check if we can then delete the project?

😄

[github-actions]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[doublest]

not stale

[github-actions]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[github-actions]

This issue was closed because it has been stalled for 30 days with no activity. If this issue is still relevant, please re-open a new issue.