[BUG] Code coverage is failing to upload on dependabot PRs

[dblock]

What is the bug?

info - 2024-04-15 07:50:46,699 -- ci service found: github-actions
Error: Codecov token not found. Please provide Codecov token with -t flag.

How can one reproduce the bug?

See PRs, e.g. #753 (review)
https://github.com/opensearch-project/opensearch-js/actions/runs/8685364111/job/23814712885?pr=753

What is the expected behavior?

Coverage to run and upload.

[nhtruong]

For context:

This was the last time the action completed successfully: https://github.com/opensearch-project/opensearch-js/actions/runs/8635756121/job/23674229038

Run codecov/codecov-action@v4
==> linux OS detected
https://cli.codecov.io/latest/linux/codecov.SHA256SUM
==> Running version latest
gpg: directory '/home/runner/.gnupg' created
gpg: keybox '/home/runner/.gnupg/pubring.kbx' created
gpg: /home/runner/.gnupg/trustdb.gpg: trustdb created
gpg: key 806BB28AED779869: public key "Codecov Uploader (Codecov Uploader Verification Key) <security@codecov.io>" imported
gpg: Total number processed: 1
gpg:               imported: 1

gpg: Signature made Wed Apr 10 17:26:31 2024 UTC
gpg:                using RSA key 27034E7FDB850E0BBC2C62FF806BB28AED779869
gpg: Good signature from "Codecov Uploader (Codecov Uploader Verification Key) <security@codecov.io>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 2703 4E7F DB85 0E0B BC2C  62FF 806B B28A ED77 9869

==> Uploader SHASUM verified (2c9804ba263ba6880d67a671c35388a72fbfe014bd814acd212d8ff38644bf91  codecov)
==> Running version v0.5.0
==> Running git config --global --add safe.directory /home/runner/work/opensearch-js/opensearch-js
/usr/bin/git config --global --add safe.directory /home/runner/work/opensearch-js/opensearch-js
==> Running command '/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov create-commit'
/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov create-commit --git-service github -C 34fed7a15188c6db4c98e3b0a4397fbfb4152c32 -Z
info - 2024-04-10 18:03:20,834 -- ci service found: github-actions
info - 2024-04-10 18:03:21,275 -- The PR is happening in a forked repo. Using tokenless upload.
info - 2024-04-10 18:03:22,057 -- Process Commit creating complete
==> Running command '/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov create-report'
/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov create-report --git-service github -C 34fed7a15188c6db4c98e3b0a4397fbfb4152c32 -Z
info - 2024-04-10 18:03:22,791 -- ci service found: github-actions
info - 2024-04-10 18:03:23,787 -- Process Report creating complete
info - 2024-04-10 18:03:23,787 -- Finished creating report successfully --- {"response": "{\"external_id\":\"c364d3fd-31b8-491a-9c22-050a4a190dfd\",\"created_at\":\"2024-04-10T18:03:23.730922Z\",\"commit_sha\":\"34fed7a15188c6db4c98e3b0a4397fbfb4152c32\",\"code\":null}"}
==> Running command '/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov do-upload'
/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov do-upload -Z -f ./coverage.lcov --git-service github -C 34fed7a15188c6db4c98e3b0a4397fbfb4152c32
info - 2024-04-10 18:03:24,510 -- ci service found: github-actions
warning - 2024-04-10 18:03:24,520 -- xcrun is not installed or can't be found.
warning - 2024-04-10 18:03:24,655 -- No gcov data found.
warning - 2024-04-10 18:03:24,655 -- coverage.py is not installed or can't be found.
info - 2024-04-10 18:03:24,798 -- Found 2 coverage files to report
info - 2024-04-10 18:03:24,798 -- > /home/runner/work/opensearch-js/opensearch-js/.github/workflows/coverage.yml
info - 2024-04-10 18:03:24,798 -- > /home/runner/work/opensearch-js/opensearch-js/coverage.lcov
info - 2024-04-10 18:03:25,843 -- Your upload is now processing. When finished, results will be available at: https://app.codecov.io/github/opensearch-project/opensearch-js/commit/34fed7a15188c6db4c98e3b0a4397fbfb4152c32
info - 2024-04-10 18:03:26,166 -- Process Upload complete

We recently upgraded to cocodecov-action from v2 to v4 because of a different error: https://github.com/opensearch-project/opensearch-js/actions/runs/8635655002/job/23674115308

Run codecov/codecov-action@v2
==> linux OS detected
https://uploader.codecov.io/latest/linux/codecov.SHA256SUM
==> SHASUM file signed by key id 806bb28aed779869
==> Uploader SHASUM verified (a66af85a0b56b2179d568067cf951e5b40c440b7cbffc305b6b08bcf9eb17dd8  codecov)
==> Running version latest
==> Running version v0.7.2
/home/runner/work/_actions/codecov/codecov-action/v2/dist/codecov -n  -Q github-action-2.1.0 -Z -f ./coverage.lcov -C d7d16b2a9223ac5bbeea469488a5f399c6a2115b
[2024-04-10T18:00:40.915Z] ['info'] 
     _____          _
    / ____|        | |
   | |     ___   __| | ___  ___ _____   __
   | |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
   | |___| (_) | (_| |  __/ (_| (_) \ V /
    \_____\___/ \__,_|\___|\___\___/ \_/

  Codecov report uploader 0.7.2
[2024-04-10T18:00:40.923Z] ['info'] => Project root located at: /home/runner/work/opensearch-js/opensearch-js
[2024-04-10T18:00:40.926Z] ['info'] -> No token specified or token is empty
[2024-04-10T18:00:40.934Z] ['info'] Searching for coverage files...
[2024-04-10T18:00:40.970Z] ['info'] => Found 1 possible coverage files:
  ./coverage.lcov
[2024-04-10T18:00:40.970Z] ['info'] Processing ./coverage.lcov...
[2024-04-10T18:00:40.974Z] ['info'] Detected GitHub Actions as the CI provider.
[2024-04-10T18:00:41.187Z] ['info'] Pinging Codecov: https://codecov.io/upload/v4?package=github-action-2.1.0-uploader-0.7.2&token=*******&branch=version_bump%2F2.7.0&build=8635655002&build_url=https%3A%2F%2Fgithub.51.al%2Fopensearch-project%2Fopensearch-js%2Factions%2Fruns%2F8635655002&commit=d7d16b2a9223ac5bbeea469488a5f399c6a2115b&job=Coverage&pr=749&service=github-actions&slug=opensearch-project%2Fopensearch-js&name=&tag=&flags=&parent=
[2024-04-10T18:00:41.351Z] ['error'] There was an error running the uploader: Error uploading to [https://codecov.io:](https://codecov.io/) Error: There was an error fetching the storage URL during POST: 404 - {'detail': ErrorDetail(string='Unable to locate build via Github Actions API. Please upload with the Codecov repository upload token to resolve issue.', code='not_found')}
Error: Codecov: Failed to properly upload: The process '/home/runner/work/_actions/codecov/codecov-action/v2/dist/codecov' failed with exit code 255

[nhtruong]

Both errors (v2 and v4) pointed to missing token. Most likely related to the breaking changes on codecov

According to this FAQ, a token is required for our use case, and it can be found at https://app.codecov.io/github/opensearch-project/opensearch-js/settings. However I don't have access to that page or the github settings page of this repo to update the workflow with a token.

Going to try OIDC first.

[dblock]

How did we configure it in other repos?

[nhtruong]

Dashboards is using codecov-action@v3

[nhtruong]

Both v3 and OIDC failed

[nhtruong]

It doesn't work for dashboards either: https://github.com/opensearch-project/OpenSearch-Dashboards/actions/runs/8693296132/job/23839779862
However their workflow does not have fail_ci_if_error: true like us. I can remove that option in our workflow unless you want add a token and keep v4?

[dblock]

However their workflow does not have fail_ci_if_error: true like us. I can remove that option in our workflow unless you want add a token and keep v4?

But that will hide the failure? :)

We want code coverage don't we, or am I missing something?

[nhtruong]

This step is only uploading codecoverage to codecov.com to track code-coverage overtime. The steps that runs code coverage and assure that the coverage clears a certain threshold still pass.

[dblock]

So uploading for PRs doesn't and shouldn't work? We should turn that off explicitly rather than ignoring failures because it will hide future real failures quietly.

[nhtruong]

It used to work. It's a nice-to-have-feature. If we have the time we should go through the trouble of adding the token for it. I think for now we can just remove it like you said.