manylinux2014 - yum install gives 502 : Bad Gateway or times out

[vegardjervell]

Description

I'm using cibuildwheel with Github Action's ubuntu-latest runner, and trying to install some dependencies with yum in the CIBW_BEFORE_ALL_LINUX step.

After pulling in the fix in #1915 (using pypa/cibuildwheel@v2.19.2) I'm getting the error

https://vault.centos.org/centos/7/os/x86_64/repodata/repomd.xml: [Errno 14] HTTPS Error 502 - Bad Gateway

It's not completely clear to me if this is an issue with vault.centos.org or an issue with cibuildwheel, but I thought I would bring attention to it in case it is in fact an issue with cibuildwheel, or if there is some known workaround.

Build log

https://github.com/vegardjervell/thermopack/actions/runs/10161895317/job/28101395655

CI config

https://github.com/vegardjervell/thermopack/actions/runs/10161895317/workflow

[theAeon]

Something's up with CentOS's origin. Given this is IBM-Red Hat we're talking about whether or not this is intentional or a mistake? Yet to be seen.

slot in some sed commands to change it to https://archive.kernel.org/centos-vault/ for a temporary workaround

looks like it came back up just as I posted this 😅

[vegardjervell]

Wow... just my luck there. I haven't set this up before, so I've spent my day believing this was just me having no idea what I was doing...

Thanks anyway, I guess I'll take solace in that I've learned a bit about manylinux containers and cibuildwheels on the way :)

[vegardjervell]

Ok, it seems to be back up, but now the install is timing out. This doesn't appear to be an issue with cibuildwheel but unless you are opposed, I'll leave it up for now so that other people having the same issue can find it more easily.

[henryiii]

Maybe the failure in pypa/manylinux#1652 is related?

[vegardjervell]

@henryiii Yes, that looks like the exact same error as I was initially seeing, It appeared to originate from an error on vault.centos.org's side and seems to be fixed now. But if you re-run the workflows, I'm assuming you will see a timeout on the downloads there, which is what I'm getting now.

[mayeut]

The updates of the manylinux images passed so I guess this is fixed ?

[Parskatt]

This is broken for me.

      + /opt/python/cp38-cp38/bin/python -c 'import sys, json, os; json.dump(os.environ.copy(), sys.stdout)'
      + sh -c 'yum install -y eigen3-devel'
  Loaded plugins: fastestmirror, ovl
  Determining fastest mirrors
  Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=container error was
  14: curl#6 - "Could not resolve host: mirrorlist.centos.org; Unknown error"
  
  
   One of the configured repositories failed (Unknown),
   and yum doesn't have enough cached data to continue. At this point the only
   safe thing yum can do is fail. There are a few ways to work "fix" this:
  
       1. Contact the upstream for the repository and get them to fix the problem.
  
       2. Reconfigure the baseurl/etc. for the repository, to point to a working
          upstream. This is most often useful if you are using a newer
          distribution release than is supported by the repository (and the
          packages for the previous distribution release still work).
  
       3. Run the command with the repository temporarily disabled
              yum --disablerepo=<repoid> ...
  
       4. Disable the repository permanently, so yum won't use it by default. Yum
          will then just ignore the repository until you permanently enable it
          again or use --enablerepo for temporary usage:
  
              yum-config-manager --disable <repoid>
          or
              subscription-manager repos --disable=<repoid>
  
       5. Configure the failing repository to be skipped, if it is unavailable.
          Note that yum will try to contact the repo. when it runs most commands,
          so will have to try and fail each time (and thus. yum will be be much
          slower). If it is a very temporary problem though, this is often a nice
          compromise:
  
              yum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true
  
  Cannot find a valid baseurl for repo: base/7/x86_64
                                                              ✕ 1.06s

[henryiii]

@Parskatt what cibuildwheel version?

[vegardjervell]

@Parskatt I believe updating to pypa/cibuildwheel@v2.19.2 should fix that, see #1915. However (@mayeut) I tried re-running this workflow and I'm still getting a timeout issue when trying to yum install lapack. Maybe the issue is on my side?

[Parskatt]

Ah, sorry. I see now that mine was pypa/cibuildwheel@v2.18.0. Thanks.

[vegardjervell]

@Parskatt No problem :) please let me know if you're also seeing the timeout issue. I'm getting

+ /opt/python/cp38-cp38/bin/python -c 'import sys, json, os; json.dump(os.environ.copy(), sys.stdout)'
      + sh -c 'yum install lapack'
  Loaded plugins: ovl
  Resolving Dependencies
  --> Running transaction check
  ---> Package lapack.x86_64 0:3.4.2-8.el7 will be installed
  --> Processing Dependency: libgfortran.so.3(GFORTRAN_1.4)(64bit) for package: lapack-3.4.2-8.el7.x86_64
  --> Processing Dependency: libgfortran.so.3(GFORTRAN_1.0)(64bit) for package: lapack-3.4.2-8.el7.x86_64
  --> Processing Dependency: libgfortran.so.3()(64bit) for package: lapack-3.4.2-8.el7.x86_64
  --> Processing Dependency: libblas.so.3()(64bit) for package: lapack-3.4.2-8.el7.x86_64
  --> Running transaction check
  ---> Package blas.x86_64 0:3.4.2-8.el7 will be installed
  ---> Package libgfortran.x86_64 0:4.8.5-44.el7 will be installed
  --> Finished Dependency Resolution
  
  Dependencies Resolved
  
  ================================================================================
   Package              Arch            Version               Repository     Size
  ================================================================================
  Installing:
   lapack               x86_64          3.4.2-8.el7           base          5.4 M
  Installing for dependencies:
   blas                 x86_64          3.4.2-8.el7           base          399 k
   libgfortran          x86_64          4.8.5-44.el7          base          301 k
  
  Transaction Summary
  ================================================================================
  Install  1 Package (+2 Dependent packages)
  
  Total download size: 6.1 M
  Installed size: 12 M
  Error: The operation was canceled.

where the runner seems to hang until timeout at the Installed size: 12 M line before cancelling.

[mayeut]

@vegardjervell,

Can you try the suggestion in that comment to see if that helps with timeouts: #1948 (comment)

For reference, if it helps, the sed commands already used when building the image are in
https://github.com/pypa/manylinux/blob/main/docker/build_scripts/fixup-mirrors.sh

[vegardjervell]

... This was a skill issue. I had managed to use yum install lapack instead of yum install -y lapack, which caused the runner to time out while waiting for install confirmation. To be clear: The docs specify to use yum install -y <package>, but if anything it might be helpful if the docpage hinted that you would get a timeout if you don't use the -y flag.

As far as I can tell this is resolved, thanks for all the response and support :)

[Parskatt]

My issues also seems resolved on 2.19.2