Jenkins using new python versions in the matrix #1174

weideng1 · 2023-08-23T22:30:43Z

No description provided.

…runner image

…rsions

…her python versions

… 3.10+ problem

test-requirements.txt

Jenkinsfile

absurdfarce · 2023-08-25T21:38:22Z

Problem reported by @weideng1 when trying to use the approach above (install ccm normally, then update PyYAML after the fact) with 3.11.x Python versions:

$ pip install "pyyaml<6.0"                                                                                                                   
Collecting pyyaml<6.0                                                                                                                                                                                                 
  Using cached PyYAML-5.4.1.tar.gz (175 kB)                                                                                                                                                                           
  Installing build dependencies ... done                                                                                                                                                                              
  Getting requirements to build wheel ... error                                                                                                                                                                       
  error: subprocess-exited-with-error                                                                                                                                                                                 
                                                                                                                                                                                                                      
  × Getting requirements to build wheel did not run successfully.                                                                                                                                                     
  │ exit code: 1                                                                                                                                                                                                      
  ╰─> [68 lines of output]                                                                                                                                                                                            
      /tmp/pip-build-env-sqpvlw_j/overlay/lib/python3.11/site-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config in `setup.cfg`                                                         
      !!                                                                                                                                                                                                              
                                                                                                                                                                                                                      
              ********************************************************************************                                                                                                                        
              The license_file parameter is deprecated, use license_files instead.                                                                                                                                    
                                                                                                                                                                                                                      
              By 2023-Oct-30, you need to update your project and remove deprecated calls                                                                                                                             
              or your builds will no longer be supported.                                                                                                                                                             
                                                                                                                                                                                                                      
              See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.                                                                                                                 
              ********************************************************************************                                                                                                                        
                                                                                                                                                                                                                      
      !!                                                                                                                                                                                                              
        parsed = self.parsers.get(option_name, lambda x: x)(value)                                                                                                                                                    
      running egg_info                                                                                                                                                                                                
      writing lib3/PyYAML.egg-info/PKG-INFO                                                                                                                                                                           
      writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt                                                                                                                                           
      writing top-level names to lib3/PyYAML.egg-info/top_level.txt                                                                                                                                                   
      Traceback (most recent call last):                                                                                                                                                                              
        File "/home/automaton/no-previous-cython-venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>                                                     
          main()                                                     
       File "/home/automaton/no-previous-cython-venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/automaton/no-previous-cython-venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-sqpvlw_j/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 355, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-sqpvlw_j/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 325, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-sqpvlw_j/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 341, in run_setup
          exec(code, locals())
        File "<string>", line 271, in <module>
        File "/tmp/pip-build-env-sqpvlw_j/overlay/lib/python3.11/site-packages/setuptools/__init__.py", line 107, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-sqpvlw_j/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-sqpvlw_j/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/tmp/pip-build-env-sqpvlw_j/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-sqpvlw_j/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 1233, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-sqpvlw_j/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-sqpvlw_j/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 319, in run
          self.find_sources()
        File "/tmp/pip-build-env-sqpvlw_j/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 327, in find_sources
          mm.run()
        File "/tmp/pip-build-env-sqpvlw_j/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 549, in run
          self.add_defaults()
        File "/tmp/pip-build-env-sqpvlw_j/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 587, in add_defaults
          sdist.add_defaults(self)
        File "/tmp/pip-build-env-sqpvlw_j/overlay/lib/python3.11/site-packages/setuptools/command/sdist.py", line 113, in add_defaults
          super().add_defaults()
        File "/tmp/pip-build-env-sqpvlw_j/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
          self._add_defaults_ext()
        File "/tmp/pip-build-env-sqpvlw_j/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext
          self.filelist.extend(build_ext.get_source_files())
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "<string>", line 201, in get_source_files
        File "/tmp/pip-build-env-sqpvlw_j/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
          raise AttributeError(attr)
      AttributeError: cython_sources
      [end of output]
   
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Underlying problem here appears to be the same as that described in yaml/pyyaml#601. PyYAML itself uses cython and it looks like certain versions of cython don't correctly hook into the build infrastructure. The workaround suggested there appears to work, at least with 3.11.x:

$ pip install "Cython<3.0" "pyyaml<6.0" --no-build-isolation
Collecting Cython<3.0
  Obtaining dependency information for Cython<3.0 from https://files.pythonhosted.org/packages/43/c9/4c8a84731c0948b408d059c2c17c95cbb8b9c2baefe1c6985c0b29a9055e/Cython-0.29.36-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl.metadata
  Downloading Cython-0.29.36-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl.metadata (3.1 kB)
Collecting pyyaml<6.0
  Downloading PyYAML-5.4.1.tar.gz (175 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 175.1/175.1 kB 5.8 MB/s eta 0:00:00
  Preparing metadata (pyproject.toml) ... done
Downloading Cython-0.29.36-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (1.9 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.9/1.9 MB 49.7 MB/s eta 0:00:00
Building wheels for collected packages: pyyaml
  Building wheel for pyyaml (pyproject.toml) ... done
  Created wheel for pyyaml: filename=PyYAML-5.4.1-cp311-cp311-linux_x86_64.whl size=45658 sha256=53aa5944120645f9f03c4edd882ca0be069c7125935c64919305d9e285cd9625
  Stored in directory: /home/automaton/.cache/pip/wheels/2f/2f/af/a062a8f866dc44812a825da11175d50d7f255900f3b38c79b5
Successfully built pyyaml
Installing collected packages: pyyaml, Cython
Successfully installed Cython-0.29.36 pyyaml-5.4.1
$ python -V
Python 3.11.5

Note that you'll also have to install the wheel package in order to avoid complaints about "bdist_wheel" missing on install:

pip install wheel

You should install wheel before trying to build the PyYAML package.

weideng1 · 2023-08-25T22:27:21Z

Thanks Bret for the latest investigation. Your suggested workaround did solve the ccm installation issue. Please review the latest commit and approve. I'll then run Jenkins tests using master branch.

…, before PYTHON-1297 gets merged

absurdfarce

Yup, this looks good to me. Only one small (and not really required) comment about documenting the weirdness with older PyYAML < 6.0 installs on Python 3.x but otherwise I think we're good here.

absurdfarce · 2023-08-28T14:39:32Z

Jenkinsfile

@@ -167,6 +167,8 @@ def initializeEnvironment() {
    sudo apt-get install socat
    pip install --upgrade pip
    pip install -U setuptools
+    pip install wheel
+    pip install "Cython<3.0" "pyyaml<6.0" --no-build-isolation


Might be worth adding a short comment here about why this command is being invoked this way

absurdfarce · 2023-08-28T14:41:56Z

Jenkinsfile

@@ -186,7 +188,7 @@ def initializeEnvironment() {
  }

  sh label: 'Install unit test modules', script: '''#!/bin/bash -lex
-    pip install nose-ignore-docstring nose-exclude service_identity
+    pip install pynose nose-ignore-docstring nose-exclude service_identity


Nice, interesting find! It's at least a better short-term bridge until we can get onto pytest

…sync_with_upstream_3.29.1 version 3.29.0 * tag '3.29.0' of https://github.com/datastax/python-driver: Documentation (and other) updates for 3.29.0 (datastax#1194) PYTHON-1331 ssl.match_hostname() is deprecated in 3.7 (datastax#1191) PYTHON-1371 Add explicit exception type for serialization failures (datastax#1193) Remove outdated Python pre-3.7 references (datastax#1186) PYTHON-1368 Avoid installing DSE deps + executing DSE tests for Python 3.12 (datastax#1188) PYTHON-1366 Handle removal of asyncore in Python 3.12 (datastax#1187) Removed backup(.bak) files (datastax#1185) docs: Fix typo in add_callbacks (datastax#1177) Remove some remnants of Python2 (datastax#1172) PYTHON-1313 Fix asyncio removals in Python 3.10 (datastax#1179) PYTHON-1364 Fix ssl.wrap_socket errors (from eventlet) for Python 3.12 (datastax#1181) Add Jenkins support for Python 3.12.0 (datastax#1180) Update redirects in docs.yaml (datastax#1178) Jenkins using new python versions in the matrix (datastax#1174) Update docs.yaml to point to most recent 3.28.0 docs changes CONN-38 Notes for 3.28.0 on PYTHON-1350 (datastax#1167) Include docs for 3.28.0 Fixed non-valid rst in README

update Jenkins matrix to use available python versions in the latest …

9e48dc2

…runner image

weideng1 requested a review from absurdfarce August 23, 2023 22:31

absurdfarce approved these changes Aug 23, 2023

View reviewed changes

weideng1 added 8 commits August 23, 2023 20:31

Use earliest and latest supported python runtimes for smoke test

ae4b5bd

Remove C* and DSE versions that are EOL'ed

c8bebf6

updated ccm and PyYAML version to avoid problem with higher python ve…

960336f

…rsions

workaround for CCM's PyYAML dependency that's not compatible with hig…

7993359

…her python versions

fix setup.py that also has pyYaml version requirement, causing python…

f79ed89

… 3.10+ problem

escape backslash to avoid groovy freaking out

cb90f6e

simplify sed script to fix setup.py

b00f3b8

fix typo in the sed script to fix setup.py

f851b46

weideng1 requested a review from absurdfarce August 24, 2023 06:49

absurdfarce requested changes Aug 25, 2023

View reviewed changes

test-requirements.txt Outdated Show resolved Hide resolved

Jenkinsfile Outdated Show resolved Hide resolved

Jenkinsfile Outdated Show resolved Hide resolved

address Bret's comments

b1a7a01

weideng1 requested a review from absurdfarce August 25, 2023 18:34

weideng1 force-pushed the jenkins-new-python branch from ae19bb7 to 8f5f00d Compare August 25, 2023 22:09

second attempt to make ccm-private work with python 3.10+

0ca0ddf

weideng1 force-pushed the jenkins-new-python branch from 8f5f00d to 0ca0ddf Compare August 25, 2023 22:21

weideng1 force-pushed the jenkins-new-python branch from 1055314 to 0ca0ddf Compare August 28, 2023 14:15

use pynose to replace nosetests so that it can run under Python 3.10+…

59b36ce

…, before PYTHON-1297 gets merged

weideng1 force-pushed the jenkins-new-python branch from d4a4e0d to 59b36ce Compare August 28, 2023 14:32

absurdfarce approved these changes Aug 28, 2023

View reviewed changes

address issue that brought in the wrong ccm-public

a3051f3

weideng1 force-pushed the jenkins-new-python branch from d8ca18d to a3051f3 Compare August 28, 2023 19:30

added a comment as Bret suggested

6c97c73

weideng1 merged commit 8ba0a5c into master Aug 28, 2023
2 of 5 checks passed

weideng1 deleted the jenkins-new-python branch August 28, 2023 21:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Jenkins using new python versions in the matrix #1174

Jenkins using new python versions in the matrix #1174

weideng1 commented Aug 23, 2023

absurdfarce commented Aug 25, 2023 •

edited

Loading

weideng1 commented Aug 25, 2023

absurdfarce left a comment

absurdfarce Aug 28, 2023

absurdfarce Aug 28, 2023

Jenkins using new python versions in the matrix #1174

Jenkins using new python versions in the matrix #1174

Conversation

weideng1 commented Aug 23, 2023

absurdfarce commented Aug 25, 2023 • edited Loading

weideng1 commented Aug 25, 2023

absurdfarce left a comment

Choose a reason for hiding this comment

absurdfarce Aug 28, 2023

Choose a reason for hiding this comment

absurdfarce Aug 28, 2023

Choose a reason for hiding this comment

absurdfarce commented Aug 25, 2023 •

edited

Loading