SQLAlchemy Version Bug

[jonathandurantalkiatry]

First Stop

• I have read and agreed to the ThoughtSpot Dev Tools EULA.
• I understand that if I have a critical issue with ThoughtSpot, I should reach out to my designated Account Executive and Solutions Consultant team instead of cs_tools.
• I searched the cs_tools documentation, with the integrated search.

Platform Configuration

Description

When running cs_tools tools searchable gather, sqlalchemy==1.4.50 is installed which is incompatible with sqlmodel. I think this causes an upgrade to sqlalchemy==2.X.X but this breaks due to deprecated libraries. Specifically, from sqlalchemy.util.compat import string_types is no longer available in sqlalchemy==2.X.X.

Traceback:

16s
Run /home/runner/.config/cs_tools/.cs_tools/bin/cs_tools tools searchable gather
/home/runner/.config/cs_tools/.cs_tools/bin/cs_tools tools searchable gather
--config default
--syncer snowflake://$HOME/snowflake-syncer.toml
shell: /usr/bin/bash -e {0}
env:
pythonLocation: /opt/hostedtoolcache/Python/3.9.18/x64
PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.9.18/x64/lib/pkgconfig
Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.18/x64
Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.18/x64
Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.18/x64
LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.9.18/x64/lib
[08:34:16] INFO registering syncer: snowflake syncer.py:49
INFO installing package: wheel register.py:51
[08:34:17] INFO Collecting wheel _updater.py:69
INFO Downloading _updater.py:69
wheel-0.42.0-py3-none-any.whl.metadata (2.2
kB)
INFO Downloading wheel-0.42.0-py3-none-any.whl (65 _updater.py:69
kB)
INFO ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ _updater.py:69
65.4/65.4 kB 2.9 MB/s eta 0:00:00
INFO Installing collected packages: wheel _updater.py:69
INFO Successfully installed wheel-0.42.0 _updater.py:69
INFO installing package: pyarrow == 10.0.1 register.py:51
[08:34:18] INFO Collecting pyarrow==10.0.1 _updater.py:69
INFO Downloading _updater.py:69
pyarrow-10.0.1-cp39-cp39-manylinux_2_17_x86_6
4.manylinux2014_x86_64.whl (35.9 MB)
[08:34:19] INFO ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ _updater.py:69
35.9/35.9 MB 276.8 MB/s eta 0:00:00
INFO Collecting numpy>=1.16.6 (from _updater.py:69
pyarrow==10.0.1)
INFO Downloading _updater.py:69
numpy-1.26.2-cp39-cp39-manylinux_2_17_x86_64.
manylinux2014_x86_64.whl.metadata (61 kB)
INFO ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ _updater.py:69
61.2/61.2 kB 224.9 MB/s eta 0:00:00
INFO Downloading _updater.py:69
numpy-1.26.2-cp39-cp39-manylinux_2_17_x86_64.
manylinux2014_x86_64.whl (18.2 MB)
INFO ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ _updater.py:69
18.2/18.2 MB 295.6 MB/s eta 0:00:00
INFO Installing collected packages: numpy, pyarrow _updater.py:69
[08:34:23] INFO Successfully installed numpy-1.26.2 _updater.py:69
pyarrow-10.0.1
INFO installing package: cryptography == 40.0.2 register.py:51
[08:34:24] INFO Collecting cryptography==40.0.2 _updater.py:69
INFO Downloading updater.py:69
cryptography-40.0.2-cp36-abi3-manylinux_2_28
x86_64.whl (3.7 MB)
INFO ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ _updater.py:69
3.7/3.7 MB 16.9 MB/s eta 0:00:00
INFO Collecting cffi>=1.12 (from _updater.py:69
cryptography==40.0.2)
INFO Downloading _updater.py:69
cffi-1.16.0-cp39-cp39-manylinux_2_17_x86_64.m
anylinux2014_x86_64.whl.metadata (1.5 kB)
INFO Collecting pycparser (from _updater.py:69
cffi>=1.12->cryptography==40.0.2)
INFO Downloading _updater.py:69
pycparser-2.21-py2.py3-none-any.whl (118 kB)
INFO ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ _updater.py:69
118.7/118.7 kB 254.7 MB/s eta 0:00:00
INFO Downloading _updater.py:69
cffi-1.16.0-cp39-cp39-manylinux_2_17_x86_64.m
anylinux2014_x86_64.whl (443 kB)
INFO ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ _updater.py:69
443.4/443.4 kB 41.5 MB/s eta 0:00:00
[08:34:25] INFO Installing collected packages: pycparser, _updater.py:69
cffi, cryptography
INFO Successfully installed cffi-1.16.0 _updater.py:69
cryptography-40.0.2 pycparser-2.21
INFO installing package: snowflake-sqlalchemy == register.py:51
1.4.6
[08:34:26] INFO Collecting snowflake-sqlalchemy==1.4.6 _updater.py:69
INFO Downloading _updater.py:69
snowflake_sqlalchemy-1.4.6-py2.py3-none-any.w
hl (32 kB)
INFO Collecting sqlalchemy<2.0.0,>=1.4.0 (from _updater.py:69
snowflake-sqlalchemy==1.4.6)
INFO Downloading _updater.py:69
SQLAlchemy-1.4.50-cp39-cp39-manylinux_2_5_x86
_64.manylinux1_x86_64.manylinux_2_17_x86_64.m
anylinux2014_x86_64.whl.metadata (10.0 kB)
[08:34:27] INFO Collecting snowflake-connector-python<4.0.0 _updater.py:69
(from snowflake-sqlalchemy==1.4.6)
INFO Downloading _updater.py:69
snowflake_connector_python-3.6.0-cp39-cp39-ma
nylinux_2_17_x86_64.manylinux2014_x86_64.whl.
metadata (58 kB)
INFO ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ _updater.py:69
58.6/58.6 kB 7.2 MB/s eta 0:00:00
INFO Collecting asn1crypto<2.0.0,>0.24.0 (from _updater.py:69
snowflake-connector-python<4.0.0->snowflake-s
qlalchemy==1.4.6)
INFO Downloading _updater.py:69
asn1crypto-1.5.1-py2.py3-none-any.whl (105
kB)
INFO ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ _updater.py:69
105.0/105.0 kB 26.4 MB/s eta 0:00:00
INFO Requirement already satisfied: _updater.py:69
cffi<2.0.0,>=1.9 in
/home/runner/.config/cs_tools/.cs_tools/lib/p
ython3.9/site-packages (from
snowflake-connector-python<4.0.0->snowflake-s
qlalchemy==1.4.6) (1.16.0)
INFO Requirement already satisfied: _updater.py:69
cryptography<42.0.0,>=3.1.0 in
/home/runner/.config/cs_tools/.cs_tools/lib/p
ython3.9/site-packages (from
snowflake-connector-python<4.0.0->snowflake-s
qlalchemy==1.4.6) (40.0.2)
INFO Collecting pyOpenSSL<24.0.0,>=16.2.0 (from _updater.py:69
snowflake-connector-python<4.0.0->snowflake-s
qlalchemy==1.4.6)
INFO Downloading _updater.py:69
pyOpenSSL-23.3.0-py3-none-any.whl.metadata
(12 kB)
INFO Collecting pyjwt<3.0.0 (from _updater.py:69
snowflake-connector-python<4.0.0->snowflake-s
qlalchemy==1.4.6)
INFO Downloading _updater.py:69
PyJWT-2.8.0-py3-none-any.whl.metadata (4.2
kB)
INFO Collecting pytz (from _updater.py:69
snowflake-connector-python<4.0.0->snowflake-s
qlalchemy==1.4.6)
INFO Downloading _updater.py:69
pytz-2023.3.post1-py2.py3-none-any.whl.metada
ta (22 kB)
INFO Collecting requests<3.0.0 (from _updater.py:69
snowflake-connector-python<4.0.0->snowflake-s
qlalchemy==1.4.6)
INFO Downloading _updater.py:69
requests-2.31.0-py3-none-any.whl.metadata
(4.6 kB)
INFO Requirement already satisfied: packaging in _updater.py:69
/home/runner/.config/cs_tools/.cs_tools/lib/p
ython3.9/site-packages (from
snowflake-connector-python<4.0.0->snowflake-s
qlalchemy==1.4.6) (23.2)
INFO Collecting charset-normalizer<4,>=2 (from _updater.py:69
snowflake-connector-python<4.0.0->snowflake-s
qlalchemy==1.4.6)
INFO Downloading updater.py:69
charset_normalizer-3.3.2-cp39-cp39-manylinux
2_17_x86_64.manylinux2014_x86_64.whl.metadata
(33 kB)
INFO Requirement already satisfied: idna<4,>=2.5 _updater.py:69
in
/home/runner/.config/cs_tools/.cs_tools/lib/p
ython3.9/site-packages (from
snowflake-connector-python<4.0.0->snowflake-s
qlalchemy==1.4.6) (3.6)
INFO Requirement already satisfied: _updater.py:69
certifi>=2017.4.17 in
/home/runner/.config/cs_tools/.cs_tools/lib/p
ython3.9/site-packages (from
snowflake-connector-python<4.0.0->snowflake-s
qlalchemy==1.4.6) (2023.11.17)
INFO Requirement already satisfied: _updater.py:69
typing-extensions<5,>=4.3 in
/home/runner/.config/cs_tools/.cs_tools/lib/p
ython3.9/site-packages (from
snowflake-connector-python<4.0.0->snowflake-s
qlalchemy==1.4.6) (4.9.0)
INFO Collecting filelock<4,>=3.5 (from _updater.py:69
snowflake-connector-python<4.0.0->snowflake-s
qlalchemy==1.4.6)
INFO Downloading _updater.py:69
filelock-3.13.1-py3-none-any.whl.metadata
(2.8 kB)
INFO Collecting sortedcontainers>=2.4.0 (from _updater.py:69
snowflake-connector-python<4.0.0->snowflake-s
qlalchemy==1.4.6)
INFO Downloading _updater.py:69
sortedcontainers-2.4.0-py2.py3-none-any.whl
(29 kB)
INFO Collecting platformdirs<4.0.0,>=2.6.0 (from _updater.py:69
snowflake-connector-python<4.0.0->snowflake-s
qlalchemy==1.4.6)
INFO Downloading _updater.py:69
platformdirs-3.11.0-py3-none-any.whl.metadata
(11 kB)
INFO Collecting tomlkit (from _updater.py:69
snowflake-connector-python<4.0.0->snowflake-s
qlalchemy==1.4.6)
INFO Downloading _updater.py:69
tomlkit-0.12.3-py3-none-any.whl.metadata (2.7
kB)
INFO Collecting urllib3<2.0.0,>=1.21.1 (from _updater.py:69
snowflake-connector-python<4.0.0->snowflake-s
qlalchemy==1.4.6)
INFO Downloading _updater.py:69
urllib3-1.26.18-py2.py3-none-any.whl.metadata
(48 kB)
INFO ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ _updater.py:69
48.9/48.9 kB 226.4 MB/s eta 0:00:00
INFO Requirement already satisfied: _updater.py:69
greenlet!=0.4.17 in
/home/runner/.config/cs_tools/.cs_tools/lib/p
ython3.9/site-packages (from
sqlalchemy<2.0.0,>=1.4.0->snowflake-sqlalchem
y==1.4.6) (3.0.3)
INFO Requirement already satisfied: pycparser in _updater.py:69
/home/runner/.config/cs_tools/.cs_tools/lib/p
ython3.9/site-packages (from
cffi<2.0.0,>=1.9->snowflake-connector-python<
4.0.0->snowflake-sqlalchemy==1.4.6) (2.21)
[08:34:28] INFO Collecting cryptography<42.0.0,>=3.1.0 (from _updater.py:69
snowflake-connector-python<4.0.0->snowflake-s
qlalchemy==1.4.6)
INFO Downloading updater.py:69
cryptography-41.0.7-cp37-abi3-manylinux_2_28
x86_64.whl.metadata (5.2 kB)
INFO Downloading _updater.py:69
snowflake_connector_python-3.6.0-cp39-cp39-ma
nylinux_2_17_x86_64.manylinux2014_x86_64.whl
(2.5 MB)
INFO ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ _updater.py:69
2.5/2.5 MB 131.3 MB/s eta 0:00:00
INFO Downloading _updater.py:69
SQLAlchemy-1.4.50-cp39-cp39-manylinux_2_5_x86
_64.manylinux1_x86_64.manylinux_2_17_x86_64.m
anylinux2014_x86_64.whl (1.6 MB)
INFO ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ _updater.py:69
1.6/1.6 MB 280.8 MB/s eta 0:00:00
INFO Downloading updater.py:69
charset_normalizer-3.3.2-cp39-cp39-manylinux
2_17_x86_64.manylinux2014_x86_64.whl (142 kB)
INFO ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ _updater.py:69
142.3/142.3 kB 288.0 MB/s eta 0:00:00
INFO Downloading filelock-3.13.1-py3-none-any.whl _updater.py:69
(11 kB)
INFO Downloading _updater.py:69
platformdirs-3.11.0-py3-none-any.whl (17 kB)
INFO Downloading PyJWT-2.8.0-py3-none-any.whl (22 _updater.py:69
kB)
INFO Downloading pyOpenSSL-23.3.0-py3-none-any.whl _updater.py:69
(58 kB)
INFO ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ _updater.py:69
58.8/58.8 kB 240.8 MB/s eta 0:00:00
INFO Downloading updater.py:69
cryptography-41.0.7-cp37-abi3-manylinux_2_28
x86_64.whl (4.4 MB)
INFO ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ _updater.py:69
4.4/4.4 MB 301.8 MB/s eta 0:00:00
INFO Downloading requests-2.31.0-py3-none-any.whl _updater.py:69
(62 kB)
INFO ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ _updater.py:69
62.6/62.6 kB 232.1 MB/s eta 0:00:00
INFO Downloading _updater.py:69
urllib3-1.26.18-py2.py3-none-any.whl (143 kB)
INFO ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ _updater.py:69
143.8/143.8 kB 273.7 MB/s eta 0:00:00
INFO Downloading _updater.py:69
pytz-2023.3.post1-py2.py3-none-any.whl (502
kB)
INFO ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ _updater.py:69
502.5/502.5 kB 305.8 MB/s eta 0:00:00
INFO Downloading tomlkit-0.12.3-py3-none-any.whl _updater.py:69
(37 kB)
INFO Installing collected packages: _updater.py:69
sortedcontainers, pytz, asn1crypto, urllib3,
tomlkit, sqlalchemy, pyjwt, platformdirs,
filelock, charset-normalizer, requests,
cryptography, pyOpenSSL,
snowflake-connector-python,
snowflake-sqlalchemy
[08:34:29] INFO Attempting uninstall: sqlalchemy _updater.py:69
INFO Found existing installation: SQLAlchemy _updater.py:69
2.0.24
INFO Uninstalling SQLAlchemy-2.0.24: _updater.py:69
INFO Successfully uninstalled SQLAlchemy-2.0.24 _updater.py:69
[08:34:30] INFO Attempting uninstall: cryptography _updater.py:69
INFO Found existing installation: cryptography _updater.py:69
40.0.2
INFO Uninstalling cryptography-40.0.2: _updater.py:69
INFO Successfully uninstalled cryptography-40.0.2 _updater.py:69
ERROR pip's dependency resolver does not currently _updater.py:69
take into account all the packages that are
installed. This behaviour is the source of
the following dependency conflicts.
INFO sqlmodel 0.0.14 requires _updater.py:69
SQLAlchemy<2.1.0,>=2.0.0, but you have
sqlalchemy 1.4.50 which is incompatible.
INFO Successfully installed asn1crypto-1.5.1 _updater.py:69
charset-normalizer-3.3.2 cryptography-41.0.7
filelock-3.13.1 platformdirs-3.11.0
pyOpenSSL-23.3.0 pyjwt-2.8.0
pytz-2023.3.post1 requests-2.31.0
snowflake-connector-python-3.6.0
snowflake-sqlalchemy-1.4.6
sortedcontainers-2.4.0 sqlalchemy-1.4.50
tomlkit-0.12.3 urllib3-1.26.18
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /home/runner/.config/cs_tools/.cs_tools/lib/python3.9/site-packages/cs_tools │
│ /cli/main.py:144 in run │
│ │
│ 141 │ app.add_typer(_log.app) │
│ 142 │ │
│ 143 │ try: │
│ ❱ 144 │ │ return_code = app(standalone_mode=False) │
│ 145 │ │
│ 146 │ except (click.Abort, typer.Abort): │
│ 147 │ │ return_code = 0 │
│ │
│ /home/runner/.config/cs_tools/.cs_tools/lib/python3.9/site-packages/typer/ma │
│ in.py:328 in call │
│ │
│ /home/runner/.config/cs_tools/.cs_tools/lib/python3.9/site-packages/typer/ma │
│ in.py:311 in call │
│ │
│ /home/runner/.config/cs_tools/.cs_tools/lib/python3.9/site-packages/click/co │
│ re.py:1157 in call │
│ │
│ /home/runner/.config/cs_tools/.cs_tools/lib/python3.9/site-packages/typer/co │
│ re.py:778 in main │
│ │
│ ... 9 frames hidden ... │
│ in exec_module:850 │
│ in _call_with_frames_removed:228 │
│ │
│ /home/runner/.config/cs_tools/.cs_tools/lib/python3.9/site-packages/cs_tools │
│ /sync/snowflake/syncer.py:9 in │
│ │
│ 6 import enum │
│ 7 import uuid │
│ 8 │
│ ❱ 9 from snowflake.sqlalchemy import URL │
│ 10 from pydantic import root_validator, Field, BaseModel │
│ 11 import pyarrow.parquet as pq │
│ 12 import sqlalchemy as sa │
│ │
│ /home/runner/.config/cs_tools/.cs_tools/lib/python3.9/site-packages/snowflak │
│ e/sqlalchemy/init.py:30 in │
│ │
│ 27 │ VARCHAR, │
│ 28 ) │
│ 29 │
│ ❱ 30 from . import base, snowdialect │
│ 31 from .custom_commands import ( │
│ 32 │ AWSBucket, │
│ 33 │ AzureContainer, │
│ │
│ /home/runner/.config/cs_tools/.cs_tools/lib/python3.9/site-packages/snowflak │
│ e/sqlalchemy/base.py:13 in │
│ │
│ 10 from sqlalchemy.schema import Sequence, Table │
│ 11 from sqlalchemy.sql import compiler, expression │
│ 12 from sqlalchemy.sql.elements import quoted_name │
│ ❱ 13 from sqlalchemy.util.compat import string_types │
│ 14 │
│ 15 from .custom_commands import AWSBucket, AzureContainer, ExternalStage │
│ 16 from .util import _set_connection_interpolate_empty_sequences │
╰──────────────────────────────────────────────────────────────────────────────╯
ImportError: cannot import name 'string_types' from 'sqlalchemy.util.compat'
(/home/runner/.config/cs_tools/.cs_tools/lib/python3.9/site-packages/sqlalchemy/
util/compat.py)

[boonhapus]

Thanks @jonathandurantalkiatry ! I see this actually as a snowflake-sqlalchemy issue. The Snowflake team has some work to do to achieve true sqlalchemy 2.0 compatibility, but I've worked around it in the 1.5.0 branch .

This branch is still in a very healthy beta phase at the moment, but should be available before the end of the month.

[boonhapus]

@jonathandurantalkiatry I think I've fixed this as part of v1.4.15 in case this is blocking you.