Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

✨ Add BigQuery source and Prefect tasks #1035

Open
wants to merge 24 commits into
base: 2.0
Choose a base branch
from
+783 −2
Open

✨ Add BigQuery source and Prefect tasks #1035

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
f3a3eec
✨ added bigquery source file.
Diego-H-S Aug 30, 2024
29641ba
✨ added bigquery task file.
Diego-H-S Aug 30, 2024
00d45c5
✨ added bigquery flow file.
Diego-H-S Aug 30, 2024
4536a4c
✨ added unit test files.
Diego-H-S Aug 30, 2024
e600ed3
📝 updated docstrings.
Diego-H-S Aug 30, 2024
fcc8ece
🔥 removed credentials in df_to_adls.
Diego-H-S Aug 30, 2024
0d7ab7f
📝 updated repository.
Diego-H-S Aug 30, 2024
e99984b
📝 added new package.
Diego-H-S Aug 30, 2024
e5b26de
🎨 updated integration test file folder.
Diego-H-S Sep 17, 2024
71e4375
📝 updated comments in file.
Diego-H-S Sep 17, 2024
a72de5e
📝 renamed some functions.
Diego-H-S Sep 19, 2024
7066674
📝 updated a comment.
Diego-H-S Sep 19, 2024
45436cc
📝 changed integration test file.
Diego-H-S Sep 19, 2024
5b571aa
📝 updated comments.
Diego-H-S Sep 19, 2024
6d46289
new case needed when query is provided
fdelgadodyvenia Sep 24, 2024
f6ba109
new case needed when query is provided
fdelgadodyvenia Sep 24, 2024
2cd3556
removing bad files
fdelgadodyvenia Sep 24, 2024
7ca8cb1
removing another 2 files crap
fdelgadodyvenia Sep 24, 2024
7753a5f
🎨 precommit changes
mgardzinski Sep 26, 2024
82fe5b1
🐛 Adding pandas_gbq to __init__
mgardzinski Sep 26, 2024
9ada6b4
🎨 fixing formating
mgardzinski Sep 26, 2024
b8788d5
♻️ Handle import with extra dependency
mgardzinski Sep 26, 2024
95c520e
➕ update requirements-dev.lock
mgardzinski Sep 27, 2024
0106a6f
🐛 Update names for methods, update extra condition in `api_connection`
mgardzinski Sep 27, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ dependencies = [
"defusedxml>=0.7.1",
"aiohttp>=3.10.5",
"pytest-mock>=3.14.0",
"pandas-gbq==0.23.1",
]
requires-python = ">=3.10"
readme = "README.md"
Expand Down
67 changes: 65 additions & 2 deletions requirements-dev.lock
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
# all-features: false
# with-sources: false
# generate-hashes: false
# universal: false

-e file:.
aiohappyeyeballs==2.4.0
Expand Down Expand Up @@ -93,6 +94,7 @@ comm==0.2.2
coolname==2.2.0
# via prefect
coverage==7.6.1
# via coverage
croniter==2.0.7
# via prefect
cryptography==43.0.0
Expand All @@ -102,6 +104,8 @@ cssselect2==0.7.0
# via cairosvg
dateparser==1.2.0
# via prefect
db-dtypes==1.3.0
# via pandas-gbq
debugpy==1.8.5
# via ipykernel
decorator==5.1.1
Expand Down Expand Up @@ -147,8 +151,32 @@ gitdb==4.0.11
# via gitpython
gitpython==3.1.43
# via mkdocs-git-revision-date-plugin
google-api-core==2.20.0
# via google-cloud-bigquery
# via google-cloud-core
# via pandas-gbq
google-auth==2.34.0
# via google-api-core
# via google-auth-oauthlib
# via google-cloud-bigquery
# via google-cloud-core
# via kubernetes
# via pandas-gbq
# via pydata-google-auth
google-auth-oauthlib==1.2.1
# via pandas-gbq
# via pydata-google-auth
google-cloud-bigquery==3.26.0
# via pandas-gbq
google-cloud-core==2.4.1
# via google-cloud-bigquery
google-crc32c==1.6.0
# via google-resumable-media
google-resumable-media==2.7.2
# via google-cloud-bigquery
googleapis-common-protos==1.65.0
# via google-api-core
# via grpcio-status
graphql-core==3.2.3
# via sgqlc
graphviz==0.20.3
Expand All @@ -158,6 +186,11 @@ greenlet==3.0.3
griffe==0.47.0
# via mkdocstrings-python
# via prefect
grpcio==1.66.1
# via google-api-core
# via grpcio-status
grpcio-status==1.66.1
# via google-api-core
h11==0.14.0
# via httpcore
# via uvicorn
Expand All @@ -169,6 +202,7 @@ httpcore==1.0.5
# via httpx
# via prefect
httpx==0.27.0
# via httpx
# via neoteroi-mkdocs
# via prefect
humanize==4.10.0
Expand Down Expand Up @@ -302,11 +336,13 @@ mkdocs-include-markdown-plugin==6.2.2
mkdocs-jupyter==0.24.8
mkdocs-material==9.5.32
# via mkdocs-jupyter
# via mkdocs-material
mkdocs-material-extensions==1.3.1
# via mkdocs-material
mkdocs-mermaid2-plugin==1.1.1
mkdocs-table-reader-plugin==3.0.1
mkdocstrings==0.25.2
# via mkdocstrings
# via mkdocstrings-python
mkdocstrings-python==1.10.5
# via mkdocstrings
Expand All @@ -330,8 +366,10 @@ nest-asyncio==1.6.0
networkx==3.3
# via visions
numpy==1.26.4
# via db-dtypes
# via imagehash
# via pandas
# via pandas-gbq
# via pyarrow
# via pywavelets
# via scipy
Expand All @@ -348,18 +386,25 @@ openpyxl==3.1.5
orjson==3.10.7
# via prefect
packaging==24.1
# via db-dtypes
# via google-cloud-bigquery
# via ipykernel
# via jupytext
# via mkdocs
# via nbconvert
# via pandas-gbq
# via prefect
# via pytest
paginate==0.5.6
# via mkdocs-material
pandas==2.2.2
# via db-dtypes
# via mkdocs-table-reader-plugin
# via pandas-gbq
# via viadot2
# via visions
pandas-gbq==0.23.1
# via viadot2
pandocfilters==1.5.1
# via nbconvert
parso==0.8.4
Expand Down Expand Up @@ -391,6 +436,13 @@ prefect-sqlalchemy==0.4.4
# via viadot2
prompt-toolkit==3.0.47
# via ipython
proto-plus==1.24.0
# via google-api-core
protobuf==5.28.2
# via google-api-core
# via googleapis-common-protos
# via grpcio-status
# via proto-plus
psutil==6.0.0
# via ipykernel
psycopg2-binary==2.9.9
Expand All @@ -400,6 +452,8 @@ ptyprocess==0.7.0
pure-eval==0.2.3
# via stack-data
pyarrow==10.0.1
# via db-dtypes
# via pandas-gbq
# via viadot2
pyasn1==0.6.0
# via pyasn1-modules
Expand All @@ -414,6 +468,8 @@ pydantic==1.10.17
# via viadot2
pydantic-core==2.23.0
# via prefect
pydata-google-auth==1.8.2
# via pandas-gbq
pygit2==1.14.1
# via viadot2
pygments==2.18.0
Expand Down Expand Up @@ -442,6 +498,7 @@ python-dateutil==2.9.0.post0
# via dateparser
# via faker
# via ghp-import
# via google-cloud-bigquery
# via jupyter-client
# via kubernetes
# via moto
Expand Down Expand Up @@ -498,6 +555,8 @@ regex==2024.7.24
requests==2.32.3
# via apprise
# via docker
# via google-api-core
# via google-cloud-bigquery
# via kubernetes
# via lumacli
# via mkdocs-material
Expand All @@ -511,6 +570,7 @@ requests==2.32.3
# via viadot2
requests-oauthlib==2.0.0
# via apprise
# via google-auth-oauthlib
# via kubernetes
# via o365
responses==0.25.3
Expand Down Expand Up @@ -538,8 +598,6 @@ scipy==1.14.0
# via imagehash
sendgrid==6.11.0
# via viadot2
setuptools==73.0.0
# via mkdocs-mermaid2-plugin
sgqlc==16.3
# via prefect-github
shapely==2.0.6
Expand Down Expand Up @@ -609,6 +667,7 @@ trino==0.328.0
typer==0.12.4
# via lumacli
# via prefect
# via typer
typing-extensions==4.12.2
# via aiosqlite
# via alembic
Expand Down Expand Up @@ -663,3 +722,7 @@ yarl==1.9.4
# via aiohttp
zipp==3.20.0
# via importlib-metadata
setuptools==73.0.0
# via mkdocs-mermaid2-plugin
# via pandas-gbq
# via pydata-google-auth
61 changes: 61 additions & 0 deletions requirements.lock
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
# all-features: false
# with-sources: false
# generate-hashes: false
# universal: false

-e file:.
aiohappyeyeballs==2.4.0
Expand Down Expand Up @@ -71,6 +72,8 @@ cryptography==43.0.0
# via prefect
dateparser==1.2.0
# via prefect
db-dtypes==1.3.0
# via pandas-gbq
defusedxml==0.7.1
# via viadot2
dnspython==2.6.1
Expand All @@ -92,8 +95,32 @@ frozenlist==1.4.1
# via aiosignal
fsspec==2024.6.1
# via prefect
google-api-core==2.20.0
# via google-cloud-bigquery
# via google-cloud-core
# via pandas-gbq
google-auth==2.34.0
# via google-api-core
# via google-auth-oauthlib
# via google-cloud-bigquery
# via google-cloud-core
# via kubernetes
# via pandas-gbq
# via pydata-google-auth
google-auth-oauthlib==1.2.1
# via pandas-gbq
# via pydata-google-auth
google-cloud-bigquery==3.25.0
# via pandas-gbq
google-cloud-core==2.4.1
# via google-cloud-bigquery
google-crc32c==1.6.0
# via google-resumable-media
google-resumable-media==2.7.2
# via google-cloud-bigquery
googleapis-common-protos==1.65.0
# via google-api-core
# via grpcio-status
graphql-core==3.2.3
# via sgqlc
graphviz==0.20.3
Expand All @@ -102,6 +129,11 @@ greenlet==3.0.3
# via sqlalchemy
griffe==0.47.0
# via prefect
grpcio==1.66.1
# via google-api-core
# via grpcio-status
grpcio-status==1.66.1
# via google-api-core
h11==0.14.0
# via httpcore
# via uvicorn
Expand All @@ -113,6 +145,7 @@ httpcore==1.0.5
# via httpx
# via prefect
httpx==0.27.0
# via httpx
# via prefect
humanize==4.10.0
# via jinja2-humanize-extension
Expand Down Expand Up @@ -169,8 +202,10 @@ multimethod==1.12
networkx==3.3
# via visions
numpy==1.26.4
# via db-dtypes
# via imagehash
# via pandas
# via pandas-gbq
# via pyarrow
# via pywavelets
# via scipy
Expand All @@ -187,11 +222,18 @@ openpyxl==3.1.5
orjson==3.10.7
# via prefect
packaging==24.1
# via db-dtypes
# via google-cloud-bigquery
# via pandas-gbq
# via prefect
# via pytest
pandas==2.2.2
# via db-dtypes
# via pandas-gbq
# via viadot2
# via visions
pandas-gbq==0.23.1
# via viadot2
pathspec==0.12.1
# via prefect
pendulum==2.1.2
Expand All @@ -208,9 +250,18 @@ prefect-github==0.2.7
# via viadot2
prefect-sqlalchemy==0.4.4
# via viadot2
proto-plus==1.24.0
# via google-api-core
protobuf==5.28.2
# via google-api-core
# via googleapis-common-protos
# via grpcio-status
# via proto-plus
psycopg2-binary==2.9.9
# via lumacli
pyarrow==10.0.1
# via db-dtypes
# via pandas-gbq
# via viadot2
pyasn1==0.6.0
# via pyasn1-modules
Expand All @@ -225,6 +276,8 @@ pydantic==1.10.17
# via viadot2
pydantic-core==2.23.0
# via prefect
pydata-google-auth==1.8.2
# via pandas-gbq
pygit2==1.14.1
# via viadot2
pygments==2.18.0
Expand All @@ -238,6 +291,7 @@ pytest-mock==3.14.0
python-dateutil==2.9.0.post0
# via croniter
# via dateparser
# via google-cloud-bigquery
# via kubernetes
# via o365
# via pandas
Expand Down Expand Up @@ -275,6 +329,8 @@ regex==2024.7.24
requests==2.32.3
# via apprise
# via docker
# via google-api-core
# via google-cloud-bigquery
# via kubernetes
# via lumacli
# via o365
Expand All @@ -284,6 +340,7 @@ requests==2.32.3
# via viadot2
requests-oauthlib==2.0.0
# via apprise
# via google-auth-oauthlib
# via kubernetes
# via o365
rfc3339-validator==0.1.4
Expand Down Expand Up @@ -346,6 +403,7 @@ trino==0.328.0
typer==0.12.4
# via lumacli
# via prefect
# via typer
typing-extensions==4.12.2
# via aiosqlite
# via alembic
Expand Down Expand Up @@ -379,3 +437,6 @@ websockets==12.0
# via prefect
yarl==1.9.8
# via aiohttp
setuptools==75.1.0
# via pandas-gbq
# via pydata-google-auth
Loading