-
Notifications
You must be signed in to change notification settings - Fork 72
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
PostgreSQL Support #257
PostgreSQL Support #257
Changes from 11 commits
655b7ad
428ed8a
5063623
bde6a96
b0dc3db
4034fc6
c6659c4
b7c49ed
6604017
95e012d
dce4117
0a3ed41
05305c5
c0e1ab4
9738fe6
f0257fc
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,80 @@ | ||
"""drivers/postgresql_meta_store.py | ||
|
||
PostgreSQL-backed metadata driver. Metadata is stored in a PostgreSQL database. | ||
""" | ||
|
||
from typing import Mapping, Sequence | ||
|
||
import sqlalchemy as sqla | ||
from terracotta.drivers.relational_meta_store import RelationalMetaStore | ||
|
||
|
||
class PostgreSQLMetaStore(RelationalMetaStore): | ||
"""A PostgreSQL-backed metadata driver. | ||
|
||
Stores metadata and paths to raster files in PostgreSQL. | ||
|
||
Requires a running PostgreSQL server. | ||
|
||
The PostgreSQL database consists of 4 different tables: | ||
|
||
- ``terracotta``: Metadata about the database itself. | ||
- ``key_names``: Contains two columns holding all available keys and their description. | ||
- ``datasets``: Maps key values to physical raster path. | ||
- ``metadata``: Contains actual metadata as separate columns. Indexed via key values. | ||
|
||
This driver caches key names. | ||
""" | ||
SQL_DIALECT = 'postgresql' | ||
SQL_DRIVER = 'psycopg2' | ||
SQL_TIMEOUT_KEY = 'connect_timeout' | ||
|
||
MAX_PRIMARY_KEY_SIZE = 2730 // 4 # Max B-tree index size in bytes | ||
DEFAULT_PORT = 5432 | ||
|
||
def __init__(self, postgresql_path: str) -> None: | ||
"""Initialize the PostgreSQLDriver. | ||
|
||
This should not be called directly, use :func:`~terracotta.get_driver` instead. | ||
|
||
Arguments: | ||
|
||
postgresql_path: URL to running PostgreSQL server, in the form | ||
``postgresql://username:password@hostname/database`` | ||
|
||
""" | ||
super().__init__(postgresql_path) | ||
|
||
# raise an exception if database name is invalid | ||
if not self.url.database: | ||
raise ValueError('database must be specified in PostgreSQL path') | ||
if '/' in self.url.database.strip('/'): | ||
raise ValueError('invalid database path') | ||
|
||
@classmethod | ||
def _normalize_path(cls, path: str) -> str: | ||
url = cls._parse_path(path) | ||
|
||
path = f'{url.drivername}://{url.host}:{url.port or cls.DEFAULT_PORT}/{url.database}' | ||
path = path.rstrip('/') | ||
return path | ||
|
||
def _create_database(self) -> None: | ||
engine = sqla.create_engine( | ||
self.url.set(database='postgres'), # `.set()` returns a copy with changed parameters | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Postgres does not allow to connect without connecting to a specific database. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That is also what I had in mind. Perhaps add a There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Good idea. Maybe There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. With a brief explaining comment, I think that is good |
||
echo=False, | ||
future=True, | ||
isolation_level='AUTOCOMMIT' | ||
) | ||
with engine.connect() as connection: | ||
connection.execute(sqla.text(f'CREATE DATABASE {self.url.database}')) | ||
connection.commit() | ||
|
||
def _initialize_database( | ||
self, | ||
keys: Sequence[str], | ||
key_descriptions: Mapping[str, str] = None | ||
) -> None: | ||
# Enforce max primary key length equal to max B-tree index size | ||
self.SQL_KEY_SIZE = self.MAX_PRIMARY_KEY_SIZE // len(keys) | ||
super()._initialize_database(keys, key_descriptions) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We might want to rename these to
SQL_USER
/SQL_PASSWORD
and then deprecate theMYSQL_
fields?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it might be nice to support
PG*
environment variableshttps://www.manniwood.com/2005_01_01/using_the_pg_environmental_variables_to_make_your_shell_scripts_more_terse.html
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see I have actually broken the
MYSQL_*
settings with theSQLAlchemy
branch :/ Seems I have somehow overlooked their existence... I'll fix that, and rename themSQL_USER
/SQL_PASSWORD
in the process, in a new PR shortly.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@chapmanjacobd I have thought about this, but I think it would be best to avoid using the
PG*
variables. Every other env var that Terracotta reads is prefixed withTC_
so for consistency I think we should stick to that.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@nickeopti Okay, I won't add them here then. If we rename the existing variables, it will break backwards compatibility for users who are already using MySQL. Since we are heading for a 1.0 release, maybe that's fine?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll leave decisions regarding backwards compatibility and deprecation behaviour to @dionhaefner, @j08lue and/or you @mrpgraae. I don't know the use cases well enough
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See #258