Add capability to get SQL statements to add custom CRS in the database #2577

rouault · 2021-03-13T16:03:16Z

Add proj_insert_object_session_create(), proj_insert_object_session_destroy(), proj_get_insert_statements(), proj_suggests_code_for() to get SQL statements to add custom CRS in the database
projinfo: add -o SQL --output-id AUTH:NAME to get the SQL statements for a CRS object
Add proj_context_get_database_structure() to dump structure of empty valid auxiliary DB
projinfo: add a --dump-db-structure switch

Example:

# Create an auxiliary database with the definition of a custom CRS.
projinfo "+proj=merc +lat_ts=5 +datum=WGS84 +type=crs +title=my_crs" \
     --output-id HOBU:MY_CRS --dump-db-structure | sqlite3 aux.db

# Check that everything works OK
projinfo --aux-db-path aux.db HOBU:MY_CRS

outputs

PROJ.4 string:
+proj=merc +lat_ts=5 +lon_0=0 +x_0=0 +y_0=0 +datum=WGS84 +units=m +no_defs +type=crs

WKT2:2019 string:
PROJCRS["my_crs",
    BASEGEOGCRS["unknown",
        ENSEMBLE["World Geodetic System 1984 ensemble",
            MEMBER["World Geodetic System 1984 (Transit)"],
            MEMBER["World Geodetic System 1984 (G730)"],
            MEMBER["World Geodetic System 1984 (G873)"],
            MEMBER["World Geodetic System 1984 (G1150)"],
            MEMBER["World Geodetic System 1984 (G1674)"],
            MEMBER["World Geodetic System 1984 (G1762)"],
            ELLIPSOID["WGS 84",6378137,298.257223563,
                LENGTHUNIT["metre",1]],
            ENSEMBLEACCURACY[2.0]],
        PRIMEM["Greenwich",0,
            ANGLEUNIT["degree",0.0174532925199433]],
        ID["HOBU","GEODETIC_CRS_MY_CRS"]],
    CONVERSION["unknown",
        METHOD["Mercator (variant B)",
            ID["EPSG",9805]],
        PARAMETER["Latitude of 1st standard parallel",5,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8823]],
        PARAMETER["Longitude of natural origin",0,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8802]],
        PARAMETER["False easting",0,
            LENGTHUNIT["metre",1],
            ID["EPSG",8806]],
        PARAMETER["False northing",0,
            LENGTHUNIT["metre",1],
            ID["EPSG",8807]]],
    CS[Cartesian,2],
        AXIS["(E)",east,
            ORDER[1],
            LENGTHUNIT["metre",1]],
        AXIS["(N)",north,
            ORDER[2],
            LENGTHUNIT["metre",1]],
    ID["HOBU","MY_CRS"]]

rouault · 2021-03-13T16:12:36Z

CC @hobu

kbevers · 2021-03-15T10:01:59Z

Interesting feature, @rouault. Do you mind sharing the motivation behind this?

I haven't really looked at the code but at least the docs looks alright to me.

rouault · 2021-03-15T11:50:48Z

Do you mind sharing the motivation behind this?

so that people can ingest custom WKTs and refer to them next with a code. @hobu might give more details about his intented use case
it should also make it easier to ingest later the IAU definitions that are provided as WKT.

kbevers · 2021-03-15T12:05:03Z

so that people can ingest custom WKTs and refer to them next with a code

Yeah, so much was clear to me :-)

@hobu might give more details about his intented use case

That would be great.

Just to be clear, I think this is a great idea and a good replacement for custom init-files.

Thinking a bit further, it might also be nice to be able to do custom transformations, e.g. by running projinfo "+proj=helmert +x=12 +y=34 +z=56 ..." -s EPSG:xxxx -t EPSG:yyyy --output-id HOBU:MY_TRANSFORMATION --dump-db-structure .

rouault · 2021-03-15T12:22:28Z

Thinking a bit further, it might also be nice to be able to do custom transformations, e.g. by running projinfo "+proj=helmert +x=12 +y=34 +z=56 ..." -s EPSG:xxxx -t EPSG:yyyy --output-id HOBU:MY_TRANSFORMATION --dump-db-structure .

agreed, but not in my current scope of work. The "easy" solution would be to put them in the other_transformation table directly as PROJ strings. You might still need to customize the SQL or add switches to specify the accuracy, scope and extent.

kbevers · 2021-03-15T12:36:29Z

but not in my current scope of work

I wasn't suggesting that you do it, just thinking out loud. One thing that comes to mind though is, how are the custom CRS added with this new feature usable without also registering transformations to this new CRS?

You might still need to customize the SQL or add switches to specify the accuracy, scope and extent.

Yeah, it could get complicated to capture everything in one projinfo call. Existing switches could be overridden as implied in my example above that uses -s and -t. Something similar could be done for --accuracy and --area which then only leaves the scope.

rouault · 2021-03-15T12:42:44Z

One thing that comes to mind though is, how are the custom CRS added with this new feature usable without also registering transformations to this new CRS?

For the use cases I was provided they use existing EPSG datums. They are typically projected CRS using us-ft as units but reyling on NAD83(2011). So the current set of EPSG transformations is fine to deal with them.

kbevers · 2021-03-15T12:50:55Z

For the use cases I was provided they use existing EPSG datums.

I guess that works because of the +datum parameter? If I were to do this using WKT2 instead of a projstring, would it still work provided that a BASEGEOGCRS(?) is set?

rouault · 2021-03-15T13:21:21Z

I guess that works because of the +datum parameter?

the actual use cases are for CRS instanciated from WKT1 or WKT2

If I were to do this using WKT2 instead of a projstring, would it still work provided that a BASEGEOGCRS(?) is set?

Yes. It will try to re-use existing intermediate objects by using the CRS::identify() method as much as possible.
For example

$ projinfo  --output-id HOBU:MY_CRS -o SQL -q 'PROJCS["my CRS",
    GEOGCS["WGS 84",
        DATUM["WGS_1984",
            SPHEROID["WGS 84",6378137,298.257223563,
                AUTHORITY["EPSG","7030"]],
            AUTHORITY["EPSG","6326"]],
        PRIMEM["Greenwich",0,
            AUTHORITY["EPSG","8901"]],
        UNIT["degree",0.0174532925199433,
            AUTHORITY["EPSG","9122"]]],
    PROJECTION["Transverse_Mercator"],
    PARAMETER["latitude_of_origin",0],
    PARAMETER["central_meridian",4],
    PARAMETER["scale_factor",0.9996],
    PARAMETER["false_easting",500000],
    PARAMETER["false_northing",0],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    AXIS["Easting",EAST],
    AXIS["Northing",NORTH]]'

outputs

INSERT INTO conversion VALUES('HOBU','CONVERSION_MY_CRS','unnamed','','EPSG','9807','Transverse Mercator','EPSG','8801','Latitude of natural origin',0,'EPSG','9122','EPSG','8802','Longitude of natural origin',4,'EPSG','9122','EPSG','8805','Scale factor at natural origin',0.9996,'EPSG','9201','EPSG','8806','False easting',500000,'EPSG','9001','EPSG','8807','False northing',0,'EPSG','9001',NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,0);
INSERT INTO usage VALUES('HOBU','USAGE_CONVERSION_MY_CRS','conversion','HOBU','CONVERSION_MY_CRS','PROJ','EXTENT_UNKNOWN','PROJ','SCOPE_UNKNOWN');
INSERT INTO projected_crs VALUES('HOBU','MY_CRS','my CRS','','EPSG','4400','EPSG','4326','HOBU','CONVERSION_MY_CRS',NULL,0);
INSERT INTO usage VALUES('HOBU','USAGE_PROJECTED_CRS_MY_CRS','projected_crs','HOBU','MY_CRS','PROJ','EXTENT_UNKNOWN','PROJ','SCOPE_UNKNOWN');

which shows that it re-uses EPSG:4326

hobu · 2021-03-15T13:54:12Z

@hobu might give more details about his intended use case

There are just too many situations that depend up on single codes where WKT of any flavor won't do. OGC services are a big culprit, but there are other cases as well.

In the US, many high precision engineering coordinate systems are either not in EPSG, are somewhere in the process of becoming registered with EPSG, or are registered EPSG CRSs that are bastardized by their users to modify things like parameters and transforms. /me shakes his fist at COMP_CS users of horizontal meters and vertical feet.

The old init file relief valve of sticking a code in there isn't such a good thing to do with PROJ 6+, and this approach gives us a more sustainable way to do such a thing. An alternative that would also be fine with me is some sort of ADDITIONAL custom proj.db being used with an environment variable denoting its location. The nice thing about the latter approach is your custom CRSs would survive software upgrades and such.

rouault · 2021-03-15T14:02:35Z

An alternative that would also be fine with me is some sort of ADDITIONAL custom proj.db being used with an environment variable denoting its location

The mechanics for that is already in place at the API level (we have a proj_context_set_database_path() function that can take a path to the main DB and auxiliary DBs). Maybe proj_context_create() could look at a PROJ_AUX_DB=/path/to/first.db:/path/to/second.db:... env varible to set that in a more user-friendly way ?

kbevers · 2021-03-15T14:05:54Z

Thanks for the elaboration, Howard.

Maybe proj_context_create() could look at a PROJ_AUX_DB=/path/to/first.db:/path/to/second.db:... env varible to set that in a more user-friendly way ?

That would be quite useful, yes. It would certainly make it easier to distribute a set of custom CRS's to users, especially in cases where it is not fitting to put it in the main PROJ distribution.

snowman2 · 2021-03-15T14:09:58Z

One thought I had was that if you are not doing so already, it would be nice to prevent users from using the official authorities such as EPSG, ESRI, .. to help clarify that this was a custom CRS and is not from one of the official authorities.

rouault · 2021-03-15T14:32:43Z

One thought I had was that if you are not doing so already, it would be nice to prevent users from using the official authorities such as EPSG, ESRI, .. to help clarify that this was a custom CRS and is not from one of the official authorities.

I would not enforce that. This is the responsibility of the users. They might need to mess with official authorities as workarounds if some software is only able to support EPSG:XXXX codes.

rouault · 2021-03-15T14:38:12Z

look at a PROJ_AUX_DB=/path/to/first.db:/path/to/second.db:... env varible to set that in a more user-friendly way ?

done

snowman2 · 2021-03-15T14:44:42Z

They might need to mess with official authorities as workarounds if some software is only able to support EPSG:XXXX codes.

Since this is a new feature, they would likely need to update the underlying software anyway. I think helping direct the users towards intended usage patterns would be a good idea. Otherwise, it gets messy/confusing.

hobu · 2021-03-15T14:49:32Z

They might need to mess with official authorities as workarounds if some software is only able to support EPSG:XXXX codes

Is it PROJ's burden to support that? I vote no. They could still manually edit the database to do what they needed. We don't really want them adding EPSG codes. EPSG:900913 wasn't such a good thing 😉

rouault · 2021-03-15T14:52:59Z

Since this is a new feature, they would likely need to update the underlying software anyway. I think helping direct the users towards intended usage patterns would be a good idea. Otherwise, it gets messy/confusing.

How would we determine which authorities are allowed to have new definitions from those who aren't ? Currently in proj.db, we have EPSG, ESRI, IGNF, OGC, NKG and PROJ. And this new capability might potentially be used in the future for the ingestion scripts that populate proj.db with some of those official authorities. I think your concern would be best addressed by some note in the documentation. I'll let you follow-up in a later PR with such doc addition if you wish.

snowman2 · 2021-03-15T14:59:02Z

How would we determine which authorities are allowed to have new definitions from those who aren't?

I was actually thinking of only allowing one. Maybe just call it PROJ_CUSTOM and then only allow the user to enter a code for their CRS.

kbevers · 2021-03-15T15:00:45Z

How would we determine which authorities are allowed to have new definitions from those who aren't ?

I'd say don't allow reuse of authorities already in proj.db. Authorities that are already in proj.db already knows how to get their stuff in the main database (or is sufficiently important that we do it on their behalf), so they can follow that path if they need to expand their data in the database. For any other use cases they can use a tempory AUTH like "NKG2". When it comes to proper integration in proj.db they auth-name is easily changed.

hobu · 2021-03-15T15:01:49Z

I was actually thinking of only allowing one. Maybe just call it PROJ_CUSTOM and then only allow the user to enter a code for their CRS.

There's still too much opportunity for clashing in this case. Your PROJ_CUSTOM:4326 and my PROJ_CUSTOM:4326 might have flipped XY axes but different units 😛

snowman2 · 2021-03-15T15:03:26Z

There's still too much opportunity for clashing in this case. Your PROJ_CUSTOM:4326 and my PROJ_CUSTOM:4326 might have flipped XY axes but different units

That's true. Maybe use it as a prefix: PROJ_CUSTOM_HOBU

rouault · 2021-03-15T15:08:06Z

The last comments are really about debating how a tool might be misused. There are so many ways that PROJ can be misused. In ancient times, people could easily add a new line to the 'epsg' file... And we won't prevent them to edit the existing proj.db with new EPSG definitions, whatever technological barrier we put, if they are sufficiently determined to do so

hobu · 2021-03-15T15:15:30Z

Let's go with this for now:

Document what is expected for adding custom registries (what is possible and what is not)
State that custom additions should not be common registries such as ESRI, EPSG, or IAU (but don't enforce that in any way)
Guide people toward using their own PROJ_AUX_DB instead of updating proj.db directly to prevent their data getting stomped on by a software update
Point to the EPSG process and tell people that they should really register real world coordinate systems that are in use.

snowman2 · 2021-03-15T15:16:07Z

There are so many ways that PROJ can be misused.

Two things can be true at the same time. It is true someone can hack the proj.db and true that it would be good for PROJ to try to get users to use it correctly. 😄

…-coded WGS84-based CRSs

…look for intermediate objects

…valid auxiliary DB

…ne or several auxiliary DBs

rouault · 2021-03-15T15:29:08Z

Let's go with this for now:

See my last commit in this PR

…ries and use auxiliary databases

rouault added this to the 8.1.0 milestone Mar 13, 2021

rouault force-pushed the insert_sql branch from 40544c8 to c96abdc Compare March 13, 2021 16:28

rouault force-pushed the insert_sql branch from 4940acb to 51f5230 Compare March 15, 2021 14:44

rouault added 2 commits March 15, 2021 16:16

Add C/C++ API to get SQL statements to insert a CRS into database

5ddddfa

projinfo: add a '-o SQL --output-id AUTH:CODE' SQL output

a234207

rouault added 7 commits March 15, 2021 16:16

CRS::identify(): take into account allowed authority to identify hard…

e33f6b5

…-coded WGS84-based CRSs

SQL output: add capability to restrict the authorities into which to …

8a67a3f

…look for intermediate objects

SQL export: add publication_date and frame_reference_epoch to datums

d618dcc

SQL export: add support for DatumEnsemble

7293d65

Add proj_context_get_database_structure() to dump structure of empty …

80ef9ce

…valid auxiliary DB

projinfo: add a --dump-db-structure switch

eda2311

Add support for PROJ_AUX_DB environment variable to set the path to o…

72f4f49

…ne or several auxiliary DBs

rouault force-pushed the insert_sql branch from 51f5230 to 72f4f49 Compare March 15, 2021 15:16

rouault force-pushed the insert_sql branch from d61e8ea to 1321e1b Compare March 15, 2021 15:36

Doc improvements regarding recomandation not to alter official regist…

eae06c9

…ries and use auxiliary databases

rouault force-pushed the insert_sql branch from 1321e1b to eae06c9 Compare March 15, 2021 15:50

rouault mentioned this pull request Mar 17, 2021

OSR: Get/Set access to PROJ auxiliary database filenames OSGeo/gdal#3590

Merged

rouault merged commit d4e5e1a into OSGeo:master Mar 18, 2021

This was referenced Mar 19, 2021

SQL output: make it possible to export non-EPSG projection methods or… #2585

Merged

Any update on IAU2000/2015 CRS support? #2601

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add capability to get SQL statements to add custom CRS in the database #2577

Add capability to get SQL statements to add custom CRS in the database #2577

rouault commented Mar 13, 2021

rouault commented Mar 13, 2021

kbevers commented Mar 15, 2021

rouault commented Mar 15, 2021

kbevers commented Mar 15, 2021

rouault commented Mar 15, 2021

kbevers commented Mar 15, 2021

rouault commented Mar 15, 2021

kbevers commented Mar 15, 2021

rouault commented Mar 15, 2021

hobu commented Mar 15, 2021

rouault commented Mar 15, 2021

kbevers commented Mar 15, 2021

snowman2 commented Mar 15, 2021

rouault commented Mar 15, 2021

rouault commented Mar 15, 2021

snowman2 commented Mar 15, 2021

hobu commented Mar 15, 2021

rouault commented Mar 15, 2021

snowman2 commented Mar 15, 2021

kbevers commented Mar 15, 2021

hobu commented Mar 15, 2021

snowman2 commented Mar 15, 2021 •

edited

Loading

rouault commented Mar 15, 2021

hobu commented Mar 15, 2021 •

edited

Loading

snowman2 commented Mar 15, 2021

rouault commented Mar 15, 2021

Add capability to get SQL statements to add custom CRS in the database #2577

Add capability to get SQL statements to add custom CRS in the database #2577

Conversation

rouault commented Mar 13, 2021

rouault commented Mar 13, 2021

kbevers commented Mar 15, 2021

rouault commented Mar 15, 2021

kbevers commented Mar 15, 2021

rouault commented Mar 15, 2021

kbevers commented Mar 15, 2021

rouault commented Mar 15, 2021

kbevers commented Mar 15, 2021

rouault commented Mar 15, 2021

hobu commented Mar 15, 2021

rouault commented Mar 15, 2021

kbevers commented Mar 15, 2021

snowman2 commented Mar 15, 2021

rouault commented Mar 15, 2021

rouault commented Mar 15, 2021

snowman2 commented Mar 15, 2021

hobu commented Mar 15, 2021

rouault commented Mar 15, 2021

snowman2 commented Mar 15, 2021

kbevers commented Mar 15, 2021

hobu commented Mar 15, 2021

snowman2 commented Mar 15, 2021 • edited Loading

rouault commented Mar 15, 2021

hobu commented Mar 15, 2021 • edited Loading

snowman2 commented Mar 15, 2021

rouault commented Mar 15, 2021

snowman2 commented Mar 15, 2021 •

edited

Loading

hobu commented Mar 15, 2021 •

edited

Loading