Feature/geopackage reader #1603
Conversation
|
As a follow up to this mr, I need to add
|
|
Thank you for this great work! My major concern is whether it works with GeoPackage files stored on cloud storage such as HDFS or S3. As far as I know |
|
@Kontinuation good point, I ll write the test to make sure it works. Integration test with minio would be enough I think. |
|
WIP |
| // skip srid for now | ||
| reader.getInt() | ||
|
|
||
| skipEnvelope(resolvedFlags._1, reader) | ||
|
|
||
| val wkb = new Array[Byte](reader.remaining()) | ||
| reader.get(wkb) | ||
|
|
||
| val wkbReader = new org.locationtech.jts.io.WKBReader() | ||
| val geom = wkbReader.read(wkb) |
There was a problem hiding this comment.
I suggest that we take SRID into consideration. val wkbReader = new WKBReader(new GeometryFactory(new PrecisionModel(), srid)) would be sufficient.
| if (pathString.toLowerCase(Locale.ROOT).endsWith(".geopackage")) { | ||
| val path = new Path(pathString) | ||
| val fs = path.getFileSystem(hadoopConf) | ||
|
|
||
| val isDirectory = Try(fs.getFileStatus(path).isDirectory).getOrElse(false) | ||
| if (isDirectory) { | ||
| pathString | ||
| } else { | ||
| pathString.substring(0, pathString.length - 3) + "???" | ||
| } |
There was a problem hiding this comment.
If I understand it correctly, if pathString ends with ".geopackage", and it is not a directory, it will be transformed to "****.geopack???". I cannot grasp the idea of this transformation.
There was a problem hiding this comment.
yeah you are right, I was sure that in this part I already have list of files in the directory, I am wondering how it would behave if I am passing list of files and if its actually needed (user specifying paths with different file formats).
| val serializableConf = new SerializableConfiguration( | ||
| sparkSession.sessionState.newHadoopConfWithOptions(options.asScala.toMap)) | ||
|
|
||
| val tempFile = FileSystemUtils.copyToLocal(serializableConf.value, files.head.getPath) |
There was a problem hiding this comment.
Can we detect if the path is a local path skip calling copyToLocal?
|
@Kontinuation thanks for the review ! |
Imbruced
force-pushed
the
feature/geopackage-reader
branch
from
767b8ca
to
5b88709
Compare
| @@ -98,6 +97,36 @@ | |||
| <groupId>org.locationtech.jts</groupId> | |||
| <artifactId>jts-core</artifactId> | |||
| </dependency> | |||
| <dependency> | |||
| <groupId>org.testcontainers</groupId> | |||
There was a problem hiding this comment.
Is there a reason why these dependencies only appear in the spark-3.3 profile?
There was a problem hiding this comment.
test testing loading from s3 (I used minio with test containers) is only in spark 3.3. I can duplicate it to other versions of spark as well
There was a problem hiding this comment.
Please move these to the spark-common pom.xml so all Spark 3.X pom.xml will share it
There was a problem hiding this comment.
Do we support other Spark versions 3.0, 3.1, and 3.2?
There was a problem hiding this comment.
I didn't test it with those I can add data sources
There was a problem hiding this comment.
Yes, if the implementation differs among the versions, we should definitely replicate the tests and test them for each version
Did you read the Contributor Guide?
Is this PR related to a JIRA ticket?
What changes were proposed in this PR?
Geopackage datasource
How was this patch tested?
integration tests
Did this PR include necessary documentation updates?