Support Copy with Remote Object Stores in datafusion-cli

[devinjdangelo]

Is your feature request related to a problem or challenge?

This unfortunately does not work in datafusion-cli currently:

DataFusion CLI v34.0.0
❯ copy (values (1,2)) to 's3://dfcli-test-bucket/test3.parquet';
Internal error: No suitable object store found for s3://dfcli-test-bucket/.
This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker
❯ copy (values (1,2)) to 's3://dfcli-test-bucket2/test3.parquet';
Internal error: No suitable object store found for s3://dfcli-test-bucket2/.
This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker

But this does:

❯ create external table remote_table2(a int, b int) stored as parquet location 's3://dfcli-test-bucket2/';
0 rows in set. Query took 0.001 seconds.

❯ insert into remote_table2 values (1,2);
+-------+
| count |
+-------+
| 1     |
+-------+
1 row in set. Query took 0.272 seconds.

❯ select * from remote_table2;
+---+---+
| a | b |
+---+---+
| 1 | 2 |
+---+---+

Describe the solution you'd like

Support "on the fly" registration of remote object store for copy statements similar to create external table.

Describe alternatives you've considered

No response

Additional context

No response

[alamb]

I think the need is documented, so this is just a matter of debugging and connecting up the existing APIs. I think it is a good first issue

[manoj-inukolunu]

Hello @alamb , can i give this a shot? Thank you.

[alamb]

Thanks @manoj-inukolunu !