Arrow-flight connector #23032

sabbasani · 2024-06-19T11:23:38Z

Description

Motivation and Context

Impact

Test Plan

Contributor checklist

Please make sure your submission complies with our development, formatting, commit message, and attribution guidelines.
PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
Documented new properties (with its default value), SQL syntax, functions, or other functionality.
If release notes are required, they follow the release notes guidelines.
Adequate tests were added if applicable.
CI passed.

Release Notes

Please follow release notes guidelines and fill in the release notes below.

== RELEASE NOTES ==

General Changes
* Add Arrow Flight connector :pr:`23032`
* Add documentation for the :doc:`/connector/base-arrow-flight`  :pr:`23032`

If release note is NOT required, use:

== NO RELEASE NOTE ==

steveburnett · 2024-06-19T14:22:18Z

Consider adding documentation for the new connector.

Suggest revising the release note entry to follow the Release Note Guidelines:

== RELEASE NOTES ==

General Changes
* Add Arrow Flight connector :pr:`23032`

tdcmeehan

It seems like there's a hardcoded presumption that the underlying datasource accepts a SQL query. Can you remove this from the PR? The service may not accept SQL.

tdcmeehan · 2024-07-16T14:53:03Z

presto-base-arrow-flight/src/main/java/com/facebook/plugin/arrow/ArrowAbstractMetadata.java

+    public List<Field> getColumnsList(String schema, String table, ConnectorSession connectorSession)
+    {
+        try {
+            String dbSpecificSchemaName = getDBSpecificSchemaName(config, schema);


Avoid "DB" references as it might not be an underlying DB

I have removed "DB" references

tdcmeehan · 2024-07-16T14:53:44Z

presto-base-arrow-flight/src/main/java/com/facebook/plugin/arrow/ArrowTypeHandle.java

+            @JsonProperty("jdbcType") int jdbcType,
+            @JsonProperty("jdbcTypeName") String jdbcTypeName,


We can't have references to JDBC in this connector.

I have removed JDBC re references

steveburnett · 2024-07-16T17:11:46Z

Suggest change to the release note entry as follows:

== RELEASE NOTES ==

General Changes
* Add Arrow Flight connector :pr:`23032`

The documentation for Arrow Flight Connector appears to be being added in #23212 , so it doesn't need to be mentioned in this release note.

BryanCutler

I took a quick pass and had a few comments. I'll take another look at the rest a little later.

BryanCutler · 2024-07-17T00:44:34Z

presto-base-arrow-flight/pom.xml

+        <air.main.basedir>${project.parent.basedir}</air.main.basedir>
+        <grpc.version>1.53.0</grpc.version>
+        <dep.okhttp.version>4.10.0</dep.okhttp.version>
+        <arrow.version>11.0.0</arrow.version>


This is a fairly old version of Arrow, can you use a more recent one?

@BryanCutler I have updated to leastest arrow version

presto-base-arrow-flight/pom.xml

BryanCutler · 2024-07-17T01:04:20Z

presto-base-arrow-flight/src/main/java/com/facebook/plugin/arrow/ArrowColumnHandle.java

+    public ArrowColumnHandle(
+            @JsonProperty("columnName") String columnName,
+            @JsonProperty("columnType") Type columnType,
+            @JsonProperty("jdbcTypeHandle") ArrowTypeHandle arrowTypeHandle)


did you mean? "jdbcTypeHandle" -> "arrowTypeHandle"

BryanCutler · 2024-07-17T01:14:12Z

presto-base-arrow-flight/src/main/java/com/facebook/plugin/arrow/ArrowFlightClientHandler.java

+            return;
+        }
+        try {
+            RootAllocator allocator = new RootAllocator(Long.MAX_VALUE);


You would usually create a RootAllocator as a class member and they should be closed when not used anymore

@BryanCutler I have made changes for closing allocator

We are not re-using the flight client. The root allocator will be closed when ArrowFlightClient is closed or auto closed.

BryanCutler · 2024-07-17T01:16:21Z

presto-base-arrow-flight/src/main/java/com/facebook/plugin/arrow/ArrowFlightClientHandler.java

+                }
+            }
+
+            logger.debug("location %s", location.getUri().toString());


Is this meant to stay?

I have removed log @BryanCutler

BryanCutler · 2024-07-17T01:21:10Z

presto-base-arrow-flight/src/main/java/com/facebook/plugin/arrow/ArrowFlightClientHandler.java

+            trustedCertificate.get().close();
+        }
+        shutdownTimer();
+        isClientClosed.set(true);


Do the calls to getClient() and close() need to be thread-safe?

@BryanCutler I have addressed comments can please review changes

linux-foundation-easycla · 2024-07-17T06:10:26Z

The committers listed above are authorized under a signed CLA.

✅ login: sabbasani (f3c57ad)

steveburnett

I reviewed the base-arrow-flight.rst documentation in #23212. If this documentation is to be included in this PR - which I think would be a good idea to do this - please address my comments in my most recent review in #23212 here.

tdcmeehan · 2024-07-24T20:15:28Z

presto-base-arrow-flight/src/main/java/com/facebook/plugin/arrow/ArrowQueryBuilder.java

+import static java.util.Objects.requireNonNull;
+import static java.util.stream.Collectors.joining;
+
+public class ArrowQueryBuilder


This might better belong in a submodule that depends on this module which implements the Flight SQL spec. I don't think it belongs here.

@tdcmeehan I have addressed comments can please review changes

sabbasani · 2024-07-25T21:12:41Z

@tdcmeehan @BryanCutler @steveburnett
we have addressed all comments above.Can you please review

steveburnett

Thanks for the docs! Minor suggestions, mostly formatting.

A local doc build returns the following warning:

/Users/steveburnett/Documents/GitHub/presto/presto-docs/src/main/sphinx/connector/base-arrow-flight.rst: WARNING: document isn't included in any toctree

To address this warning,

Add connector/base-arrow-flight to
https://github.com/prestodb/presto/blob/master/presto-docs/src/main/sphinx/connector.rst so the new page is present on the Connector page in the Presto documentation.
Add the new line in alphabetic order for the new page title Arrow-flight Connector.

presto-docs/src/main/sphinx/connector/base-arrow-flight.rst

steveburnett · 2024-07-29T16:12:08Z

Suggest update of the release note to include the PR number in both entries, and to link to the new doc from the release note.

== RELEASE NOTES ==

General Changes
* Add Arrow Flight connector :pr:`23032`
* Add documentation for the :doc:`/connector/base-arrow-flight`  :pr:`23032`

tdcmeehan

Have you investigated how to make an end to end test with an example implementation? It would be great to demonstrate real queries working with this connector.

tdcmeehan · 2024-07-30T18:46:08Z

presto-base-arrow-flight/pom.xml

+            <version>2.0.65.Final</version>
+        </dependency>
+
+        <dependency>


Can you help me understand why so many exclusions are required in this POM?

we excluded dependencies to resolve security issues.should retain the dependencies instead of excluding?

The excludes are very brittle, if possible please don't add them

tdcmeehan · 2024-07-30T18:46:42Z

...to-base-arrow-flight/src/main/java/com/facebook/plugin/arrow/ArrowAbstractFlightRequest.java

+        query = Optional.empty();
+    }
+
+    public ArrowAbstractFlightRequest(String schema, String table, Optional<String> query)


I still see schema, table and query being referenced as if this were JDBC. Can you please remove?

tdcmeehan · 2024-07-30T18:47:26Z

presto-base-arrow-flight/src/main/java/com/facebook/plugin/arrow/ArrowExpression.java

+
+import static java.util.Objects.requireNonNull;
+
+public class ArrowExpression


What is an arrow expression? Is this a SQL query?

tdcmeehan · 2024-08-23T19:58:09Z

@sabbasani is this PR still being worked on?

sabbasani · 2024-08-23T20:12:26Z

@sabbasani is this PR still being worked on?

Yes @tdcmeehan I am working on testcases as discussed earlier

sabbasani · 2024-09-02T11:09:29Z

@tdcmeehan I have added testcases for arrow flight connector. Can you please review.

tdcmeehan

Thanks for adding these tests.

I'm wondering if we can use a more robust dataset, and then use a more robust test suite, such as by extending from AbstractTestQueries.

Please take a look at H2QueryRunner, which can be used to generate the complete TPCH dataset required by AbstractTestQueries. We should be able to generate the same dataset between the expected query runner and the Arrow query runner, and verify they return identical results in a new test suite that derives from AbstractTestQueries.

tdcmeehan · 2024-09-04T13:53:05Z

presto-base-arrow-flight/src/test/java/com/facebook/plugin/arrow/TestArrowFlightSmoke.java

+    @Test
+    public void testShowSchemas()
+    {
+        assertEquals(queryRunner.execute("SHOW SCHEMAS FROM arrow").getRowCount(), 3);


Can you validate the contents as well?

tdcmeehan · 2024-09-04T13:53:39Z

presto-base-arrow-flight/src/test/java/com/facebook/plugin/arrow/TestArrowFlightSmoke.java

+        String query1 = "SELECT * FROM testdb.example_table1";
+        String expected1 = "1, John Doe, 1990-05-15, 50000.00, true";
+        String expected2 = "2, Jane Smith, 1985-11-20, 60000.00, false";


Please inline these variables

tdcmeehan · 2024-09-04T13:53:58Z

presto-base-arrow-flight/src/test/java/com/facebook/plugin/arrow/TestArrowFlightSmoke.java

+        String expected2 = "2, Jane Smith, 1985-11-20, 60000.00, false";
+
+        String result1 = queryRunner.execute(query1).toString();
+        assertTrue(result1.contains(expected1), "Expected row not found: " + expected1);


Please also check that this is the entirity of the result set, or at least check on rowcount.

tdcmeehan · 2024-09-04T13:54:16Z

presto-base-arrow-flight/src/test/java/com/facebook/plugin/arrow/TestArrowFlightSmoke.java

+    {
+        // Ensure the catalog and schema names are correct
+        String query = "SHOW TABLES FROM testdb";
+        String[] expectedTables = {"example_table1", "example_table2", "example_table3"};


Just use one line for each

tdcmeehan · 2024-09-04T14:19:09Z

presto-docs/src/main/sphinx/connector/base-arrow-flight.rst

+* ``ArrowAbstractFlightRequest.java``
+  Implement this class to define the request data, including the data source type, connection properties, the number of partitions and other data required to interact with database.
+
+* ``ArrowAbstractMetadata.java``


Suggested change

* ``ArrowAbstractMetadata.java``

* ``AbstractArrowMetadata.java``

tdcmeehan · 2024-09-04T14:19:19Z

presto-docs/src/main/sphinx/connector/base-arrow-flight.rst

+* ``ArrowAbstractMetadata.java``
+  To retrieve metadata (schema and table information), implement the abstract methods in the ArrowAbstractMetadata class.
+
+* ``ArrowAbstractSplitManager.java``


Suggested change

* ``ArrowAbstractSplitManager.java``

* ``AbstractArrowSplitManager.java``

elbinpallimalilibm · 2024-10-10T05:55:14Z

presto-base-arrow-flight/src/main/java/com/facebook/plugin/arrow/ArrowFlightClientHandler.java

+
+    public ArrowFlightClient getClient(Optional<String> uri)
+    {
+        return initializeClient(uri);


We are not re-using the flight client. This is a change from the approach mentioned in https://github.com/prestodb/rfcs/blob/main/RFC-0004-arrow-flight-connector.md. While testing the connector with multi pod IBM flight server on an open shift cluster, we noticed that re-using the flight client does not distribute the flight get info requests to the different flight server pods. But opening new flight connections for every request distributed the client requests among the different flight pods.

elbinpallimalilibm · 2024-10-10T06:38:07Z

presto-base-arrow-flight/src/test/java/com/facebook/plugin/arrow/TestArrowFlightSmoke.java

+import static org.testng.Assert.assertTrue;
+
+public class TestArrowFlightSmoke
+        extends AbstractTestQueries


@tdcmeehan Following your suggestion we are running all the 500 or so tests in AbstractTestQueries apart from the ones defined here.

Co-authored-by: sai bhaskar reddy <sai.bhaskar.reddy.sabbasani1@ibm.com> Co-authored-by: SthuthiGhosh9400 <Sthuthi.Ghosh@ibm.com> Co-authored-by: lithinwxd <Lithin.Purushothaman@ibm.com>

sabbasani requested a review from a team as a code owner June 19, 2024 11:23

sabbasani requested a review from presto-oss June 19, 2024 11:23

sabbasani marked this pull request as draft June 19, 2024 11:42

tdcmeehan self-assigned this Jul 15, 2024

tdcmeehan requested changes Jul 16, 2024

View reviewed changes

tdcmeehan requested a review from BryanCutler July 16, 2024 14:55

BryanCutler reviewed Jul 17, 2024

View reviewed changes

steveburnett requested changes Jul 17, 2024

View reviewed changes

sabbasani force-pushed the arrow-connector branch 4 times, most recently from a69b92e to 7bd4ec9 Compare July 24, 2024 13:38

tdcmeehan reviewed Jul 24, 2024

View reviewed changes

sabbasani force-pushed the arrow-connector branch 8 times, most recently from 04cdc53 to 824175b Compare July 25, 2024 21:05

tdcmeehan requested a review from BryanCutler July 26, 2024 14:11

sabbasani force-pushed the arrow-connector branch from 6d9d75b to 11bada5 Compare July 29, 2024 13:16

steveburnett requested changes Jul 29, 2024

View reviewed changes

sabbasani force-pushed the arrow-connector branch from 3b73ee3 to c4e77f1 Compare July 29, 2024 17:28

sabbasani force-pushed the arrow-connector branch 2 times, most recently from 123ab6b to e24a904 Compare July 30, 2024 15:45

tdcmeehan requested changes Jul 30, 2024

View reviewed changes

sabbasani force-pushed the arrow-connector branch from 63b388b to fa50f08 Compare August 1, 2024 17:38

steveburnett mentioned this pull request Aug 13, 2024

Document the base arrow module #23212

Draft

6 tasks

sabbasani force-pushed the arrow-connector branch 2 times, most recently from 7fafbd2 to e0daebb Compare September 2, 2024 11:07

sabbasani force-pushed the arrow-connector branch 4 times, most recently from 8b97b44 to e824379 Compare September 4, 2024 09:10

tdcmeehan requested changes Sep 4, 2024

View reviewed changes

elbinpallimalilibm force-pushed the arrow-connector branch from 1bbf201 to d329e9f Compare October 2, 2024 05:23

elbinpallimalilibm force-pushed the arrow-connector branch from a9a311a to 43e72a2 Compare October 10, 2024 05:40

sabbasani marked this pull request as ready for review October 10, 2024 05:54

sabbasani requested a review from elharo as a code owner October 10, 2024 05:54

elbinpallimalilibm reviewed Oct 10, 2024

View reviewed changes

elbinpallimalilibm force-pushed the arrow-connector branch 2 times, most recently from b274a2d to b450554 Compare October 10, 2024 06:36

elbinpallimalilibm reviewed Oct 10, 2024

View reviewed changes

elbinpallimalilibm force-pushed the arrow-connector branch 4 times, most recently from 82adf97 to 32c9c08 Compare October 10, 2024 11:19

Arrow flight connector

f3c57ad

Co-authored-by: sai bhaskar reddy <sai.bhaskar.reddy.sabbasani1@ibm.com> Co-authored-by: SthuthiGhosh9400 <Sthuthi.Ghosh@ibm.com> Co-authored-by: lithinwxd <Lithin.Purushothaman@ibm.com>

elbinpallimalilibm force-pushed the arrow-connector branch from c3de72f to f3c57ad Compare October 10, 2024 12:57

		@JsonProperty("jdbcType") int jdbcType,
		@JsonProperty("jdbcTypeName") String jdbcTypeName,


		import static java.util.Objects.requireNonNull;

		public class ArrowExpression

	* ``ArrowAbstractMetadata.java``
	* ``AbstractArrowMetadata.java``

	* ``ArrowAbstractSplitManager.java``
	* ``AbstractArrowSplitManager.java``

Arrow-flight connector #23032

Are you sure you want to change the base?

Arrow-flight connector #23032

Conversation

sabbasani commented Jun 19, 2024 • edited Loading

Description

Motivation and Context

Impact

Test Plan

Contributor checklist

Release Notes

steveburnett commented Jun 19, 2024

tdcmeehan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

steveburnett commented Jul 16, 2024

BryanCutler left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

linux-foundation-easycla bot commented Jul 17, 2024 • edited Loading

steveburnett left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sabbasani commented Jul 25, 2024 • edited Loading

steveburnett left a comment

Choose a reason for hiding this comment

steveburnett commented Jul 29, 2024

tdcmeehan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tdcmeehan Jul 31, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tdcmeehan commented Aug 23, 2024

sabbasani commented Aug 23, 2024

sabbasani commented Sep 2, 2024

tdcmeehan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sabbasani commented Jun 19, 2024 •

edited

Loading

linux-foundation-easycla bot commented Jul 17, 2024 •

edited

Loading

sabbasani commented Jul 25, 2024 •

edited

Loading

tdcmeehan Jul 31, 2024 •

edited

Loading