Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support FileSourceRelation to load CSV in PPL #677

Draft
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

LantaoJin
Copy link
Member

@LantaoJin LantaoJin commented Sep 18, 2024

Description

This PR is for the syntax option B described in #638
The PR for option A is #678.

Syntax of option B

search file=<tempTableName> <fileUrl> [predicate]

Usage:

search file=table1 "s3://bucket_name/folder1/folder2/flights.csv" FlightDelay > 500
search file=table1 "s3://bucket_name/folder1/folder2/flights.csv.gz" FlightDelay > 500
search file=table1 "s3a://bucket_name/folder1/folder2/flights.parquet" FlightDelay > 500

Only single file is supported now.

PS: the current search command syntax is

search index=<indexName> [predicate]
search source=<indexName> [predicate]

Issues Resolved

Resolves #638

Check List

  • Updated documentation (ppl-spark-integration/README.md)
  • Implemented unit tests
  • Implemented tests for combination with other commands
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Lantao Jin <ltjin@amazon.com>
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@YANG-DB YANG-DB added the Lang:PPL Pipe Processing Language support label Sep 18, 2024
@LantaoJin LantaoJin changed the title Support load CSV in PPL Support FileSource relation to load CSV in PPL Sep 18, 2024
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin LantaoJin marked this pull request as ready for review September 18, 2024 15:56
@LantaoJin LantaoJin changed the title Support FileSource relation to load CSV in PPL Support FileSourceRelation to load CSV in PPL Sep 18, 2024
@YANG-DB
Copy link
Member

YANG-DB commented Sep 19, 2024

@LantaoJin I would go with option A as shown in #678
Thanks again for u'r support and contributions

@YANG-DB YANG-DB added the 0.6 label Sep 25, 2024
@YANG-DB
Copy link
Member

YANG-DB commented Sep 25, 2024

@LantaoJin can u resolve the latest conflict plz ?

@YANG-DB
Copy link
Member

YANG-DB commented Oct 1, 2024

@LantaoJin can u plz change to PR into draft mode ?

@LantaoJin LantaoJin marked this pull request as draft October 2, 2024 01:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
0.6 Lang:PPL Pipe Processing Language support
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[FEATURE] Support load CSV in PPL (inputlookup or search)
2 participants