Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exploring streaming #133

Merged
merged 50 commits into from
Apr 21, 2022
Merged

Exploring streaming #133

merged 50 commits into from
Apr 21, 2022

Conversation

Jolanrensen
Copy link
Collaborator

As mentioned here: #122
This WIP pr creates a withSparkStreaming() {} function and KSparkStreaming scope.
It makes it easier to work with Apache Spark Streaming since the boilerplate code is already handled.
See Streaming.kt for an example.

@@ -78,18 +86,82 @@ inline fun withSpark(builder: Builder, logLevel: SparkLogLevel = ERROR, func: KS
KSparkSession(this).apply {
sparkContext.setLogLevel(logLevel)
func()
sc.stop()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did we decide to remove it?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yup! in the RDD branch, will remove it here as well

Jolanrensen and others added 25 commits February 24, 2022 15:16
… like this could form the basis of easy streaming support :)
…ing-streaming

# Conflicts:
#	kotlin-spark-api/3.2/src/main/kotlin/org/jetbrains/kotlinx/spark/api/SparkHelper.kt
…eaming pair-functions, decided to go tuple-first instead of arities
…ode), new notation for entering KSparkSession scope inside streaming scope
# Conflicts:
#	kotlin-spark-api/3.2/src/main/kotlin/org/jetbrains/kotlinx/spark/api/Conversions.kt
…to normal "exploring-streaming" after vacation
Added DataStreamWriter forEachBatch helper
added KotlinStatefulNetworkCount.kt
cleaning up
added KotlinSqlNetworkWordCount.kt
@Jolanrensen
Copy link
Collaborator Author

@asm0dey What do you think of the notation like this?
I'll add some more tests for the tuple functions btw. Also, are there any other streaming examples you know of that I can test to see whether we support them?

@Jolanrensen
Copy link
Collaborator Author

@asm0dey checkmarks all around, finally. Any other tests or checks I need to do before merging streaming support?

@Jolanrensen Jolanrensen merged commit 8ca1be7 into spark-3.2 Apr 21, 2022
@Jolanrensen Jolanrensen deleted the exploring-streaming branch April 21, 2022 17:27
@Jolanrensen Jolanrensen added this to the 1.1.0 milestone May 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants