Rdd related helpers #132
Rdd related helpers #132
Conversation
…nction taking a SparkConf
Adds sc JavaSparkContext to KSparkSession
examples/src/main/kotlin/org/jetbrains/kotlinx/spark/examples/Streaming.kt
Outdated
Show resolved
Hide resolved
kotlin-spark-api/3.2/src/main/kotlin/org/jetbrains/kotlinx/spark/api/SparkHelper.kt
Outdated
Show resolved
Hide resolved
kotlin-spark-api/3.2/src/main/kotlin/org/jetbrains/kotlinx/spark/api/SparkHelper.kt
Outdated
Show resolved
Hide resolved
kotlin-spark-api/3.2/src/test/kotlin/org/jetbrains/kotlinx/spark/api/ApiTest.kt
Show resolved
Hide resolved
… tests, removed streaming example
|
So, can I merge it? :) @asm0dey |
| inline fun <reified T> List<T>.toDS() = toDS(spark) | ||
| inline fun <reified T> Array<T>.toDS() = spark.dsOf(*this) | ||
| inline fun <reified T> dsOf(vararg arg: T) = spark.dsOf(*arg) | ||
| inline fun <reified T> RDD<T>.toDS() = toDS(spark) |
There was a problem hiding this comment.
That doesn't hold for all right? For instance primitives, are those Serializable?
There was a problem hiding this comment.
But there can't be generic with primitive inside
There was a problem hiding this comment.
Wait, so Int and other primitves that do not implement Serializable are allowed when having inline fun <reified T : Serializable> RDD<T>.toDS() = toDS(spark)? That's news to me. But useful!
There was a problem hiding this comment.
I believe they will be boxed to serializable Integer
There was a problem hiding this comment.
And I'll add it to broadcast too!
There was a problem hiding this comment.
no wait that doesn't hold. You can broadcast a non Serializable List for instance. And... you can also make an RDD of a non Serializable List
There was a problem hiding this comment.
List itself is not serializable, but its implementations are
There was a problem hiding this comment.
Ah right, but enforcing it to be Serializable requires users to for instance wrap their list like ArrayList(listOf(1, 2, 3)) which is not ideal...
There was a problem hiding this comment.
Nono, if "just" listOf doesn't work — let's not implement it.
| @@ -593,10 +598,63 @@ class ApiTest : ShouldSpec({ | |||
| it.nullable() shouldBe true | |||
| } | |||
| } | |||
| should("Easily convert a (Java)RDD to a Dataset") { | |||
There was a problem hiding this comment.
I think this huge test should be split into smaller ones?
|
You will hate me, @Jolanrensen, but I've added more comments :) |
|
Haha they're always welcome ;) |
Added kotlin official code style prop for later
|
Alright, after this is done checking I'll merge it. |
adds sc: JavaSparkSession to KSparkSession
adds (Java)RDD.toDS() functions
includes tests