add monotonically increasing id and inputFileName functions #229
Conversation
Codecov Report
@@ Coverage Diff @@
## master #229 +/- ##
==========================================
+ Coverage 96.15% 96.16% +<.01%
==========================================
Files 52 52
Lines 937 939 +2
Branches 15 14 -1
==========================================
+ Hits 901 903 +2
Misses 36 36
Continue to review full report at Codecov.
|
| def prop[A : TypedEncoder](xs: List[X1[A]])(implicit x2en: Encoder[X2[A, Long]]) = { | ||
| val ds = TypedDataset.create(xs) | ||
|
|
||
| val result = ds.withColumn[X2[A, Long]](monotonicallyIncreasingId[X1[A]]()) |
There was a problem hiding this comment.
Would this work if we omitted X1[A] and just type monotonicallyIncreasingId()?
There was a problem hiding this comment.
No, and it's terribly frustrating. Being that monotonicallyIncreasingId doesn't take a TypedColumn[T, A] like so many other functions do, there's no way that I know of to infer the T needed for the methods on TypedDataset[T] that require the Ts to align, e.g. select, withColumn etc.
Any ideas on that? Very open to suggestions here!
There was a problem hiding this comment.
if you help the compiler by specifically adding .apply it can infer the type. I run into this few other times. For example, ds.withColumn[X2[A,Long]].apply(monotonicallyIncreasingId()) should work.
There was a problem hiding this comment.
The "problem" here is the existence of two overloaded methods withColumn. I am thinking maybe is a good idea to rename the withColumn that does replacement of existing column as withColumnReplaced. I tried it and with that the expression in question works without having to help the compiler with .apply.
There was a problem hiding this comment.
Agreed. I prefer renaming to withColumnReplaced over having to use types everywhere.
| @@ -462,6 +461,72 @@ class NonAggregateFunctionsTests extends TypedDatasetSuite { | |||
| check(forAll(prop[Int] _)) | |||
| } | |||
|
|
|||
| test("inputFileName") { | |||
There was a problem hiding this comment.
Didn't even knew this function exists 👍 . I know that adding them to terget means they should be cleaned up on sbt clean, but I think it would be better if we just delete the temp files on exit. What do you think?
There was a problem hiding this comment.
Agreed! I'll try to get on this sometime in the next few days. Holidays etc getting in the way. Done
frosforever
force-pushed
the
i#164-missing-functions
branch
from
b0a9ace
to
9d47027
Compare
|
@imarios could you do another review pass on this PR? |
Connects to #164