spark 3.4 support - replacing dataTypeFor logic #698
Comments
|
adding to a ~/.sbt/repositories above maven-central is enough to reproduce along with this changed sbt. I'll attempt to verify if just adding the previous impl back is sufficient next week but I'll happily look at other alternatives. |
|
Commit that removes the function: apache/spark@030c1ba#diff-aa90bc536778701618f741339d52ce249bb50f808b138e96a481f2b31b462b65L69-L116 (Leaving it here so we have more context) |
|
MapGroups is a change from after 3.3.0 introducing order from SPARK-38591 |
|
managed to sneak in some time over lunch - copying the small bits of impl over works fine against the test suite and Quality's tests where I detected it. The branch on my fork has the impl, but clearly the MapGroups constructor needs a work around for cross version and although an empty ordering "works" I've no idea if it's meant to. I'll not easily be able to test on databricks until there is a published snapshot, but checking the MapGroups constructor shows 5 params e.g. MapGroups(null, null, null, null, null)(null, null, null) compiles but 4 params doesn't and shows the dataOrder param. There may be other dragons there but at least these two are consistent with 3.4. I'll see if I can get a nice cross compile support for MapGroups next week. |
|
per 1186e5b I've added changes to manage the MapGroups across versions - verified against 3.4 RC5. Also tested with Quality locally. Per the rc5 vote it's likely we'll see at least another RC. I'll update and test again against them as they drop. Should I raise a pull now, knowing it'll only fail until after 3.4.0 is officially published? |
|
Hey @chris-twiner why not? If an open PR hanging around is not bothering you, it could be a nice reminder for us. |
|
Awesome progress you've done btw 🎉 I'm super happy to see it. |
|
more than welcome, #699 is raised for it. |
…arrays are derived in TypedEncoder so no need for that either
…re for the avg error
…han tests, and attempt to remove that from coverage
* #698 - build changes to see 3.4 errors * #698 - copied over dataTypeFor impl and empty ordering 'works' * #698 - split out mapgroups impl * #698 - regen workflows * #698 - cross build working * #698 - bump to rebuild as 3.4 is out * #698 - regenerated * (cherry picked from commit 31c8db5) * #698 - regenerated has windows path for mdocs * #698 - work with new 3.4, needs cross testing * #698 - kmeans only works locally, extra info for CI builds * #698 - tidy up before pull * #698 - fix rlike test, removing unused code from spark c+p, arrays are derived in TypedEncoder so no need for that either * #698 - jumped the gun on mima version checks * #698 - jumped the gun on mima version checks - with 'column' method ignore * #698 - should also be current version * #698 - clearly the wrong thing to do, reverting * #698 - tell mima to look for a different name * #698 - tell mima to look for a different name * #698 - retry x times if the array error occurs, even more rare for the avg error * #698 - regex is a, not b... * #698 - refactor to be minimal change at import only, other than tests, and attempt to remove that from coverage * #698 - force testing of code gen on DisambiguateLeft/Right * build.sbt cleanup * regexp code cleanup --------- Co-authored-by: Grigory Pomadchin <grigory.pomadchin@disneystreaming.com>
ScalaReflection.dataTypeFor has been completely removed in 3.4 rc4 (and seemingly in Databricks 12.2 as well) assumed as part of SPARK-41045. MapGroups also has a change but I've not yet tracked that down.
The text was updated successfully, but these errors were encountered: