Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for explode map #383

Closed
asm0dey opened this issue Jul 29, 2019 · 5 comments · Fixed by #488
Closed

Add support for explode map #383

asm0dey opened this issue Jul 29, 2019 · 5 comments · Fixed by #488

Comments

@asm0dey
Copy link

asm0dey commented Jul 29, 2019

Currently, it looks like that explode on Maps won't work. Kindly add support

@OlivierBlanvillain
Copy link
Contributor

Is this supported by Spark? If so, it should only be a mater of adding an implicit for Map alongside the other ones:

trait CatalystExplodableCollection[V[_]]
object CatalystExplodableCollection {
implicit def explodableVector: CatalystExplodableCollection[Vector] = new CatalystExplodableCollection[Vector] {}
implicit def explodableArray: CatalystExplodableCollection[Array] = new CatalystExplodableCollection[Array] {}
implicit def explodableList: CatalystExplodableCollection[List] = new CatalystExplodableCollection[List] {}
implicit def explodableSeq: CatalystExplodableCollection[Seq] = new CatalystExplodableCollection[Seq] {}
}

@asm0dey
Copy link
Author

asm0dey commented Jul 30, 2019

Of course it does. It produces 2 columns: key and value (comparing to existing implementation)

@OlivierBlanvillain
Copy link
Contributor

Cool :) If you want to give this a try it should be fairly straightforward first contribution!

@asm0dey
Copy link
Author

asm0dey commented Jul 30, 2019

I'm absolutely have no idea on how to implement this: level of knowledge about spark is too low

@ayoub-benali
Copy link
Contributor

@OlivierBlanvillain CatalystExplodableCollection has the same issue as CatalystSizableCollection
It has only one hole for the type while Map needs two.
The easiest fix I see is to implement a second explode that accepts a Map instead of a CatalystExplodableCollection. Just like I did here #255

If that works for you I can make a PR

ayoub-benali pushed a commit to ayoub-benali/frameless that referenced this issue Jan 20, 2021
pomadchin pushed a commit to ayoub-benali/frameless that referenced this issue Nov 8, 2021
pomadchin pushed a commit to ayoub-benali/frameless that referenced this issue Nov 8, 2021
pomadchin pushed a commit to ayoub-benali/frameless that referenced this issue Nov 11, 2021
pomadchin pushed a commit to ayoub-benali/frameless that referenced this issue Nov 11, 2021
pomadchin added a commit that referenced this issue Nov 27, 2021
* Allow Frameless to explode Map[A,B]

Fixes #383

* Fix explodeMap implementation and add explodeMap tests

* Change implicits ordering in explode functions

Co-authored-by: Grigory Pomadchin <gr.pomadchin@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants