Null handling in some transformers. #1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Kafka source create df if new data arrive Signed-off-by: David Ignjic <[email protected]>
redsk
reviewed
Aug 23, 2018
|
|
||
| val linearCombination = udf { | ||
| (linearWeights:Row, values: Seq[Float]) => { | ||
| (values, linearWeights.getSeq[Float](1)).zipped.map(_ * _).sum / linearWeights.getFloat(0) |
Collaborator
There was a problem hiding this comment.
So what is the problem here? that linearWeights is shorter than values? In that case I think it should be addressed before calling the udf as it might drive you to believe this is behaving correctly when it's not.
Collaborator
Author
There was a problem hiding this comment.
Problem is that i want ignore null columns and also remove weights of it
Signed-off-by: David Ignjic <[email protected]>
Signed-off-by: David Ignjic <[email protected]>
Signed-off-by: David Ignjic <[email protected]>
Signed-off-by: David Ignjic <[email protected]>
redsk
force-pushed
the
master
branch
3 times, most recently
from
965de82 to
885c872
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Kafka source create df if new data arrive
Signed-off-by: David Ignjic [email protected]