LAE

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: kursattaskin

Hello everyone

Do you have any previously developed data quality and cleansing algorithms on hand?

let me give some bullets

1- I have a lazy word list and I want to remove them from the text fields that I choose
2- Full profiling of all fields, duplicates, null, frequency, partially duplicates etc.
3- check all words of one address with other and find a scores inside a field

any idea will be appreciated
Kursat

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: rboccuzzi

Hello Kursat

So we have some nodes that could help these things, but nothing specific that does exactly what you are looking for. You can do most any of this using our foundation nodes or possibly with some assistance using the interface nodes such as Python, R, or Java. I don't think you would need to, generally, however.

For 1, if the list of words is known a priori, you could probably use Replace Text Rules, and enter them as part of the configuration of the nodes. If the words can be entered as part of the configuration of the node, certainly. If the list is coming from some other data source, then you would probably need to create a node using the above interface nodes mentioned.

For 2, we have a few nodes that can help, the Data Analyzer node in core, and the Quick Stats node in LAL. Those should give you a start on what you want, and probably with a composite containing those nodes and perhaps another could of foundation nodes, you could add the full set you are looking for?

For 3, we don't have exactly what you describe, but we do have the Fuzzy Join and Fuzzy X-Ref nodes which I think could be similar to what you are describing and certainly designed to help out with what sounds like the problem you are trying to solve.

And of course, anything you build along these lines to further enhance your user experience, I would be certainly interested in seeing if you are willing to share. They could become candidates for future LAL inclusion, if you don't mind.

Cheers
Rich