LAE

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: Anusha.Gunasekaran

Hi,

I'm new to Lavastorm and I'm using Lavastorm BRE V4.6.1.0. I need to filter out items from a list that has different naming convention for the same item.

For Instance:
Original Item - "City of Toronto"
Duplicate Item- "Toronto City"

There are lots of instances like this in various combinations for the same item such as "10 dollars" (Original), "Ten Dollars" (Duplicate), $10 (Duplicate) and so on.

Is there any logic that can be used where Lavastorm displays these items (both Original and its Duplicates) separately from the rest of the list?


Thanks

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: stonysmith

The best thing to do is use a set of filters (or other tools) to standardize the data, then use the Duplicate Detection node to identify what is duplicate.
====
Assuming that you have a large amount of data, you can use the Histogram node to help identify the candidate values in any one column, and then use that information to help build the filter(s) you need.