LAE

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: sree

Hi Team,

how to eliminate duplicate record, source is xlxs file.
eg .

column1 column2 column 3
2164 5 66
1234 7 10
2164 8 65
----
----
----
2164 77 66


1.how to check if a record duplicate or not.
2.if a record is deplicate,how to send first record to one file and remaining records to other file
3. suppose in duplicate records,if suppose any date field there in those records,how to check which one latest created record based on date filed.

Thanks in advance for all your support.


Regards,
Sree.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: ejones

Hi Sree,

There are a few options you can use depending on what you want.

In the attached graph the first example uses a Sort and an Agg that splits any record that is not firstInGroup to the 2nd output.

The 2nd example uses a LAL1 node that just removes the dups. By default it removes using all columns but you can specify column1.

The 3rd example uses the core Duplicate Detection node and sends any records that are duplicates to the 2nd output. By default it will also error if a duplicate is found but you can turn that feature off. You could then use the nodes in the first example above to further process the duplicates and then use a Cat node to put them back together with the non-duplicates.
Attachments:
DuplicateHandlingOptions.brg