LAE

Welcome to the LAE community!  Please feel free to start a discussion in the discussion tab or join in a conversation.

Discussions

Members

Resources

Events

 View Only

  • 1.  Remove Duplicate, Tally through Multiple Files

    Employee
    Posted 03-03-2015 13:51

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: jpstory

    Hi

    I need to loop through 30 files for the records that meeting certain criteria and then remove duplicate on 'Model' field to find the file that has the highest count.

    I can do it on a single file (import data, and set up filter etc.), but could not figure out how to it on 30 files and keep track of which file has the highest volume.

    csv file, same format, received daily
    File Content/Format: Region:String (US, UK, HK), Product: String (Cellphone, Tablet, PC), Quantity: Int ( >=0 ), Producer: String ( A, B, C, D)
    Region:String Product:String Quantity:int Producer:String Model: String
    US Cellphone 119 A A1G356G33
    UK PC 176 B A1G356G34
    HK Tablet 199 C A1G356G33
    UK PC 56 D A1G356G34
    US Cellphone 61 B A1G356G37
    UK Tablet 121 C A1G356G34
    US PC 102 A A1G356G39
    HK Tablet 136 D A1G356G34
    UK Cellphone 113 D A1G356G41
    US PC 123 C A1G356G34
    HK Cellphone 135 A A1G356G43
    US Tablet 114 B A1G356G33


  • 2.  RE: Remove Duplicate, Tally through Multiple Files

    Employee
    Posted 03-03-2015 21:22

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: stonysmith

    Add a CSVFile node like below and connect it to your Directory List.
    It will read all 20 files, but will add the Filename to each record.
    Then, use the logic that works for one single file, and include "FileName" as part of your grouping options.


    node:CSV_File
bretype:core::CSV File
editor:sortkey=54f687d82c6e0ca3
input:54f687d827390083/=
output:@41e6c6cd11e613c6/=
prop:FilenameExpr=<<EOX
FileName
EOX
prop:FilenameOutputField=FileName
editor:XY=190,110
end:CSV_File


  • 3.  RE: Remove Duplicate, Tally through Multiple Files

    Employee
    Posted 03-09-2015 13:39

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: jpstory

    Hi Stonysmith

    Thanks for the response, I did not work. Not sure if I did it correctly, here is what I did:
    I dragged a 'CSV File' node onto the graph and then right clicked to open it in SciTE and replaced the existing codes with those you provided and save.
    However, when I re-open it, the codes reverted back to the inital... ... please kindly advise.


  • 4.  RE: Remove Duplicate, Tally through Multiple Files

    Employee
    Posted 03-10-2015 10:09

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: ryeh

    Even simpler - copy the code and paste directly onto the graph. It will create the node.