LAE

Welcome to the LAE community!  Please feel free to start a discussion in the discussion tab or join in a conversation.

Discussions

Members

Resources

Events

 View Only

  • 1.  Removing unwanted characters from output file

    Employee
    Posted 06-29-2017 03:31

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: linnoinen

    Hi everyone,

    I'm tackling with a specific problem regarding an file output.

    I've connected two nodes to a file, which are collected together to form a file (using the CAT node).
    On node creates a custom header row, the other is the body of the file.

    3,3,X,,
Transaction1,1,X,,
Transaction2,1,X,,
Transaction3,1,X,,
    This file contains a header row, which has previously calculated the number of rows below (3), the sum of the second column (3) and then a flag for an indicator (X).
    Transaction rows below the header contain the transaction number (Transaction#), a value (1), an indicator flag (X) and a blank column.

    The system this file is being read into requires that the first row does not have this blank column.

    I have two options on how to output this file:

    1) As the file is in CSV format, I can manipulate the indicator flag column to include the commas and drop the blank column: "X,,"
    The problem with this is that as my delimiter is a comma, this column will be escaped by quotation marks, which I would need to remove.

    2) Create the file as in the above example and then run a node to postprocess the file, and remove the two commas from the first line of the file.

    I have a snippet of Python code which should do the job for option 1, but I'm unsure of which Python node to use. Also, where should the code be inserted in the node?

        with open('c:\tmp\test.csv', 'r') as infile,
        open('c:\tmp\test_mod.csv','w') as outfile:
            data = infile.read()
            data = data.replace("\"","")
            outfile.write(data)
    Any suggestions are more than welcome!

    /Karri


  • 2.  RE: Removing unwanted characters from output file

    Employee
    Posted 07-03-2017 02:18

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: awilliams1024

    Hi Karri,

    I was a little unsure about what you were trying to achieve so I may have misunderstood.

    I think using the 'Python' node could be a bit of overkill for your use case, you should be able to process the data using other standard nodes.
    Also, the snippet of code you suggested would read the entire file into memory which could cause an out of memory condition if your file was large enough.

    Attached is a data flow that shows how the data can be read in from your temporary file using the Delimited File node which is configured to read in the entire record as a single field by setting the delimiter to a value that will not appear in the data (Hex '07' in the example). The name of the output field is set using the 'FieldNames' property on the Optional tab. The Filter node uses a regular expression to strip of the trailing commas from only the first record of the data - if you wanted to replace the commas in all records then the first line would need to be just:

    tmpData = {Data.regexSubstitute(",,$","")

    The Output Delimited node is then configured to write the data to a file without writing the header record (meaning that your custom header record becomes the first line in the file).


    For completeness, the lower section of the example shows the operation of the nodes using sample data from a Static Data node rather than a file.

    Modify_Custom_Header.brg

    Regards,
    Adrian


  • 3.  RE: Removing unwanted characters from output file

    Employee
    Posted 07-03-2017 06:44

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: linnoinen

    Hi Adrian,

    Awesome! Thanks!

    /Karri