LAE

Welcome to the LAE community!  Please feel free to start a discussion in the discussion tab or join in a conversation.

Discussions

Members

Resources

Events

 View Only

  • 1.  CSV Node Error

    Employee
    Posted 10-19-2015 07:58

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: jpstory

    Hi All

    I have a CSV node been working for the past months until today. I came across the following error message while trying to import 25 csv files via CSV node.
    "WARN: Node reported the following messages to stderr" "input buffer overflow, can't enlarge buffer becasue scanner uses REJECT"

    Each file has identical columns/fields and has around 1200 columns in it.
    There are some extra lines (File Generation Date, and Title etc.) before the column headers, thus CSV Node won't take the files directly.
    I had to populate all the field names (Account ID, Order ID, Product ID etc.)in the 'FileNames' section under 'Optional' tab in order for the node to work.

    Strangely, it works if I populate the field names as "Field1, Field2, Field3 ...... Fieldn"

    Any idea what might have caused this error, please kindly advise, thanks.


  • 2.  RE: CSV Node Error

    Employee
    Posted 10-19-2015 14:14

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: jpstory

    Hi

    Appreciate it if someone could please look at this.


  • 3.  RE: CSV Node Error

    Employee
    Posted 10-20-2015 06:16

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: lnason

    Can you clarify a few things?

    You say your file has 1200 columns? Did you mean to say rows?

    Also, you mentioned that the Input CSV node you are using has been working for a few months and that you are trying to import 25 files. Are these 25 files all new or is that they have been accumulated over the past few months? If they are accumulated, I'm wondering if you can recreate the error using just the latest file or two.

    In general I'd want to try and figure out if the node is failing on a particular file(s) out of the 25 files and then inspect the contents of the suspect files to see if there is some kind of anomaly that needs to be treated. You could use a HashSplit or a series of split nodes to do this assuming you are reading in a list of file names from a DirectoryList node.

    If there is any way to send some sample data along that could be used to recreate the error, I could take a look.

    Luke


  • 4.  RE: CSV Node Error

    Employee
    Posted 10-21-2015 06:28

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: jpstory

    Hi Inason

    Thanks for looking at this. Apologies, I can not provide any sample data as these are classified information.

    It's a very lengthy report with 1200 columns, not rows.
    I tried the graph on the old files (which the graph processed successfully last month with no issues) as soon as I ran into the error, unfortunately I got the same error message.
    Therefore I don't believe the issue is related to any anomaly within the 25 files I am trying to process.

    These 25 files contains exactly same column but with some daily updates.

    I tried one single file, but still got that error.

    Not sure if this rings the bell, when I input some dummy field names such as "Filed1,Field2,Field3....... Filed1200" it worked.... but when I input the real field name such as Client ID, Client Name etc. it would fail.


  • 5.  RE: CSV Node Error

    Employee
    Posted 10-21-2015 07:52

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: stonysmith

    In your original post, it says that you put the column names into the parameter of "Filename". I assume you meant "Fieldnames".

    The only reason I can think of for the out of memory error is if the CSV file is badly formed, such as a beginning double-quote that is not properly matched to an ending quote, etc.

    Always carefully read the error messages.. sometimes they change when you introduce new conditions. It might be that you're no longer getting the memory error, but instead you're getting a message such as "Found 1199 columns - expected 1200"

    =======
    Here's one thing to try.

    Take the file that is failing, and setup a Delimited File node to read it.
    Use FieldSeparator=\n and RecordSeparator=\n and Optional-FieldNames=Text

    This will read all the records and give you an idea how long the file is.

    Then, use a filter on that output with just this code:
    emit execCount as Rownum
emit strlen(Text) as Chars
    Then either sort by Chars or use the histogram node - see if you can find any records that exceedingly long (megabytes in length).

    If all the records look to be a reasonable length, then it's time to check for missing quotes.

    ======
    Again, add a filter off of the delimited file node and use this code:
    a1=strlen('Text')
t='Text'.replace("\"","")
a2=strlen(t)
emit execCount as RowNum
MatchedQuotes=((a1-a2)%2 == 0)
emit MatchedQuotes
where not MatchedQuotes
    Run that and see if you get any records out of it. If you do, then it's likely that you need to get the source files repaired.