LAE

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: zon

Hi,

I'm trying to load a file, that has 563.678 lines, but the node dellimited runs, and it only retrieves 76.366 lines.
I've check the file, there's no issue, the file is normal.

Why is that?
Has LAE reach some limit?
If so, how can I solve this?

Regards,
ZON

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: mlowe

Hi ZON,

This could potentially be due to escape characters that are not closed/escaped properly causing the reader to think the rest of the line is a record, including the carriage return, etc.

Try looking for records that have extra long fields or perhaps just the last record to try and narrow down the specific character/etc that is causing the issue.

Cheers
Mike

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: zon

I already thought about that, but is has nothing to do with the records.
In order to understand that, I deleted the first 76.366 lines of the file, to see if it wouldn't load.
But it loaded.

So, it has to do something with some limitation, memory issue (not disk space, because that I have, a lot).

Cheers,
ZON Optimus

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: stonysmith

To diagnose such issues, try another copy of the Delimited File node using \n as the field delimiter, and \n as the record delimiter.
This will tell it to read just one big field out of each record.

If you only get the 76k records, then something could be going wrong with the node.
But if you get all 563k records, then the problem is with the content of the file.

If you do get all 563k records, then use Head/Tail nodes to focus in on the records around 76366 - I would use Head=76370 followed by Tail=10

Then, you can look at just those 10 records and see if you can determine what might be going wrong.

===
The Delimited File node should not suffer from any memory issues unless the delimiters are messed up - it writes one output record for every input record, without queueing any of it into memory.

You also can add a second output pin to the Delimited File node - any badly formed records will be written to the 2nd (Error) pin.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: zon

I did what you said, and still I have 76366 records.
Can this have something to do with Heap Allocation?

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: zon

Help!
No one knows what's happening?
See my last post.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: Tim Meagher

Hey,

Just checking, the node runs and reports as having successfully completed right (i.e. green tick)?
If it was something with heap allocation, the node should fail with a memory/heap error.

When you tried with to run the node with a second output pin were any records on the second output?

Tim.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: zon

Hey,

Yes, green tick.
But previously, it gave that kind of error, about memory/heap.
But then I tried to change something in file ls_brain.prop, according to some posts here, but it didn't work. So I've put back the file ls_brain.prop as it was, and restarted the machine. The error about heap haven't return, but now it doesn't load all info.

I'm not understanding what you want me to do about the second output? Can you explain?

ZON Optimus

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: Tim Meagher

You should be able to just add a second output to the node.
Then, when you run the node, see if any records appear on that output.