LAE

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: mikewillett

Hi there,

I've got this error message when reading in some tab delimited files. It certainly must be specific to a select number of records as the node runs for a little while and then errors out.

No doubt there is something unexpected in the files I am reading in but as I'm reading in 2.5M rows, how can I go about either isolating the records that are in error (put them to a different output pin) or find them in the dataset so I can fix them directly in the files?

Apologies if this question has been asked before but my quick search of the forum couldn't find an answer for this one.

Mike

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: rboccuzzi

Mike, have you tried creating a second output pin? On the delimited node, if it can't process records, it will error...unless you create a second output. Then it will attempt to recover and put those records out on that pin.

Cheers
Rich

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: mwillett

Hi Rich, I think I thought I'd done that but perhaps I've not done it right.

Assuming I have all the workings for output 1 and am ok on those, can you confirm how I need to confirm this for output 2 (the error pin)?

Mike

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: mwillett

Let me quickly clarify the above - the error is not on the delimited node (this works fine) - rather on a later filter note where I'm changing formats of the data, doing some preparatory calculations etc.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: rboccuzzi

Sure. You should be able to go into the node editor, click on the IO tab, and then click the duplicate button on the right side, next to the outputs list. This should add a second output to the delimited node, which should contain all recoverable errors on your next run.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: rboccuzzi

Mike, can you provide a small (obfuscated if necessary) example of what you are encountering? Ideally, the Delimited file should detect this.

Another way that might tell you how to find the error, right click on the input pin of the node that is failing, and select "view all records". This will pull all the records into the brdViewer (which will take a while), but should tell you all the records it can't read properly when it loads. What version are you using?

Cheers
Rich

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: mwillett

Sure, let me post this in the next day or so.

I just ran the "view all records" so can't check my version but think its 4.1.5 (?)

Thanks,
Mike

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: rboccuzzi

Ok...there were some enhancements done to the viewer that provide better feedback as to which records might be flawed and where...so your version might not have that, as it is a bit out of date...unless you mean you are 4.5.1, not 4.1.5, and if that's the case, you should be fine

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: mikewillett

Thanks Rich, I've split the incoming files and run them all through and then "cat" them together and for some reason, this seems to work - even though there is no change in the underlying file itself.

I'll have a chat to the local MDA here team but let's close this one off for now.

Mike