LAE

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: zon



Hello,

I have a filter node, that always worked, doing a split.
Due to some issues in the original file, some rows have less columns that they should.
So the filter node is returning error saying he can't finish processed the node because he found a line with less columns that it was expecting.

How can I exclude those lines, and continue the processing?

Cheers,

ZON Optimus

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: ejones

You can get the number of entries in an array doing using the len() function. So, maybe you can put the processing inside an if statement and only process results where the split() function returns a large enough number.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: cpeckham

Another option would be to add a filter before yours which adds a dummy field called "columnCount" (int) or "columnFlag" (boolean) and populate that field appropriately. Next, split the file based on that output into valid (post process with your filter) and invalid entries (return to them later to figure out why).
Regards,
Charles

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: zon

How can I populate that field with the number of columns?


The erros is:

ERROR processing data:
invalid number of columns in line 7129393: expected 25, got 6

Installing log (05/15/2014 15:11:25) ...
2014-05-15 15:14:49; Level: 2; Type: 0; Desc: "invalid number of columns in line 7129393: expected 25, got 6"; File: ".\DelimitedLineReader.cpp"; Line: 82; Context: ""
Uninstalling log (05/15/2014 15:14:53) ...


I found a previous post, and I hope what the end of this post says isn't true, because is my case:
http://community.lavastorm.com/threa...itedLineReader

Is says that the error could have something to do with the fact that BRAINScratch is in a portable drive.
Do you confirm?

Regards,
ZON Optimus

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: ejones

Hi Zon,

This sounds like a different problem than I originally understood. It sounds like the input data to the filter has gotten messed up.

In the context where I've seen this error, it was clearly the last record in the file that was messed up because the file was truncated at that random point. In other words, the last record in the file was truncated and the records that should have followed it were not in the file at all.

So this makes me wonder, did the disk drive where BRAINScratch is located fill up? But this usually would cause the node before it to fail because it is unable to write the complete file. Maybe with a portable drive the OS made a mistake and didn't cause that error, not sure though.

One thought is that if you can edit the input data using a good text editor and remove line 7129394 which I'm expecting is the last line in the file, the filter would then work.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: cpeckham

columnCount = datafield.split(",").len()

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: zon

Hi ejones,

My portable drive has a lot lot of space. Its strange because I've run several nodes, with no problems.
But then, is this one node, turns this error.

I'm enable to open the file and remove the line. This info comes from 279 files (with header and footer), put together.
First I have a node to gather all the info into one, then I take out headers and footers, after I split the records, treat the info (conversions), test for duplicated records.
And when I try to split the records, that's when the error comes.

Why it didn't come in the other nodes, and came now?

There's a node that allows me to retrieve data from the first rows, and another for the last rows.
Is there a way to do the same, but in the middle of the rows? Like "from row 100 to 150"?

Regards,
NOS

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: stonysmith

Add a filter with this code:

c=count()
emit c as row1
emit *
where c > 99
if c>=150 then consume(1,true)

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: zon

I tried, but not even that he allows me to do.
It turns out the same error, because that one line with the problem is in the middle.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: zon

I need help.
This clearly looks like a problem with the fact that I now use a portable disk for BRAINScratch.
Can anyone help me solve this?

Cheers,
NOS

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: ltolleson

Sounds very similar to this issue

http://community.lavastorm.com/threa...light=portable

My guess is that if some nodes run, but others don't the issue is a latency or even dropped connection between your PC and the external drive.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: zon

And what's the solution?
Local drive only?

The portable disk is where I have more space to keep the files

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: ltolleson

Zon,

LAE was not designed to be fault tolerant with external/portable devices plugged into a desktop/laptop via a USB port and we do not officially support these devices although this configuration does work most of the time. I have tried to repeat the error you are experiencing, but have not been able to repeat this issue. My suspicion is that the USB connection to the drive is getting dropped or is timing out, which is causing the data to not write completely to the drive. It could also be the speed of the connection, the USB driver or drive itself that is causing this issue.

Here are a few things that you should look for...

• What version of USB are you using. If you are using USB 2.0 you could try to upgrade to USB 3.0.
• Upgrade any USB drivers to the latest.
• Scan your computer for any virus or malware that could be affecting the performance of your computer

Open your "Device Manager" and look at the properties for each USB device or USB Hub. If there is a "Power Management" tab make sure the "Save Power" option is disabled. See the attached screenshot. Make sure that option is not checked.

Screenshot 2014-05-22 10.59.30.jpg

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: zon

Hi,

I try to run the process all over again, and it turned out error in a previous node, that in my first run worked fine.

So I checked what you said and:
I'm using USB 3.0.
This is a new laptop, so its running the latest drivers.
I have a antivirus running on my PC constantly, so there no virus or malware.

Also, I've open Device Manager and unchecked all options for Save Power.

After that, I try to run the process all over again, but it turned out the same type of error in a different node, again.

I don't know what else to do.
My portable disk has a cable to connect to my laptop with two USB's (due to energy).
Randomly, nodes fail...it doesn't make much sense.

I'm seeing that the only way is to put BRAINScratch in local drive.

Regards,
NOS

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: stonysmith

Does the drive have an option for using external AC power?
Also check your Power Setting options - windows may be shutting down the drive to save power. Try setting it to "never turn off drives"

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: ltolleson

If you have access to a different drive you might try to switch external drives and see what results you get. I have a 64GB USB thumb drive that I'm testing with and it seems to be working fine. I guess it could be the drive itself.