LAE

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: ryanherb

Hi there

I'm getting constant errors with the message "invalid number of columns in line X, expected Y, got Z". I'm not too sure why this is occurring. Perhaps it has something to do with my server configuration?

I'm sure it's not the data:
* I was working with it yesterday and had no issues.
* It's happening on simple aggs or filters well after the data is actually imported.
* I tried reimporting the data using the BRAIN 4 CSV node and outputting badly formed records to a second output - nothing came out.
* I tried uninstalling and reinstalling BRAIN 4 to no avail

Some info about the system I'm working with:
* BRAIN 4.1.4.2 running BRAIN 3 nodes from the basic library
* BRAINscratch directory is on a portable hard disk (F

Any urgent assistance would be greatly appreciated as I'm working to a very tight deadline and don't want to lose any more time on this.

Ryan

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: ryanherb

ERROR processing data:
invalid number of columns in line 887494: expected 14, got 13

Installing log (09/08/2009 15:24:18) ...
2009-09-08 15:24:18; Level: 0; Type: 0; Desc: "Node character set to 'windows-1252'"; File: "\src\brain\brain_4_1_4\ra_dev\code\source\brain\N odeControl.cpp"; Line: 692; Context: "Salmon::NED::NodeControl::getCharacterSetFromProp erties()"
2009-09-08 15:25:31; Level: 2; Type: 0; Desc: "invalid number of columns in line 887494: expected 14, got 13"; File: "\src\brain\brain_4_1_4\ra_dev\code\source\tableio \DelimitedLineReader.cpp"; Line: 82; Context: ""
Uninstalling log (09/08/2009 15:25:31) ...

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: ryanherb

Attachment added

Attachments:
Support.zip

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: mlowe

Hi Ryan,

Is it possible to see the 3 lines around the particular line referenced? i.e the line before 887494 (887493) and the line after (887495) including the line referenced.

I know you don't think there is an issue with the data but I'm just wondering if you are now referencing columns that weren't previously used.

Mike

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: mermacora

I'm a bit confused.
This error typically comes out of a Delimited File reader node when an input line is encountered with more field delimiters (e.g. commas) than there are fields defined in the first line of the file. Yet the attached screen dump indicates that it's the join node which has failed.
Please confirm exactly which node is failing with the error you've described.
The first thing I'd suggest you do is to replace the three "Convert Delimited" nodes with Brain 4 "Delimited File" nodes from the core library, making sure that the node parameters: Field Delimiter, Record Delimiter and Quote Characters are all are specified based on the format of source data you trying to import. E.g if the source data uses Unix style EOL markers, you need to specify "\n" as the Record Delimiter.
I hope this helps. Let me know how you get on.
Cheers,
Mario

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: ryanherb

Hi Mario/Mike

I was getting these errors in sorts/filters/aggs, and it never happened on any import nodes. Very strange indeed. I have reinstalled BRAIN 4 and set the BRAINscratch directory to be on my local disk and everything now seems to be working fine.

I'm guessing that it's something related to having BRAINscratch on a portable hard drive.

Is the ability to move/change BRAINscratch directories to different locations (in a easy/reliable way without having to hack the setbrainenv.bat file) being considered for future releases of BRAIN? It would definitely be worthwhile, as it would come in handy where you have data that's just too big for your laptop HDD and need to work off a portable drive.

Cheers - Ryan

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: rbeliz000

Was there a solution addressing this error?

I am getting the same error from an x-ref node, filter node, a head node for 148,865 records.
"-- Errors for Sort_2
WARN: invalid number of columns in line 148866: expected 5, got 3
Error Code: brain.node.DelimitedLineReader.cpp.82

ERROR: Node execution terminated while performing sort by error:
invalid number of columns in line 148866: expected 5, got 3
Error Code: lae.node.executionTerminated"

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: ejones

The solution above was to change to a more reliable disk drive for the BRAINscratch directory. Evidently there was an issue creating the output temp file cleanly and the next node that ran and read this temp file had issues. I have not yet heard of this issue occurring outside of a portable hard drive scenario.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: ejones

Oh. There is one other way I have seen this error. This is when reading the temp file while still being written by the node. This only works with a Linux or Unix Lavastorm Server. Here is how:

While waiting for a longer running node to complete and while it is still running.
1. Edit the running node and go to the Status tab. In the Log Files section identify the output corresponding to the output you are waiting on that is only partially produced. Copy the filename to the pasteboard. The name would look something like this, /home/lae/Lavastorm/LAE6.1/tmp/admin/unnamed/default/temp.unknown5ce0c5a1e208.attlocal.net.55828.11267. 1495561706487.592474e6008c6215.log
2. Add a new BRD File node to the graph and paste the filename into the File parameter.
3. Go to the Optional tab and set "NoWrite" and "NoRecordCounts" to true. (This NoWrite option does not do the same thing in Windows and is basically ignored)
4. Connect a Filter or some other node to the output of the BRD File node.
5. Run both nodes.
So:
If the file is still being written it is highly likely that a part of the last record in the file has not been completely written and, at least when I do it, I receive this error.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: ejones

I was just challenged by @stonysmith to try to find a way of doing this that he had seen before. The thinking is that if the data contains the tab character then it might throw the next node off. I was unable to reproduce that issue directly.

But I was able to create it when I created a bad BRD file. Note that BRD files are pretty much tab delimited files with the column data types specified in the first record with column names. So I created a bad tab delimited file and read it back in with the BRD File node. The BRD File node succeeded but the next node fails. Attached is my work producing this information.
Attachments:
badBrdFile.brg