LAE

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: kursattaskin

Hi All
I got such an error when processing 770million record. do you have any idea about that?

ERROR performing sort:
Error opening file: D:/BRAINscratch/tmp\brain\Planogram_MarketBasket\default\sort.miat sdtspell.5380.14736.1358248110688.brd
2013-01-15 21:02:51.000; WARN: ...exception seen
id: 0 chain: 0 group: 0
cppDetail: context: Salmon::NED::SimpleIo::FileInput::initialize() .\SimpleIo.cpp@170

2013-01-15 21:02:51.000; WARN: ...exception seen
id: 0 chain: 0 group: 0
cppDetail: context: Salmon::NED::SimpleIo::FileInput::FileInput() .\SimpleIo.cpp@149

2013-01-15 21:02:51.000; ERROR: Error opening file: D:/BRAINscratch/tmp\brain\Planogram_MarketBasket\default\sort.miat sdtspell.5380.14736.1358248110688.brd
id: 0 chain: 0 group: 0
cppDetail: context: openInputFiles() .\BrainSort.cpp@308

2013-01-15 21:02:51.000; WARN: ...exception seen
id: 0 chain: 0 group: 0
cppDetail: context: openInputFiles() .\BrainSort.cpp@312

2013-01-15 21:03:06.000; ERROR: ERROR performing sort:
Error opening file: D:/BRAINscratch/tmp\brain\Planogram_MarketBasket\default\sort.miat sdtspell.5380.14736.1358248110688.brd
id: 0 chain: 0 group: 0
cppDetail: context: Salmon::NED::NodeMain::errorMsg() .\NodeMain.cpp@28


the structure of the data as follow

Recno Liste
2 ('28091000', '30130327', '30580535')
2 ('28091000', '30130327', '37990020')
2 ('28091000', '30130327', '37990110')
2 ('28091000', '30130327', '37991012')
2 ('28091000', '30580535', '37990020')
2 ('28091000', '30580535', '37990110')
2 ('28091000', '30580535', '37991012')
2 ('28091000', '37990020', '37990110')
2 ('28091000', '37990020', '37991012')
2 ('28091000', '37990110', '37991012')
2 ('30130327', '30580535', '37990020')

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: stonysmith

Can you look in your folder "D:/BRAINscratch/tmp\brain\Planogram_MarketBasket\default" and count the number of files that start with "sort....brd"? Knowing how many of those files there are might help with diagnosing this.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: kursattaskin

there are 17.505 items that start with sort and end with brd.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: kursattaskin

Any update for this issue?
why it does not sort?

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: stonysmith

1) delete all those sort temp files (just to clean up things)
2) find the file {LAE_install_directory}/conf/brain/ls_brain_node.prop

Toward the bottom of that file you will find this line:
ls.brain.node.sort.maxBufferSize=30000000

Try changing that to:
ls.brain.node.sort.maxBufferSize=50000000

and see if it helps.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: rboccuzzi

I think Stony is correct, this is the cause. Under Linux or Solaris, you would generally get a "too many files open" error as seen in this thread, but I believe under windows you may not get as graceful an error, unfortunately.

So I think the answer is to do as he suggested...if the above number does not help, you can make it larger. If you read the other thread, on the second page, there is a good explanation of this number and what you are doing, hopefully that will help clarify as well.

Cheers
Rich

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: nnaemeo

Hi,

if a sort node fails while sorting over 500 million records due to some network disconnect, would it be possible to start it with the play button to continue from where it stopped or would it start right from the beginning?

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: stonysmith

Depending upon certain circumstances, the sort might continue to finish to completion. You should use the techniques above to see if the node finished.

Go to the Status tab of the failed sort node, make note of the Output temp file name, and see if you can find that file on the server, and the check the number of records in it. If it exists, you can re-read the sorted data back in with a BRDFile node.

Outside of that, you will have to restart the node, and yes, it always starts at the beginning.