LAE

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: spalluru

Hi Team,
May I know the meaning of parameters ls.brain.node.javaMaxHeapSize vs ls.brain.server.javaMaxHeapSize
We are getting No space left of device error while excel node writes to a file system. Will increase in the value help in resolving the issue ?

I am sure that we have got enough free space on file system when this graph runs.
Looking for suggestions on resolving this issue ? appreciate your quick response on this.

Also note we are running version 4.6.1 on VM SuSe Linux verion 11,patchlevel 2 64 bit version

Present heap/buffer settings on my system are as below

1)
>> /opt/lavastorm/lae/4.6.1/bin/linux-x86-64/getProperties -d | grep heap
2)
>> /opt/lavastorm/lae/4.6.1/bin/linux-x86-64/getProperties -d | grep buffer
ls.brain.node.sort.maxbuffersize = "104857600"
ls.brain.node.lookup.maxbuffersize = "1073741824"
ls.brain.node.join.maxbuffersize = "1073741824"
lavadmin@lavasb02:/opt/lavastorm/lae/4.6.1/conf/brain>

-Sreekanth

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: xathras

Sreekanth,

ls.brain.node.javaMaxHeapSize is the maximum JVM size for a specific node
ls.brain.server.javaMaxHeapSize is the maximum JVM size for the server. I believe by default its using 1Gb

How big is the excel file you're writing?

Wayne

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: spalluru

Hi Wayne,
xls file size is 1.6GB , our problem got resolved after increasing /tmp filesystem to 4GB.
I see that lavastorm is temporarily using /tmp to create the intermediate xml file as part of output excel node and later the actual xls file will be copied
to designated location.
Is there any way we can direct lavastorm server to use LAE temp location rather than OS /tmp location for such operations ?

-Sreekanth

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: xathras

Hi Sreekanth,

In Path /opt/software/lae/4.6.0/conf/brain, replacing /opt/software/lae/4.6.0 with your install path, locate file ls_brain.prop. What is the value of the following properties:


ls.brain.tempDir= "/work/tmp"
ls.brain.outputTempDir= "/work/tmp"
ls.brain.localTempDir= "/work/tmp"

Wayne

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: spalluru

Hi Wayne,
Above suggested parameters are set to a dedicated temp location ( not /tmp) see below , but still output excel node writes into /tmp
Any Idea why ?

To be more specific output excel node write intermediate file in format of xml and after output excel node completes writing the file then the final
file in xlxs format will be copied the designated destination. Since xml file is more than that of our /tmp location we are getting this issue

Appreciate your quick response

ls.tempdir = "/export/gcs1/lavasb02/lae/tmp"
ls.brain.logtempdir = "/export/gcs1/lavasb02/lae/tmp"
ls.brain.localtempdir = "/export/gcs1/lavasb02/lae/tmp"

-Sreekanth

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: xathras

Hi Sreekanth,

I tested by created a huge data output and get the same issue. It appears a large file in tmp is created and removed once the node is complete

However, after looking at the release notes and node help this is called out:

NOTE: If using a template input file, this should contain a very limited amount of data and the size of the file should be relatively small. Using a large template file may result in the node running out of memory as all template data needs to be loaded into memory.

My recommendation would be to output to CSV or split the output over multiple CSV files.

Also, contact one of the Lavastorm resources to see if the functionality is planned to change

Wayne

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: DanRosenblatt

Originally posted by: xathras
					

 contact one of the Lavastorm resources to see if the functionality is planned to change
			

I can confirm that we have identified this as an issue with the Excel File node when a template file is used, and we currently plan to address it in a future release of the Lavastorm Analytic Library (LAL).

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: spalluru

Thanks for the confirmation

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: svimes

Hi has this been fixed? getting this on a 4MB excel file, 13k rows only.

if i reduce the number of rows it goes through fine. temp directory has 1 GB of space...