LAE

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: sdb

I would like to avoid creating files on the server in a multi-user environment, because there is no easy way to determine which files are still in use, and manage the disc space.
Furthermore, and especially the case, it is impossible to audit the server held data: we keep our data (i/o) under revision control on a web file system that also makes workgroup sharing and metadata processing easily accessible.

So such things as manual file transfer, and ftp are not helpful. It is also quite painful and unhelpful to force unnecessarily the conversion of flat files to/from a networking database, without losing much of the above facility or forcing its reinvention elsewhere.

There are several situations where the contents of a file (which can be csv, xls in particular) are already present on an output pin (web file get, email mime attachment). There is even an Input Raw node that produces similar. While JSON and XML could process and make sense of certain structures, unfortunately the Fixed, Delimited, CSV and Excel acquisition nodes can only read from a file, not from a field.

Without writing all these instances to a temporary file, how can the data be processed directly in LAE?
If it has to be a temporary file - how may it be cleaned up without losing state?

Simon

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: stonysmith

I have a node that downloads an XLS excel file from a remote (secured) web server.
In my HTTP node, I select

DataOutputMode=File
DataOutputField=FileName

the output file is readable, but the filename format itself is undesirable.

I simply send the output from the HTTP node into a filter and use moveFile() to rename it.

It would be just as easy to feed the FileName into the Excel Reader, and then use deleteFile() in a filter to remove it once it's been read.
Simply clock the delete filter after the excel reader, and the interim file would not be delete until the Excel node has read it.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: sdb

Stony

Thanks for the response. It seems then that the only way is to create a temporary file for the acquisition node to read and then delete.

What would it take to have the acquisition nodes read data from the pin rather than a temporary file store?

Simon

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: Tim Meagher

Hi Simon,

For a lot of the nodes that have been around for a long time, the options to process the data from an input field doesn't exist.
For most of the newer acquisition nodes (like XML Data, JSON Data, etc) they have been written to handle data from an input field.

For the Excel File node, it probably would not be too difficult to add an option to process the data from data in an input field rather than only allowing it to specify a filename field.

A lot of the other older nodes utilize a legacy C++ node architecture which is more difficult to modify to use this filed-based processing approach.
There are plans on the roadmap to migrate all of these nodes from the legacy node architecture and at that point, we may end up also allowing for this field-based processing option.

For delimited and CSV files, I've written a (unsupported, very minimally tested etc and a lot slower than the Delimited File/CSV File nodes) prototype node that overcomes some of the limitations of the Delimited and CSV File nodes regarding character set support and also supports processing the data directly from an input field.
See the post here: http://community.lavastorm.com/threa...g-unicode-data

Note: If you are not a member of the "Lavastorm Labs" group, you'll need request access. To request access, you need to go to your User CP, and select Group Membership, and request access from there.

I've logged the ability to process data from input fields as an enhancement request in our issue tracking system - as issue LAL-1548.

Regards,
Tim.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: Tim Meagher

Hey,

Attached is an example of an Excel File node that can be used to process data from an input field.
It's pretty quickly hacked together and just a prototype, so not supported etc etc.
However, I think this is more or less what you were after for the Excel side of things isn't it?
It just wraps the existing Excel File node, but instead of specifying a Filename or Filename Field, you can specify Data Field, Filename Field or Filename.
When Data Field is used, then you specify the name of the input field containing the data to process.
Since this will not normally be easily encoded in a field within the LAE, it also allows for the data to be base64 encoded.
When Data Field is used, the Base64DecodedField parameter on the Optional tab can be used to set whether or not the data on the incoming field is Base64 encoded and requires decoding (default true).
This parameter is ignored if the Excel Data parameter is not set to Data Field.

In the case that you've specified Base64DecodedField (or not set it to false), it will also check whether or not the data field contains the "<<@application/base64@>>" prefix (used by Input Raw and other nodes to denote base64 encoding) and will handle this appropriately.

In the graph is an example of how it could be used (although a not particularly useful example), where an excel file is simply read by an Input Raw node (you need to enter the filename in the "File" parameter for this to work) and then processed by the Multi Excel File node.
I'm Not sure why I called it Multi Excel File... Now that I think of it, the name doesn't make much sense.
Oh well it's just a prototype (and there are probably bugs and issues with it).

Regards,
Tim.
Attachments:
MultiExcel.brg

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: links123

It wraps the existing Excel File node, one can specify Data Field specificly. The name will be specified at the time of input field containing data to process.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: zeeshan.sabir

Hi Tim,

I have a BASE64 encoded data field that i want to decode directly without using any temp files etc. Can you please help out?

Regards
Zeeshan