LAE

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: jpstory

Hi All

Certain lines went missing when I tried to import a text file (attached) into lavastorm with Delimited File ( used comma as Field Delimiter and \n as the Record Delimiter).
Here is the line that is missing: 1234 12345678:12:34ABCD000000012340

Also, it rendered me the same result when I tried the Fixed Format File node, the above line went missing.

Could you please help explain why this is happening? What would be the solution? Thanks!
Attachments:
Test.1234.txt

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: stonysmith

Sometimes, the problem is with the BRDviewer, and not with the data that the server actually processed.
Did you get a rowcount of 4 out of the DelimitedFile node?

Can you provide a bit more information about your environment?
Are you running the nodes on an enterprise server, is it windows or linux?

Sometimes there are issues with newlines and carriage returns, and the forum here may be altering the file you uploaded. We'd need to see the "raw" file.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: AdamParker

If you zip the file before uploading it, it should preserve the newlines/carriage returns.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: jpstory

Hi stonysmith

Thanks for the prompt response.

Yes, I got a rowcount of 4 out of the DelimitedFile node, the first line is blank in the data viewer ( but when I open up the file in the note pad or Excel, I could see that line isn't blank)


Here's the raw file - zipped
Attachments:
MyFile.zip

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: stonysmith

Okay.. line 2 of the file has a NUL (0x00) character in position one.
Unfortunately, our software has issues with that particular character. The data did not get read into the BRD temp file.

The question becomes:
- How often do you have to deal with such data?
- Is this a one time thing or do you need a repeatable solution?
- Can the NUL be simply removed?
- Is it possible that this is unicode data?


===============
One solution, if the data is stored on linux, is to use the TR program from a command line:

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: jpstory

Hi StonySmith

This is something I am facing on daily basis therefore I need a repeatable solution. Further, this missing line contains a piece of critical information to my process therefore can not be removed.
Unicode Date or not ? This is too technical for me to answer already.

Apologies for missing one of your earlier, I am running on Windows 7, not Linux.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: AdamParker

Stony's solution using the "tr" command doesn't remove the whole line (I don't think ***).

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: stonysmith

Correct.. the "tr" command would only remove the one NUL character.. not the whole line.
But. it's not going to work on Windows.

We'll have to seek out another solution.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: awilliams1024

Hi,

one solution is to use the embedded R functionality provided by the Power R node.

In the attached data flow, the Power R node in the composite reads in the contents of the specified file, ignoring any Nul characters. The R 'read.table()' function is configured to align with the original settings you had for your delimited node, i.e. to expect a header record and use the comma character as the field separator.


Read_Delimited_File_with_Nul_Values_Power_R_Node.brg

Regards,
Adrian

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: awilliams1024

If you don't have the node pack license for the Power R node an alternative would be to use the R node in conjunction with an Open Source R installation. However, this approach is a little more involved than simply running the Power R node.

You can get the R Node installer from our website:

http://www.lavastorm.com/current-rel...orm-libraries/

There are a number of prerequisites to using the R node (i.e. the installation of Open Source R and some CRAN library packages). These prerequisites and installation instructions are described in the R Node Release Notes which can be found by following the above link. The installers and release notes are also available for previous versions of Lavastorm here:
http://na3.salesforce.com/_ui/selfse...0150000000pH2G



Here is the data flow that provides the R Node version of the node:

Read_Delimited_File_with_Nul_Values_OS_R_Node.brg

The R Node in the composite is configured in the same way as the original Power R node. The R node leverages an external R installation - the R Node uses the default configuration to access the Rserve instance which assumes it is installed on the local machine, is listening on port 6311 and does not require authentication. If your Rserve instance is configured differently you will need to modify the appropriate properties on the R node. Obviously, the Rserve instance must be up and running - follow the instructions in the R Node Release Notes.

Regards,
Adrian