LAE

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: sdb

Using an Output Delimited node to write records out as a text file, with record delimiter = \n, does not do the job correctly.
The problem is what it perceives to be escape characters - possibly because the node is biased towards CSV output.
For example a newline embedded in a record (or a double quote) triggers quoting the whole record - pretty useless if one is trying to write out records as a shell script.
Turning off quoting then requires an escape character to be defined, which has to be something other than whitespace. So this does not work.
How can one write out records as-is without quoting or escapes?
Alternatively, I have used a Reduce to List with \n delimiter and followed by an Output Raw, but this is not very attractive, and also seems to trigger unexpected EOF on occasion.
How can one write a simple text file???

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: stonysmith

Have you tried the Output Raw node?
You may still have to use the ReduceToList to get what you want, but it may do what you need.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: sdb

Like I said in my original post
"I have used a Reduce to List with \n delimiter and followed by an Output Raw, but this is not very attractive, and also seems to trigger unexpected EOF on occasion."
Output Raw will strip newline from the end of each record, and this causes unexpected EOF and upsets processes like 'diff' etc.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: belliveaujd

I'm not quite sure how you've structured your data flow, but I've had success in the past in putting the file contents into one column and then outputting to delimited. I use a delimiter I know will not exist in the data (e.g. *) and \n for the record delimiter. See example below:

node:Static_Data
bretype:core::Static Data
editor:sortkey=568eb9184ead2af5
output:@40fe6c55598828e5/=
prop:StaticData=<<EOX
batchFile:string
net start LavastormH2Database6.1-9961
net start LavastormJettyServer6.1-8861
net start LavastormLAEServer6.1-7761
EOX
editor:XY=110,110
end:Static_Data

node:Output_Delimited
bretype:core::Output Delimited
editor:sortkey=568eb93b623d2520
input:@43c680084b824568/=Static_Data.40fe6c55598828e5
prop:FieldDelimiter=*
prop:FieldNames=None
prop:File=C:\Users\jbelliveau\Desktop\Temp\TestTxt Output.bat
prop:RecordDelimiter=\n
editor:XY=220,110
end:Output_Delimited


The output file then contains the expected text, e.g.:
net start LavastormH2Database6.1-9961
net start LavastormJettyServer6.1-8861
net start LavastormLAEServer6.1-7761

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: sdb

OK. Lavastormers - here is your challenge.

I would like to see the first graph that can reproduce the example input file Input.txt and write it to a new file Result.txt, that is identical character by character in all respects. The judge of that is the unix diff command.

Should there be more than one way (I reckon there is zero), then the tie-break will be the graph with the fewest nodes, and then the fewest lines of script. Python, Java and R are cheating.

Basically either this issue screams bug, or over sight in terms of the capabilities of the core library acquisition and publishing nodes.

Here is the file input.txt:

input.jpg

#!/bin/sh
echo "this is a statement output over\nmore than one line"
# a comment
non-shell compliant text but plain, and simple!@~:}{][!"�$%^&('#<>.,?!

echo "above there is a blank line"
echo 'this line uses single quotes'
echo 'this line is indented and \ttabbed'
echo 'this is the last line of the file terminated newline, EOF'

~
~



And my example graph - Good Luck - Simon

node:Delimited_File
bretype:core::Delimited File
editor:sortkey=568fb4f803990ee0
output:@41e6c6cd11e613c6/=
prop:FieldDelimiter=*
prop:FieldNames=line
prop:File=/tmp/input.txt
prop:RecordDelimiter=\n
editor:XY=450,200
end:Delimited_File

node:Reduce_To_List
bretype:core::Reduce To List
editor:sortkey=568fb9e61f4a08d9
input:@3fba65925f181f9b/=Delimited_File.41e6c6cd11e613c6
output:@3fba65920bf11dfc/=
prop:Delimiter="\n"
prop:GroupBy=<<EOX
1
EOX
prop:ItemField=line
editor:XY=550,320
end:Reduce_To_List

node:Output_Delimited
bretype:core::Output Delimited
editor:sortkey=568eb93b623d2520
input:@43c680084b824568/=Delimited_File.41e6c6cd11e613c6
prop:FieldDelimiter=*
prop:FieldNames=None
prop:File=/tmp/result.txt
prop:RecordDelimiter=\n
editor:XY=640,200
end:Output_Delimited

node:Add_final_new_line
bretype:core::Filter
editor:Label=Add final new line
editor:sortkey=568a6f7a44950f50
input:@40fd2c74167f1ca2/=Reduce_To_List.3fba65920bf11dfc
output:@40fd2c7420761db6/=
prop:Script=<<EOX
new_line_list = 'line-list' + "\n"

emit new_line_list as "line-list"

EOX
editor:XY=660,320
end:Add_final_new_line

node:Add_final_new_line_2
bretype:core::Filter
editor:Label=Add final new line
editor:sortkey=568a6f7a44950f50_2
input:@40fd2c74167f1ca2/=Add_final_new_line.40fd2c7420761db6
output:@40fd2c7420761db6/=
prop:Script=<<EOX
new_line_list = 'line-list' + "\n"

emit new_line_list as "line-list"

EOX
editor:XY=770,320
end:Add_final_new_line_2

node:Add_final_new_line_3
bretype:core::Filter
editor:Label=Add final new line
editor:sortkey=568a6f7a44950f50_3
input:@40fd2c74167f1ca2/=Add_final_new_line_2.40fd2c7420761db6
output:@40fd2c7420761db6/=
prop:Script=<<EOX
new_line_list = 'line-list' + "\n"

emit new_line_list as "line-list"

EOX
editor:XY=880,320
end:Add_final_new_line_3

node:Output_Raw
bretype:lal1::Output Raw
editor:sortkey=568fb9640b556b22
input:@5106ebc83f035045/=Add_final_new_line_3.40fd2c7420761db6
output:@4ffae39e126d75ee/=
prop:DataField=line-list
prop:Directory=/tmp
prop:File=rawresult.txt
prop:FileExistsBehavior=Overwrite
editor:XY=990,320
end:Output_Raw

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: stonysmith

If you could, please attach the sample file as an attachment... the forum removes some characters such as the tab.

I tried it with just the three nodes:
- Input Delimited
- Reduce to List
- Output Raw

The only thing that had any trouble was the � character - which is not part of 7 bit ascii text.
If you remove that character, then unix diff says there are no changes.

I will admit, the Delimited Text node needs a unicode option, but I don't know if we should qualify 8-bit ascii as a "bug".

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: sdb

input.txt - I had to WinSCP this off the Linux platform - so the linefeeds may be bastardised by Windows.

Sorry no compromise on the character set - � is standard on my keyboard - but the point is to show the principle that the delimited output either provides superfluous quotes or superfluous escapes. And it is not the only character that does it, standard ASCII does too by the looks of it.

If the issue is lack of Unicode - then this oversight is long overdue. Casting field type is poor if that will be a requirement. Note that � and all the input exist happily as a field of type string on the input.

And the other route out is also blocked: Output Raw is still stripping \n from the end of records

Therefore there remains no way to reproduce any text file from LS...

Originally posted by: stonysmith
					

If you could, please attach the sample file as an attachment... the forum removes some characters such as the tab.


I tried it with just the three nodes:

- Input Delimited

- Reduce to List

- Output Raw


The only thing that had any trouble was the � character - which is not part of 7 bit ascii text.

If you remove that character, then unix diff says there are no changes.


I will admit, the Delimited Text node needs a unicode option, but I don't know if we should qualify 8-bit ascii as a "bug".
			

Attachments:
rawresult.txt
result.txt

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: lnason

Hi - I was able to pass the "diff" test by using a combination of the "Execute BRX" node and the "Output Fixed" node. I've attached the 2 graphs that implement this solution. The Execute BRX node is used so we can reset the Field Length value used in the Output Fixed node each time we need to save a new file.

Obviously, it would be nice if we could handle this in a more elegant fashion. I'm going to open a support request to address the two issues that seem to be present in the Output Raw node:

- Removing of newline chars at the end of the file
- Addition of "garbage" char (�) before characters like �.

I will append the support request id to this thread once I've opened it.

ExecuteSaveSimpleText.brg
SaveSimpleText.brg

Here are some items to note when testing this out:

- You'll need to update the DataDirectory parameter in both graphs
- In the "ExecuteSaveSimpleText" graph, I think you'll need to update the "Roll Up Data into 1 Record" agg node so that the RecordDelimiter no longer includes "\r"
- Save the "SaveSimpleText" graph as a brx file and upload it to the DataDirectory

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: sdb

I have to admire your ingenuity and also exploitation of existing building blocks to achieve the goal. But without a certain amount of study even the man skilled in the art will take a while to determine this is a solution to writing out text without undue alteration!

BUT NOTE - it also cannot be run on a standalone desktop version - as the BRX node is locked.

Frankly I could achieve the same and be arguably more obvious by a couple of lines of Python in the initialise (open file), pump (write record) and finalise (close file) of a standard Python node, and being OS file commands should be equally system portable, albeit one has to distribute the library node.

Without question this underscores my point - there is no direct one node solution to writing out text at present in Lavastorm core library. This is a major oversight.

Thanks for the support request for the Output Raw - these are bugs.

But it would be nice to have a support request for the Output Delimited or ano node to reproduce text intact, so that there is no need to pre-aggregate the records - maximising the utility of sort and data examination at all stages. And one which works equally across desktop, server and is portable without any other library dependency.

Simon