LAE

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: henk thijssen

Capture1.JPG

The above is what I have on an output pin after execution of a few nodes.
I want to save this in a .csv file using the node "Output Delimited". The required format should be UTF-8 without BOM.
In the .csv I get:

CSV.JPG

This is not what my customer wants M\xd6 in stead of �.
Please help with saving text in UTF-8 encoding scheme.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: Wizardous

As I understand converting that field to the character coding you want will solve your problem.

override emit name.unicode("UTF8") as name

Then you can just output it.

Regards,

�zg�n

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: henk thijssen

Thank you for your comment.

Sadly enough the advice you gave did not solve the problem. I already have unicode type detected, as can bee seen in the first screenshot.
override emit name.unicode() as name does not help. The format before and after the filter is unicode.
The problem is, to my opinion, with the node "Output Delimited". This nodes writes '\xd6' where it should be "�".

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: Wizardous

Sorry for the late reply and confusion. I use the convertion below to correctly extract turkish characters to csv files. Hope it works for also the letters you are looking for (as they are also in turkish). If those hex numbers(like u011e) does not work for you, you can just copy and paste the "name" to View>Unicode Converter section on BRE, and see the unicode reflection of that text. Then you need to change the replace function with your unicode form. Hope these work for you too.

override emit 'name'.unicode("UTF8").replace(u"\u011f ","ğ")
.replace(u"\u011e","Ğ")
.replace(u"\u0131","ı")
.replace(u"\u0130","İ")
.replace(u"\u00f6","�")
.replace(u"\u00d6","�")
.replace(u"\u00fc","�")
.replace(u"\u00dc","�")
.replace(u"\u015f","ş")
.replace(u"\u015e","Ş")
.replace(u"\u00e7","�")
.replace(u"\u00c7","�")
.str() as name

outputting this new 'name' field directly with output delimited node works for me (on attached graph).

Regards

�zg�n
Attachments:
convert.brg

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: henk01

Hello Henk,

I was having a good look at language support recently and that�s why this reply is a bit more extensive.

First of all, without any modifications you should be able to retrieve data and write data to output without needing any conversions. If the input file is provided in UTF8 encoding then all temporary files and output files will be in UTF8 too.

This can be most easily checked when using Notepad++ or on Unix (with file �i <filename>).

However, having a file in UTF-8 is not always handy as extended characters are presented as double byte characters, i.e. two characters. Care must be taken when applying string functions such as left(20) as extended characters will cause issues (strings become longer and extended characters could be chopped in half).

When for example applying string functions it would be better to convert the character encoding from UTF8 to windows-1252. LAE uses by default WIN-1252 character encoding, an 8-bit single-byte encoded character set, and a superset of ISO 8859-15. In a BRE viewer you can always change the coding to utf-8 (Preferences -> Set codepage), but that would be simply change the way you view the data.

�zg�n has the right idea by using name.unicode("UTF8"), but then unfortunately the Unicode characters are not converted correctly to file when using the �Output delimited� node. In his graph �zg�n mentions name.unicode("UTF8").str() which is the conversion is back to string format, and the string are now displayed in the windows-1252 encoding. When checking the encoding of the temporary files, this has now changed to ANSI (on Windows using Notepad++) or �iso-8859-1� (on Unix). If the file no longer does contain extended character (e.g. filtered out) then the encoding of the file on Unix will be �us-ascii�. The �Output Delimited� node should now write the data correctly. Attached is a graph demonstrating the conversion which hopefully clarifies the options.

There are a number of threads discussing the limitations of the �Output Delimited� node:
http://community.lavastorm.com/threa...ht=decode+file
http://community.lavastorm.com/threa...ht=decode+file

I hope this helps. So definitely the �blocks�, or �\xd6� can be avoided, and the file will be generated correctly, but in �us-ascii� or �iso-8859-1� encoding rather than �UTF-8�. If the file definitely needs to be in UTF-8 format then on Unix the �iconv� command can be used (using an ExecuteSystemCommand node), and there should be an Windows version as well (and I can provide a sample for that as well if needed).

Best regards,

Henk
Attachments:
UTF8.zip

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: henk thijssen

Henk,

Thank you for helping on this topic. The additional info was very helpful. I better understand the problems than can occur with string processing.
In his last sentence you also addresses the real big problem I am facing: The delimited output node has problems generating UTF8. It will generate “us-ascii” or “iso-8859-1”

In your example only the delimited output node with the text ‘No errors, and still UTF8’ is OK.
All other delimited notes, in your example, create ANSI. For no reason I would say.
And I cannot get it to produce anything else.

I do need a node that will create delimited output (.txt or .csv) in UTF-8 without BOM.

Best would be the ‘standard’ Output Delimited’ had an option to control the output encoding. (UTF-8, UTF-8 without BOM, ANSI, UCS-2,..)
Or using an external command node to have access to a tool that does the job.
But that node (ExecuteSystemCommand) I do not have in my Trial package.. or do I?

As systems integrator receive lots of data collected from print devices all over the world. This data I intend to validate with BRE (pre-processing) before it is to be used by other analytical and back office systems. These systems need UTF-8 without BOM as we have declared this encoding as standard.

Regards,
Henk Thijssen