LAE

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: Tim Meagher

**Nodes released into this community are prototypes. They have gone through a minimum set of tests; therefore, we cannot guarantee that they will work as designed nor are they supported by any existing maintenance contracts.**


Attached is a prototype �Delimited Data� node which is able to read delimited data with an optionally specified character set.

This is specifically designed to deal with things like Unicode (UTF-8, UTF-16 etc) delimited data.
A pre-configured CSV Data Node is also in the library, along with examples of using the Delimited Data node.

While this node is not supported, if you find any issues with the nodes, please post them in this thread. If possible I will try and address them and update the nodes.


REQUIREMENTS:

• One of:
  
  • Lavastorm 4.6 Professional Plus
  • Lavastorm 4.6 Workgroup with Java Node support (Developer Node Pack)
  • Lavastorm 4.6 Enterprise

• LAL 2.11.0 4.6 or later

INSTALLATION

• ​Download the attached BRG

RUNNING THE NODE

• Open the "Delimited Reader" graph in BRE
• Run the example nodes attached
• In order to use this node in another graph add the DelimitedReader via Graph->Edit Base Libraries in BRE

COMPARISON WITH DELIMITED FILE NODE IN CORE.BRG:

• Improvements
  
  • Handles any character set with defaults to correctly handle data in a Unicode/string field
  • Includes a ConcatenationType for use when reading from multiple data sources with different metadata.
  • Follows a Literal/Filename Field/Data Field paradigm to allow for reading data from an input field as well as just input filenames
  • PassThroughFields & PassThroughFieldPrefix options to allow data on an input pin to be passed through to the output with an optional prefix applied to prevent name collisions
  • SubstituteInvalidCharacters option provided for cases when the field names would be invalid according to BRD metadata

• Disadvantages
  
  • Does not allow for the user to provide BRAINScript to specify the filenames
  • EscapeCharacter parameter only allows for a single character as an escape, rather than multiple escape sequences in a comma separated list that are allowed in the EscapeCharacters parameter in the Delimited File node
  • QuoteCharacter parameter only allows for a single character as a quote, rather than multiple quote characters in a comma separated list that are allowed in the QuoteCharacters parameter in the Delimited File node
  • In the Delimited File node, it is possible to add a second output pin, where errored records are written. This isn't possible with the Delimited Data node.
  • Performance is not as good as the existing Delimited File node.

• Other Limitations
  
  • Currently no handling of Byte Order Mark (BOM)s
  • The SubstituteInvalidCharacters parameter only handles reserved metadata characters such as ":" and newlines. It doesn't yet handle auto-converting Unicode characters in the metadata to something in the metadata's character set

Attachments:
DelimitedReader.brg

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: aop

Is there some known limitations in the quotecharacter support as I cannot read in any data set with quotecharacters (having no problems with the delimited file node in core library).

Is it a planned feature or current limitation that field names with dots in them are not supported?

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: Tim Meagher

Hi,

Both of those are oversights (bugs) in the node.
I think that both issues should be relatively straightforward to resolve though....
I'll take a look at it in the coming days and will post an update here when any fixes are available.

Regards,
Tim.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: Tim Meagher

Hi,

I updated the node to fix those issues.
It should now correctly handle the quote characters and dots in the field names.

Regards,
Tim.