LAE

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: techm_aritra

Hi,

Please provide example for ways to use directory list.

Regards

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: andycooper

Hi,

A simple example is that you can use the directory list node to quickly load multiple files held in a directory. If you have a folder with five csv files (one for each day of the week) you could load these with five separate nodes. If you simply drag the folder from Windows Explorer onto the BRE canvas, this will create a directory node automatically and this will list all the files in the folder (you can use wildcards to exclude file names and/or file types).

If you now point a csv node to the directory list it will add the data from all five files into the csv node (using FileName in the FilenameExpr field).

It's a quicker process if you want to load and concatenate multiple files, plus if new files are added to the folder these can be loaded without further changes to the node.

Hope this helps.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: techm_aritra

Hi Andy,

please provide an example graph..

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: andycooper

Example attached.

Directory List.brg

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: techm_aritra

Thanks for the example

but the problem i am having is reading each file in that directory after getting the list of the file using directory list node.

please let me know how to read each file.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: Tim Meagher

Hey,

Files can have many different formats.
There is no way to just "read a file" without knowing what the actual format of the file is.
Are these excel files, or delimited files, or fixed width files, or binary files, or CSV files, or XML files, or JSON files, or ...... ?

You really need to know *what* you are trying to read first.

Once you've figured that out, then look in the "Acquisition" category and you'll likely find a corresponding node (like "Delimited File", "Multiple BRD Files", "Fixed Format File", "Excel File", "XML Data", etc) where the name if the file pretty much indicates the type of format that it is designed to read.

For some of these nodes (e.g. XML Data, JSON Data) the nodes are part of the Lavastorm Analytics Library, so make sure you have that installed.

In any case, if the name of the node is not clear enough, each of these nodes will also have documentation available which describe exactly what their purpose is, and how to use them to read the data you are trying to read. Just click on the node in the Acquisition section, and then hit F1 to bring up the node help.

Regards,
Tim.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: techm_aritra

I am trying to read 100 CSV files from a directory and process them from error records...

please help me.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: Tim Meagher

Hi,

Then you should look at using the "CSV File" node.
Create an input to that node & connect the output of the "Directory List" node to the input of the "CSV File" node.

Consult the "CVS File" node help for more information.
Note that if you create a second output on the CSV File node, any error records (records that don't match the delimited format) will be written there.
Note also that the node expects that the format will be the same for all of the input files - i.e. all files would have to have the same number of fields.

Regards,
Tim.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: techm_aritra

i am reading the files from a directory and there are 100 files. so I am using Directory List Node to read all of them at a time. Now I want to process them one by one. please tell me how i should be able to read each file separately from the directory list node.

already did what you said but its giving the below error
Error while starting execution of node 'CSV+File'
Error executing node: Process exited before ENDIO message received.
2014-06-02 12:05:21.000; WARN: Property ls.brain.node.convert.filenameExpr not set
id: 0 chain: 0 group: 0
cppDetail: context: Salmon::NED::Convert::ConvertNodeControl::getSetti ngsFromProperties() .\ConvertNodeControl.cpp@127

2014-06-02 12:05:21.000; WARN: ...exception seen
id: 0 chain: 0 group: 0
cppDetail: context: Salmon::NED::Convert::ConvertNodeControl::getSetti ngsFromProperties() .\ConvertNodeControl.cpp@142

2014-06-02 12:05:21.000; WARN: ...exception seen
id: 0 chain: 0 group: 0
cppDetail: context: Salmon::NED::NodeControl::initialize() .\NodeControl.cpp@318

2014-06-02 12:05:21.000; ERROR: ERROR reading settings:
Property ls.brain.node.convert.filenameExpr not set
id: 0 chain: 0 group: 0
cppDetail: context: Salmon::NED::NodeMain::errorMsg() .\NodeMain.cpp@28

Note: all files are in .csv format

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: Tim Meagher

I'm not clear on what you mean by reading them "one at a time".
Do you want this all to be done by one node which reads all of the files?
Or do you want to process each file separately, and have the data from each node on a separate output?
If it's the latter, then you will need 100 different nodes, or to move your CSV node into it's own graph, save it as a BRX and have an Execute BRX node execute the CSV File node for each record from the Directory List node.

If it's the former, then you will need to leave the Filename parameter empty and set the FilenameExpr parameter to the name of the input field which contains the filenames.

As per the node help (which is generally a good idea to read if you're having trouble with understanding how a node works), the node:

• Reads and converts a delimited file for input given a designated set of field and record delimiters. If there is one input, then FilenameExpr is used to determine the names of the files to process. The list of files is constructed using the current row of the input file, and applying the specified expression. If there are no inputs, then File is used to determine the file to process. Both FilenameExpr and File may not be specified at the same time.
  When using the one input version, all of the files must conform to the same specification, that is, have same number of fields in the same order and format.
  If there are two outputs, the second ouput is an error output.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: techm_aritra

Thanks

it is done