Data360 Analyze

I have a config file with a list of tables. These tables should be merged and written into one table.

The Cat node does the job of merging resp. the different tables.

But I have up to 100 tables for the Cat union and the list of tables needs to be a config file or parameter rather that individual Input nodes for each table and adding the new Input node to the workflow when adding a table.

I guess I need some Do While node to iterate over the list of tables from the config file, but I don't know, how to connect it with the Cat node.

Any ideas how to solve this problem?

Best regards
Holger

It sounds like a Do While would work and you could keep on appending your logic to a brd file that you finally read when the loop has completed.

I'm attaching an example of how I would configure a Do While that would cycle through a list of 5 tables.

Attached files

Loop - 6 Sept 2018.lna

Thank you very much for providing a demo workflow! :-)

I notice, that you are using the Output BRD File node and not the CAT node.

But my input tables have slightly different column structures.

CAT has ConcatenationMode = Union ("Outputs all of the fields across the inputs, filling in nulls for fields not present in a given input"), but Output BRD File with Append = True ("If true and the output file exists, the field names and types of the existing file and the input stream are compared to make sure they are identical.") does not accept different columns.

Any ideas?

Best regards
Holger

Rather than using Append, I added reading back in the BRD file and then using Cat (union) to join back up to the current iteration. Then we will overwrite a new BRD file each time. That will get you past that error.

Attached files

Loop with Cat (Union) - 7 Sept 2018.lna