Data360 Analyze

Concatenate / Merge multiple Datasets based on dynamic number of inputs.

Hi All,

I want to do something really simple but I am finding a lot harder to achieve than I thought it would be. Hoping someone (<x-zendesk-user data-user-name="Adrian Williams">366746295667</x-zendesk-user>..) can help.

I have an API call that I have to split into groups to abide by the API endpoint batch size, therefore each GET may return any number of records and to make it slightly harder, the number of group I send to the endpoint may also vary.

This results in a number of datasets/output that need to be merged together using a "cat" node. The problem being that if I create a 9th, 10th or 11th input and there is then no data for this input, the node never runs.

What I need is something like the cat node, with the functionality to ignore the fact that a defined input is empty and merge all the records anyway...

Any ideas...?

Thanks

John

Hi John,

I'm not entirely sure I understand what you mean by "a defined input is empty". If the input has zero records but the metadata is defined then the Cat node should concatenate the remaining inputs with non-zero records.

Perhaps you could try writing each of the non-zero data sets from your API responses to a set of temporary files in a specified directory and then, once all response files have been written you could use a run dependency to enable a Directory List node to run against the directory. The filename list can then be passed to the relevant input connector node (e.g. CSV/Delimited or JSON Data) which will then concatenate the data from the individual files. When the node has run you could use another run dependency to enable a Transform node that is configured to delete all the listed temporary files.

Thanks Adrian,

I'll consider the best way approach this. Currently if you try and connect a number of JSON nodes to a CAT and one of those JSON nodes are empty (or a meta check doesn't output) then the CAT node doesn't like to run.

I was hoping there was a direct to do this without having to write out to a file.

Thanks, John

Morning Adrian,

Just a follow up note on this one. 

Turns out it was more to do with the JSON node not outputting a value for the cat node to consume rather than the cat node doing anything odd.

Tweaking the JSON node that is connected to the HTTP response to ignore errors then allowed for an input to the cat node and then setting the the cat type to "Union" allowed me to do exactly what I needed to do.

Thanks for your help - it got my mind on the right track to solve this one :)