Data360 Analyze

Hi,

I'm dividing and grouping data flows into relevant projects and scopes and subsequently into relevant Execute Data Flow Nodes , like 7 flows for one specific scope, 8 for another and so on, total 80 flows (which will increase in the future). Those dataflows will be scheduled to run daily after checking if certain tables are up or have been updated using meta check and then triggering the Execute Data Flow Node. 
Now the difficult part is how to make each data flow inside the same Execute Data Flow to run in 1-2 minutes after the previous data flow in the same Execute Data Flow because I want to avoid putting too much strain on data warehouse with simultaneous sql queries.  apparently scheduling individual flow is not a good idea since the general flow containing execute data flow will be scheduled itself. 

Is there any function or method for synchronizing this kind of chain which you have solved in the past?

Thanks,
Irakli

------------------------------
Irakli Chitishvili
Degroof Petercam SA/NV
------------------------------

Hi 

The first idea that comes to mind is to use the Sleep node to wait the 2 minutes before starting a dataflow.  This attached screen shot shows how it might be structured.

Disclaimer: I can't guarantee this will work because it relies on the controller that runs nodes inside dataflows to determine when dataflows are run.  It was not built with the idea of scheduling dataflows optimally.  Many years ago I was trying to schedule SQL queries so that one would be delayed. The node scheduler did not reliably cooperate.  But in the context you are describing, perfection is not completely necessary so this might work well enough.

Ernest

Hi,

I'm dividing and grouping data flows into relevant projects and scopes and subsequently into relevant Execute Data Flow Nodes , like 7 flows for one specific scope, 8 for another and so on, total 80 flows (which will increase in the future). Those dataflows will be scheduled to run daily after checking if certain tables are up or have been updated using meta check and then triggering the Execute Data Flow Node. 
Now the difficult part is how to make each data flow inside the same Execute Data Flow to run in 1-2 minutes after the previous data flow in the same Execute Data Flow because I want to avoid putting too much strain on data warehouse with simultaneous sql queries.  apparently scheduling individual flow is not a good idea since the general flow containing execute data flow will be scheduled itself. 

Is there any function or method for synchronizing this kind of chain which you have solved in the past?

Thanks,
Irakli

------------------------------
Irakli Chitishvili
Degroof Petercam SA/NV
------------------------------