Data360 Analyze

 View Only

  • 1.  Synchronization of flows within Execute Data Flow Node

    Posted 01-26-2024 04:54
    Edited by Irakli Chitishvili 01-26-2024 05:15

    Hi,

    I'm dividing and grouping data flows into relevant projects and scopes and subsequently into relevant Execute Data Flow Nodes , like 7 flows for one specific scope, 8 for another and so on, total 80 flows (which will increase in the future). Those dataflows will be scheduled to run daily after checking if certain tables are up or have been updated using meta check and then triggering the Execute Data Flow Node. 
    Now the difficult part is how to make each data flow inside the same Execute Data Flow to run in 1-2 minutes after the previous data flow in the same Execute Data Flow because I want to avoid putting too much strain on data warehouse with simultaneous sql queries.  apparently scheduling individual flow is not a good idea since the general flow containing execute data flow will be scheduled itself. 

    Is there any function or method for synchronizing this kind of chain which you have solved in the past?

    Thanks,
    Irakli



    ------------------------------
    Irakli Chitishvili
    Degroof Petercam SA/NV
    ------------------------------



  • 2.  RE: Synchronization of flows within Execute Data Flow Node

    Employee
    Posted 01-29-2024 10:49

    Hi 

    The first idea that comes to mind is to use the Sleep node to wait the 2 minutes before starting a dataflow.  This attached screen shot shows how it might be structured.

    Disclaimer: I can't guarantee this will work because it relies on the controller that runs nodes inside dataflows to determine when dataflows are run.  It was not built with the idea of scheduling dataflows optimally.  Many years ago I was trying to schedule SQL queries so that one would be delayed. The node scheduler did not reliably cooperate.  But in the context you are describing, perfection is not completely necessary so this might work well enough.

    Ernest



    ------------------------------
    Ernest Jones
    Precisely Software Inc.
    PEARL RIVER NY
    ------------------------------



  • 3.  RE: Synchronization of flows within Execute Data Flow Node

    Posted 01-30-2024 05:13

    Thanks, sleep nodes seems to the functionality which might solve my problem.



    ------------------------------
    Irakli Chitishvili
    Degroof Petercam SA/NV
    ------------------------------

    Original Message