Data360 Analyze

 View Only
  • 1.  Scheduling of graphs

    Posted 06-14-2021 23:57

    We have a schedule of 15 graphs where there is dependency between the graphs that first graph produces output which is input for second graph and so on.

    And we have scheduled the graphs one after other with some buffer time.

    But what happens is, at times some graphs take more than expected time to finish. Then the schedule starts failing as next graph doesn't get its input on time.

    Is there any way we can introduce dependency between all these.

    Also important point if in these 15 graphs 7th graph fails due to some error, can we start the run again from 7th graph itself till end? Beacuse of dependency I dont want that if any graph fails in middle of sequence we have to run entire schedule again from first graph.

     

    Thanks in advance



  • 2.  RE: Scheduling of graphs

    Employee
    Posted 06-15-2021 03:25

    You may want to consider using a 'parent'/'driver' node that leverages the Execute Data Flow node to run each of the dependent data flows using a set of Execute Data Flow nodes that are configured with Run Dependencies.

     

    The parent data flow would be run under a schedule. Depending on the nature of the error (e.g. unavailable external system) it may be possible to open the failed scheduled run and re-run it from the point of failure. 



  • 3.  RE: Scheduling of graphs

    Employee
    Posted 06-15-2021 03:26

    See the 'Node help' > 'Interfaces and Adapters' > 'Execute Data Flow' topic in your Help documentation or the online documentation, here:

    https://d3sa-preview.infogixsaas.com/docs/dist/help/Default.htm#e-node-help/Interfaces_and_Adapters/execute-data-flow.htm



  • 4.  RE: Scheduling of graphs

    Employee
    Posted 06-15-2021 14:43

    Hi Adrain,

     

    This is something I am looking for to solve similar issue. I tried following instructions, it is throwing an error. Please have a look and let me know what I am doing wrong here.

     

    Attached files

    Testing_graph - 15 Jun 2021.lna

     



  • 5.  RE: Scheduling of graphs

    Employee
    Posted 06-16-2021 02:02

    It looks like the value of the DataFlow property is incorrect. The value needs to the the resource locator for the data flow. If you clear the current contents of the property and click the folder icon

    The 'Choose a Data Flow' dialog will be displayed.Navigate the the location of the data flow to be run and select it. Click on the 'Choose' button. 

    The canvas will be displayed again and the 'DataFlow' property will be populated with the resource path of the selected data flow:

    Please refer to the node Help documentation for the other configuration options for the Execute Data Flow node.



  • 6.  RE: Scheduling of graphs

    Posted 10-14-2021 09:23

    Hi Adrian

    Just taking Jagdev's simple example attached above - why didn't it work? I thought it was possible to pass a field in as follows using the from Field selector which looks like what he had done? I cant get it to work either. See screenshot below and attached. thanks

     

     

    Attached files

    executedataflow.png

     



  • 7.  RE: Scheduling of graphs

    Employee
    Posted 10-14-2021 11:10

    See the attached document for an explanation and an example of how the Execute Data Flow node can be used.

     

     

    Attached files

    Execute_Data_Flow_Node_Example.docx

     



  • 8.  RE: Scheduling of graphs

    Posted 10-15-2021 02:41

    Thanks Adrian for the document, that helps to bring it to life. 

    For anyone looking at the example above and wondering why it isn't working - the problem was with the DataFlow field. Need to click the drop down and set it to DataFlow (from Field). Also make sure your graph location has the #graph suffix at the end e.g. //admin/SubGraph1#graph

    When you just manually select the graph in the dataflow field rather than using (from Field) it doesn't add the #graph and it wont work without it when using (from Field)

    also (as it says in the node help), you can copy the path to data flows in your system from the Directory. The Resource Path is listed in the details panel on the right of the screen - like this