Data360 Analyze

 View Only
  • 1.  Accessing flow runtime/statistics outside of Analyze

    Posted 01-09-2023 05:09
    Hi, 

    I'm exploring the possibility of accessing the data flows run times and exporting or visualizing them in PowerBI in order to see which flows take more or less time to run? I know there are statistics available in Analyze for individual flows but with couple of hundred flows I really want to find more versatile way to have them somewhere automatically exported/API called for grouping then into run time groups in order to analyze and optimize further. 

    Thank you

    ------------------------------
    Irakli Chitishvili
    Degroof Petercam SA/NV
    ------------------------------


  • 2.  RE: Accessing flow runtime/statistics outside of Analyze

    Posted 01-10-2023 03:05

    Hi Irakli,
    Look into the Analyze API. 
    When logged into your server, press the help (i) button on the top right near your user name and look for "System Administration" / "API Developer documentation" in the menu. The look for the tutorial in that section

    For me this looks like: 

    https://<analyze server>/docs/dist/api/index.html
    https://<analyze server>/docs/dist/api/index.html#tutorials

    Essentially you can create a dataflow to access the analyze REST API similar to what is done for Govern and post that information back to Govern or do whatever you want with it. 

    Good luck.



    ------------------------------
    Peter Sykes
    Vontobel Holding AG
    ZUERICH
    ------------------------------



  • 3.  RE: Accessing flow runtime/statistics outside of Analyze

    Posted 01-10-2023 13:53
    Thank you, this is very useful!

    ------------------------------
    Irakli Chitishvili
    Degroof Petercam SA/NV
    ------------------------------



  • 4.  RE: Accessing flow runtime/statistics outside of Analyze

    Employee
    Posted 01-10-2023 10:27
    I agree that AnalyzeCli is the way to go.

    I've been able to use the following code in a Transform/Generate Data node to run AnalyzeCli from inside Analyze.

    import subprocess
    p = subprocess.Popen([analyzeCli, '-f', 'json', "runs"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)​

    The analyzeCli variable either references the installed analyzeCli executable, or the analyzeCli.bat file in Windows.

    The output in this example is a JSON list of JSON objects in this form.

    {
      "scheduledTaskName" : "The Schedule Name",
      "status" : "COMPLETED",
      "duration" : 2031,
      "startTime" : "2023-01-06T23:04:10.699Z",
      "modifiedByName" : "username here"
    }


    Ernest



    ------------------------------
    Ernest Jones
    Precisely Software Inc.
    PEARL RIVER NY
    ------------------------------



  • 5.  RE: Accessing flow runtime/statistics outside of Analyze

    Posted 01-10-2023 13:56
    Great, thank you. What is the time granularity of 'duration' field?  ​​​​​

    ------------------------------
    Irakli Chitishvili
    Degroof Petercam SA/NV
    ------------------------------



  • 6.  RE: Accessing flow runtime/statistics outside of Analyze

    Employee
    Posted 01-10-2023 14:24
    I do not know the granularity of the duration internally.  This output shows the units in whole seconds.

    ------------------------------
    Ernest Jones
    Precisely Software Inc.
    PEARL RIVER NY
    ------------------------------