Data360 DQ+

 View Only
  • 1.  Setting resource manager queue

    Posted 09-13-2019 12:57

    Hello,

    I would like to change the default resource manager for DQ+. Right now it is running under the default queue but we would like to change it to a different YARN queue that has been set up. Is there a way to configure this in DQ+?



  • 2.  RE: Setting resource manager queue

    Employee
    Posted 09-17-2019 14:31

    Hi Blair,

      There is an approach that we think can get you there, but let me confirm it first. 

     

    Thanks,

    Jeff



  • 3.  RE: Setting resource manager queue

    Employee
    Posted 09-20-2019 07:33

    Hi Blair,

     After speaking with the team, they confirmed that it is possible change to a different YARN queue, but starting in DQ+ 3.3. To do it, you can add an Execution Property at the Analysis Level:  spark.yarn.queue and set it to the name of the custom queue. You can get to Execution Properties through Edit Stage / Settings (gear in top right) / Execution Property.

    Thanks,

    Jeff



  • 4.  RE: Setting resource manager queue

    Posted 11-12-2020 12:53

    Jeff, the method you suggest requires creating a setting for each individual analysis created. Is there a way to set spark.yarn.queue to a global default so it doesn't have to be set for each analysis? Perhaps something added in 4.3?



  • 5.  RE: Setting resource manager queue

    Employee
    Posted 11-13-2020 07:06

    Hi Rob, there is not a way to alter the global default for each analysis.  One alternative would be to create an Execution Profile with the default execution property settings that you wan to set, then apply the Execution Profile on a per analysis basis.  This would mean setting this once at each analysis, but after that you can change the default anytime. 



  • 6.  RE: Setting resource manager queue

    Employee
    Posted 11-13-2020 07:16

    Hi Rob... I did create an enhancement for this inquiry.   One idea that I can think of would be to create a Default Execution Setting execution profile that could be applied to each analysis chain (by defualt), then allow the Admin to update that profile as necessary.



  • 7.  RE: Setting resource manager queue

    Posted 03-04-2021 07:20

    Hi Jeff:  We also are experiencing this issue. Can you give us a time frame (month/year) when this enhancement will be implemented or was this resolved in Analyze 3.6.8?



  • 8.  RE: Setting resource manager queue

    Employee
    Posted 03-04-2021 08:55

    Hi Hal, This enhancement is related to DQ+ and not Analyze.  If you are having issues with this topic in Analyze, please feel free to post here:

    https://support.infogix.com/hc/en-us/community/topics/360001073633-Data360-Analyze