Data360 Analyze

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

The Run Parameter values are not migrated when migrating a legacy Lavastorm graph. We need the ability to migrate the Run Parameters over and keep the multiple environments (Dev, test, production). So in Lavastorm we utilize one graph with Run parameters for each environment. This workflow is important for total dev, test and production analysis of data and processing. 

Currently no Run Parameters are migrated at all.

There are two aspects to your request.

Data3Sixty Analyze can import a legacy data flow (graph) from a specified .brg file. The run property definitions (i.e. names) are imported as part of the data flow. However, run property values are stored separately in a corresponding .brp file. The .brp file is not currently read when importing the legacy data flow. We will consider whether this functionality will be introduced in a future release. This is typically a one-time/ low frequency activity that is performed during data flow migration activities. The values in the .brp file can be accessed using a text editor to open the file.

Data3Sixty Analyze does not currently provide the ability to define multiple property groups, where different values can be assigned to each run property (e.g. for different environments). This is under consideration for the roadmap.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Adrian - thank you for your reply. I agree that Run parameter values are stored in the brp. I also agree that it is a one time activity but I disagree on low frequency (at least at our site where we have to interact with multiple environments from the same graph. One graph (brg) I am working with now goes to 7 different environments and has 25 Run parameters and 0 graph parameters since a database we query could be different or where we write a file could be different or the name of the file is built dynamically. Lavastorm handles that scenario with out issues but what I am being asked to do with Data3Sixty is create 7 dataflows that are exactly the same except for parameters and input 7X25 (175) run parameters. and that is for just 1 graph. I might look at reading the brp file into Data3Sixty and to somewhat mirror the current process but then passwords I have stored as a password is encrypted. I will need to write a run parameter import dataflow to bring the values in and will need to talk to someone in dev to figure out where they are stored in Data3Sixty.

Thank you for your time.

Hi Donald,

I take your point about how the number of property values can rapidly increase with when dealing with multiple properties and environments. I will add your use case into the discussions.

Many thanks,

Adrian

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Adrian

Has there been any more discussion on Run Parameters. One user will have to take 4 graphs and create and maintain 24 dataflows. I am also looking into having the dataflow read a brs file or having the user put Run Parameters in a database table and reading the vaules from there based on the env they are running.

Can the Run properties be populated from within a dataflow. Where I could read a brs or database table and then update the Run Properties and thus the user would not have to modify the dataflow with the exception of putting a node to pull the data and then a node to populate the Run Properties. 

I can not find any documentation on this.

Thank you

Hi Donald,

We will aim to introduce support for run property groups in a future release in a few month's time.

The run property values are constants for a data flow execution and cannot be modified from within the data flow (i.e. they are not global/environment variables).

The relevant sections of the  documentation are:

-  Advanced topics ->  Defining properties

- Advanced topics ->  Using derived property values

As an alternative, if you are using scheduled execution of the data flows using an external scheduler, you may want to consider using the external scheduler to execute a scheduled run via the defined API and pass in the required runtime property values from the external scheduler.

The API documentation is in the following section of the server help:

System administration -> API developer documentation

(See Ad-hoc run of a simple scheduled task)

Regards,

Adrian

Hi Adrian,

We have a similar issue in that run parameters are softcoded and graphs are run via a shell script (support ticket #69181 raised).  I'm interested in looking at how the API functions and if we can utilise this going forward.

However, the above links do not appear to work?  Have they moved to a new address?

Thanks.

Hi Stu,

I have just tried each of the links in my previous message and they all are resolving correctly. Is it a DNS issue from within your network (due to the use of the lavastorm.com domain)?

You can access the same information within the integrated Help of your installation. From the Designer main page click on the 'i' button on the main menu and then click on 'Advanced topics' item which should display that topic together with the navigation pane on the left. You can then expand the Advanced topics section to access the 'Defining properties' and 'Using derived property values' topics, and the 'System administration' section to access the 'API developer documentation' topic.

Regards,

Adrian

Thanks Adrian - didn't consider the potential DNS issue.

I've managed to get the information as per your instructions - Thanks again!