Data360 Analyze

Hi, 

Currently I notice large periods of latency when a I'm editing a workflow with a large number of nodes in it. To give an example, in one of the biggest files I have, it can take 10 minutes (sometimes longer) to simply disconnect a connection to a node and re-connect it to a different existing node. This makes editing very frustrating as even when I know exactly what I need to do, it can sometimes take 1-2 hours to do something where I'm really only making active changes for maybe 10 minutes and the rest is just waiting for each change to go through. I'm trying to figure out if there's any good practices I can do to minimize the amount of time I'm waiting during the editing process.

Are there any best practices I could follow to specifically improve this latency?

For example, is there any best practices surrounding composite nodes to improve speed? I don't use composite nodes at all anymore because I've used them sparingly in the past and sometimes just entering a composite node will take 5-10 minutes and if I accidentally click into the wrong composite node and need to go back to the main workflow and then re-enter the correct one, that's an instant 20-30 minutes of waiting just to enter the right part I need to edit without actually doing any editing.

However I'm wondering now if categorizing my entire workflow into many composite nodes would reduce overall time (maybe separating my entire workflow into many smaller composite nodes means there's less things to load on each level? I'm not sure if it actually works like that, but if it did, this would be the kind of tip I'd be interested in!)

Any help would be greatly appreciated!

A support ticket has been raised to enables us to discuss this in more detail.

Is there any information that can be shared with the community on this? No one I work with has had the delays that are mentioned above however users do complain about the latency compared to how Lavastorm ran.

If the data flow contains many nodes then, in general, the design-time responsiveness of the UI may be improved if Composites are used to group related nodes as there are fewer nodes to render on the canvas in any one 'level'.

In the case where Run Dependencies (Clocks) are not used the Composites are only graphical objects that do not affect the run-time performance of the individual nodes. When changes are made to the data flow, the use of Composites restricts the scope of work required by the system to reflect those changes - which will decrease the time required to compose the data flow model.

When Run Dependencies are used to synchronize the running of two Composites (A & B) and the Composites each contain a large number of nodes (#m, #n) then the would be n * m implicit Run Dependencies created which may degrade performance. I this situation it is recommended that explicit run dependencies are made to nodes within each of the composites or that a Run Dependency is created between, say, Composite A and a single node and a separate Run Dependency is created between the single node and Composite B - which would result in the creation of n + m Run Dependencies.

The Help documentation provides information on how explicit Run Dependencies can be created for nodes within Composites

Some suggested tips are:

• Group nodes for each input data source into its own Composite
• Group logical strings of nodes for transformation and analysis into their own Composites
• Multiple output sources should have their own Composites
• Improve the readability of the data flow by keeping the number of nodes on a level to a relatively small number that can be viewed without the need to pan the canvas, say around 20
• Add names to your nodes and Composites - this will make any error messages more readable. You can now navigate directly to an errored node by clicking on the error in the error panel - the canvas will pan to the node; if the node is on another level to the current displayed level, the canvas navigation will drill-in/out and pan to the node.