Data360 Analyze

is there a plan to reimplement the data-names pivot and  names-data pivot nodes in the new D360 Analyze? as of 3.6.2, i notice the underlying functionality is still in brainscript / java nodes, and is not particularly fast.

for example, i did a 3-way hash-split on 220 million records and performed a data->names pivot on each branch (70m records each), and d360 took 2.5 hours to complete on a 3GHz Intel 4-core device.

can the roadmap or releaseplan of added and enhanced functionality be made available?

 A replacement node for the Pivot Data to Names node will be available in the 3.7.x release branch and, subsequently in the 3.8.x LTS branch. The replacement node does not utilize BRAINScript. The existing node will be superseded but will contine to be available and operate unchanged in these releases.

The Pivot Names to Data node is already implemented using Java technologies and does not utilize BRAINScript. No changes are planned for this node at this time.

While the Pivot Table node is implemented as a Composite library node, it does not expose BRAINScript to the user. It is likely that this node will be replaced by a new node in a future release, but the timeframe for this has not been confirmed at this time.

The application currently provides the Transpose node, which is implemented using Java and can be used for a variety of simple data rotation use cases.

In terms of performance - the new node is dramatically faster.  In my brief testing typically 15 to 100 times faster.  For example, the new node processed 60 million rows in ~18 minutes.

Thanks for the update. I look forward to the new nodes.