Data360 Analyze

Hi, 

I'm currently trying to migrate an existing file initially written in Brainscript into Python-based nodes and am having some trouble finding an appropriate analogue to 2 specific functions: "inputFields" and "field". Is there an easy way to use these functions in Python that I'm overlooking? More information is below if it helps at all:

The "inputFields" function seems to take in 1 parameter, a number, and returns the column name for the given index number. Eg. inputFields(3) will give the name of the 3rd column.

The "field" function seems to take in 1 parameter, a string, and returns the value for the given column name string. Eg. field("FavouriteColor") will effectively return the value in the "FavouriteColor" column. However, the strength of this function is in its ability to be dynamic. For example, if there's a column A that's filled with a variety of column names (such as B, C, D, etc.), using the "field" function in a different column X will allow column X to return values from B, C, D, etc.

Essentially these 2 functions can be used together in a Do While Conditional loop to take a file with many columns and convert it into a file with less columns and many rows. Eg. if there's a file with columns such as Value_A, Value_B, Value_C, ..., Value_Z, these 26 columns could be replaced with 2 columns. One called Column Name which uses the "inputFields" function on a counter variable and one called "Value" which uses the "field" function on the value returned from the "inputFields" function.

If it helps, the below is effectively the exact code used to do this:

Fields = inputFields()

emit *

override emit Fields['Counter'] as "Column Name"
override emit (field(Fields['Counter'])) as "Value"

The closest equivalents in a Transform node would be to use the fields.todict() function in the ConfigureFields to create a dictionary representation of the input fields and then use the keys() function on dictionary object to get the list of input field names.

Within the ProcessRecords script you can use fields['Some_Field'] to access the value of the field named "Some_Field" in the current record. If you are outputting the values for multiple input fields with different data types then you will need to cast the field values to an appropriate common data type e.g. a string.

See the 'Python scripting' > 'API and script bindings' topic in the integrated Help documentation or the corresponding online documentation for the latest version of Data3Sixty Analyze here. 

To output the cells for each input record on a different output record you will need to explicitly write the output records using say the node.write() function as described in the 'API and script bindings' topic and see the example in the default script of the Transform node.

Also see the attached data flow that provides an example of how the cells in an input data set can be 'pivoted' into a set of key: value pairs. The example extends this to also include: a field indicating which input record the key: value pairs are associated with; a field indicating the data type of the input field; a field indicating whether the original value was Null (to allow you to disambiguate a Null value in a string from the string "Null").

Requires Data3Sixty 3.4.x or above.

Attached files

Pivot Data Cells_3.4.2--share - 29 Jul 2019.lna