Data360 Analyze

Hi All,

I am trying to perform a data manipulation which requires selecting a single cell in a particular column, instead of selecting the entire column.

Currently i am getting TypeError: 'long' object is unsubscriptable Error executing "ProcessRecords" at line "9

Below is a snippet from my code

if fields.Key=='OTCFINDRA' and fields.key is not Null:
out1['Total DRA']= in1['MTM_1'][1]

I am not sure if Data360 supports this functionality because it requires the column to be converted into an array

Any help will be great and appreciated.

You cannot access data in individual records of the input data as an array. The Transform node (and the majority of other nodes) operate on a record-by-record basis. This allows you to process an arbitary volume of records without requiring large amounts of RAM as all of the data is not stored in-memory. 

If you want to access the value in a particular record in the input data you can use the in-built node.execCount variable as an index for the number of records processed (1-based). For example:

The above script  also outputs a field 'Canary' even if the test fails and acts as a flag to indicate whether the 'CellValue' field contains valid data. This is useful because if the test fails then zero records would be output and typically it is easier to code downstream logic to operate by testing a value rather than attempting to handle the situation where a node outputs metadata but has zero records.

If you are certain that you will always have sufficient RAM for the maximum number of input records you could read in all the values for a particular field into a list and then you can test the built-in node.lastExec variable to determine all records have been read before performing subsequent processing on the list.

Hi Adrian,
Thank so much for the detailed explanation. I have one follow-up question?

Can we also place the record of one cell at a particular row position of a pre-existing column, instead of placing at the start of the column.

Example
If fields.Key=='Red' and fields.key is not Null:
out1['Total DRA'][5] = myList[1]

Normally records are processed one at a time. You only have access to values in the current input record and can ony write values for the current output record. You cannot write values to fields that have already been processed or set field values in records that have yet to be processed.

If you want to retain access to a field's value while processing a subsequent record your script would need to store the value in a variable or a list, as described above. If you wanted to output a stored/calculated value in a specific output record then you would need to use the node.execCount value as part of the 'if' test for outputing a record.

You can use the node.write() function to ouptut multiple records for a particular input record (the default text in the Transform node shows one example).

In the example below records are only output once all the input records have been processed (in this case none of the input data is actually used). A list of color values and an empty list are initialized in the ConfigureFields script. The Idx list has the execution count appended for each record (just as a proxy to indicate some 'work' being done for each record processed). Once all records have been processed the node outputs the multiple records comprising the values of the elements in the Idx list and the color values from the rainbow list. 

Note that this is just an example of how to write multiple records for a particular input record (the last in this case - but you can do something similar for the first or the Nth). It is not intended to be a robust example as if there were less than seven input records it would have generated an index out of range error for the Idx list.