LAE

Welcome to the LAE community!  Please feel free to start a discussion in the discussion tab or join in a conversation.

Discussions

Members

Resources

Events

 View Only
  • 1.  Dynamic creation of new columns

    Employee
    Posted 02-07-2014 04:06

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: feanor0

    I've got a variable number of input columns called 'Asset0', 'Asset1', ..., 'Asset[NumAssets]' where NumAssets > 0

    My data looks a bit like this for NumAssets = 3, say.

    Asset0, Asset1, Asset2, Asset3, Term1, Term2
    0.1, 0.3, 0.4, 0.5, 0.9, 0.1
    0.2, 0.4, 0.5, 0.6, 0.8, 0.2

    I want to do the following computation for each of n = 1, 2, ...

    value1 = Term1 * Asset0 + Term2 * Asset1
    value2 = Term1 * Asset0 + Term2 * Asset2
    ...
    value[NumAssets] = Term1 * Asset0 + Term2 * Asset[NumAssets]
    and emit each value as new column called Value1, Value2, ...

    (Note that Asset0 always participates in the computation.)

    How could I do this? If this is not possible, I don't mind overwriting the Asset1, Asset2, ... columns with the newly computed corresponding Value1, Value2, ...

    I tried the following code in a Filter node, but I'm getting errors:

    output 1 {
    	emit wildcard "Asset*"
    
    }
    i = 1
    while i <= NumAssets {
    	fieldName = strcat("Asset", i)
    	value = Asset0 * Term1 + field(fieldName) * Term2
    	do output 1 {
    		emit value as strcat("Value", i)
    	}
    	i = i + 1
    }
    Please advise.


  • 2.  RE: Dynamic creation of new columns

    Employee
    Posted 02-07-2014 09:54

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: Tim Meagher

    Hi,

    The problem you are running into is that there is currently no way to specify field names dynamically via a variable.

    This can be done way how you can achieve what you want by having these values output once per record.
    Then you can use the Pivot - Names to Data node to transform these values to field names.

    I've attached an example that shows how it can be done.
    It's pretty similar to what you had already, just with the change that all values get output first to a field "Value" before being transformed back via the pivot node.

    Hope this helps,
    Tim.
    Attachments:
    assetValueExaple.brg


  • 3.  RE: Dynamic creation of new columns

    Employee
    Posted 02-10-2014 09:58

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: feanor0

    Thanks for this, Tim, very useful.

    What I find, though, is that when there are, say, 12000 rows with 5 different names (Asset0, Asset1, ..., Asset4), then the Data-to-Names pivot takes a really long time. Is there a way to speed it up? I thought sorting the input in the Data-to-Names node would help, but it appears to be the same as 'default'?

    I'm looking at much larger tables of assets - perhaps 40, and each with 20,000 rows, so any tips on executing more efficiently will be very welcome.


  • 4.  RE: Dynamic creation of new columns

    Employee
    Posted 02-11-2014 11:41

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: Tim Meagher

    Hi,

    Given that BRAINscript has the limitation of not being able to specify the field name to output as a variable, I can't think of a way to speed this up significantly unless you went down the path of writing a custom python or Java node for what you are trying to achieve.
    As a side note, setting the "SortInput" parameter on the data to names node to true will actually force it to sort the input.
    If you look at the documentation for the parameter, you'll see that the default is also true.

    However, you should be able to speed things up a little bit by setting SortInput to false.
    The node groups by "execId" therefore, only needs to be sorted by this field.
    Since the data coming into the node will already have the records in the correct order (i.e. in terms of increasing execId) then you can safely change the "SortInput" parameter to false, which will mean that the step of sorting the data will be skipped and speed things up somewhat.

    Regards,
    Tim.