LAE

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: Andrey Kateshov

I have a python node with a custom configured output. Below is the code i use to initialize the output:

m = self.newMetadata()

m.append("GID","int")
m.append("RAS_GBV","double")
m.append("IFRS_GBV","double")
m.append("IFRS_NBV","double")
m.append("Yrly_Charge", "double")
m.append("Dep_ed_to_Month", "int")
m.append("DEPRECIATE","boolean")
m.append("RUL","int")
m.append("Res_Val","double")
m.append("IFRS5","boolean")
m.append("IAS40","boolean")
m.append("FULLDISPOSAL","boolean")
m.append("AccDepn","double")

self.output.metadata = m

I have a total of 4 mln lines sended to this output and the python node takes around 30 minutes to processs. I figured that although I have a very time consuming code to prepare the actual data, which is then sended to the output, 90% of these 30 minutes relates to writing result to the output itself.

i.e. the following line of code slows the whole thing:
self.outputs[0].write(self.record1)

Is there any way how I could speed up the output process? Maybe something like a batch output? Please help!

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: stonysmith

Are you sure that the delay is in the self.outputs[0].write(self.record1) statement?

How are you copying your fields from the input record to the ouptut record?

If you are using Named fields, instead of Ordinals, Python can take a very long time to process the data. Python seems to be slow when searching a list by name.

This code will run very slowly:
self.inrec=self.inputs[0].read()
self.record1["IFRS5"] = self.inrec["RAS_GBV"]
self.record1["IFRS5"] = self.inrec["IFRS_GBV"]
self.record1["IFRS5"] = self.inrec["IFRS_NBV"]
....
....
self.record1["AccDepn"] = self.inrec["AccDepn"]
self.outputs[0].write(self.record1)

Whereas this code will run significantly faster:
self.inrec=self.inputs[0].read()
self.record1[0] = self.inrec[0]
self.record1[1] = self.inrec[1]
self.record1[2] = self.inrec[2]
....
....
self.record1[12] = self.inrec[12]
self.outputs[0].write(self.record1)

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: Andrey Kateshov

I use ordinals, exactly as in your second example. So this should not be the reason.

I read records from a single input. For each input record I do certain computations and then write the computed fields to the custom output. The same type of calculation is repeated for each line of input.

I noticed that for every additional column I add to the output there is a noticeable drop in performance. I.e. with a three column output it takes about 15 minutes to process and with four columns it could be already 30 minutes.

self.record1[0] = JE_ENTRY_ID_0
self.record1[1] = TYPE_0
self.record1[2] = IFRS_NBV * sigma
self.record1[3] = IFRS_GBV * sigma
self.record1[4] = not FULLDISP
self.record1[5] = (IFRS_GBV - IFRS_NBV) * sigma
self.record1[6] = GID_0

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: Tim Meagher

Hey,

When you increase the number of columns, are you talking about increasing both the number of input & output columns?

Also are you increasing the processing performed prior to writing the records or are the variables for these fields already in existence in the 3 column output, where the only difference is that you actually write them to the output in the 4 column case?

Essentially, you have to look at all of the processing that is being performed to understand why this is occurring.

Every time you access a record, and try and read fields off an input record, this record and the fields themselves need to be deserialized into variables accessible within python.

Each operation you perform will also obviously add to the processing time.

In addition, the size of the data you are reading and writing - particularly when processing large numbers of records - will impact performance. This includes naturally the number of fields within the metadata, but also the width of each field. If your first 3 fields were simple integer or boolean fields, and the fourth field is a string which can have a lot of data, then performance will also be impacted.

If in the addition of adding fields, you are introducing additional floating point operations (as in the case of IFRS_GBV * sigma in your example), then this too will have an impact.

If there are no additional calculations introduced, and you are simply adding additional fields whose values have already been calculated within the data, then the jump from 15 minutes to 30 minutes does seem quite large - unless the fourth column contains large string data.

Perhaps posting your python node (or if you don't want to post this publicly, then PMing this) would allow someone to look at what exactly is going on here.

I did a little bit of performance anaylsis of the python node using 5 million input records, where each record contained 26 very simple single-character String fields ("a","b","c",...,"z").

For these tests, each node was to simply to read it's input records, and write them to the output pin. In addition, the nodes output some statistics relating to processing time.
This was done for an unoptimised python node (using String based indexing), an optimised python node (using integer based indexing), a simple filter node and an optimised java node.

The results are in the attached chart - with the elapsed time in seconds.

From this, you will see that the python node performance is significantly worse than that of a filter node (indeed it will be significantly worse than a lot of the brainscript based nodes). The obvious reason being that brainscript allows only a restricted set of operations that can be performed within well defined and restricted nodes, thus allowing for optimized binaries to be used for each node.

However, you might also notice that the Java node (which is available as of LAE 4.5 and onwards) has much better performance than that of the python node, and approaches that of the filter node.

For this reason from 4.5. onwards, when attempting to perform complex calculations/logic which you are unable to do within the standard LAE nodes, the Java node is probably the best option if you want to improve performance compared with a Python node.

While this may not help you at the moment, it is certainly something to consider in the future.
Attachments:
NodePerformanceComparison.jpg