Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.
Originally posted by: stonysmithOriginally posted by: jpstory
This is awesome!
A minor issue is that except for _RowNum all the rest field names need to be input into Pivot - Names To Data node.
The files I am dealing with between 1500 and 2000 fields ( I know, it's crazy), it'll be difficult to keep inputing the field names for every different report.
That is why Python come up as an option, hope you understand.
I do understand fully. Been there, bought the t-shirt.
The code below should handle your data. There is an "opportunity" with our python node in that it can't read all the records, count them, and then re-wind for a second pass. So, I added a "cheat" where I first use an Agg to count the records and then send that count into Python as a secondary input. Ugly, but it works.
Not trying to beat a drum here, but this method will work as long as you don't fill up RAM memory.. it is usually a 4gb limit.
Separate issue: I would expect troubles with the BRDViewer if you have thousands of columns. Remember.. when you click on "View Data", then the file contents have to be transferred down to your PC - that can take time, and the BRDviewer may have some difficulty if the individual rows are too wide (too many columns).
node:ArrayOperation_2
bretype:core::Python
editor:Label=ArrayOperation
editor:sortkey=57b31c7215cd3f94
input:57b31da559f00ea6/Input1=Static_Data.40fe6c55598828e5
input:58dd0f8102d1537d/out1=Agg.40fd2c744c862db0
output:57b32c5a64eb261c/Out1=
prop:MaxRecdCnt=10
prop:Python2Implementation=<<EOX
import braininfo
def setup(brainNodeControlObj, BrainNodeClass):
class BrainNode(BrainNodeClass):
def initialize(self):
super(BrainNode, self).initialize()
inRec = self.inputs[1].read()
self.rowCount = inRec["__Rows"]
metadata = self.newMetadata()
for i in range(self.rowCount):
metadata.append("Row_"+str(i+1),"String")
metadata.append("FieldName","String")
self.outputs[0].metadata = metadata
self.CurrentRow = 0
self.inRec = self.inputs[0].read()
self.colCount = len(inRec)
self.myArray = [["x" for C in range(self.colCount)] for R in range(self.rowCount)]
def finalize(self, val):
outRec = self.outputs[0].newRecord()
for row in range(self.colCount-1):
fn=self.inputs[0].metadata[row]
outRec["FieldName"] = fn[0]
for col in range(self.rowCount):
outRec[col] = self.myArray[col][row]
self.outputs[0].write(outRec)
super(BrainNode, self).finalize(val)
def pump(self, quant):
while quant.permitsRunning(self):
if not self.inRec:
return False
for col in range(self.colCount-1):
self.myArray[self.CurrentRow][col] = self.inRec[col]
self.CurrentRow = self.CurrentRow+1
self.inRec = self.inputs[0].read()
return BrainNode
EOX
prop:RecordLength=5
editor:XY=990,210
editor:Notes=This node will evenly distribute the records in the input pin among all output pins until all output pin reaches the max threshold.
editor:Notes=All the remaining records will be dumped into the last output pin
editor:propdef=RecordLength|string|Input|ls.brain.node.transpose.RecLen|None
editor:propdef=MaxRecdCnt|string|Input|ls.brain.node.transpose.MxRdCnt|None
end:ArrayOperation_2
node:Agg
bretype:core::Agg
editor:sortkey=58dd0f7c04c62348
input:@40fd2c7427456e5b/=Static_Data.40fe6c55598828e5
output:@40fd2c744c862db0/=
prop:GroupBy=<<EOX
1
EOX
prop:Script=<<EOX
c=groupCount()
emit c as __Rows
where lastInGroup
EOX
editor:XY=900,240
end:Agg
node:Static_Data
bretype:core::Static Data
editor:sortkey=58d532be43b56c18
output:@40fe6c55598828e5/=
prop:StaticData=<<EOX
color:string, id:int,type:string,rand:int,junk:string
Red,1,primary,27959,okay
Green,2,primary,7138,"""Let's Go"", he said"
Blue,3,primary,15790,maybe
Yellow,4,secondary,17809,nope
Cyan,5,secondary,16901,Not NULL
Magenta,6,secondary,15488,null
Orange,7,tertiary,14584,NULL
Chartreuse,8,tertiary,3465,True
Aquamarine,9,tertiary,18650,1234
Azure,10,tertiary,26377,23.4
Violet,11,tertiary,27272,06/25/70
Fuchsia,12,tertiary,6803,"June 25, 1970"
Grue,13,fictional,7373,2:46 PM
Bleen,14,fictional,22852,1.00001
Octarine,15,fictional,11228,3.1415
Garrow,16,fictional,26567,1.2.3.4.5
Gendale,17,fictional,30176,
Hooloovoo,18,fictional,12092,WieRD CaSED stRINg
Fire,19,NULL,12345,14:46
Ice,20,NULL,12345,2:46 PM 06/25/1970
EOX
editor:XY=650,290
end:Static_Data