LAE

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: rhallmark3

I've got a CSV file with over 100 fields and was wondering if there was a way to only use the first 20 fields and discard the rest?

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: timonk

Good Day,

Unfortunately, there is not a quick and dirty way of doing this. The file input nodes do not have a "range" ability on input fields, nor is there a similar direct BRAINscript function that you could use in a filter node.

You will have to input the entire file, then use a filter to emit the fields you want to work with (or exclude the ones you don't)

For example:
emit Field1, Field2, Field3....Field20.

This can be a bit tedious. An option is to create a graph parameter, which lists the fields you want to input, then reference it wherever needed. Say you have a string parameter FOO, with value "field1,field2,field3" (ignore quotes when actualy using). Then use this in a filter node:

emit {{^FOO^}} # will emit field1, field2 and field3.

The parameter gives you the advantage of only needing to setup your string of input fields once. If for some reason you wanted to drop one of the fields in one place, but not another you can use the exclude statement

emit {{^FOO^}}
exclude field3

This also works in reverse:
emit *
exclude {{^FOO^}} # will exclude fields1-3, emitting all the rest.


The only other real method to approximate what you are asking, involves setting up a rather large (in your case) Do Output statement construct that would take more time in the long run, (as you would still have to type and list out each field you wanted, assign it a default value...then tie it into the Do Output) and would still require the use of a filter node.

Do you forsee having more complicated needs than simply emitting a leading subset of your data?

Regards,
Timon Koufopoulos,
MDA Support

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: rhallmark3

No, it's not any more complicated. I was hoping that there may be a way to not import the fields that I don't need as there are so many of them and all it's doing is taking up space.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: rboccuzzi

If the fields have relevant names, you can either emit or exclude fields based on both wildcarding and pattern matching... this is documented under BRAINscript, Language Elements, pattern or wildcard.

Essentially, if the fields are named with some sort of pattern, you could do.

emit pattern "field[1-9]"
emit pattern "field1[0-9]"
emit pattern "field20"

or if the extra all had something in it, like "empty"
emit *
exclude wildcard "*empty*"

I hope this helps.

Cheers
Rich

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: ltolleson

If you want to eliminate fields as you are reading the data could possibly do this with a custom Python node.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: rhallmark3

Yeah I was looking into possibly doing this yesterday. I'm not sure where to begin though when I start as none of the existing python nodes we have are like this.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: rboccuzzi

If you use the Python Filter in core, it is really trivial. I modified only the initialize function, to the code below:

''' Called at node initialization, before first handleRecord.'''
super(BrainNode, self).initialize()
outputMetadata = self.inputMetadata
while len(outputMetadata) > 3:
outputMetadata.remove(len(outputMetadata) - 1)

for output in self.outputs:
output.metadata = outputMetadata

This will only give you the first three fields. Not sure what order it is populated, but I think it is the correct one, so this would be the first 3 found.

Cheers
Rich

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: rhallmark3

I was looking at creating a custom python node to use for the import into the graph. I believe the python filter node would only be used if I already had the data in the graph.