LAE

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: jpstory

Hi

I am trying to extract some elements out of a XML file with Tabular XML File node and XMLpy File node, but running into what appears to be a namespace issue:

For example, I can extract the "isCorrection" element only if I manually remove the namespace (bold below). If I don't remove it, my result will be nothing / blank.

Could you please explain why this is happening and what is the workaround ? Thanks !

Here is the code I used

data = {}
@elementHandler('/abcReport')
def EntryHandler(element):
data['isCorrection'] = str(element.isCorrection)
outputRecord(data, 0)

Removing Namespace:
<?xml version="1.0" encoding="UTF-8"?>
<abcReport>
<header>
<creationTimestamp>2015-06-04T16:23:52Z</creationTimestamp>
</header>
<isCorrection>false</isCorrection>
</abcReport>

Original Format:
<?xml version="1.0" encoding="UTF-8"?>
<abcReport xmlns="http://www.fpml.org/FpML-5/transparency" fpmlVersion="5-5">
<header>
<creationTimestamp>2015-06-04T16:23:52Z</creationTimestamp>
</header>
<isCorrection>false</isCorrection>
</abcReport>

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: ejones

Strange that it worked for me. I wonder what I did different.

I took the Original Format above and found the isCorrection value is being put into a field named abcReport.isCorrection. I am using the XML Data node with nothing changed except specifying the input field containing the XML to process.

node:XML_Data
bretype:lal1::XML Data
editor:sortkey=55847f2d25750787
input:55847f2d48115f26/=Input_Static.51ae0881326a4c48
output:@51781d234c465db7/=
output:@522743291aa237ba/=
output:@522743281dab2bc6/=
prop:XmlData=_RawData
prop:XmlDataParamType=Data Field
editor:XY=200,180
end:XML_Data

node:Input_Static
bretype:lal1::Input Static
editor:sortkey=55847f0c16c203d2
output:@51ae0881326a4c48/=
prop:StaticData=<<EOX
<?xml version="1.0" encoding="UTF-8"?>
<abcReport xmlns="http://www.fpml.org/FpML-5/transparency" fpmlVersion="5-5">
<header>
<creationTimestamp>2015-06-04T16:23:52Z</creationTimestamp>
</header>
<isCorrection>false</isCorrection>
</abcReport>
EOX
editor:XY=120,180
end:Input_Static

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: jpstory

Hi

I used XMLpy File node not XML Data, the reason why I did not use XML Data node was because there was only record in each XML file.

However the structure of the record is fairly complicated, some elements have 8 layers of childnodes and each layer have multiple attributes as well.

I need to pull out multiple specific elements and attributes from the file, but XMLpy could not process the file properly once there's name space in it.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: Tim Meagher

Hi,

So is the problem with using the XML Data node that you can't exclude the elements and attributes which appear multiple times & then cause the node to output multiple records?
If so, while you can't exclude them, you can create an additional output pin on the node, and name that output pin with the path to the elements you want to exclude.
Then, the repeating elements which are causing the problem would be output to that new output pin, and you can just ignore that output pin - obviously, it would be better to be able to have a parameter to be able to explicitly exclude certain element paths in the provided XML, which I believe is an existing enhancement request.

As an example if you had an XML format like:
Then if in the node, you create an additional output (so it has 4 outputs) and call the new output "root.B", Then the B, B1, BB1, BB2 elements would be written to that output, and it would have multiple records since these repeat in the data.
However, the main "Data" output would only have one record containing the elements A and C under root.


Tim

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: jpstory

Hi Tim

Thanks, that works well!

Unfortunately it solves only part of the puzzle. Some elements I am trying to extract are repeated elements.
And I am supposed to piece these repeated elements back with the single record in the main Data pin to create one single record that contains all the information.

Using your example above, the end result should be something like this:

A, B.B1.BB1, B.B1.BB2, B.B1.BB1, B.B1.BB2
foo, something, somethingElse, somehtingNew, somethingElseNew


I have multiple such structure within the source file at different level of childnode, which adds the complexity.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: Tim Meagher

Hi,

The XMLpy File node doesn't support namespaces.
I think what you're going to need to do is to treat the extraction of individual elements of the repeating fields & getting them onto the same single record as a post acquisition phase via either filtering or agging the data, followed by a lookup.

It is difficult to say exactly which the best approach is without knowing which of the elements of the repeating fields are to be extracted and how.

Regards,
Tim.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: jpstory

Hi Tim

I found a manual workaround for this, to get XMLpy node to work I manually open the source XML file and replace "xmlns" with "ABC" (where bolded below).

My question now is, how can I get the replacing done in lavastorm?
Tried importing the file via CSV node and performing replacing with a filter node, but do not know how to output the results with proper format, could you please advise.

If the above works, another question would be, how to extract repeated elements, I believe the default setting is to only extract the first repeated element / overwrite previous repeated element.


<?xml version="1.0" encoding="UTF-8"?>
<abcReport xmlns="http://www.fpml.org/FpML-5/transparency" fpmlVersion="5-5">
<header>
<creationTimestamp>2015-06-04T16:23:52Z</creationTimestamp>
</header>
<isCorrection>false</isCorrection>
</abcReport>

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: Tim Meagher

Hi,

I haven't used the XMLpy File node much so can't help on the question of extracting repeated elements.
I always just use the XML Data node.
For the question of outputting the data, I *think* that the Output Raw node should give you what you need, though again I haven't used that much either.
Hopefully someone else here can shed more light??

In any case, if you have to extract repeated elements in the XMLpy File node aren't you going to have to specify that in the python code snippet used in the node...
In which case, if you have to specify which fields to extract, *and* need a pre-processing step to remove/modify the namespaces, I'm wondering if it wouldn't be simpler using the XML Data with some pivoting as shown in the attached example. Anyway, if you really need the XMLpy File, I think someone else on here might need to provide the help sorry.

Tim.
Attachments:
Example using XmlData.brg

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: Bryt Park

Hi Tim

The solution file you posted is in 6.0 version, I can not open it. Could you please resave it v4.6.1.2 version compatible, thanks.

I am afraid I have to stick with XMLpy File node, as XML Data node would create 13 records for each xml record. Also there are some complicated logic I need to apply in order to piece the 13 records back properly and I need to process 100+ files daily.....

Could someone else please take a look at:
1. how to handle repeated elements
2. remove the word 'xmlns' from the source file


Or can someone demonstrate how to extract an element from an XML via Python node so I can start, thanks!

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: Tim Meagher

Hi,
See attached for a BRG5 version of the graph previously posted.
I think this should open in 4.6.1.2.

If you want to use XMLpy File though, I guess this won't help.

Tim.
Attachments:
Example using XmlData.brg

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: jpstory

Hi

Could anyone provide an example of using Python node to parse repeated elements from an XML file, thanks.