LAE

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: Akintosh1

Hello im completely new at Lavastorm

Right now im testing and trying to learn the basics,



For example, i want to compare fields, and calculate a probability of how well they match each other by fieldname. Which node do i start with and what functions in BRAINscript can i use ?

Example :

Field 1 xyz

Field 2 xyz-SE

Match-probability = 60%

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: ltolleson

What algorithm are you using to determine Match Probability? There is a node called "Fuzzy Join" which can determine the number of edits it takes to make field1 = field2. This node will return a score, which is basically the number of edits. I don't know of a single BRAINScript function that will do this, but if you share the algorithm then we can easily write the BRAINScript code to accomplish this.

Thanks,
Larry

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: ryeh

Here's an example of what Larry was describing with our fuzzy nodes. It's basically checking for exact matches first (X-Ref node), and then running through the non-matches through a series of Fuzzy X-Ref (using the Levenshtein Distance to score the match). The fuzzy "score" is the number of edits to go from one string to another. You can sort of use it to calculate a percentage, but it's not quite what you've described. For example, both XYZ/XYZA and XYZ/XYA will have a score of 1. The first pair requires one edit to add the extra letter A where the second pair requires one edit to change the Z to an A.

If it's always the case of one string is contained in another another, i.e. Field1 will also be contained in Field2 or vice versa, then we could potentially use 'strFind' to find a match (and then 'strlen' to calculate the percentage).
Attachments:
FuzzyCompare.brg

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: Akintosh1

Thanks for reply,

I have not yet created an algorithm for the match. My intention was to use strlen as Ryeh suggested, but as i am new to LS i put my hope to solve this problem from the community.

Ryeh, your example is great, but when i run it , on the first threshold it fails and sais : The following classpath entry for the node does not exist: C:/Program Files (x86)/Lavastorm/LAE6.1/lib/java/lal/hsqldb-2.2.5.jar.

Errorcode: brain.execution.java.node.classpathNotFound

I have LAE 6.1 paid license

What is the problem here ?

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: ryeh

My guess is that some of the libraries were not installed for some reason. I will have our support team reach out to you.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: Akintosh1

Hello Ryeh, the thresholds are now working after reinstallation,

My question is : If i have the fields in one column that i want to compare with each other , not several columns, how do i do this ? The matching needs to be by fieldname, and probability of it, i could not get this to work by Fuzz X ref as it compares two columns with each other.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: ryeh

Hi, Ahmet. So are the two values separated by something, like a comma or a space? In that case, I would split those up into two fields using the 'split' function. See the attached file.
Attachments:
FuzzyCompare.brg

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: Akintosh1

Hi Ryeh, the fields are not separated by anything, they are random numbers and text, for example needs to look like this :

Product: Match:

1AC74B1E9A4CE496BD51569E774CF1C4 17%
02EA52E5121A13D28D2D9FBF275D1F46 9%
301C5450EA7E011D808E72FD4D95E3B8 5%

so the highest Match field is the field that has most matches by each fieldname in the Product table.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: ryeh

I'm not following... So you have a field (or column) called Product. And you and to match that against other records in the same column? Your example that you gave above, how're the 17%, 9%, and 5% calculated?

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: Akintosh1

In Qlikview there is a function called strcmp, which returns the first mismatch position of two strings.

So something similar like this, so the percent should be calculated by the mismatch or the match by each field , where the top, 17 % is the highest match or mismatch of fields.

I also wanted to ask about " BRAINscript " , i want to learn more about the basic syntax, i found " Quick Reference Guide " , but if you have more files about basic brainscript syntax i would be glad to check it out.

https://community.qlik.com/docs/DOC-4651

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: ryeh

Take a look at this submission:
https://infogix.zendesk.com/hc/en-us/community/posts/360050709434

For BRAINscript syntax, another good resource is: Help -> BRE Help -> BRAINscript.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: Akintosh1

Hi Ryeh, thank you for your help