LAE

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: kursattaskin

Hi there
Do you have any idea about how to make Market Basket Analysis using LAE?

Kind Regards,
Kursat

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: Tim Meagher

Hey,

I have never looked into this (or heard of market basket analysis) before reading this post, so someone can probably give you better advice.

That being said ...

The first thing that comes to mind is that it would probably make sense to look and see if there are any publicly available implementations of this sort of analysis in R, via the R community.
If so, you could then use the R node: download here: http://www.lavastorm.com/softwaredow...ndtrials#rnode - and make sure you also download all of the required additional R software as mentioned in the Release Notes.
From a quick Google search, it looks like there are a few resources.
I'm sure there are others, but the following seemed to jump out:

• http://cran.r-project.org/web/packag...les/index.html
• http://prdeepakbabu.wordpress.com/20...ackage-arules/

I believe there would also be some additional work required within the R node to ensure that you are passing the data from the LAE to R correctly.
The R tutorial can be used to go through these steps.

This looked like an interesting problem so I thought I'd take a quick look at seeing how this could be done using "standard" LAE nodes.
I've attached my attempt to get this sort of analysis done in the LAE.

The logic of all of this is performed in the "Analyze Basket" node.
The node takes one input, where the input needs to contain a unique identifier for each transaction (or "basket") and a field which identifies an item purchased in the transaction.
Each of the items within a given transaction need to be on a separate record.

The node is then parameterized by providing the names of these fields.

The node then outputs the following:

• "IfItem" - First item in the permutation
• "ThenItem" - Second item in the permutation
• "OccurrenceCount" - Number of times this item pair appeared in a transaction together
• "ItemCount" - Number of times the "IfItem" appeared in any transaction
• "Probability" - If the "IfItem" appears in a transaction, this identifies the probability that the "ThenItem" would appear in that same transaction
• "Support" - probability of a transaction containing the "IfItem"

The Analyze Basket node is a composite, so you can drill into it to see how it's implemented and then modify the details to suit your needs.
I've added some additional parameters to the node to allow for thresholding for the minimum number of occurrences of a given item pair, the minimum probability value, and the minimum support value.

As mentioned previously, I haven't really looked at this type of analysis before, and I just put this together relatively quickly. It's most likely very inefficient, but it should give the basis to try and do what you want.

There are a lot of smarter people than me who have spent a lot more time than me on this sort of problem who have implemented this sort of functionality in R though, so I think that trying to put together an R based solution is probably the best way forward if you are looking at using this for large data quantities. The implementation attached is really just to give you an idea of how this sort of thing can be done using only the LAE "core" nodes.

Tim.
Attachments:
MarketBasket.brg

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: kursattaskin

Hi Tim
Great work thanks a lot. At the same time, by spending my Sunday  actually I have tried to implement the algorithm according the pdf that I have attached. I do not have enough energy to explain all I did since it is 01:00 AM in Turkey. But, you are better and did a good work. Thanks again.

I have just attached mine work to yours. They give different results but I have to check what makes difference, in performance site, yours is better. I could not do this with Brain Script, however I tried python, even I do not know anything about python before. I did the combinations with python. So from this work, I learned something about python, how to implement in LAE  these are lessons.

For R, actually with small data sets it works however, for large datasets like 500 million, it gives memory error. R is not good for large sets.

Market Basket Analysis is a good application especially in retail sector, however it has good use cases in fraud detection also.

Thanks again

Kursat

MarketBasket-Combined.brg
07apriori.pdf