LAE

Welcome to the LAE community!  Please feel free to start a discussion in the discussion tab or join in a conversation.

Discussions

Members

Resources

Events

 View Only
  • 1.  Data Profiling

    Employee
    Posted 08-19-2016 01:29

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: cpapageo

    I am trying to use a field to create distribution intervals eg 5 units and distribute my records to these intervals for data profiling (count per interval etc). Is there a node to use? Does anyone have an idea how to proceed?


  • 2.  RE: Data Profiling

    Employee
    Posted 08-19-2016 06:24

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: ltolleson

    If you have the LAL Lavastorm library installed there are some nodes under the "Profiling and Patterns" node section that might help. Specifically look at the "Interval Inspection" and "Data Profiling" nodes. You can download the libraries from http://www.lavastorm.com/current-rel...orm-libraries/. Select the appropriate library for your client and server. You must install the same version of LAL on both client and server.


  • 3.  RE: Data Profiling

    Employee
    Posted 08-19-2016 11:43

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: prasmussen

    Sounds like 'Band By Strata' node is what you need if the band remains constant. In the example below, the band is increment is 5 so there are 4 bands between 1-20. The output field _Band puts them into the band.

    node:Static_Data
    bretype:core::Static Data
    editor:sortkey=57b752757b1d1b45
    output:@40fe6c55598828e5/=
    editor:XY=340,320
    end:Static_Data

    node:Band_by_Strata
    bretype:lal1::Band by Strata
    editor:sortkey=57b75259564f0969
    input:@40fd2c74167f1ca2/=Static_Data.40fe6c55598828e5
    output:@40fd2c7420761db6/=
    prop:Increment=5
    prop:InputExpr=<<EOX
    'id'
    EOX
    editor:XY=430,320
    end:Band_by_Strata

    node:Quick_Stats
    bretype:lal1::Quick Stats
    editor:sortkey=57b753cb01484bc5
    input:@4b4668c040aa5a85/=Band_by_Strata.40fd2c7420761db6
    output:@4b4668e708143fb4/=
    prop:FieldList=<<EOX
    'id'
    EOX
    prop:GroupByExpr=<<EOX
    '_Band'
    EOX
    prop:SortInput=true
    editor:XY=550,320
    node:Bypass
    bretype:::Bypass
    editor:shadow=5170486774e16b95
    input:@4b467f7e129d45c1/=
    input:@4b467f830ffe047b/=
    output:@40fd2c7436717256/=
    end:Bypass

    node:Sort
    bretype:::Sort
    editor:shadow=51704867723c35b0
    input:@40fd2c743ebf4304/=
    output:@40fd2c746a2a3b47/=
    end:Sort

    node:Agg
    bretype:::Agg
    editor:shadow=517048675e0a6211
    input:@40fd2c7427456e5b/=
    output:@40fd2c744c862db0/=
    end:Agg

    end:Quick_Stats


  • 4.  RE: Data Profiling

    Employee
    Posted 08-23-2016 01:43

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: cpapageo

    Dear both, thank you for your responses. Let me elaborate on what i would like to do: The scope is to create a dimensional table (2 dimensions 16x16) with revenue and costs, then populate the table with each interval’s count number (256 in total)… Please have a look to the partial filter node code:

    output 1

    {emit *
    where revenue6month >= 0.00 and revenue6month <= 5.00 and cost6month >= 0.00 and cost6month <= 5.00
    }
    output 2

    {emit *
    where revenue6month >= 0.00 and revenue6month <= 5.00 and cost6month > 5.00 and cost6month <= 10.00
    }
    output 3

    {emit *
    where revenue6month >= 0.00 and revenue6month <= 5.00 and cost6month > 10.00 and cost6month <= 15.00
    }

    and so on, 256 outputs in total (the node is running for two days&^&^(*....)


  • 5.  RE: Data Profiling

    Employee
    Posted 08-23-2016 07:46

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: stonysmith

    I think this graph will do what you are asking for, and it will run much faster.
    256 discrete outputs is probably the wrong way of achieving a result.
    Profiling.brg


  • 6.  RE: Data Profiling

    Employee
    Posted 08-24-2016 06:41

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: cpapageo

    I agree not efficient at all...i have tried band by strata twice and then pivot table and the result was satisfactory. I will try the node you suggest as well..i suppose it is a test node...?


  • 7.  RE: Data Profiling

    Employee
    Posted 08-24-2016 06:46

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: cpapageo

    Unfortunately i work on LAE 5.1 Server edition, could not open the brg file...


  • 8.  RE: Data Profiling

    Employee
    Posted 08-24-2016 07:12

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: stonysmith

    Here is a version5 copy:
    Profiling_v5.brg