LAE

Welcome to the LAE community!  Please feel free to start a discussion in the discussion tab or join in a conversation.

Discussions

Members

Resources

Events

 View Only
  • 1.  Issue with double 0.3

    Employee
    Posted 02-07-2012 11:13

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: jodycrutchfield

    I am having a problem with a filter node not finding 0.3. It finds 0.1 just fine.

    Version is 4.5.2. Here are the nodes i am testing with.

    The filter only outputs the 0.1 record.

    node:Static_Data
    bretype:core::Static Data
    editor:sortkey=4f316561030d2a65
    output:@40fe6c55598828e5/=
    prop:StaticData=<<EOX
    amount:double
    0.3
    0.1
    EOX
    editor:XY=570,570
    end:Static_Data

    node:Filter
    bretype:core::Filter
    editor:sortkey=4f3164b922510308
    input:@40fd2c74167f1ca2/=Static_Data.40fe6c55598828e5
    output:@40fd2c7420761db6/=
    prop:Script=<<EOX
    emit * where
    amount == 0.3
    or amount == 0.1
    EOX
    editor:XY=710,570
    end:Filter


  • 2.  RE: Issue with double 0.3

    Employee
    Posted 02-07-2012 11:43

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: jodycrutchfield

    After discussing with Rich and reading the help about Floating Point Arithmetic I have a solution.

    Instead of using "double == double" I am using the round and convert to int method.

    This makes my filter node look like.
    round(int(10000000000 * amount),1)/10000000000 == round(int(10000000000 * 0.3),1)/10000000000

    My max precision is 10 digits therefore the 10000000000 number.

    The other option would be using strings.
    str(amount) == "0.3"