LAE

Welcome to the LAE community!  Please feel free to start a discussion in the discussion tab or join in a conversation.

Discussions

Members

Resources

Events

 View Only
  • 1.  stop rounding

    Employee
    Posted 04-12-2010 22:40

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: PGeee

    I have a case of spradic unwanted rounding in a str conversion

    Value = div(double(Amount),1000)
    Val = str(Value)

    with amount as follows
    node:BRD_Examiner
    bretype:core::Filter
    editor:Label=Value = 1020.715
    prop:Script=<<EOX
    emit * where
    'Value' == 1020.715
    EOX
    end:BRD_Examiner

    the result in Val is 1020.720

    The weridest thing is I process tens of thousands of records like this with vastly varying amounts, amd this occurs about once every three months, so the statmenet as it stands is 99.99999 percent ok


  • 2.  RE: stop rounding

    Employee
    Posted 04-12-2010 23:46

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: mermacora

    Hi Paul

    The problem here is that you're doing an equality test on two floating-point values, and this is inherently unreliable due to the nature of floating-point implementation in all modern computer systems.
    See the below post that attempts to explain this in detail:
    http://community.mda-data.com/showthread.php?t=82

    Hope this helps,
    Mario