Showing posts with label floating-point arithmetic. Show all posts
Showing posts with label floating-point arithmetic. Show all posts

Monday, 17 September 2007

float precision

So, I want to test that some floating-point helper function returns a certain value - but I only want it to check to a certain precision (so I don't get hit with negligible rounding errors).

Add this to config/environment.rb (and why doesn't this already exist in Ruby?)

class Float
  def precision(pre)
    mult = 10 ** pre
    (self * mult).round.to_f / mult
  end
end

Then you can do things like this:

assert_equal -5087.35, order1.profit_loss.precision(2)

[Edit: I've replaced 'truncate' with 'round' - it's the only change I've made in two years and I'm still using this code... maybe I should put it into the Rails core-extensions instead?]