Monday, 17 September 2007

float precision

So, I want to test that some floating-point helper function returns a certain value - but I only want it to check to a certain precision (so I don't get hit with negligible rounding errors).

Add this to config/environment.rb (and why doesn't this already exist in Ruby?)

class Float
  def precision(pre)
    mult = 10 ** pre
    (self * mult).round.to_f / mult
  end
end

Then you can do things like this:

assert_equal -5087.35, order1.profit_loss.precision(2)

[Edit: I've replaced 'truncate' with 'round' - it's the only change I've made in two years and I'm still using this code... maybe I should put it into the Rails core-extensions instead?]

9 comments:

kajinski said...

exactly -- why isn't this in ruby? I couldn't find any way to limit float's precision or significant digits, other than converting to a string and using string formatting, or changing the class constant (DIG), which seems like overkill in most cases. thanks for this one....

Taryn said...

Hey no worries.
Yeah, string-conversion feels like overkill. Even this does!

I've recently updated this to go on "Numeric" instead - so it will work with BigDecimals too, with a few error-conditions being checked.

class Numeric
def precision(pre)
return self.truncate if pre == 0
mult = 10.0 ** pre
(self * mult).truncate / mult
end
end


If it were to go into Rails, I'd even consider changing truncate to allow me to pass in an optional precision.

Paul Battley said...

There is already a way to do this in Test::Unit:

assert_in_delta expected, actual, epsilon

Where epsilon is a small number, e.g. 0.0001. This is equivalent to:

(expected - actual).abs < epsilon.

Taryn said...

Yes, you're quite right - you can use assert_in_delta while testing - but since then, I've actually used this method in real code to truncate a floating point number to a specific precision.

Even so, I'd still want a convenience function. I don't care if my test number needs 0.001 epsilon, I want to know if it's correct to two decimal places. I'd like the parameter to be 2, because that actually makes sense to me. :)

Sohan said...

Thanks for the code.
You could use the round method of Float class instead. The round method takes as input the number of decision places you want from it.
you can see the ruby-doc for more.

Taryn said...

Hi Sohan,

thanks for the comment.
You might not have noticed, this blog article was posted almost two years ago and Float didn't have precision then.

Good to know they've added it in since ;)

Cheers,
Taryn

Taryn said...

Sohan:
irb(main):002:0> 1.23456.round(1)
ArgumentError: wrong number of arguments (1 for 0)
from (irb):2:in `round'
from (irb):2

prashant.vincere said...

In Rails Console
>> 1.23456.round(1)
=> 1.2
>>

In Irb
1.23456.round(1)
ArgumentError: wrong number of arguments (1 for 0)
from (irb):1:in `round'
from (irb):1

Taryn said...

Hi Prashant,

yes, that's weird... it worked for a little while there.

I guess that means it's a rails extension instead of a Ruby one.

Good spotting!