🔒 Locked

conversion problems

Started by nextgengamer Jan 16, 2003 at 6:05 PM 1 replies 200+ views

Original Post

Jan 16, 2003 at 6:05 PM

Hello all, I''m having a weird bug that I think is because of converting var types but I cant seem to catch it in the debugger. Let me show you the code. double insert = currOC * 100.0; insert += 0.01; int woah = insert; -currOC can be anything from 0.01 -> 0.24 -the line with insert += 0.01 fixes the problem is a hack that works but I would sill like to know whats going on. when currOC is 0.15 -> 0.24 my woah var becomes 15 -> 24 for 0.11 -> 0.14 woah becomes 10 -> 13 (it rounds down) then from 0.03 -> 0.10 it works correctly 0.01 -> 0.02 it rounds down again. This seems weird to me because the value of insert always equals a whole number like 15.000000000, or 9.00000000000 so I dont see where the rounding down is comming from? Could it be because 14.00000000000 is really 13.99999999999999 or something? Because that hack of adding .01 to the value fixes everything. Please help if you can. Thanks, nextgengamer

Miserable

Jan 16, 2003 at 6:10 PM

Floating-point values, unfortunately, are not completely accurate. You will have to deal with little things like this.

On another note, you may already be aware of this, but if you want to round a floating-point number to an integer, a cast is not going to perform the rounding off for you, but merely truncate at the decimal point. A hack (which I frequently employ) to correctly round off a number is to add 0.5 before casting.

nextgengamer

Author

nextgengamer

Jan 16, 2003 at 7:29 PM

Thanks for the reply Miserable. I remember hearing something about floating point numbers not being completly accurate. I knew that when you convert from a float or double to an int it just drops the decimal place. However, the number the debugger is telling me is that the value is a whole number like 14.0000000 but when it drops the decimal it didn''t round to 14 it round to 13 making me think that floating point error was saying it was 14.0000000 but in fact it may be 13.9999999999999 or something like that.

-nextgengamer

Topic Locked

This topic has been locked by a moderator. New replies are not allowed.