Welcome to the Second Life Forums Archive

These forums are CLOSED. Please visit the new forums HERE

Dividing with float still rounds number?

Jasmin Summers
Registered User
Join date: 16 Feb 2006
Posts: 23
08-09-2009 12:54
Hello,

Suppose the following:
CODE

float num = ((13 / 10) - 1) * 10;
llOwnerSay((string)num);



Can someone tell me why num always equals 0.000 rather than 3 and how to get the correct answer out of lsl? The correct answer is 3.


Thanks.
Hewee Zetkin
Registered User
Join date: 20 Jul 2006
Posts: 2,702
08-09-2009 13:10
The literals '13' and '10' are integers. When dividing an integer by an integer, integer (truncating) division is performed. It doesn't matter that later you plan to assign the result to a float; it only matters what types you are operating on in the expression. You'll want to convert (at least one of) those numbers in the division to floating point numbers before doing the division. So something like:

CODE

float num = ((13.0 / 10.0) - 1.0) * 10.0;


or even:

CODE

float num = (( ((float)13) / 10) - 1) * 10;


since the result of the first operation is a float, and each operation thereafter will result in a float since one of the operands is a float.
Rolig Loon
Not as dumb as I look
Join date: 22 Mar 2007
Posts: 2,482
08-09-2009 13:14
All of your numerical operations are on integers. The order of operations is from inside to outside, so you divide the integer 13 by the integer 10 and you get 1.3, which rounds down to 1. Then subtract 1, which makes it zero. Then multiply it by the integer 10, which is still zero. Then cast the integer result to a float, which makes it 0.0.

Have you tried float num = ((13.0/10.0) - 1.0) * 10.0 ?
_____________________
It's hard to tell gender from names around here but if you care, Rolig = she. And I exist only in SL, so don't ask.... ;)

Look for my work in XStreetSL at
SuezanneC Baskerville
Forums Rock!
Join date: 22 Dec 2003
Posts: 14,229
08-09-2009 13:21
I don't see how you expect to get 3 out of it.
_____________________
-

So long to these forums, the vBulletin forums that used to be at forums.secondlife.com. I will miss them.

I can be found on the web by searching for "SuezanneC Baskerville", or go to

http://www.google.com/profiles/suezanne

-

http://lindenlab.tribe.net/ created on 11/19/03.

Members: Ben, Catherine, Colin, Cory, Dan, Doug, Jim, Philip, Phoenix, Richard,
Robin, and Ryan

-
Hewee Zetkin
Registered User
Join date: 20 Jul 2006
Posts: 2,702
08-09-2009 13:23
From: SuezanneC Baskerville
I don't see how you expect to get 3 out of it.


((13/10)-1)*10 = (1.3-1)*10 = 0.3*10 = 3
Jasmin Summers
Registered User
Join date: 16 Feb 2006
Posts: 23
08-09-2009 13:25
Thanks guys, appreciate your time and help. I should have known better, I just forgot about the float/integer and adding a decimal to let lsl know its a float. I've probably asked this same question years ago lol.


@ SuezanneC... ((13 / 10) - 1) * 10 = 3
Look at google calculator http://www.google.ca/search?hl=en&q=%28%2813+%2F+10%29+-+1%29+*+10&btnG=Google+Search&meta=&aq=f&oq=
SuezanneC Baskerville
Forums Rock!
Join date: 22 Dec 2003
Posts: 14,229
08-09-2009 13:27
From: Hewee Zetkin
((13/10)-1)*10 = (1.3-1)*10 = 0.3*10 = 3

Thanks, I guess it's been a while since I've read any math outside the context of scripts and programs.
_____________________
-

So long to these forums, the vBulletin forums that used to be at forums.secondlife.com. I will miss them.

I can be found on the web by searching for "SuezanneC Baskerville", or go to

http://www.google.com/profiles/suezanne

-

http://lindenlab.tribe.net/ created on 11/19/03.

Members: Ben, Catherine, Colin, Cory, Dan, Doug, Jim, Philip, Phoenix, Richard,
Robin, and Ryan

-