Number length and accuracy in SL

Hi,

I was measuring a land height today with llGround() and got a result of 11.718750. This intrigued me, so i took a few more. I noticed that all the land heights had 6 digits after the decimal place, but the sixth was always zero. Why?

I then rezzed a cube, and set it's size and location, using as many digits as the entry box in the Edit menu would allow. It allowed values of:

52.6666 666666 666666 666666 66666 (30 digits for location)
0.6666 66666 66666 66666 66666 66666 6 (31 digits for size)

Does SL really use this amount of precision?

All decimal pointed values in SL use the IEEE single precision (32 bits) "float" format. See http://en.wikipedia.org/wiki/IEEE_754.

A normalized float uses 24 bits (the first is always assumed to be 1) for the mantissa. That translates to +/- 8 million, about 7 significant decimal places.

Actually, I don't think this is QUITE true. All positions, scales, rotations (the internal quaternion), and quite possibly colors do seem to use 32-bit floats, but lots of primitive parameters seem to use highly restricted floats that seem like they might well be implemented internally as small-range integers. The latest update did increase the precision of quite a few primitive parameters, though.

One interesting part SL using 32-bit floats for things like position is that the position box might say "125.000", but the object is actually at 125.000012... and this affects seams. Sometimes the only way to set this correctly is with a script.

There are 6 digits after the decimal because casting a float to a string, which is necessary to present it to you in a human-readable form, always includes 6 digits after the decimal, even if they're all 0. The last digit always being zero in your tests, however, is probably coincidental.

Ground height is based on a grayscale heightmap, so it is likely that it is locked into discrete "steps" of some binary fraction of a meter at max resolution. As such, it will not use irrational decimal fractions.

The values of height in between the meter points wil be irrational. Some brave soul needs to dissect the client source to see what exactly this fnction is that extrapolates between the points of the raw heights map. Afaik, the only info sent to the client about a sim is a 256 x 256 table of heights (0 - 255) , a similar array of multipliers, parcel flags.

Then again, LL functions that return ground information (height and slope) are run on the server so perhaps there is no extrapolation on the client, just approximation. We might never find out what the extrapolation function is on the server.