Second Life Forums Archive - Object compression?

SuezanneC Baskerville

Forums Rock!

Join date: 22 Dec 2003

Posts: 14,229

08-01-2006 19:24

Is the object compression already in effect or does it start tomorrow?

_____________________

-

So long to these forums, the vBulletin forums that used to be at forums.secondlife.com. I will miss them.

I can be found on the web by searching for "SuezanneC Baskerville", or go to

http://www.google.com/profiles/suezanne

-

http://lindenlab.tribe.net/ created on 11/19/03.

Members: Ben, Catherine, Colin, Cory, Dan, Doug, Jim, Philip, Phoenix, Richard,
Robin, and Ryan

-

Infiniview Merit

The 100 Trillionth Cell

Join date: 27 Apr 2006

Posts: 845

08-01-2006 19:49

_____________________

http://slurl.com/secondlife/puea/126/10/60/

SuezanneC Baskerville

Forums Rock!

Join date: 22 Dec 2003

Posts: 14,229

08-01-2006 21:03

To answer my own question:

From: someone

Release Notes for Second Life 1.11.2(2) July 31, 2006
=====================================
Other changes:
* Asset compression expanded to cover objects

It's not implemented yet. There is no mention of removing it in the most current version of the preview release notes so I assume it will be implemented tomorrow.

_____________________

-

So long to these forums, the vBulletin forums that used to be at forums.secondlife.com. I will miss them.

I can be found on the web by searching for "SuezanneC Baskerville", or go to

http://www.google.com/profiles/suezanne

-

http://lindenlab.tribe.net/ created on 11/19/03.

Members: Ben, Catherine, Colin, Cory, Dan, Doug, Jim, Philip, Phoenix, Richard,
Robin, and Ryan

-

CrazyMonkey Feaver

Monkey Guy

Join date: 1 Jul 2003

Posts: 201

08-02-2006 02:11

Awsome.. Aren't they transmited as XML(ish) files?
That scares me greatly... Id prefer structs(binary) to
save bandwidth plus the overhead of text-to-variable conversion.
Is that what the compression is? or is it just still XML'ish + compression?
(and is it RLE or Zerocoded?)

They need two sets of release notes, the normal one plus the geek version.

Dr Tardis

Registered User

Join date: 3 Nov 2005

Posts: 426

08-02-2006 09:48

If you really want to g33k 0ut, download libSecondLife and check out the demo programs that come with.

One is a debugger that dumps session data so you can get a good look at how all that works.

You're right, Crazy: sending binary data IS faster. The down side is that you have to deal with all the niggly little problems with different platforms' implemenation of binary data, including alignment issues that come with structures.

The banking software company I worked at changed compilers twice during my tenure. The first time was to shift platforms (obsolete DG system to IBM AIX) and the second was to switch to a compiler that was still supported (old compiler publisher went out of business).

The biggest problem we had? The compiler did weird things to align data members on boundries. When you have data elements smaller than 1 word (4 bytes), you have to decide how to position them for fastest access. The convention on the new compiler was that objects align on their size: so chars aligned on any byte boundry, ints (16 bits) aligned on even addresses, and long ints (4 bytes) aligned on 4 byte boundries. (Believe it or not, you don't use floats... you use integers with an assumed decimal point). The old compiler just aligned all variables on 4 byte boundries, even single char values. It was important to know how this worked, since we stored data on disc by simply writing the binary data from the struct variables.

Then there are Endian issues to consider: Motrola chips are "Big Endian", meaning that the most significant byte is first in a multi-byte number. Intel chips are small endian - the smallest byte is stored first, then the next largest byte.

Then we consider bytecode compilers: Java, .Net. These are pretty much guaranteed to implement variables differently than C++ does. I'm not completely sure if there is even a supported way to get a bitstream from a C# variable.

So if you consider the differing optimizations between different compilers and the different internal implemenations of binary numbers (thank God that at least floats are an IEEE standard), it becomes apparent pretty quickly that sending binary C structs is not the smartest way to go - either for disc storage or for network transfer. Packed binary streams aren't so bad, but it's important to pack them manually, rather than just send a memory image of the structure.

Infiniview Merit

The 100 Trillionth Cell

Join date: 27 Apr 2006

Posts: 845

08-02-2006 09:54

From: Dr Tardis

If you really want to g33k 0ut, download libSecondLife and check out the demo programs that come with.

One is a debugger that dumps session data so you can get a good look at how all that works.

You're right, Crazy: sending binary data IS faster. The down side is that you have to deal with all the niggly little problems with different platforms' implemenation of binary data, including alignment issues that come with structures.

The banking software company I worked at changed compilers twice during my tenure. The first time was to shift platforms (obsolete DG system to IBM AIX) and the second was to switch to a compiler that was still supported (old compiler publisher went out of business).

The biggest problem we had? The compiler did weird things to align data members on boundries. When you have data elements smaller than 1 word (4 bytes), you have to decide how to position them for fastest access. The convention on the old program was that objects align on their size: so chars aligned on any byte boundry, ints (16 bits) aligned on even addresses, and long ints (4 bytes) aligned on 4 byte boundries. (Believe it or not, you don't use floats... you use integers with an assumed decimal point).

Then there are Endian issues to consider: Motrola chips are "Big Endian", meaning that the most significant byte is first in a multi-byte number. Intel chips are small endian - the smallest byte is stored first, then the next largest byte.

So if you consider the differing optimizations between different compilers and the different internal implemenations of binary numbers (thank God that at least floats are an IEEE standard), it becomes apparent pretty quickly that sending binary C structs is not the smartest way to go. You could create a binary bistream that's packed on one end and unpacked on the other end in some arbitrary fashion, but that would defeat the "open standards" part of LL's mission with SL.

Yeah!

So it makes stuff smaller?

_____________________

http://slurl.com/secondlife/puea/126/10/60/

Dr Tardis

Registered User

Join date: 3 Nov 2005

Posts: 426

08-02-2006 09:56

uhm... yeah.

objects take less bandwidth to transfer, so they transfer faster.

Alan Edison

Ty Zvezda

Join date: 28 Jun 2004

Posts: 420

08-02-2006 09:56

From: Infiniview Merit

Yeah!

So it makes stuff smaller?

OBVIOUSLY... duhhh lol jk

_____________________

Ty Zvezda

Fenrir Reitveld

Crazy? Don't mind if I do

Join date: 20 Apr 2005

Posts: 459

08-02-2006 10:02

From: Dr Tardis

Packed binary streams aren't so bad, but it's important to pack them manually, rather than just send a memory image of the structure.

Seriously, if you are just doing fwrites of your C structs, you deserve whatever cross-platform/cross-compiler problems you encounter.

One thing that strikes me about libsecondlife is the ability to write our own client to interface with SL. For example, an IM client or even a "MUD style" system that would let you do very basic things such as manipulate your landmarks and then chat/IM with people -- Without the overhead of having a 3D graphics client.

As strange as that sounds, it is something I would really dig as I tend to like to chat with people but actually don't care about the visual elements. Having SL running prevents me from bringing up other applications.

Infiniview Merit

The 100 Trillionth Cell

Join date: 27 Apr 2006

Posts: 845

08-02-2006 10:04

Oh ok, data transfer, got it.

I was thinking of some kind of in world affect like as in
visually, faster is good.

_____________________

http://slurl.com/secondlife/puea/126/10/60/

Persephone Milk

Very Persenickety!

Join date: 7 Oct 2004

Posts: 870

08-02-2006 10:12

I am so excited to read about object compression! Now we can make very tiny things without the need for prim torture ...

_____________________

~ Persephone Milk ~

Please visit my stores on Persenickety Isle
Musical Alchemy - Pianos, harps and other musical intruments.
Persenickety! - Ladies Eyewear, Jewelry and Clothing Fashions

Lordfly Digeridoo

Prim Orchestrator

Join date: 21 Jul 2003

Posts: 3,628

08-02-2006 10:31

I can't wait until the asset server "compresses" an asset (ie a house) and then can't uncompress it ever again.

Which will happen in about... 2 weeks.

_____________________

----
http://www.lordfly.com/
http://www.twitter.com/lordfly
http://www.plurk.com/lordfly

Ron Overdrive

Registered User

Join date: 10 Jul 2005

Posts: 1,002

08-02-2006 10:44

From: Lordfly Digeridoo

I can't wait until the asset server "compresses" an asset (ie a house) and then can't uncompress it ever again.

Which will happen in about... 2 weeks.

2 hours tops, by then the asset server will will overloaded and crash again.

CrazyMonkey Feaver

Monkey Guy

Join date: 1 Jul 2003

Posts: 201

08-02-2006 12:23

Yeah, I do programming just as a hobby, but I know about all them alignment/endian issues. But if i were doing it and the compiler would'nt make the struct id use assembly

-- Im a speed nut when it comes to programming.

And I glanced at libSecondLife already of course

I don't know C# so well, I know c/c++ better.
But it looks nice and clean, much more so then my code, hehe.

So, is it "zero coded"

compressed) like the other stuff?
with XML'ish data I would'nt think it would help much, if any.
RLE either.. Id think you'd need some form of "real" compression..
I hope the compression/decompression wont hurt performance, lol.

*Edited to add the answer: Its gzip(its like winzip)..
It was asked in SL answers, he said it compresses an average of 8:1, nice

Object compression?
SuezanneC Baskerville Forums Rock! Join date: 22 Dec 2003 Posts: 14,229	08-01-2006 19:24 Is the object compression already in effect or does it start tomorrow? _____________________ - So long to these forums, the vBulletin forums that used to be at forums.secondlife.com. I will miss them. I can be found on the web by searching for "SuezanneC Baskerville", or go to http://www.google.com/profiles/suezanne - http://lindenlab.tribe.net/ created on 11/19/03. Members: Ben, Catherine, Colin, Cory, Dan, Doug, Jim, Philip, Phoenix, Richard, Robin, and Ryan -
Infiniview Merit The 100 Trillionth Cell Join date: 27 Apr 2006 Posts: 845	08-01-2006 19:49 _____________________ http://slurl.com/secondlife/puea/126/10/60/
SuezanneC Baskerville Forums Rock! Join date: 22 Dec 2003 Posts: 14,229	08-01-2006 21:03 To answer my own question: From: someone Release Notes for Second Life 1.11.2(2) July 31, 2006 ===================================== Other changes: * Asset compression expanded to cover objects It's not implemented yet. There is no mention of removing it in the most current version of the preview release notes so I assume it will be implemented tomorrow. _____________________ - So long to these forums, the vBulletin forums that used to be at forums.secondlife.com. I will miss them. I can be found on the web by searching for "SuezanneC Baskerville", or go to http://www.google.com/profiles/suezanne - http://lindenlab.tribe.net/ created on 11/19/03. Members: Ben, Catherine, Colin, Cory, Dan, Doug, Jim, Philip, Phoenix, Richard, Robin, and Ryan -
CrazyMonkey Feaver Monkey Guy Join date: 1 Jul 2003 Posts: 201	08-02-2006 02:11 Awsome.. Aren't they transmited as XML(ish) files? That scares me greatly... Id prefer structs(binary) to save bandwidth plus the overhead of text-to-variable conversion. Is that what the compression is? or is it just still XML'ish + compression? (and is it RLE or Zerocoded?) They need two sets of release notes, the normal one plus the geek version.
Dr Tardis Registered User Join date: 3 Nov 2005 Posts: 426	08-02-2006 09:48 If you really want to g33k 0ut, download libSecondLife and check out the demo programs that come with. One is a debugger that dumps session data so you can get a good look at how all that works. You're right, Crazy: sending binary data IS faster. The down side is that you have to deal with all the niggly little problems with different platforms' implemenation of binary data, including alignment issues that come with structures. The banking software company I worked at changed compilers twice during my tenure. The first time was to shift platforms (obsolete DG system to IBM AIX) and the second was to switch to a compiler that was still supported (old compiler publisher went out of business). The biggest problem we had? The compiler did weird things to align data members on boundries. When you have data elements smaller than 1 word (4 bytes), you have to decide how to position them for fastest access. The convention on the new compiler was that objects align on their size: so chars aligned on any byte boundry, ints (16 bits) aligned on even addresses, and long ints (4 bytes) aligned on 4 byte boundries. (Believe it or not, you don't use floats... you use integers with an assumed decimal point). The old compiler just aligned all variables on 4 byte boundries, even single char values. It was important to know how this worked, since we stored data on disc by simply writing the binary data from the struct variables. Then there are Endian issues to consider: Motrola chips are "Big Endian", meaning that the most significant byte is first in a multi-byte number. Intel chips are small endian - the smallest byte is stored first, then the next largest byte. Then we consider bytecode compilers: Java, .Net. These are pretty much guaranteed to implement variables differently than C++ does. I'm not completely sure if there is even a supported way to get a bitstream from a C# variable. So if you consider the differing optimizations between different compilers and the different internal implemenations of binary numbers (thank God that at least floats are an IEEE standard), it becomes apparent pretty quickly that sending binary C structs is not the smartest way to go - either for disc storage or for network transfer. Packed binary streams aren't so bad, but it's important to pack them manually, rather than just send a memory image of the structure.
Infiniview Merit The 100 Trillionth Cell Join date: 27 Apr 2006 Posts: 845	08-02-2006 09:54 From: Dr Tardis If you really want to g33k 0ut, download libSecondLife and check out the demo programs that come with. One is a debugger that dumps session data so you can get a good look at how all that works. You're right, Crazy: sending binary data IS faster. The down side is that you have to deal with all the niggly little problems with different platforms' implemenation of binary data, including alignment issues that come with structures. The banking software company I worked at changed compilers twice during my tenure. The first time was to shift platforms (obsolete DG system to IBM AIX) and the second was to switch to a compiler that was still supported (old compiler publisher went out of business). The biggest problem we had? The compiler did weird things to align data members on boundries. When you have data elements smaller than 1 word (4 bytes), you have to decide how to position them for fastest access. The convention on the old program was that objects align on their size: so chars aligned on any byte boundry, ints (16 bits) aligned on even addresses, and long ints (4 bytes) aligned on 4 byte boundries. (Believe it or not, you don't use floats... you use integers with an assumed decimal point). Then there are Endian issues to consider: Motrola chips are "Big Endian", meaning that the most significant byte is first in a multi-byte number. Intel chips are small endian - the smallest byte is stored first, then the next largest byte. So if you consider the differing optimizations between different compilers and the different internal implemenations of binary numbers (thank God that at least floats are an IEEE standard), it becomes apparent pretty quickly that sending binary C structs is not the smartest way to go. You could create a binary bistream that's packed on one end and unpacked on the other end in some arbitrary fashion, but that would defeat the "open standards" part of LL's mission with SL. Yeah! So it makes stuff smaller? _____________________ http://slurl.com/secondlife/puea/126/10/60/
Dr Tardis Registered User Join date: 3 Nov 2005 Posts: 426	08-02-2006 09:56 uhm... yeah. objects take less bandwidth to transfer, so they transfer faster.
Alan Edison Ty Zvezda Join date: 28 Jun 2004 Posts: 420	08-02-2006 09:56 From: Infiniview Merit Yeah! So it makes stuff smaller? OBVIOUSLY... duhhh lol jk _____________________ Ty Zvezda
Fenrir Reitveld Crazy? Don't mind if I do Join date: 20 Apr 2005 Posts: 459	08-02-2006 10:02 From: Dr Tardis Packed binary streams aren't so bad, but it's important to pack them manually, rather than just send a memory image of the structure. Seriously, if you are just doing fwrites of your C structs, you deserve whatever cross-platform/cross-compiler problems you encounter. One thing that strikes me about libsecondlife is the ability to write our own client to interface with SL. For example, an IM client or even a "MUD style" system that would let you do very basic things such as manipulate your landmarks and then chat/IM with people -- Without the overhead of having a 3D graphics client. As strange as that sounds, it is something I would really dig as I tend to like to chat with people but actually don't care about the visual elements. Having SL running prevents me from bringing up other applications.
Infiniview Merit The 100 Trillionth Cell Join date: 27 Apr 2006 Posts: 845	08-02-2006 10:04 Oh ok, data transfer, got it. I was thinking of some kind of in world affect like as in visually, faster is good. _____________________ http://slurl.com/secondlife/puea/126/10/60/
Persephone Milk Very Persenickety! Join date: 7 Oct 2004 Posts: 870	08-02-2006 10:12 I am so excited to read about object compression! Now we can make very tiny things without the need for prim torture ... _____________________ ~ Persephone Milk ~ Please visit my stores on Persenickety Isle Musical Alchemy - Pianos, harps and other musical intruments. Persenickety! - Ladies Eyewear, Jewelry and Clothing Fashions
Lordfly Digeridoo Prim Orchestrator Join date: 21 Jul 2003 Posts: 3,628	08-02-2006 10:31 I can't wait until the asset server "compresses" an asset (ie a house) and then can't uncompress it ever again. Which will happen in about... 2 weeks. _____________________ ---- http://www.lordfly.com/ http://www.twitter.com/lordfly http://www.plurk.com/lordfly
Ron Overdrive Registered User Join date: 10 Jul 2005 Posts: 1,002	08-02-2006 10:44 From: Lordfly Digeridoo I can't wait until the asset server "compresses" an asset (ie a house) and then can't uncompress it ever again. Which will happen in about... 2 weeks. 2 hours tops, by then the asset server will will overloaded and crash again.
CrazyMonkey Feaver Monkey Guy Join date: 1 Jul 2003 Posts: 201	08-02-2006 12:23 Yeah, I do programming just as a hobby, but I know about all them alignment/endian issues. But if i were doing it and the compiler would'nt make the struct id use assembly -- Im a speed nut when it comes to programming. And I glanced at libSecondLife already of course I don't know C# so well, I know c/c++ better. But it looks nice and clean, much more so then my code, hehe. So, is it "zero coded"compressed) like the other stuff? with XML'ish data I would'nt think it would help much, if any. RLE either.. Id think you'd need some form of "real" compression.. I hope the compression/decompression wont hurt performance, lol. *Edited to add the answer: Its gzip(its like winzip).. It was asked in SL answers, he said it compresses an average of 8:1, nice

Welcome to the Second Life Forums Archive

Object compression?