Discussion: Prim Persistant Data Storage

Hi

Many folk have stored data in the description and title of a prim. Main advantage is that the description is not affected by resets, rezzes and inventory storage.

This one compacts the data a bit so the 127 byte description can store up to 26 values, including string (chopped at 5 characters), integer (a bit over a million max), float, vector and rotation. The latter two take up three locations each; rotations are stored as euler vectors.

The storage script does not check the validity of what is stored at each location nor keep track of the type. With 5 bytes for each value, the tradeoff was increased storage space.

The way it generally works is that for floats, 24 mantissa bits are stored in 4 bytes, and the fifth byte stores the exponent in 5 bits and the sign in the sixth bit. A string of 64 printable characters supplies the 6 bit values that are actually stored in each location.

This script is the main part:

// The mantissa consists of 24 bits or 4 x 6 bit sextets, add one for guard
// Use one character for power of ten + sign
// Makes 6 characters/value
// Description is limited to 127 chars max, hence can have 21 numbers
// This code will not validate the values


integer debug = TRUE;
// numbers for link values

integer     INIT = 64;
integer     GET  = 128;
integer     PUT  = 256;

integer     STRING   = 512;
integer     INTEGER  = 1024;
integer     FLOAT    = 2048;
integer     VECTOR   = 4096;
integer     ROTATION = 8192;

integer     RE_STORAGE   = 16384;

// use the least significant 5 bits for number position in description, 0 to 20
integer MAXPOS = 25; // 0 to 25 = 26 available positions

// 64 characters used to represent 6 bits
string k = "0123456789ABCDEFGHIJKLMNOPQRSTUVWabcdefghijklmnopqrstuvwxyz!@#$%";
integer CPE = 5; //characters per entry
string SNULL = "00000";  // integer 0, multiplier 1 (10 ^ 0)



// put one string into description.
putString (integer n, string v) {
  if (llStringLength (v) > CPE)
    v = llGetSubString (v, 0, (CPE - 1));
  string vd = llGetObjectDesc();
  integer len = llStringLength (vd);
  integer insPos = n * CPE;
  if (len > insPos)
    llDeleteSubString(vd, insPos, insPos + CPE - 1);
  while (llStringLength (vd) < insPos)
    vd += SNULL;
  vd = llInsertString (vd, insPos, v);
  llSetObjectDesc(vd);
}

convert (integer n, integer i, string p) {
  if (debug)
    llOwnerSay ("convert n=" + (string) n + " i=<" + (string) i + ">");
  integer j = i % 64;
  string  s0 = llGetSubString (k, j, j);
  i = i >> 6;
  j = i % 64;
  string  s1 = llGetSubString (k, j, j);
  i = i >> 6;
  j = i % 64;
  string  s2 = llGetSubString (k, j, j);
  i = i >> 6;
  j = i % 64;
  string  s3 = llGetSubString (k, j, j);
  putString (n, p + s3 + s2 + s1 + s0);
  if (debug) {
    llOwnerSay ("convert n=" + (string) n + " val=<" +  p + s3 + s2 + s1 + s0 + ">" );
    integer test = revert (p + s3 + s2 + s1 + s0);
    llOwnerSay ("got reverted to " + (string) test);
  }


}



putInteger (integer n, string v) {
  integer vi = (integer) v;
  string p;
  if (vi < 0) {
    p = llGetSubString (k, 32, 32);
    vi = -vi;
  } else
    p = llGetSubString (k, 0, 0);
  
  convert (n, vi, p);
}



putFloatFloat (integer n, float vf) {
   if (debug)
     llOwnerSay ("putFloatFloat n=" + (string) n + " vf=<" +  (string) vf + ">");

   integer neg = FALSE;
   string p;

   if (vf < 0.0) {
     vf = -vf;
     neg = TRUE;
   }

   integer m = 0;
   if (vf < 0.0009)
     vf = 0.0;
   else
     while ((vf < 8388608) && (m < 31)) {  // 2 ^ 22
       vf *= 2.0;
       ++m;
     }

   if (neg == TRUE)
     m += 32;

   integer vi = (integer) llRound (vf);

   p = llGetSubString (k, m, m);
   if (debug == TRUE)
     llOwnerSay ("putFloatFloat vi=" + (string) vi + " m=" + (string) m + " p=" + (string) p);
   convert (n, vi, p);
}


putFloat (integer n, string v) {
   float vf = (float) v;
   putFloatFloat (n, vf);
}
   

putRotation (integer n, string s) {
  rotation r = (rotation) s;
  vector v = llRot2Euler(r);
  putFloatFloat (n + 0, v.x);
  putFloatFloat (n + 1, v.y);
  putFloatFloat (n + 2, v.z);
}

putVector (integer n, string s) {
  vector v = (vector) s;
  putFloatFloat (n + 0, v.x);
  putFloatFloat (n + 1, v.y);
  putFloatFloat (n + 2, v.z);
}




// get one string from description.
string getStringString (integer n) {
  string v = SNULL;
  string vd = llGetObjectDesc();
  integer len = llStringLength (vd);
  if (len > (n * CPE)) 
    v = llGetSubString(vd, n * CPE, n * CPE + CPE - 1);
  if (debug)
    llOwnerSay ("getStringString n=" + (string) n + " string=<" + v + ">");
  return v;
}



//get the numerical value part as an integer
integer revert (string s) {
  integer i;
  string ch;
  integer chv;
  integer ival = 0;
  integer mult = 1;
  
  for (i = CPE - 1; i > 0; --i) {
      
    ch = llGetSubString(s, i, i);
    chv = llSubStringIndex(k, ch);
    if (chv < 0) 
      chv = 0;
    ival += (chv * mult);
    
    mult *= 64;
  }
  if (debug)
    llOwnerSay ("revert ival=" + (string) ival);
    
  return ival;
}


getString (integer n, string id) {
  string s = getStringString (n);
  llMessageLinked (LINK_THIS, (integer) id, s, "");
}



getInteger (integer n, string id) {
  string ch;
  string v = "";
  integer ival = 0;
  string s = getStringString (n);
  if (s == SNULL)
    v = "0";
  else {
    ival = revert (s);
    ch = llGetSubString(s, 0, 0);
    if (ch != "0") 
      ival = -ival;
  }
  v = (string) ival;
  llMessageLinked (LINK_THIS, (integer) id, v, "");
}



//ppk npk

float getFloatFloat (integer n) {
  string ch;
  integer chv;
  float vf = 0.0;
  integer ival = 0;
  string s = getStringString (n);
  if (s == SNULL)
    vf = 0.0;
  else {
    ival = revert (s);
    ch = llGetSubString(s, 0, 0);
    integer chv1 = llSubStringIndex(k, ch);
     chv = chv1 & 31;
     if ((chv1 & 32) != 0)
       ival = -ival;
     float pow = llPow (2.0, chv);
 
    vf = ((float) ival) / pow;
    if (debug) 
      llOwnerSay ("getFloatFloat chv=" + (string) chv + " pow=" + (string) pow);
  }
  return vf;
}


getFloat (integer n, string id) {
  float vf = getFloatFloat (n);
  string v = (string) vf;
  llMessageLinked (LINK_THIS, (integer) id, v, "");
}



vector getVectorVector (integer n) {
  float x = getFloatFloat (n + 0);
  float y = getFloatFloat (n + 1);
  float z = getFloatFloat (n + 2);
  vector r = <x,y,z>;
  return r;
}



getVector (integer n, string id) {
  vector v = getVectorVector (n);
  string vs = (string) v;
  llMessageLinked (LINK_THIS, (integer) id, vs, "");
}


getRotation (integer n, string id) {
  vector v = getVectorVector (n);
  rotation r = llEuler2Rot(v);
  string rs = (string) r;
  llMessageLinked (LINK_THIS, (integer) id, rs, "");
}

doInit () {
  integer i;
  for (i = 0; i <= MAXPOS; i++)
    putString (i, SNULL);
}


default
{
    state_entry() {
    }

    link_message(integer sender_num, integer num, string str, key id)
    {
    
       if (num > RE_STORAGE) {
         if (debug)
           llOwnerSay ("storage link_message " + (string) num + "/" + str + "/" + (string) id + 
                   "/"  + (string) llGetFreeMemory ());
         if (num & INIT) {doInit (); return;}
          
         integer numPos = num % 32;
         if (debug)
           llOwnerSay ("numPos=" + (string) numPos);
       
         if (numPos <= MAXPOS) {
           if (num & PUT) {
             if (num & STRING)   {putString   (numPos, str); return;}
             if (num & INTEGER)  {putInteger  (numPos, str); return;}
             if (num & FLOAT)    {putFloat    (numPos, str); return;}
             if (num & VECTOR)   {putVector   (numPos, str); return;}
             if (num & ROTATION) {putRotation (numPos, str); return;}
           }
           if (num & GET) {
             if (num & STRING)   {getString   (numPos, id); return;}
             if (num & INTEGER)  {getInteger  (numPos, id); return;}
             if (num & FLOAT)    {getFloat    (numPos, id); return;}
             if (num & VECTOR)   {getVector   (numPos, id); return;}
             if (num & ROTATION) {getRotation (numPos, id); return;}
           }
         }
       }

    }
    
} // End of default state and end of script.

This is the test script. It shows how to call each kind of storage action with message linked calls. There is no call back on these calls.

A typical num argument is:

RE_STORAGE + PUT + FLOAT   + 2

where the equates request the script to store a float at location 2.

To read a value, typically send this linked message:

RE_STORAGE + GET + FLOAT   + 2, "", "122"

where a call back to our messagelinked will have 122 as its num, the string will be the float value stored at location 2.

// test script
integer debug = FALSE; // set to TRUE for diagnostics


// numbers for link values - should match application header

integer     INIT = 64;
integer     GET  = 128;
integer     PUT  = 256;

integer     STRING   = 512;
integer     INTEGER  = 1024;
integer     FLOAT    = 2048;
integer     VECTOR   = 4096;
integer     ROTATION = 8192;

integer     RE_STORAGE   = 16384;



default {

  touch_start (integer n) {
    llMessageLinked (LINK_THIS, RE_STORAGE + INIT + 0, "", "");
    llMessageLinked (LINK_THIS, RE_STORAGE + PUT + INTEGER + 0, "7654321", "");
    llMessageLinked (LINK_THIS, RE_STORAGE + PUT + FLOAT   + 1, "0.001", "");
    llMessageLinked (LINK_THIS, RE_STORAGE + PUT + FLOAT   + 2, "27654.321", "");
    llMessageLinked (LINK_THIS, RE_STORAGE + PUT + FLOAT   + 3, "17654.321", "");
    llMessageLinked (LINK_THIS, RE_STORAGE + PUT + FLOAT   + 4, "10054.321", "");
    llMessageLinked (LINK_THIS, RE_STORAGE + PUT + INTEGER + 5, "-12345", "");
    llMessageLinked (LINK_THIS, RE_STORAGE + PUT + INTEGER + 6, "12", "");
    llMessageLinked (LINK_THIS, RE_STORAGE + PUT + STRING  + 7, "STRIN", "");
    llMessageLinked (LINK_THIS, RE_STORAGE + PUT + FLOAT   + 8, "-12345.678", "");
    llMessageLinked (LINK_THIS, RE_STORAGE + PUT + VECTOR  + 9, "<0.5, 0.5,0.5>", "");
    llMessageLinked (LINK_THIS, RE_STORAGE + PUT + ROTATION+12, "<0.5, 0.5, 0.5, 0.5>", "");

    llMessageLinked (LINK_THIS, RE_STORAGE + GET + INTEGER + 0, "", "120");
    llMessageLinked (LINK_THIS, RE_STORAGE + GET + FLOAT   + 1, "", "121");
    llMessageLinked (LINK_THIS, RE_STORAGE + GET + FLOAT   + 2, "", "122");
    llMessageLinked (LINK_THIS, RE_STORAGE + GET + FLOAT   + 3, "", "123");
    llMessageLinked (LINK_THIS, RE_STORAGE + GET + FLOAT   + 4, "", "124");
    llMessageLinked (LINK_THIS, RE_STORAGE + GET + INTEGER + 5, "", "125");
    llMessageLinked (LINK_THIS, RE_STORAGE + GET + INTEGER + 6, "", "126");
    llMessageLinked (LINK_THIS, RE_STORAGE + GET + STRING  + 7, "", "127");
    llMessageLinked (LINK_THIS, RE_STORAGE + GET + FLOAT   + 8, "", "128");
    llMessageLinked (LINK_THIS, RE_STORAGE + GET + VECTOR  + 9, "", "129");
    llMessageLinked (LINK_THIS, RE_STORAGE + GET + ROTATION+12, "", "132");
  
  
  }

  link_message(integer sender_num, integer num, string str, key id) {
    if ((num >= 100) && (num < 200)) {
         if (debug)
           llOwnerSay ("testLinkMsg " + (string) num + "/" + str + "/" + (string) id + 
                   "/"  + (string) llGetFreeMemory ());
    }
  }
}

Ooh, very cool. I was looking over the feasibility of getting the most out of the characters in a string, and I discovered the base64 functions. It seems like they use a 64-character set and store 24 bits as 4 characters, much like you. You can do llInteger2Base64 and llBase64ToInteger to convert back and forth... I wonder if that'd make yours faster, since more of the string processing would be done in the sim itself?

Hi Lex

I wish I had looked at base64 and RFC 1341 before I started so I would have used their arrangement of 64 characters (theirs starts with "A", mine starts with "0". As long as you use to the same code for writing and reading that does not really matter.

Ah well, I might rework the functions to incorporate the lsl base 64 functions. It will not be lot faster because only putIntegerInteger, getIntegerInteger, revert and convert directly convert integers. The bulk of the code is still about converting floats. The base64 also does not help me convert the exponent and sign. I would also need to figure out how negative integers are handled and then revert the 8 character string back to 5 or 4 characters. This might turn into a long term project. Meanwhile, I will use what I have.

I envisage this being used to store rotations and positions for doors, windows and other such devices, where the time taken is not so important. Running the 10 test cases does nto seem to take any significant amount of time.

The Timeless door is an excellent example of where this might be applied. That door stores a position vector, a rotation vector and a scale vector for both open and closed positions directly into the description and title of the prim. The description part (127 max) is ok but the title part (63 max) overflows and cannot be fully read back. I really think the TImeless door is another brilliant concept and I will try to apply this storage mechanism to it. I might also be able to store one or two intermediate positions between closed and fully open.

Anyway, feel free to use, adapt or whatever with this code!

Ed