llRezObject() fails!

I'm seeing what looks like a new problem with llRezObject(), used by MLPV2. After a while, it starts failing. Resetting the script seems to make it start working again (usually, but not always!)

I'm absolutely sure I'm actually calling the function, with correct args. I put debug chats right before the call to print the argumens, and the chat appears, but no object appears.

It's failing silently, without logging a script error.

Seems to happen if the scripts have been running for several hours, leaving balls rezzed.

Here are some relevant facts:

The object IS in inventory.
It's copiable.
The position argument is the object's center. Velocity and rotation are zero.
The land is group owned; has 3000 prims available. Build permission is allowed to all.

I added an object_rezzed handler to see if that gets triggered even though the object isn't appearing. I'll edit this post when I find out what happens.


Does anyone know why this might be happening? Is there something I'm overlooking? Has the gray goo fence gone mad?

Thanks
Lear

I think the fence will yell at you if you touch it..

Have you checked 'pending downloads' on the stats bar?

Ah, good to know, thanks.

Good idea. Stays at zero, though. Also, others see the same thing.

I've seen something like this recently and it seems to be more a problem with SL than with any particular script. Yesterrday in particular I tried to rez a ball from my MLPV2 item and it would not rex, silent fail however the balls appeared suddenly about 10 minutes later. At the same time that this was occuring another item in the same area that rezzes a die acted exactly the same way and the rezzed die appeard at the same time as the balls. Lately SL has been having many issues with its database and asset servers and i would be more likely to blame that for this problem.

If you get this happening then drag something old (ie: with no chance of being cached somewhere on the sim) from your inventory, does that rez?

Thanks, Ravanne -- much to my dismay, I'm beginning to think it's an SL problem and not a script bug. I would far rather it be a script bug, because those I can fix!

Meade, no problems rezzing from inventory.

Oh wait, something old, why? The MLP is rezzing the same object it rezzed an hour or so ago. Wouldn't rezzing something that might be cached a better test case?

In any case, I don't see a problem rezzing. But I'll do more testing with a bit more care about whether it might be cached.

Thanks again. Sure hoping there's something I can fix!

can you try it out in a different sim?

The idea, which may or may not have been a good idea, was to try and see if it was a problem related to the sim or something closer to the asset servers.

Andrew's office hour is tonight - maybe he'd have some tips for what might be wrong or what you can try. A reproducable test case will, of couse, go a long way towards getting his attention...

Update:

I put an object_rez() handler in the rug (the MLPV2 prim). Interestingly, this event handler gets fired, passing the key of an object that is not available by the time the handler runs. llGetObjectDetails() returns the empty string for name, and <0,0,0> for position.

Using this same script on a rug that's still working, I get the ball's name and position. (Note, however, that it's after the ball gets the chat message to reposition and rename itself, so this handler is somewhat delayed. That's probably a good thing for most applications.)

Also, I added two messages to the ~ball script; one to IM me on rez and the other to IM me when it gets the message to reposition itself. These work fine when the rug works. For the borked one, I get no IM, leading me to suspect that the ball is not getting rezzed.

JIRA time?

Oh, dragging in a new ~menu script does not fix the problem. Oddly enough, every time in the past that I've done a pos reset, the problem has gone away. I don't want to disturb this broken one, though, by testing that. If another one starts failing, I will. This bit really makes it seem like a bug in my scripts -- or it would, if it weren't for the fact that I get no message from the ball's on_rez handler. And I have to reproduce it to make sure it's really true.

I have a sneaky suspicion that I won't be able to reproduce this problem with a trivial script. Hmmm, though, I may be able to see if there's a Mono/LSL difference.

No difference compiling with LSL. Also, Pos reset isn't a sure-fire fix -- I just did one on a borked rug and it's still borked.

Now I'm going to try a trivial rez script.

At one point I was into testing the grey goo fence and saw silent failures on a regular basis with no notification. But in this case I would have to agree with Ravanne and would suspect an asset server failure. Asset server has been pretty rough since we hit about 55K. In the past we used to see hiccups when there were large increases in logins, at one point when there were jumps from 5,000 to 6,000 for instance. Eventually it changed to new concurrency high point multiples of 5K or so and then this changed to 10K. From 40K to 55K everything was fairly smooth but I am afraid that LL has finally hit a wall and they just can not figure out a way over the hurdle.

I have a sneaking suspicion that this is part of the reason for the push to upgrade to a fiber optic system to tie the asset servers together. We finally reached a point where tweaking/optimizing code just isn't enough. The amount of data they are having to juggle is absolutely mind boggling and is the price they pay for everyone being able to create their own scripts, textures, animations, builds and sculpties. Nothing else you can really compare this to, sure WOW has hosted a million concurrency before but that is a much different creature. Although I would not be surprised to see LL experiment with some of the affordable super computers that are out on the market now.

I asked about this at Andrews office hour tonight, though Andrew himself couldn't make it..

The short answer (with permission to quote) is:

His suggestion was that you add a support ticket and ask if they can take a look at the logs. Make sure to include the region name, exact date/time and the object name. Didn't think to ask if he meant the rezzer or the rezee - probably best to just include both.

The whole transcript will end up at http://wiki.secondlife.com/w/index.php?title=User:Andrew_Linden/Office_Hours/2009_01_15 before too long, though there wasn't a ton else said..

can i do a support ticket as a freeloader?

In any case, it's totally reproducible. Fails for one item, succeeds for another just like it next to it. It was working earlier, and will probably work again if I beat on it enough.

I can give location, repro steps, and it chats the keys of the rezzed objects (that don't appear). The logs could really help pinpoint the problem. I sure do wish it were my bug, but I just can't imagine how it could be at this point.

Er.. I think you can. They may ignore or shoot you down, though.

If you can get it into a state where it's happening, leave it in that state then come to his next office hour, Andrew or Simon might take a look at it. Or maybe ask you to run home and cause the rez failure then send them the details so they can check out the logs.. I certainly can't speak for them though, and they're usually pretty busy, but it's worth a try. Or maybe somebody like Pathfinder or Zero, if you can't make Andrew's hours.

If all that fails, a JIRA with the details in it might also work..

http://wiki.secondlife.com/wiki/Office_hours

edit: how easy is it to make happen? I had the impression it took some time before it started failing on you.. Also, this happens in multiple sims?

It's happened in different sims, but I don't know if it happens in any sim.

Anyway, I have a rug that's borked and stays borked even if I drop a new ~menu script in (which is the same as doing a menu reset). And another one that stayed borked when I dropped in a ~memory script, which is the same as doing a pos reset -- I didn't want to unbork the first one. And one sitting there that works fine; only difference is different owner, plus a few debug changes.

I also have a totally trivial rezzer script sitting there, in the hopes that it gets borked too.

Well, tonight is not a good night for rezzing things in SL anyway. People around me have been having trouble rezzing and transferring objects, and TPs are all messed up. I just gave up and logged out. Your script may be borked anyway, but you'd never be able to tell right now.

I suspect this issue is related to the rolling restart.

Unfortunately, I don't think I can recreate the problem now. We'll see if it happens again.

The one where it was recreatable was owned by my alt. While it was borked, I replaced a couple scripts and ~ball, though NOT as the owner av. It continued to be borked, in the same manner.

Today, as me (not the alt that owns it), I shift-copied it. The copy rezzed one ball and then none. The ball appeared in inventory, but not when I renamed a notecard in there to force an update. Now, the ball had full permissions for every avatar who touched it. But for some reason, the shift-copy created a copy with a no-copy ~ball (at least, that's how it looks because it rezzed the first ball and then none.)

As the owner av, I replaced ~ball and the problem went away. I replaced it using a ~ball given me by my main av (which was full perms, in inventory and when rezzed, and the one script inside was full perms, for both me and next owner). Anyway, now the problem doesn't recreate. Maybe it will after time -- this problem generally happens after some time elapses; on the order of hours it seems.

However, there are a number of other identical rugs (except not updated with debug scripts). These all would be borked, and I'd do various things to un-bork them, like full resets. Well, now, none of these are borked, even though it seemed to happen rather regularly before, after a few hours.

I'll check again later, but sure don't know what to make of it.

OK, it failed again, so it wasn't related to dropping the ball into contents from another owner.

Still, it's odd that when I shift-copy my alt's rug, the new one's ball acts like it's no-copy. Yet, if I drag it out, it's full perms.

This is happening on Second Life Server 1.24.10.106829.

I detect a common element of permissions screw ups happening on the new sim versions. This *cannot* be good, and I hope I am wrong.

well, mine is goofing up on the old sim.
But in another sim it's working fine. Doing another test there eliminating a few independent variables.

Might be related to which asset server is handling that particular sim.

New data point.

All the ones that fail are owned by my alt. The ones that don't fail are owned by me.

This dispite the fact that the alt completely reset the scripts, and all scripts and ~ball and its enclosed script are full perm.

I got one transferred back from my alt, interested to see whether that fails. I'll probably try a few other variations on the owner / transferred theme and see what happens.

A trivial script fails:

string objname = "~ball";

default
{
    state_entry() {
        llSetTimerEvent(60.);
    }

    timer() {
        llSay(0, "LIVE");
    }

    touch_start(integer total_number) {
        vector pos = llGetPos();
        llSay(1, "DIE");
        llWhisper(0, "llRezObject(\""
            + objname
            + "\", "
            + (string) pos
            + ", ZERO_VECTOR, ZERO_ROTATION, 1");
        llRezObject("~ball", pos, ZERO_VECTOR, ZERO_ROTATION, 1);
    }
    
    object_rez(key id) {
        list details = llGetObjectDetails(id, [OBJECT_NAME, OBJECT_POS]);
        llWhisper(0, "Object '"
            + llList2String(details,0)
            + "' rezzed at "
            + (string) llList2Vector(details,1)
            );
    }
}

It works at first.

Later, if as a different avatar I touch it, no ball. Output:

[20:03] test1 whispers: llRezObject("~ball", <104.40074, 164.90468, 43.86046>, ZERO_VECTOR, ZERO_ROTATION, 1
[20:03] test1 whispers: Object '' rezzed at <0.00000, 0.00000, 0.00000>

This is repeatable.

OK, here's the fun part. I log in the owner avatar, and just stand nearby, while the first (non-owner) alt touches again. IT WORKS.

[20:13] test1 whispers: llRezObject("~ball", <104.40074, 164.90468, 43.86046>, ZERO_VECTOR, ZERO_ROTATION, 1
[20:13] test1 whispers: Object '' rezzed at <0.00000, 0.00000, 0.00000>

Owner goes away. IT FAILS! (not immediately, but after a minute or so)
Owner returns. IT WORKS!

ACK!

JIRA, here I come. I'll post it; please vote!

http://jira.secondlife.com/browse/SVC-3145 ?

Still, it's interesting that you get the object_rez callback. That'd be good to add to the jira..

SVC-3695

Oh, looks like I did a duplicate -- though I did search first. Thanks for the post!