Rolling Restart for 1.25.0, 2008 Nov 12 and 17-18

How is it looking, Prospero?

So far, mostly OK. There was that login problem for a while, which was a glitch of intermediate configuration on the central hosts (some on 1.25, some on 1.24), that I worked around quickly once I had it. And, there was a brief landstore problem due to the transition that we've worked out.

Have yet to get to sim nodes....

I am really confused on when the rolling restarts are going to happen. Is it starting tonight or is it being pushed back? Do I need to stay up until midnight to keep an eye on my av? hehe Please, someone help me with my confusion. Thanks!!

The *Pilot* roll is still supposed to happen tonight, although it looks to be behind schedule. If all goes well, the first half of the grid will get restarted next Tuesday morning, with the other half getting done on Wednesday morning.

Thank you so much for helping me clear that up. I was really confused for a bit. I think I can get some sleep now and not worry. hehe Thanks again!

The pilot roll has begun. It should be done within the hour.

I'm excited about the new version. Thank you, Prospero, for all your work.

Cerura region has been down for 15 min since the warning/countdown, normally it comes back in a few min, but I have memory of some rolling restarts that took extra long to happen.

Update: restarted after 20 min, but for some reason our land description was partly blanked. This happens *every* time our sim reboots, and is still happening with this latest version, so I guess its time for a support ticket

Yeah, normally what I say is "if your region is down 20-30 min, please contact support", which lots of you have figured out is a "Scotty" estimate, because for many regions it's often just a few minutes. Sometimes it takes longer if the host has a hard time shutting down (as is often the case, for example, on openspace hosts that have regions that have been overused).

However, I also made a goof this time that's leaving some regions down for 15-20 minutes before they restart. So, don't panic until it's been 30 minutes -- some regions really will be down that long this time around.

Hi, I notice that this time around the updated regions identify as "Second Life Pilot". Is there any special sauce in the pilot version, or is that just a heads up kind of thing?

My region, Triple Labs, is holding at a steady 22.2ms Total Frame Time, with nothing showing that would account for it.

Also, object-to-object communications to at least one other sim has been cut off since before the restart, I assume when you updated the central servers is when that happened. Should that fix itself?

EDIT: I just talked to another sim owner who's sim got restarted, and they are also showing 22.2ms total frame time.

My total frame time was 4ms to 5ms before getting restarted. Curious thing is I am not noticing any lag though.

EDIT #2: Also, my llDialog menus have my name missing in the title again, as do the blue payment windows.

EDIT #3: Deeded objects show (???) (???) for the name in llDialog menu titles. "About Land" DOES show the proper group name, it does NOT say (waiting).

Triple -- the ??? stuff, is it repeatable? Please add comments to the relevant JIRA. (I don't remember the JIRA offhand, and it's way too late at night (3AM) after a full day, and I don't trust my ability to find it....)

Yes, it is repeatable. Unless I restart the region, then it may not be.

I will add comments to this JIRA:

http://jira.secondlife.com/browse/SVC-3441

EDIT: Object-to-Object communications between sims appears to have been restored.

I'm seeing the same things as Triple. Names missing from llDialogue boxes. Frame Time and Script Time stuck. Agent Update count is incorrect (always off by -1). EPS count has doubled. Active Script count is incorrect. Performance otherwise seems to be stable. I guess I need to go around and do my usual check of every object and every script to see what else is broke after a rollout. I don't like rollouts much less being part of a Pilot.

Strange thing: If you increase the script time (in my experiment, by firing a gun full-auto, which involves a decent bit of script events), the total frame time is remaining the same, unaffected by it.

So I'm wondering if this is a problem with the readout itself giving the wrong information, rather than the frame time actually being 22.2...especially since that's definitely a non-random number, being the maximum frametime that a sim can process before instituting time-dilation.


Edited to add: The script time does change when running script events, it's just the total frame time that remains still.

Also, I am unable to reproduce the llDialog() issues in my sim running 1.25.2 (Pure Combat), nor am I able to reproduce the llSetPayPrice() issues.

Owned objects, attached objects, and deeded objects are all returning normal dialog menus.

I'd love to hear feedback as to whether these are happening elsewhere, or whether this particular problem might be confined to Triple's sim.


Edited to add: Went to Triple Labs, there's a demo of the security system there, so it's deeded to group and has dialog menus, seem to work fine.

The problem with missing names has gotten worse since the pilot roll.

Btw. the JIRA is https://jira.secondlife.com/browse/SVC-3441

I agree with Jahar on the frame time issue. It looks to me like idle time is being included in the frame time readout. That causes it to never read below 22.2.

The script times, eps, and all other times look normal to me.

I just used an alt in my sim to check the dialog menus, and my name is missing in the title for objects owned by me. If the object is owned by my alt, the name in the title shows normally. I think Jahar meant he got the menus, but he didn't notice the missing name in the title.

Interestingly, the menu title from deeded objects now has changed from (???) (???) to showing a null name.

OK, to confirm -- have you seen the llDialog problem on 1.25 regions, or only on 1.24 regions?

I'm only seeing the (???) (???) thing on group-deeded objects' dialogs under 1.25.

Jahar -- can you file a JIRA with your complete observations about the "frame time" issue?

I have the problem on both server versions. And even though it is changed to a null name for now, I have seen the (???) (???) on both versions also.

Personally, I believe these missing names on menus and payment boxes (among other places) are related to the central servers, not the simulator code. This problem started weeks ago, the first time the central servers were updated to 1.25, and remained even when they were reverted to 1.24. It seems that it is possibly getting worse each time the central servers are kicked or updated.

Vektor Linden was in my sim a couple of weeks ago, was able to reproduce the problem, and grepped some logs for some other engineers to look at. I haven't heard since then what, if anything, has been done with that data.

JIRA created for the Total Frame Time issue:

http://jira.secondlife.com/browse/SVC-3520

Came back, confirmed that it is not showing the owner, and when I edit the object, I can't get it to show the owner, either, but this is only in Triple Labs sim, not in Pure Combat. Very Strange.

Also confirmed that the security system scanner in Triple's store does not show group it's deeded to in edit window, shows (Loading..). The control panel does not show the group it's set to (but not deeded) in edit window, but shows owner name for me. Dialogs from the control panel do not show the owner name, just blank.

I tried setting (not deeding) an object of my own in two sims, Pure Combat and Therianation (both 1.25), to all groups I'm part of and all groups shows properly in edit window.

I then deeded 5 objects to different groups in Therianation and had a simple llDialog in them. 3 of them showed (???) (???) for the owner in the dialog, two others showed the group name. All 5 show the group name properly in edit window. When I tested in Pure Combat, all deeded objects showed the correct group name in the dialog and edit window.