Rolling Restart for 1.25.0, 2008 Nov 12 and 17-18

More information in the blog post here: http://status.secondlifegrid.net/2008/11/11/post312/

Can I ask why the pilot rollout has been delayed 4 times now?

It hasn't been delayed, really... there was a "tentative" on the calendar for a reason.

We only made the blog post announcement today because today was when we were finally confident that we really knew when the pilot roll would be. We had earlier days before because we were hopeful, and would have done them earlier if that was possible. However, we kept "tentative" on the calendar because that's what it was.

well nothing much is working now, I highly recommend you send out a notice to stop ppl from purchasing things...
TP borked
Profiles borked
Region tools borked
people search borked
.....................................
cant even tp to visible adjacent sims.

SVC-1467 at last. *swoon*

I hope this will fix some of the latest Lag Problems since the last update. )

Yeah, I'm happy about SVC-1467 too My alt just discovered it *today* in a scripted object he was working on that required the camera permission in a child prim on a HUD....

And VWR-6714, "Camera oscillates forward and back."

/me does the happy dance!

We hit a hitch with the update of the central servers (that's why logins were broken for ~20 minutes this morning). As a result, we're going to have to postpone the pilot roll. I will update the blog shortly.

Wait... I just had a breakthrough... we may still be on schedule...

Tease

OK, we're delaying it at least a day. We've reproduced the problem on a development grid, but have yet to fully understand it, or how to work around it. We will resume working on that tomorrow morning.

I'm confused... On the status blog, there's mention of a *viewer* (not server) upgrade which caused the logins to be broken. It's a separate entry from the one about the pilot roll problem. What was that all about?

[RESOLVED] Problems with Viewer Upgrade

[10:15 AM Pacific - RESOLVED] Since rolling back to the previous version residents are able to log in again without any difficulty. The cause of the problem was that a central server stopped telling logins what the right viewer version was.

[10:00 AM Pacific] There is a problem with the new viewer upgrade, which is stopping residents from being able to login.

We are rolling back to our previous version for now in order to stop this from happening.

This entry was posted on Wednesday, November 12th, 2008 at 10:15 am and is filed under SLGrid Status.

Beer Bongs anyone?

Yeah, there was no viewer upgrade.

There was a communication problem between the login servers and the version manager, that caused the login servers to think that all versions had required upgrades to a version ".". That version, of course, wasn't downloadable... so nobody was able to log in.

It was, however, a server problem with the 1.25 code. (Actually, it's not the 1.25 code; the 1.25 code is fine by itself. But when the versions are partly 1.25 and partly 1.24, the communication problems arise. And, since we don't take down the grid for updates any more, we have to fix that before we can go forward with the deploy, because we *will* have mixed versions on the central servers.)

OK, things are looking good.

I'm going to try redoing the central servers at 7pm tonight (well past concurrency maximum). Once those are done and look stable, I'll do the pilot roll. The pilot roll may slide a bit past 9:30, since I don't think I'll get started right at 8-- indeed, it may not start until 9 itself.

OK, I lied.... the central deploy is taking a long time ("slow and sure wins the race", so the pilot roll will start rather late. At the moment, I predict a 10:30PM start for the pilot roll.

jimmy dean :0

jimmy dean :0

rock on! :0

This is probably related to the central server work you are doing. My sim is on the pilot roll list, but has not yet been restarted.

Object to object emails between my region and some other regions are failing. Object to object emails within my region is normal.

Just a FYI.

yeah, central boucnes cause fails of object-to-object emails

going to fix them right now... they should be better in a few minutes.

I am FINALLY going with the pilot roll

My sim Pure Combat just got restarted, and now I'm getting errors on every attempt with llHTTPRequest(), each time getting "Unsupported or unknown character set." returned through the http_response() event handler.

I'm using the POST Method, been working fine for years. Not sure yet if the message is actually getting to the website or not.

After my region restarted, I have some llDialog menus that are not showing the object's owner. For example, instead of saying "Triple Peccable's GWS Server V1.22" at the top of the blue menu it says "'s GWS Server V1.22". The edit window does show me as the owner. Some objects are not displaying this behavior.

Is this something you think will clear up, or is it JIRA material?

Triple -- that sounds like JIRA material. if you can make a simple repro and put that in the JIRA, that will make it much easier for us to track down the issue.

Kayla -- that also sounds like JIRA material. If there's not one on this issue yet, please make one.

Also tested in another region that got updated to v1.25 and it's also getting the same error. It's working normally in a v1.24 region though. The message does seem to get through ok, just responds to the http_response() event handler with the error.