Rolling Restart for 1.25.0, 2008 Nov 12 and 17-18

I believe this to be a showstopper:

http://jira.secondlife.com/browse/SVC-3436

Script i had compiled in 1.25, put in inventory, then put in object on 1.24 got this error.

System.MissingMethodException: Method not found: 'LindenLab.SecondLife.LslRunTime.ToInteger'.
at (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (object,object[])
at System.Reflection.MonoMethod.Invoke (System.Object obj, BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00000]

Script runs fine in 1.25 or when compiled again on 1.24.

Also, Prospero, i had requested to be removed from the pilot roll out for 1.25.

Please take my region Xidd, off the list and put me back to 1.24.

Thank you

Quick question...

The grid status page keeps telling of memory upgrades to the 501 servers. I can't find any references to that, so my question is: What is a 501 server, and what kind of regions are running on it?

/me was wondering about that, too. Had tried to post a question to Prospero about it here a few days ago but the forums were pretty unhappy that day..

I'll bet they're some sorta gateway/hub for sim hosts.

501 servers are the class 5 servers in Phoenix. A fraction of them were shipped with only 2GB of memory instead of 4GB of memory. The memory ugprades are fixing that.

Re: the rest:

Today's sadness was tangentially related to the 1.25 deploy. The original trigger was a network outage somewhere external to LL this morning. That kind of thing happens from time to time, and generally when it does we see some problems, and then they clear up when the problems go away. However, this time, because of a config chage we had to make for the 1.25 rollout, when we got the post-problem flood of logins, our login servers got overwhelmed, and that sent the entire system into a death spiral for a couple of hours. Eventually we got it all sorted out. (Basically, we didn't have enough throttling of logins on the login servers for them to survive a huge spike. This was a simple configuration change, and we have now made it.)

However, mostly unrelated to this morning's fun, another set of central servers are still leaking memory. This doesn't cause immediate problems, but after 10 or 12 hours it will. We can just kick them to keep them happy, but that's not a good long-term solution. So, we're going to have to revert the central servers to 1.24, and we won't be able to fully roll out 1.25 until we find the problem. This will put off the central server deploy to next week at the earliest, and the full roll to the week after (December 1) at the earliest.

I do not know at the moment if we're giong to keep the pilot regions on 1.25 or roll them back to 1.24.

Thanks to SVC-3436, we will be rolling back.

:0 kk, persevere! i've had to backgrade my own vendor plenty of times after 'everything worked right on the workbench' lol... sometimes there's just no way of finding the buggies until it's out on the street. (lol the unthrottled log files does kinda crack me up ;0 hate when that happens... sometimes the error reporting causes worse havoc than the error ;0)

i've still seen a few 'blank content' emails in networking as sims go off and on, but it seems to be much much fewer than before. is the update working on this prob? (i.e., it seems after sim resets that the first email message in or out will have its subject line correct, but the email body is blank. been jira-ing about it before; maybe i'll post that little bit of info for an update... i don't know if related to a previous update where email filtering was affected in script-read; either there or in the actual email server pass...)

p.s. re SVC-3436: yoikies yah... good times, good times lol... well, roll with it! ;0

Prospero, please let us know when the central servers are back on 1.24. I wouold like to see if that changes this weird missing name problem (which now seems more widespread).

Seems more frequent now to me too. For reference, JIRA entry is here: https://jira.secondlife.com/browse/SVC-3441

Does this rolling restart thing has anything to do with the fact I can't log in now?
I keep getting stuck at the screen connecting to region. Someone help please?! I tried to log in in the morning from 5am to 8amSLT and now but the same thing keep happening. I tried logging in at different location yet I'm still stuck the the connecting to region screen. HELP please thanks!

rollbacks ongoing, may be that... try punching a different region into the 'start location' box at client startup??

http://status.secondlifegrid.net/
1200+ Regions down

Posted by Status Desk on November 19th, 2008 at 06:52 pm PST Due to an issue with the 1.25 server deploy, we are rolling back effected regions to the 1.24 servers. The current action to do so have cause 1200+ regions to shot down. They are expected to be back within the hour.

The "missing name" thing has been happening to me for the past several days now, so I think think it's due to the 1.25 change. I'd never mentioned much about it because I'd assumed it was client-side, and I'm using a 3rd party client.

Has anyone tried editing a HUD that is located in a margin of around 1/8th the screen width, either left or right hand sides?
The only workround I found is to, edit something else first, then click the HUD. I have moved most of mine inside this margin, but many are preset and flick back next time you wear them.

Same goes when you try editing a prim, or openning someones profile in these margins.

I am using the latest standard client, only noticed it recently since the server deploys.

Could be this: http://jira.secondlife.com/browse/VWR-10351

Yes.. this is exactly it, and as pointed out, it effects virtually anything close to the edge of the viewer, including avatars.

Please add Sandbox Goguen and/or Sandbox Cordova to pilot roll.

An earlier instance of the problem comes from July: http://jira.secondlife.com/browse/VWR-6879

It's been driving me bats. The rightmost quarter, almost a third, of my screen, is totally unpickable except secondhand. I have to open the build tool and click something on the right side before I can do anything to it, even DELETE it! I can't even request someone's profile or send them an IM unless they're in front of me or to my left!

Update 11-12 05:39 : because of a problem updating the central servers this morning (that led to the 20-minute login outage), we are not able to proceed with the pilot roll this evening. The anticipated schedule has been updated below.

There will be a rolling restart this and next week to deploy server 1.25.0 to Second Life. We will follow the schedule below:

Tue, Dec 9, 5am-noon : deploy to central hosts, and a pilot roll to ~3000 regions.

Mon, Dec 15, 5am-9am : half of Second Life will be restarted

Tue, Dec 16, 5am-9am : the remainder of Second Life will be restarted.


Did I sleep through a month ?

.. as I submitted a JIRA ticket on a rather large, gaping security hole that I'm hoping they actually fixed. If the fix is in this new server update, it'll make IP bans actually work again and stop most of the more serious griefing pretty much permanently.

Taff -- newer updates are closer to the top. The December rolls have been on the schedule for a while. I've just refreshed the blog post to indicate that we still have those scheduled times as the estimate.

Oops -- now everybody knows who my alt is.

My apologies, I meant to post that as me.

Heh... Alts are people too! =^-^=

If the pilot is now going to be rolled out on Thursday evening, wouldn't it make sense to wait until after the weekend to roll out the rest in 2 installments instead of 1/2 the installment before the weekend and the other 1/2 after the weekend?

That is in fact the planned schedule on the blog post -- Thursday pilot roll, full roll early next week.

Your secret is safe with me Prospero, ( cant promise for the other million or so though ) )