Rolling Restart for 1.25.0, 2008 Nov 12 and 17-18

At the risk of information overload:

I have noticed some names, which have been missing from the payment windows for the last few weeks, are now showing up.

Some other names, which have been showing just fine for the last few weeks, are now missing.

Are you going to roll us back to 1.24.9 Prospero? This 1.25.2 is borked for a full rollout don't you think?

New Update

How many rollout version in a row has it been now. I know 1.25.0 was one but I don't remember before that.

This is the third attempt; 1.25.0, 1.25.1, and now 1.25.2. Other than the high load on the central server, which as far as we know is not fully diagnosed yet, I don't think 1.25 is 'borked'. Names not always appearing at top of dialog menus and payment dialogs isn't really a showstopper bug.

And Simon Linden has confirmed that the Total Frame Time issue is actually an intended new feature, not a bug. The way that frame times are being displayed are simply being updated:

http://jira.secondlife.com/browse/SVC-3437


1.25 has some really important and long-awaited fixes that were delayed by the long rollout time for 1.23 and the Mono deployment. Let's not be too quick to call it "borked" and push for rollbacks.

We are going to have to revert to 1.24.9. We've already rolled back the login machines to 1.24, and the web dataservice cluster of machines, and we're seeing a higher load on the central database server. So, we don't know what is causing the higher load on the database server.

We'll revert the simulator nodes tonight at about 8PM. That will leave part of the central machines on 1.25; if the spiky database load doesn't go away, then I'll undo the rest of the central machines as well.

This means we almost certainly will not be doing the full deploy next week. We have to find what is causing the database load before we can go forward with 1.25 again. We got through the day today with the higher load, but I doubt we'd make it through the weekend.

I have wasted too much time and am calling it a night! Good grief!

Thank you Lindens for pulling back this "clearly not ready" release. I've got "you paid _____ 79L$" transactions going on, and _____ isn't getting the money. So it's really getting screwy with the actual people involved in the transactions, not just the "name displayed".

Anyways, I hope you can get this solved soon, and I know you guys have a thankless job. Thanks for realizing that this wasn't ready yet. (I just wish my store's sim wasn't in the pilot roll this time).

Well I hate to say it.... but my store is not in the pilot roll..and I was having the same problem with transactions.

I realize a botched roll is bad for everyone... but I've been waiting for a mono fix for months now... when are you going to get this server release sorted and rolled out?

By looking at the release notes it does not appear to be anything revolutionary with this release... what is the issue?

I've had the "___ paid you L$300" happen on and off for the past month or two, I don't think it's related to this particular rollout, or if it is, it's definitely a database issue and not an issue with the sim code.

It's too bad we can't have the 1.25 sim code and keep the 1.24 database code until that's sorted out.

Just out of curiousity, does the 1.25 database code get rolled out on the Beta grid as well, or no?

This release is full of fixes for stuff that was broken by H4, or broken by fixes for stuff that was broken by H4. In general it has piles of stuff that was backed up waiting for the QA pipeline blocked by by the Mono rollout, and Mono fixes, and fixes for the Mono fixes.

I do hope that the " higher than expected load on our central database" prompting the 1.25.2 rollback doesn't turn out to actually be the result of the entire population needing to clear their caches after the viewer 1.22.3 rollout catastrophe. Or maybe I hope it was.

Maybe you've been waiting for a Mono fix for "months"...but we've been waiting for things like the SVC-2511 fix since before Mono was rolled out.

It seems that IM to email (when not online) is also suffering of the missing name syndrom,
I got many IM with the text : "object" is owned by (avatar name missing). A pity, I was using that text with the outlook rules.
However, we are waiting for a long time the mono bugs fixes, maybe a minor server update, but fixing existing bugs would be better than a major one which brings new bugs.

Hi there,
I read a little here and there, and came across the so called

"missing name syndrom"

Are you form Linden Lab aware of this issue?

As it started with _____ payd you (L$) in the beginning,

some of us dicover, that names are missing in CHAT windows or IM´s

And today, it started, that whole names are missing in the Group interface.

This issue seems to grow in its way, like I have never seen before.

Please LL, take notice, thanks Randolf

Well there are lots of things I've been waiting for that have been years in waiting... but this is a critical one that has a fix pending that is fixed in 1.25 if they ever get it rolled out.

Additionally while objects failing to collide with avatars is inconvenient it is not fatal like the script error caused by SVC-2975.

I'm not going to go through the entire thread more than the last few days and this is new for me this morning, but damn near half the grid is unavailable by reason of "privacy settings" for this payment enabled, age certified avatar. Any vendors selling small waterfalls, 10 LL meters or less, you didn't get a chance today. And these aren't places off line - and it includes the sim where my parcel is located (which was up and running and I had no problem walking around once I was there) and three places that I was able to TP to, NP. So it looks like the "privacy settings" is bogus - just one more cracked or broken from the last few weeks.

Of course, unlike any other web based area I work with, LL has no provision to provide immediate feedback that stuff is neffed. They presume their brilliant team is on top of everything, minute to minute, and of course, way too busy to actually put anything on their way behind, what you needed to know yesterday, grid status blog. So we can go put it in the JIRA, which is ignored when it serves LL purposes to ignore it and justifies what they choose to do, or log out and do something else altogether, since we have no idea if anyone knows about the problems or is doing anything about it - until it is actually fixed (or not) and they take one minute to update the blog about what is no longer of interest...

Wonder what changes take place if there is ever any competition? Us mushrooms would like to know.

I can't login. 2 AVs, both get "Login Failed" Beta Grid login failed too but the first time it gave me a message about an inventory fetch failing and a request to contact [email]support@secondlife.com[/email]

4 minutes later: Now logins fail with a "logins disabled" message and tell me to consult the status page (which shows no information on this).

Ok LL Guys, don't get me wrong, but every serverupdate in the last 3 month was a desaster. Whats about: we will test more properly before a new serverrelease? Ok, you really dont care about the shit you said yesterday, but be serious, don't you think its time for a SL 2.0? Don't try any longer to fix the holes in the boat, it's sinking if you dont "rezz" a new one!

I wont say that SL is much better with all the "new" features as Windlight, Voice, etc... It worked (more or less) before. But it's not really getting more stable with a million of more features. Start agion from scratch and release it when it's really ready and don't try to be a second Microsoft, releasing every shit as soon it's good enough to earn money with... "Hey you could patch the scrap later... we don't have to work with it..."

Sorry if I forgot the elaborated code in this posting, but in the meantime I'm really pissed...

I am getting missing names on a lot of transactions still.

To address a question above -- there's a slight misconception in referring to the "1.25 database code". The database code is MySQL, and is a standard database code supported by Sun, not something we write ourselves.

The Second Life code that we deploy goes on the central machines and the sim nodes. The central machines are not the database machines; they are the machines that handle grid-wide functions like assigning regions to sim hosts, keeping track of agent presence, processing group chat, logins, etc.

The increased database load was something in the sim code. We know this because it started when we put the sims on 1.25, and it stopped when we took the sims off of 1.25. We previously tried taking some central nodes (logins, back-end web-ds machines) off of 1.25 to see if that made the load go away, but it didn't. Some of the central nodes are still on 1.25-- and that's the likely source of the "missing name" problem.

Yes, of course, we put the centrals on the Preview Grid on 1.25, and on other grids as well as we're developing and testing. The additional load on the central database is not a lot, and is impossible to see without a full production Second Life environment's worth of load; this is why we didn't catch it ahead of time. Likewise, we have yet to come up with a clean, reproducible, force-it-to-happen reproduction of the "missing name" problem, which is making tracking it down very hard. We know it's there, yes, we're not denying that-- but it happens sometimes, in some regions, to some avatars, and we have yet to find a prescription to make it happen. We're working on it, but if anybody stumbles upon a repeatable prescription to make it happen (as opposed to just a report of "it is happening right here and right now", *please* add that information to SVC-3441.

Re: Second Life not getting more stable-- in fact, it is. Yes, we're having trouble getting server version 1.25 out, and, yes, it took a few tries before 1.24 came out. But our simulator crash rate is a whole lot less than it used to be. I posit that those who say Second Life was more stable a few years ago are remembering the history with an optimistic filter.

Prospero, there was a clean, reproducible, "missing name" issue in my sim that lasted for weeks. I posted about it early on in the SVC-3441 comments, and it wasn't long before Vektor Linden showed up in my sim to check it out. We reproduced it at will, and we conducted transactions that enabled him to capture some data. I haven't heard back from him since, but he should have some valuable data on this issue.

Now that my sim has been restarted, my name in dialog window titles (always missing previously) is now showing up properly. If I get the reproducible-every-time scenario to return I will let you know here in this thread.

Prospero, any luck tracking down what was actually causing the excess load in the 1.25 code and/or an ETA on a new rollout?

Nice and cool that sl is updating the client on a regular base, but does it have to be in the weekend on prime time for most European countries that they are shutting it down ??? I know, it will never be an accurate time to do so, but please if possible avoid doing this during weekends

I have the grid status bookmarked and its a waste of bits. That splash screen message has been around for a while and not once - not even once - has the grid status page given any useful information about unexpected problems. No doubt it has to go up and back down the chain before anyone can admit a mistake or problem at LL. On Thursday I got three warning messages from my web host ISP about problems and I couldn't log on fast enough to actually look at them. One, preemptive warnings; LL, explanations of why nothing worked yesterday. Not a technical issue but simply that one treats me like a customer and has competition; the other (LL) figures no real competition, so why invest two minutes in communication on the grid blog?

At this point its like annoying music when you are on hold - if they have no intention to ever respond to the unexpected problems or sporadic issues until it is resolved, then get rid of the message to check the grid status and rename the page to 'roll out status' or 'maintenance releases'.