Rolling Restart for Server 1.30

Uhm... would this change have any effect on say, the typing areas in the chat window and main tool bar on the bottom of the window, or the world map going all black? It just started this about 5 mins ago... Also, the map is taking an eternity to load everything...

Here, I took a screen capture of what it looks like.



Just thought I'd throw this out if no one else has seen this issue yet... or if I'm not the only one... Oh and my windows colors are black, if that makes any diff...

EDIT: oh and just discovered my preferences aren't working as they should... Trying to see black on black isn't easy

Sythe

I left a comment and unassigned the pjira, as that reflects the status of the internal jira.

Thank you. I appreciate it, while being simply amazed that it was ever closed... out here, outside the labs, we have just assumed it was just too hard to fix without breaking things, grumbled about it, put up with it, never for one moment letting the idea that it was off the table pass through our silly little heads.

In the case of an owner rezzing something on their own land, then a simulation pause of up to 30 seconds is something we allow. That's the limit we selected after receiving complaints that 15 seconds was too short of a limit for owners on their own land. Obviously, the ideal situation is that there would be no significant pause. However, as I mentioned in that KB article, it's inherent in how the simulator works, so it's not a small bug fix. It needs to be a project with enough resources to be done in a timely manner, and at this moment, we don't have those resources to dedicate to that project.



Pausing simulation to rez objects has been around before 1.26, so this is where the issue starts to get muddled. Anything that consumes memory (and possibly forces the system to start swapping) could be responsible for increased delays in everything the simulator does, including rezzing objects. That could be the result of a memory leak anywhere in the simulator or anything else running on the simhosts, increasing memory footprints for regions since 1.26 (say, due to code changes or more scripts or prims being used on average, either in a region or on avatars teleporting into regions), something else on the system using up memory, etc.

Over time, as more people are logged into Second Life, there will (in theory) be more teleports per minute, which could mean more pauses per region per minute (depending on the distribution of teleports and how the grid grows). If the average avatar is becoming more attachment-heavy, the pauses could get longer, as well.

That was a long way of saying that while the rezzing pause is a known and currently accepted behavior, there are a lot of reasons why the pauses might get longer, and some might be code related and some might not be code related. If anyone has a repro step like "I choose setting X in the estate tools, and the rezzing pause for this object jumps from 0.5 seconds to 7 seconds," that would be interesting to know.

If you have more questions about the rezzing pause, you could try bringing them up in Andrew's office hours.

Uh..

You mean that my mainland home locking up every couple minutes was something LL did on purpose?

Are you saying that LL deliberately made it so that everybody moving, everybody chatting, every script relying on chat, gets screwed up any time somebody TPs into a region?

You're telling us that SVC-4616 is a feature?

(That KB article you linked needs work, BTW.)

Someone please refresh my memory on how to get to the actual article from a KB link, which just leaves me at the support home page (in both FF and IE).

TIA

Search for what you want then do Copy Shortcut (or whatever right-clicking on the link says in your browser of choice) instead of following the link. That or get to the page you want then right-click the page and do Properties and copy the Address.

edit: that's how you *get* the link. From the link Dante posted, you can only get the KB home. Not enough in there to get to the article he wanted to reference.

Ah, I see. How useful.

He probably didn't know that about the KB. I think I did it myself a bunch of times before people yelled at me enough to break me of the habit.

Anyway, back to a few posts up, were these delays that have been DESTROYING my SL for the last few months really on purpose? I have been screaming about this for months and this is the first I've heard that it was anything but a nasty bug LL hadn't figured out yet.

/me would very, very much like an answer to this, please.

Can I haz bear plz? *winks and giggles*

So, when you 'buy' a sim, hosted on a server... isn't there a set quantity of resources allocated to that? Every web host I know of state what you get for your money. This does sound a bit open ended, and which allows for moving goalposts, effecting those that pay for the service.
I understand there are shared resources within the system, but thats why the costs of operating within SL are so high compared to standard webhosts. What I sense here is the formation of a cloud server system... LL considering the Hyper V server OS to host SL?

That maybe a sensible strategy, and should result in cheaper costs for users.

You're kidding, right?

Hyper-V is "Microsoft coming late to the table". There's several better tested, cheaper, and more reliable virtualization solutions. Xen, one of the market leaders, is open source. Why on earth, assuming someone was even dealing with a situation where virtualization makes sense (and that's not even vaguely the case here) would one add the overhead and licensing issues of Windows Server to the mix?

Servers in SL are already fungible. Migrating a sim to a new server is the same fundamental operation (copy a snapshot of the sim, start an instance of the application) as migrating a virtual machine (copy a snapshot of disk and memory, start an instance of the virtual machine) except that the copying step is faster because the application itself doesn't need to be copied. Virtualization would add nothing but overhead. It would reduce performance and increase costs.

Things may be rock-solid now compared to way back when; but if you have not noticed the rapidly-deteriorating conditions grid-wide over the past week or so, then you either have not been logging in, or are being deliberately oblivious. Just because things are better than they were in the days of asstachments does NOT make a trending back to those times excusable.

Compared to how things were even as recently as two weeks ago, the grid IS pulling itself apart at the seams, with frequent asset server failures, huds randomly disappearing off-screen (and from inventory), lag spikes that make it feel like one is crossing a sim boundary over and over and over again (and this is on an otherwise quiet and healthy sim), and LOCAL chat beginning to be afflicted with the problems plaguing group IMs. I don't give a rat's ass what things were like "in those days". This is unacceptable (and so far unexplained) behavior NOW!

Is it possible to add my sim, "Deshima" to the pilot regions list?

If so, how would I go about doing that?

The issue you're seeing might be the combination of two things: one by design, one not.

1) By design, the simulator will pause simulation to rez an object. This was an old design choice, and we know it needs to be revisited. To make adequate progress on this, it needs to be a project with people assigned to it, and at this time, it is not.

2) There are some things, such as a lot of swapping, that can cause degraded performance over time. One of the ways performance might degrade is that rez times might slow down, which would mean longer simulation pauses. Degraded performance is not "by design," but there are so many areas that could be responsible for it that knowing it's happening doesn't help point out where the actual problem is. Unless more details are discovered that really narrow down a source of degraded performance, I think the best bet for this particular issue is to focus on #1 and re-work how the simulator rezzes objects. I don't know when that will happen, though.



KB article, take two:

In case that doesn't work for some reason, the name of the article is "Why are there prim limits on coalesced objects?"

That article is not exactly on topic, but it mentions the simulation pause, which is why I had linked to it.

Yes, it's possible. The process is documented here:

http://wiki.secondlife.com/wiki/Agni_Pilot_Regions

Thanks for the reply, Dante. That KB link does indeed work.

Why do you insist on taking a simple sentence out of a point I was trying to make and deflecting it?

I will refocus my question aimed at LL, not you Argent.

So, when you 'buy' a sim, hosted on a server... isn't there a set quantity of resources allocated to that? Every web host I know of state what you get for your money. This does sound a bit open ended, and which allows for moving goalposts, effecting those that pay for the service.
I understand there are shared resources within the system, but thats why the costs of operating within SL are so high compared to standard webhosts. What I sense here is the formation of a cloud server system...

I didn't. To repeat what I wrote in the second half of the message you just ignored: LL does not need to virtualize, because they're already getting the same capability by simply running the same program on all the servers. They don't need to run on a cloud because they already *are* a cloud. Virtualization would simply add overhead and increase costs.
Yes, 800MB of shared memory, one core of a quad core CPU, disk space, network services, access to the asset servers, maintenance, support, and R&D. Virtualizing would reduce the available CPU by 5-10%; reduce the available RAM by, oh, 200MB or so; reduce the available disk space by adding copies of currently shared files to virtual disks; add overhead to all I/O due to the networking layer. It doesn't matter whether they use Xen, VMWare, Hyper-V, or any other product... these overheads are inherent in the technology.

For some bizzare reason, you are under the impression you work for LL... my question was directed at the poster and LL in particular. I don't want your assumptions, opinions etc.. I specifically asked the poster and LL.

I'm under the impression that I know the answer to the question you raised. If you're not interested in what I have to say, you're welcome to put me on ignore.

Okay, let me make this simple for you... if I want you to answer my question.. I will add your name to the post.. if LL, I will add theirs, which btw, I did.

You "buy" a named map region, which is a complex data construct. The Simulator is the program which "runs" the region when its up. That includes keeping track of all the contents of the region, simulating physics, running scripts, and communicating with all the users who are connected to that region (in it or can see it).

The physical server hosting the region can change at any time. Full regions normally are 4 per server (one dedicated core of a dual core/dual CPU server). Homesteads and openspace regions, which can have less stuff in them, are packed more per server.

So what you get for your money is 1/4 of a server basically, though its performance can be affected by the other regions running on the same box. If they hog all the communications capacity (to users, and to the other databases), it can affect your performance.

In addition, you are paying for a share of networking, support of 260TB of asset databases, customer support for 750,000 active users, etc. Oh, and Concierge support if you have 1/2 region or more.

Just looking at the server cost is like just looking at the satellite costs for DirectTV. Sure, they are a large cost item, but theres a lot more than just that to run a TV service, or second life, than the bare hardware.

As far as "cloud computing", SL is already a cloud type service, you get resources based on how much "land" you own. Buy more land, get a bigger fraction of the server capacity. Not in a strict "I get 5GB storage and 50GB transfer" kind of contract, but in a "I get one prim per 4.3 meters of land, and a reasonable ratio of scripts and avatars".

It might appear to make sense from an efficiency standpoint to have a variable number of regions per host server except for two things: Load can vary dramatically if a club, for example, has an event, they can go from 0 to 40 avatars and back again. Moving a region to a new server if the load rises means shutting it down.

Second, us users can write all manner of scripts, or do other things and crash the region (I have done it myself when testing something for lag). By isolating each region on a separate core, you are less likely to take out another region when yours dies.

As far as why I am I answering this, well, Im writing the SL user's manual wiki pages (https://wiki.secondlife.com/wiki/User%27s_Manual), I used to test and document software for a living, and there's 750,000 of us, and 300 or so Lindens, so in aggregate the SL users have more time and know more about it than they do, with a few exceptions like Server software internals.

What's with the Release Notes already?

you mean these?

https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_Beta_Server/1.30
https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_Beta_Server/1.30