Babbage Linden - Limit scripts PER PARCEL

Huh? Why this noise now? Script limit was known to come, before xmas (check out Babbage's office hour logs). One big reason why open space SIMs didn't perform well was/is scripts that are consuming too much SIM's memory (like 255 prim hair that have script in each prim).

Yes it is a known issue but this is not an easy fix. The fix itself is going to change the overall SL experience, and since it has not been worked with yet, we do not know just how big of a change. Whimper, barely noticeable or all the joy is gone unless you own a full sim? Just as in the MONO roll out, this will be worked on in Aditi for MONTHS before it is ready to make it to the MG.

Guilty, I'm afraid. I'm a huge fan of Damselfly hair...except that they have that awful resizing-by-menu script in them. I asked the maker about this, and she said that they had plans to introduce hair without this soon. I surely hope so, because I love my Damselfly hair, but I hate being a Bad Citizen lagwise, and I'd hate to have to throw out my favorite styles too...at least unless I could re-buy improved versions.

From http://wiki.secondlife.com/wiki/User:Babbage_Linden/Office_Hours/2009_03_18:

Wake up and smell the 21st century coffee, Hewee. For example, virtual servers like VMWare. Google "virtual linux server" for lots and lots of info.

This is a big thing these days. You get a virtual server with a specified amount of resources (depending on your service level agreement). The hardware resources for that come from a pool of hardware servers. You share these hardware resources with others, but as long as your service provider keeps their end of the SLA, nobody sharing those resources is ever allowed to impinge on your guaranteed allotment.

Welcome to the modern day of computer engineering! Enjoy your timeslice!

Since Meade's the one that added SVC-835, I'm sure she was hoping to read somebody ask "and that will work on the mainland, right Babbage?" in the chat log..

Babbage implies it will work but doesn't actually say it. Until I actually see it work at my land on the agni mainland, I'm going to have a hard time believing it.

Transcript of today's Babbage office hour is on the wiki:
http://wiki.secondlife.com/wiki/User:Babbage_Linden/Office_Hours/2009_03_25

Starting around 3:37 is some info about Mono vs LSO memory usage, and how Mono's dynamic memory allocation complicates the way scripts may be limited. (Fortunately I was wrong, and the actual memory usage of Mono scripts can be less than LSO's fixed 16KB, but it's not yet clear how they'll be able to reward that in measuring usage against the limits.)

Earlier, at 3:27 I brought up vehicles, but Babbage couldn't remember how that was planned to work.

Elsewhere in the log some interesting data about rate of Mono penetration, and about possible long-term introduction of Mono-only features and eventual phasing out of LSO compilation for new scripts while retaining execution ability.

I'm not precisely sure what it means when he says they're going with a model like prims and like llHTTPRequest. Either way, it seems like a big shame to me. The better solution would be some kind of time slicing/fair priority queue where each parcel gets a bigger slice of time or higher priority to run scripts based on it's size.

Take this example, one whole sim rented out, a 16 cut out with a rental box on it. The renter with 65520sqm uses hardly any script time. That 16 should get as much script time as it wishes but, if the larger parcel wants to use some then it should get priority. The large parcel would have the ability to almost (but not totally) choke the 16 of script time but if the big parcel isn't using the resource then give it to the 16.

But at the moment, the larger impact of scripts is not on CPU time per frame on the sim, but swapping on the whole server due to memory size, so it's memory that has to be constrained.

The analogy is actually to allocation of URLs to prims by parcel or avatar for handling the http_request() event (kind of the opposite of llHTTPRequest). As I mentioned earlier, http://wiki.secondlife.com/wiki/LSL_http_server#Resource_Limitations has some details of how this works.

I think I agree that for sharing CPU time, some priority queue approach would be preferable to simple quotas. For memory... not so sure. And in either case, whatever they do has to be very low overhead or we'll end up worse-off than we are now.

OK, I see. Welll in that case: Buy more memory guys, the stuff is crazy cheap right now.

Otherwise I guess you're right, it's not possible to have dynamic memory limits. i.e. you can't give someone 5MB of memory to use and then say, sorry you've been preempted someone else needs that now. Tough situation I guess because pretty much any script that bumps against a physical memory limit will fall over in unpredictable ways yet a limit per parcel does /seem/ kinda fair.

Quotas work fine, but they can't be fixed proportional quotas, because actual script usage is very lumpy, and overcommitment quotas work VERY well for lumpy usage patterns.

OK, wait, how can these memory usage limits be anything like llHTTPRequest limits then? http requests are dropped if you make too many in some time period. What will the memory limiter do that could be in anyway similar to that?

No, they're like llRequestURL()... if you don't have one available, you don't get one. Supposedly if you don't have enough script resources available, the scripted object won't rez (or something like that--no idea how that's supposed to work with dynamically allocated memory, hence the "malloc failed" spectre.)

He said that existing Mono scripts would be assumed to take up to 64k, so I assume that means there will be a UI or API you can use to set a script's quota lower, and then you'll get a stack-heap collision when you hit 8k, 24k or whatever you set.

It's not really a stack-heap collision in Mono, of course. It's like "dial tone" and "bus error"...

It seems clear from (at least) the last two Babbage hours that sim swapping due to script memory demands is a killer.
There are three sources of script memory demand
1. The scripts in prims in the sim
2. The scripts in avatars/attachments
3. The scripts in vehicles not owned by a sim landowner

If the primary purpose of limits is to avoid swapping, then there are two main ways of implementing limits within a fixed amount of sim memory.
a. Fixed reservations and limits for the memory pools of Land, Avatars and Vehicles
b. Dynamic allocation of memory to each of 1,2,3 above from an overall sim memory pool up to a maximum limit for each individual entity in each category.

(a) Would give us a situation in which sims will never start swapping and no script will ever break because of memory unavailability. We would have predictability. Avatars could move from sim to sim, whether on vehicles or not, and 'stuff' would always work.
The cost for that benefit would be a restriction on scripting that in many sims would be 'unnecessary'.

(b) Would point us back in the direction of square one. What happens when the dynamic demands can not be met?
Do we start swapping? - in which case why bother to do anything at all?
Do we reject the demand and cause scripts to break randomly?
Perhaps in this situation we could have a hybrid solution, with permanent allocation of memory to scripted prims based on land area owned, and dynamic allocation to avatars and prims.
That would ensure that sim content would always work, but avatars and vehicles would have to take their chances of scripts breaking randomly.
That would lead to a situation in which we check Map for green dots before TPing to a sim, and thinking "Nah! That's probably going to do my head in - with my stuff acting up".
"My-Scripts-breaking" would be the new "lag".
Sim landowners would be on a search-and-destroy mission for avatars that used a lot of script memory, even if those individual avatars were under the maximum memory dynamically allocatable to an avatar. Ditto situation for vehicles.
In favour of this new situation would be that a loaded sim would not be swapping three other 'innocent' sims.


Memory is cheap, so add more memory?
How much additional would be enough?
Would it simply get consumed, and we'd be back to where we started?
Is memory still cheap when multiplied by the x-ten_thousand sim servers currently existing and projected?
Even if it is cheap, the most expensive part would be the meat-space project to physically visit every rack of servers, reboot the sims running on the rack to shift them to another rack, then install the chips and test.

When I first saw that LSO/Mono - 16K/64k thing, my immeditate reaction was goodbye Mono and its benefits.
If the LSO is a fixed 16 and Mono can use less than 16 (or 64), then it's the reverse.

However, I have a feeling that much of the scripting in SL has been produced cookbook-wise by non-tech people.
Right now, a new script will be compiled in Mono by default. An edit of an existing LSO script will be recompiled in LSO by default.
If 64K is a fixed default allocation, then I'd guess that we'll see a lot of 64k scripts that actually only need 8k. If the Mono default were 4k or 8k, and calls were necessary to expand that, then maybe things might be better.
Although... if the default were to prove to be less than optimal, then we might see the first instruction in a scripting cookbook that went along the lines of : "You're going to run out of memory, so use this instruction at the start to allocate 64k."

Whatever gets implemented, it's important that the tech-heads designing the interfaces bear in mind the many of the users will not be tech-heads.


We'll probably continue to see a lot of legacy LSO scripts that will never go away. Why would they? They're in the existent standard freebies circulating.

Oh, but others WILL be impinging on your allotment. MORE SO with the script limits in place. It is also a vastly different business model. Service providers bend over BACKWARDS to make sure you have more resources than you really should need for your scenario. LL is in less of a position to do so without some careful engineering. They have to balance things carefully rather than simply saying, "Hmm. Our customers' requirements are excalating; let's just buy more hardware." But still they need to a little of both, and simply hamstringing user functionality is not a very positive answer.

How do you think those service providers give everyone those (huge) limits? Tradeoffs between different constraints: memory, CPU time, network use, database access, etc. It is a balancing game with conscious tweaking used to ensure people GET WHAT THEY NEED. From what I can see LL is not doing that. They are doing the opposite; impose fixed limits on each type of resource rather than managing and tweaking things consciously on their end; set limits that you KNOW just about EVERYONE is going to bump into and work like crazy to try to niggle their way around; take the easy way out and stick their difficulties to their customers rather than doing some careful engineering and architecture to try to IMPROVE the user experience all the way around.

We can differ in this, but as time goes on I'm being convinced more and more of this standpoint. It's very difficult for me to see this as a positive change, though I certainly do understand the motivation. And I'd certainly like to help, which is why I contribute positive suggestions at the same time I'm offering my scathing critique. Maybe someday my personal situation will provide me with the opportunity to apply to LL and help them work on their architecture first-hand. I'd love to be digging into server design documents (if they exist; MAN I'd love to have some UML to go with the viewer source code) and helping to unravel the spaghetti rather than trying to voice half-assed, unheard suggestions from the outside.

http://aws.amazon.com/ec2/#pricing

I guess Amazon disagree.

Er.. It's not _that_ crazy cheap to buy & install when you have to multiply the numbers by 5000 or so..

A GB of RAM is about $40. So $200,000 for 5000 of them. For a company making whatever it is, millions per month, this is still not a large one time cost.

Mono scripts on average are about 3x less space efficient than LSL. This is due to more complex objects, looser code, Unicode-only strings, and less compact data structures. Therefore they gave them 4x the memory to allow for that. For some scripts that turned out not to actually be enough, but nobody noticed because they weren't strictly enforcing the limits until the recent scheduler change they had to roll back for now.

Mono scripts can allocate memory more granularly. LL uses an 8k boundary. So a VERY SMALL Mono script can be half the size of an LSL script.

The 64k allocation does not represent how much space the script takes up. It represents how much space the script has been promised it is allowed to take up.

Other way around. "You get 64k, but if you want people to buy your scripts after the limits are in place, and if you can make them run in 8k, use this instruction/compile option/pulldown to set the script limit on this script to only 8k".

Plus a LOT of scripts will be 3-4x smaller as LSL2 code than Mono.

They currently have 4GB in the servers. They have two or three kinds of servers, with different ability to accept more RAM, and they need to at least double the RAM to make much difference. So that's US$800k plus a couple of MONTHS of slow rack-by-rack rolling restarts to actually go around and put the RAM in, plus replacing some percentage of the servers, and ... it's a BIG DEAL.

Well apart from the fact that you made up the "double the RAM" statement I agree. Except I still think it's NOT A BIG DEAL.

LOUD NOISES!

I've been a system administrator for 20 years. It is rarely cost-effective to roll out a big across-the-board upgrade for less than 2x capacity increase. I was actually being conservative, because technically they can get a meaningful benefit from quadrupling the ram before they start having to think about rebuilding all the servers software for 64-bit. If I went to the CEO with something like this, I would expect him to come back and tell me to put as much RAM as I could.

The big deal is not the RAM, it's going around with a NOC monkey and a crash cart to all 5000 servers and actually doing the physical upgrades. That's enough man-years of work that the difference between $40 and $160 per server for the RAM itself is not the major cost.

Look, just upgrading the OS on all the servers... which didn't involve any hardware changes... took two weeks of grid-wide degraded performance. LLnet, which is a less aggressive hardware upgrade than this, is a months-of-planning job. You can't go "I can get another GB from Best Buy for $40" and multiply that by 5000.

I was thinking more of the recreational/amateur/quickie scripter than a merchant scripter.
A new script is going to be Mono by default unless the person unchecks the option.
However, if it's the case that a Mono script starts off requiring 8k (at minimum) and can dynamically grow in 8k increments up to 64k - or a lower limit set via a future drop-down option - then my concern about a 64k allocation by default goes away.