Grey Goo Fence

The rez rate limit is measured in calls to llRezObject() not measured in prims.

Yes, some build-seeds, a castle rezzer for example, will be affected if they build too fast (240 llRezObject()'s in 6 seconds). It could be that the finished castle may be missing a tower or two when it thinks it is done because some of the rez events could fail.

The rez/give limit is keyed off of asset_id XOR owner_id, so rezzing a particular asset and giving it to another object contribute the same key to the throttle count. Distributing the rezzing among multiple assets will not spare a single object or its children from hitting the throttle because there is an internal key we call the 'family_id' that is also counted. When you rez an object from inventory it gets a new family_id, and any object it creates is part of the family. Since we also throttle rez/give events based on family_id, as soon as the family as a whole hits 240/6 then the entire group will be throttled.

Oh yeah, I forgot to mention... some throttle events are also recorded for scripted unlink operations, so it is possible that objects that have hit the limit for other reasons may fail to unlink as long as they try to do so while the trottle is still active.

I've got a few ideas on how to improve things, mostly intelligent comparison and records of what is being rezzed/given and by whom. If necessary I'll add a per-owner throttle, perhaps it will have different magic numbers instead of 240/6. I hope to eventually increase the rate limits when the fence gets smarter, and maybe reduce or remove the implicit sleep built into the llGiveInventory() call which was really just an anti-grief hobble feature that may be made obsolete by the new fence.

Some of the holes in the fence will be easy to fix, others will be hard. Some may be impossible or require heavy restrictions on script functionality. I want to minimize how much of the SL script possibility space I need to cut out, so I'll proceed with caution.

Right, I see. It sounds like only house rezzers that don't rez a series of big linked chunks of house - which AFAIK is how most of them work, though I'm not an expert - will be affected.

Yes, an army of vendors could be affected, however I think the rate limit I've chosen is rather high and I doubt many will be affected. If the army of vendors were rezzed and placed from your inventory (rather than rezzed out of the inventory of a vendor builder) then they will be less likely to hit the throttle since they would all have different family ids, unless they were all owned by the same person and were rezzing lots of copies of the same asset.

So, if you have a 'mall builder' that also rezzes the vendors, and the vendors proceed to rez more than 40 objects a second, then yes they will hit the fence. I do not know of a sim that rezzes that many temporary objects for sustained times. None showed up on my scan of the creation rates from mid April which I mentioned in the other thread, which is why I picked the magic numbers 240/6.

Yes, scripts will get an error that says something about hitting the grey goo fence, however those errors are also throttled to 8 every 2 seconds, so typically only the first 8 or so errors will be reported. I think the 8th message is something like: "too many errors... no more will be sent until the food clears".

Edited -- split up a run-on sentence

Try reading?

"but all pushes and most movement methods (hover, apply implulse) will become severely attenuated."

Sounds very reasonable. Removing artificially imposed time limits on script
calls is always a good thing in my book. I'd rather be able to rez (or give)
six things in one script nearly instantly than 300 things over 900 seconds.

Can a goo not be created by rezzing 39 objects per second? Is a goo with that densiity more tolerable for the occupants of a sim? Or is it just easier for LL to clean up?

I've asked myself what can be done to tackle this problem. I've no answers.

Oooooooh! Get you! And that's about the energy level of objects hitting the fence, nothing to do with parcel push flags.

Starax, a goo of 39 rezzes per second might not hit the fence. The hope is that we will be able to respond to the slower goos and stop them manually. The fast goos will still have to be cleaned up manually, but the fence should help to keep them contained before they eat the entire grid.

Sounds great. Well if I see anything gooey about, I'll give you a shout. Oh, and who's the cleaner at Linden Lab?

Seriously - do the cleaners at the LL offices have a Linden surname too?

I started using this freebie that like rez-foo. I have a arena setup in "The Forest" for boxing terniments. We recently held a wedding there and I used the rezzer tool to help. I placed copies of the wedding alter / pews inside it along with the boxing ring / arena billboard. Since my rezzer only rezzes 4 Objects, each are several prims, some going over 100, will they be effected?

I also like the sound of "Army of Vendors" LOL. I know what you meant but it still sounds funny. Attack of the JEVNs! You must destroy there server prims to take them down!!!

If I understand the idea correctly, this should cut it to 39 rezzes per second total for all copies of the object. If that is the case, then this should cut back the long term replication rate from exponential growth to a constant 39 per second. Which sounds like a great idea.

From my perspective, I can see house-rezzers easily adapting to a throttle; couple of sleeps in a script... minor issue. I can see weapons makers adapting as well (put a bit of RL in the ability to shoot that much, that fast!), also minor. I like that research was done before a solution was proposed, it takes the 'but what abouts' down a bit.

I'd encourage release to Siva so those who are most concerned can have an opportunity to test it out... that said, if this delays the release of this most called for measure until that version is released, I'd risk it and take it live right away.

If you want an amendment to the proposal, could the "throttle fence" automatically IM the Linden group online, as well as the person who was just throttled (ala: Fence in <sim> reporting throttle enaged for <resident>. ? Rather than wait for someone (who can't do anything) to call for help, why not allow those with the ability to do something about it know right away? This would also help those who have made an honest mistake to take steps to correct thier actions.

Lastly, and this is more policy than script, let's put right out there that fence-bangers will have measures as spelled out in the TOS applied.

This is probably a noob question, but I thought it seemed relevant.

Why cause llRezObject to fail at all? Why not just give it exponentially longer pre/post-call sleep times based on the same criteria? I mean, I can understand having to wait longer for something to rez if I am doing too much rezzing, but to have it fail seems to me like the wrong way to go about it, precisely because it has the potential to break stuff.

This is a VERY good question.

How about it, Andrew?

My guess would be that storing a hundred thousand rez requests in the event of an actual grid attack could cause some problems.

I'm guessing that such a list of delayed requests would require some new behind the scenes architecture - the only similar sort of situation I can think of is the script scheduler. That will leave off running some script to do something else then come back to the script later.

Hmmm... well, I just tested this in Mauve sandbox (please don't ban me), but there are still more ways to cause major slowdowns. Notably having a massive amount of scripts in an item, and then duplicating it, will cause severe problems. My apologies if my testing caused any problems.

Any ideas on how to fix THIS problem?

D'oh, that was a really stupid thing to do. I was only doing proof-of-concept, and now I can't log in. Please accept my apologies.

EDIT: That was way too easy. I crashed a sim with very little effort, and by making a self-replicating prim to do this... Oh god...

Afterthoughts: The sim gets overwhelmed when asked to duplicate large amounts of scripts, most likely because it requires copies to/from the asset server. This manages to make crashing sims by HAND possible, which is very bad, because if it can be done by hand, just imagine what can be done with scripts.

Heh, I probably shouldn't have posted this here, but hopefully not too many griefers will see... It would be much too easy to exploit this. And thus the 'central server' problems arise...

I like this also but have another idea for Andrew's fence.

Your 240 limit seems like a good BURST limit. But its still too high for a sustained rate. I'm assuming you on any given call check how many calls have been made in X previous time and if its above your threshold you just fail. This is good. But the rate you allowed in itself is abusable.. so if the first check passes you should test for a slightly longer time frame, with a lower rez/time allowed.

Example:

200 rezzes in last 5 seconds. (current limit)
375 rezzes in last 10 seconds.
525 rezzes in last 15 seconds.
650 rezzes in last 20 seconds.

Effectively it means you can hit a burst limit of rez calls to allow for some quick single-use calls but if the rezzes continue at that rate it gets clamped. And im not saying check the speciic thresholds i gave, just examples of increasted throttle.

End result should be that over time almost NO high sustained rate of rezzing should be possible. And any grid attack that has to slow itself to cope with perpetually tighter restrictions.

There should also be a 'warning' flag raised on your family_id groups when they get within a certain distance of the fense before the barrier is hit. So that an object cant be scripted to spam out 230 objects in 6 seconds.. sleep for 10 seconds.. rez 230 more.. thus perpetually skimming the inside of the limit without breaking it and sitll doing damage.

You guys are bound and determined to make it even harder for me to fix my fractal trees, aren't you? It's not bad enough there was not giant announcement about this change. I had to find out the hard way that my legit object is broken.

Will LSL ever be updated to return a success return value to llRezObject()? It won't break any existing scripts, and would help out a lot of advanced scripts like Rez-(Foo/Faux) or Jeffrey Gomez's BVH importer know if the call succeeded or not, to take measures like dissolving the existing rezzes or sleeping and slowing down the creation.

Delaying the scripts instead of making them fail...

This might work, I'm not sure yet. It seems like it would cause a goo army too oscillate, occasionally descheduling a few members into next year.

The first goo soldier hits the fence and gets descheduled six seconds, the next 12 seconds, 24 seconds, 48 seconds etc. Very quickly the army has descheduled the vast majority of itself into the distant future and you'd think it was 'stopped', however it actually scattered itself, with a few members in the near future. As each member is allowed to run again unthrottled (and it could be the 12 or 24 second one) it will burst a new army to about the same size, unless the deschedule growth rate is increased each time...

Hrm... (gears in head spin -- ding! light bulb goes on) If the deschedule period were given a half-life, and since each NEW goo child would NOT be descheduled yet, then what would happen is the goo would hit the fence once hard, then would hit it again a little bit later, not as hard but with the fence still halfway to next year. You'd get a double rez and the deschedule rate would be pushed back into next year. The 48 second one would hit the fence 1/4+ th the way too next year and so on... the ultimate outcome would be some final slow rate determined by the size of the army when it first hit the fence.

I'll experiment with the idea and see how well it works.

Multiple simultaneous rez-rates... that might help catch the slower but longer growths. I'll think about that one too.

One problem with rez limiting in general is that the sim lags, slowing the rez rate by the sheer number of scripts and often physical objects. It could be that some slow rates which you would expect to hit the fence would actually go under due to the lag.

How about this:

When hitting the fence limit, drop down an llDialog() like: "You're rezzing too fast, please click "continue"." After clicking, the script will pause for another 10 seconds, then resume as if nothing had happened.

Any manual / user controlled legit rezzing will still work, any automated behaviour would stop dead in operation, waiting for Lindens to take it away.

You could make the fence limit even smaller then, users can still simply confirm the dialog and merely have to wait some extra seconds.

On a related note: Why is it so hard to do clean ups after a grid flooding? Isn't there a way to deploy a "delete from grid where object_owner=griefer_id" grid-wide? I mean ... those guys don't deserve to keep any of their stuff And if this sounds too drastic, use a time limit of one hour or something. Can you clear up the issue a little for us curious kitties?

Thanks!

The way I see Andrew explaining this, my Rez-Faux won't be affected by this at all. Anything can call llRezObject in a hard loop forever, and not hit this fence, due to the 0.2 second delay in llRezObject. It might be affected if a higher-order limit is set as Seronis suggests, but such a limit would still have to equate to 5 or fewer llRezObject calls per second before it'd affect something like Rez-Faux... or probably Foo, for that matter.

Well, the way I see it from my noobish PoV is that the idea is to slow down the goo attacks so they aren't so "gooey" and degrading to the performance of the grid and the individual sims themselves.

I would think that the concept of a simple timer for an agent/object/family which starts out at 0.2 seconds (the default sleep for llRezObject). Every RezObject increases the timer by some percentage (say 10%, for example), then sleeps for that new time. Every so often, like every 6 or 12 seconds, the rate is reduced by the system by another percentage, with the lowest being the 0.2 second default.

So, basically, here's how it would go:

Rez#   Delay
    1   0.22
    2   0.24
    3   0.27
    5   0.32
    7   0.39
   10   0.52
   15   0.84
   20   1.35
   25   2.17
   35   5.62
   50   23.48
   75   254.38
  100  2756.12

(apologies if the numbers aren't exact, my mouse button may have double-clicked the button a few times).

As you can see, it can get prohibitively expensive to rez a lot of items, even with a 10% increase (isn't amortization fun? ). You could use any number of simple curve functions to smooth out the increase, or just simply cap it at some "max value". Lots of options here.
With a commensurate periodic decrease, you would eventually end up at some equilibrium point which would keep the spawns to a controllable rate.

Even with failing the RezObject (which can be detected), it doesn't abort the script, and goo objects will just keep trying anyway, so it is no better or worse a solution either way.

One function that would be kinda nice would be a "Kill all objects of mine in range with this name / ID (like a PIN, for example)". For those who are experimenting with self-replicating objects for valid uses, it would be nice to be able to terminate our experiments which get out of hand before they can do too much damage.

Got room in that boat? I also got slammed by this one. My forest rezzer is busted. And since it rezzes objects that rez objects that rez objects I don't even have an idea how to implement a throttle that can count all the rezzes esp since there are random numbers of rezzes and traveling prims involved. This limit hits way too fast. Also this isn't simply a burst rezzer it sustains the rezzing rate for about 2 minutes, but gets hit by the fence in the first few seconds.