And I was just going to post about Black Wednesday...

I came to the forums with an idea of posting a 'beware, another Black Wednesday is coming up tomorrow in SL' (Europeans getting short end of stick again)... But lol, only to find that SL is down. Again. Asset server problems. Again.

Linden Labs says: come to SL, make money, do business!
I say: are you joking? With all these outages, the only reliably working thing in SL is the software that deducts the monthly fees from my credit card

I wanted to work on a product of mine, but I'm glad I didn't log in to work on it, because it would've probably been lost time and work...

This is not good.
SL needs to be better, more stable... This is not a platform. This is a joke.

PS: At least the asset server is back up now, and running! Woot!

I bet you have some good ideas for how to fix it. I would be very interested in hearing your suggestions for ways that this could be done.

Eh I don't have to be a firefighter to see and point out when there's a fire.

How to fix it? Black Wednesdays could be eliminated by rolling updates -- okay, I read this from the Linden blog, and it is a good idea. But when will they implement it? After Havok 2?

How to fix the asset server? Another question. The only public info about the asset server is that it runs on many machines and some custom software. Which doesn't give too many information that would aid in getting suggestions. If it was open source, at least I could take a look, eh Not that this would help much: to stress test it, there would be a test lab needed with many machines, which is not in everyone's homes.

Y'know, Anshe Chung should be required to wear a T-shirt that says *Results Not Typical.

Very Few of Us will make any real kind of money on SL...And of those, I doubt *any* of us, even Anshe, will make enough money that a asset server crash is going to hurt us financially.

Bingo!



Someone should write a more realistic article with the title "buy your groceries (once in a while) with Second life!"

For whatever reason, I LOVE this idea. Anshe has been remarkably quiet lately. Must be counting her millions.

I made enough money to buy a couple songs for my Ipod.

For one thing they could actualy test releases rather than jsut plug them in to see how screwed up they are.

Okay, fair enough. What is your proposal for how to test the releases as if 8000+ users were all in world?

LOL cowhand..I have made some money too now if they would just make slingo n shopping illegal...id get to keep some of it....

Well, I made a scripted thing in SL for an event (I think a quiz machine). I scripted it, tested it a bit, and it ran fine... until one day, more people came and used it than memory was available. It gave me a script stack-heap collision error.

So what did I do? I decided I need to test it better, better than just clicking on it.

I created (scripted) really really simple 'bots' that acted like humans, that randomly did actions, and interacted with the quiz machine. I rezzed about 20 of these bots, then 40, to see what's the limit, how the quiz machine reacts.

The bots told me the limits too: on another project, I determined that my scripted object works fine up to 90 ppl. doing 210 'actions'. (After that, it goes to a fail-safe mode, not loosing any data, just not accepting more actions.)

Also, if you create (program) a website, you can do a stress test with simulated load and simulated clients. The test helps you determine how the site will bear with the load, and help you adjust and fine tune the software.

While simulating 8000+ users with bots is not really doable, using a smaller swarm of bots on a test system might help, too. To test one aspect of the system, maybe not even bots would be needed, just 'dummies', who just put load on the asset server, somewhat randomly, but mimicing the user patterns.

One problem is, it's not easy to program SL bots, there are many different actions, scenarios. And possibly no matter how good those bots are, people can be even more creative when finding bugs. But nonethless, I think a stress and regression test with bots would not be a bad idea.

But this is just one example.

Of course, testing is not cheap! In fact, it's pricey! But it might be worth it.

Let's say I buy something for L$ 2000 (8 USD). Asset server crashes. 8 USD goes down into the toilet. Not good.

And just on a side note, found this while googling for SL buisiness:

SL Business Magazine
http://slbusinessmag.com/

Wow, it's a 60 page PDF magazine (about business in SL)!

That's a totally void point made, I'm afriad. We the paying customers pay LL to do a job and provide a service. It's not down to us to know any more than it's your job to know how to change the behaviours of teenage kids that are difficult to manage (that's what I do).

The deal I signed up for was I pay, LL delivers and lately that deal has been looking a little one sided.

Bulldog

I'm still waiting for it to pay for a gallon of milk. The day that happens, I'm throwing a party. We're gonna put chocolate in that milk, baby. Chocolate milk. The "Linden Labs bought me a sexy gallon of chocolate milk" party.

One can only dream.....

Rolling updates? Can't be done at this time due to the messaging system SL uses. The communication between regions and servers at this time is very strict and has to match an exact pattern or the items aren't able to communicate. LL has reported they are working to make a more flexible system to allow for rolling updates and other such things in the future, but trying to fix what we have now is more important than trying to re-engineer the messaging system without opening up any security holes. After all, if 2 sims aren't using the same messaging, maybe someone can slip something through the gap?

Asset server?

NAS cluster of Isilon Systems servers. They operate essentially as giant webservers.

Maybe if servers all went first to a 'frozen/read only' mode (no items can be rezzed, nothing can be changed), then went down and up (restart) for the upgrade, it would still be better for people. At least they could wander around, chat, just not change things in the sim until it gets in sync with the newer sims.

This same system also manages in-world communication. Scripts would need to be disabled in every region, which can possibly break scripts. IM's would be down as well. You would only be able to communicate with those near you, and in the same sim.

Really, at this point, the downtime is the best way.