Welcome to the Second Life Forums Archive

These forums are CLOSED. Please visit the new forums HERE

Weekend Grid Outages

Smokey Newman
Registered User
Join date: 12 Feb 2007
Posts: 6
01-19-2009 20:54
From: FJ Linden
Thanks for the feedback and suggestions. I wanted address a couple more questions that seem to be recurring. First, we are looking at the affect of bots on our load. Right now there is no immediate indication that bots are a major contributor of load, but we if we find bots that are especially bad, we'll deal with them.QUOTE]

Can I ask where you are looking its quite obvious where Bots are by looking on the Map and seeing the green dots all in rows of 10s and 20s. I cannot beleive that the amount of bots online does not affect the load. Like most people I beleive you are not doing anything about them as it makes the figures look good. Yes 75,000 on line looks great. Pity you cant count the real people that cant get inworld and are getting more disatisfied (by the way these are the people that bring real money in world). People pay for accounts, pay for land, free accounts bring money in to as thay rent have businesses buy stuff etc bots clog up the place bring in no money boost and basically make trafic figures and residents on line figures totally inacurate.

Sorry but I think there may be a problem here.
Viktoria Dovgal
Join date: 29 Jul 2007
Posts: 3,593
01-19-2009 20:58
From: Bob Bunderfeld
But Frank, Philip, Robin, and the rest of you down at Linden Lab, isn't pulling in the Big Boys of mySQL exactly what I suggested you do LAST YEAR when the Asset Server was throwing up all over the place?

The asset servers aren't where MySQL is. Those are file servers not databases.
Traci Yiyuan
Registered User
Join date: 10 Jan 2008
Posts: 1
How to Handle Bots
01-19-2009 21:01
I'm very happy to see the discussion about the problems associated with Bots and artificial traffic numbers. I have seen many good shops, clubs and sims go away because of the unfairness of artificially engineered search results.

It would be very good to see the Lindens fix this, however, my experience in SL is that there is always someone ready to game the new system the moment they change it. Regardless, I can see no good reason to make it easy for artificial AVs and artificial traffic numbers to mess up the system for us real people.

We residents can do something about this. When I search on a type of sim and get results, I immediately pass over those sims with traffic over 20k, assuming that they're cheating the system. Even if they're not, lag there is sure to be horrid.

Try visiting the mid- and low- traffic sims. There are many gems there, and unless you've gotten a referral to a high traffic sim the goods are likely to be no better than anywhere else.

Dear Lindens, since you instituted voice chat I have answered every survey with "worse" when you ask if SL is getting better or worse. I don't know if voice played a role in the deterioration of the system. I will give you credit though, over the past month I have come to believe that you're trying to fix things and be up front about the problems that need to be solved.

Dear fellow SL residents, don't reward sim owners who game the traffic system. Avoid high traffic sims unless you have a referral from another resident.

Traci Yiyuan
Khamon Fate
fategardens.net
Join date: 21 Nov 2003
Posts: 4,177
01-19-2009 21:14
From: AWM
I have to wonder why, with the millions of dollars generated by SL, only open source database systems are considered?


The point of this graduate style project is to prove that a gridded world can be built and maintained by a ragtag group using open source and in house software, for the fun of it, why not.

Avoid the trap of believing that serving you as a customer, giving you your "money's worth," or creating, much less distributing, viable virtual world software for the masses to use in a practical manner are goals of the lab. Those concepts are wholly foreign to the process.
_____________________
Visit the Fate Gardens Website @ fategardens.net
Tsuitengu Kuramoto
Registered User
Join date: 24 Apr 2007
Posts: 6
01-19-2009 21:43
From: Argent Stonecutter
I don't care for MySQL myself, and think they should have picked PostgreSQL.


Have you given thought to the pains with replication of a PostgreSQL vs. a MySQL implementation?
Katheryne Helendale
(loading...)
Join date: 5 Jun 2008
Posts: 2,187
01-19-2009 22:03
From: Torri Merlin
As a non-paying member i PAY 40 US to live on my Island. I have myself and my alts (only one in sl at a time) to keep in the newest skin, clothes, hair and shoes. I'm a huge consumer of what all you wonderful creators make and easily drop 100 US or more monthy on "stuff". And i dont think i'm alone....We shopaholics are what keep SL business alive and it wouldnt do at all to give paying members login advantage over people like me. I know you are frustrated but keeping us non payers out wont solve the problem....now about those bots...do they come in male and work as cabana boys??? :-)

So many of these "paying" members of our society have taken on such elitist attitudes about the fact that they "pay" that it seriously sickens and infuriates me whenever one of these egotists suggest that non-paying members be shoved to the back burner!

I would seriously like to know what it is about being a payed member that makes you all so special? As a paying member, you get a free plot of mainland and a weekly stipend. You also get the right to buy more land on the mainland, buy sims, and set up a lucrative for-profit business on that land.

As a non-paying member, I do NOT get any free land, do NOT get a weekly stipend, BUT I am still expected to pay my tiers for land I have to buy from someone else (and will never fully own). Further, I am expected to make my money through hard work, since there is a limit to how many Lindens I can buy. Somehow, I am still able to do all this AND spend my hard-earned Lindens for products and services likely created/provided by paid members!

I am bold enough to say that I believe the non-paying members are the backbone of the SL economy. Don't believe me? Maybe all of us non-paying members who apparently don't contribute in any way to SL except to lock out paying members should stage a one-month boycott of all paid-account-owned businesses and services! Let's see how well they survive without us!

Be careful what you wish for. Be REALLY careful!
Ryker Jacobus
Registered User
Join date: 7 May 2007
Posts: 4
Unable To Log
01-19-2009 22:38
From: Leonardo Zimring


I sympathise with your problems but I'm sure that LL is correct. Your problems were local, not to do with LL. I live in nearby Australia and I have had NO problem logging in at any time OTHER than the weekend when the grid was down.

I knew when I posted this that I probably would generate at least one comment like this. Just logic suggests that when a problem occurs like mine at a time when the SL grid is in a state of serious dysfunction, it is at least in part due to the dysfunction. Simple geographic locality doesn't mean that we are being served by the same servers.
This, after further investigations by the support team, was LL's conclusion:
"which leads me to believe that the login problems you have been experiencing are as a result of the updates which Linden Lab is currently installing."

I admire all that LL has achieved with Second Life and applaud their continued efforts to restore the grid to a state of functionality. We live in a BIG world that LL tries to span and our experiences and the functionality of the SL experience of course will be different for everyone but we need to keep it real, as does LL and from the opening discussion of this thread I believe they are, there are serious issues facing the Second Life Grid as it stands and as paying customers we need to express our dissatisfaction and have a right to do so. Credit where credit is due but making excuses at this time is a failure to front up to the problem.
shug Maitland
Registered User
Join date: 15 Apr 2006
Posts: 30
First, Congratulations!
01-19-2009 22:40
80,000+ ppl !!!!!!!!
okay, it did not work perfectly. The grid did not totally crash as it would have with half that only a year or 2 ago!
Those of us who have been here a while (8/24/05 oldest alt rez date), have listened to gloom and doom all along. We are still here, we can do more, see more, experience more than most of us would have ever believed back then.
It has not been an easy road, and it will continue to be bumpy, but we will continue on toward the Metaverse Philip envisioned at the start.
Good going Lindens, keep up the good work!
Tsuitengu Kuramoto
Registered User
Join date: 24 Apr 2007
Posts: 6
Time to start thinking about scaling out
01-19-2009 23:01
Hey Frank,

I think it is great that you are focusing on realtime analysis, without them you wont know where the key problem areas are. As I'm sure you are aware there are no one size fits all approach to scalability and high-availability, nor is switching out your DB server at this stage for another vendor going to be a silver bullet... that's far from the problem.

It sounds like the central datastore is way too big if it's taking 1 hour to warm up and comeback online after a crash. While moving some of those intensive queries to the slaves in the short-term is a good idea, that could have some problems like replication lag and depending on the sensitivity of the queries might not be what your looking for. Is caching (Memcached) included in your architecture at least for some areas of the global datastore?

IMHO long-term, I think it's time to start thinking about a sharding strategy to scale horizontally instead of vertical. You already do key generation by UUID so storing content in shards should not be a problem as your guaranteed a unique identifier, just have to figure out how you would slice all your data.

Sadly, no DB vendor has sharding technology out-of-box, nor is it readily available unless of course you look into Hibernate, HScale, or use MySQL Proxy as a glue... don't think there is anything out there for Python at the moment although last time I checked there was a new entry available on google code, still in it's infancy. At any rate, generic solutions hardly have your business logic in mind and the options mentioned are not even 100% complete.. pieces are still missing (ie. rebalancing). Any way you look at it, you'll most likely end up rolling your own, probably a good thing, much better to design it in-house to taste.
Rhaorth Antonelli
Registered User
Join date: 15 Apr 2006
Posts: 7,425
01-19-2009 23:01
From: FJ Linden
Thanks for the feedback and suggestions. I wanted address a couple more questions that seem to be recurring. First, we are looking at the affect of bots on our load. Right now there is no immediate indication that bots are a major contributor of load, but we if we find bots that are especially bad, we'll deal with them.


you mention you are looking at the affect of bots on the load, but what about the affect bots have on people. People who are left standing at the door, because the amount of logins has reached the max thus turning off logins.

remove the ability for folks to use bots, or at least bots in excessive numbers (maybe limit the amount of logins per IP) then you might find that the numbers won't change too much as to how many are logged in, however the amount of customers who are happy they can actually log in will increase.

more real people on the grid = more spending on the grid (be it shopping, renting, going premium to own land, thus buying land, etc


do you think you guys will ever do something to limit the amount of bots or at least the amount of logins per IP?
_____________________
From: someone
Morpheus Linden: But then I change avs pretty often too, so often, I look nothing like my avatar. :)


They are taking away the forums... it could be worse, they could be taking away the forums AND Second Life...
papa Tulip
Registered User
Join date: 17 Feb 2007
Posts: 59
01-19-2009 23:30
From: Rhaorth Antonelli

more real people on the grid = more spending on the grid (be it shopping, renting, going premium to own land, thus buying land, etc



Additional: I know a lot of people who deny spending any money to traffic cheaters or to shop owners that give the money they earn to camping bots.
As more people do this, as less the traffic cheating has any benefit - we don't always need linden rules
Cristopher Lefavre
Registered User
Join date: 28 Sep 2007
Posts: 12
Bots
01-20-2009 00:43
Great update, thanks Frank!

There are a lot of talk about bots taking up resources, but I wonder how much really.

A typical bot that sits still in a skybox out of view from anyone moving in a sim will still make the physics engine work, but since they will be out of view from most people it should be possible to optimize them out.

A bot will not be rezzing objects, TP between sims, wearing lots of attachments needing to be fetched from the asset server on login, it actually will not log in often.

Actually, traffic bots that I guess are the most unpopular bot types because they are only intended to con people should be the least resource hogging bots.

Land sale robots TP-ing all over the mainland, making lots of calls to the system to gather data, sending data via email or XML, should be a lot more strain on the central servers.

Frank, would it be easier on the central servers if you opened up API's people could use instead of logging on bots?
Proxima Saenz
Registered User
Join date: 22 Jun 2008
Posts: 107
01-20-2009 01:29
From: Katheryne Helendale
So many of these "paying" members of our society have taken on such elitist attitudes about the fact that they "pay" that it seriously sickens and infuriates me whenever one of these egotists suggest that non-paying members be shoved to the back burner!

I would seriously like to know what it is about being a payed member that makes you all so special? As a paying member, you get a free plot of mainland and a weekly stipend. You also get the right to buy more land on the mainland, buy sims, and set up a lucrative for-profit business on that land.

As a non-paying member, I do NOT get any free land, do NOT get a weekly stipend, BUT I am still expected to pay my tiers for land I have to buy from someone else (and will never fully own). Further, I am expected to make my money through hard work, since there is a limit to how many Lindens I can buy. Somehow, I am still able to do all this AND spend my hard-earned Lindens for products and services likely created/provided by paid members!

I am bold enough to say that I believe the non-paying members are the backbone of the SL economy. Don't believe me? Maybe all of us non-paying members who apparently don't contribute in any way to SL except to lock out paying members should stage a one-month boycott of all paid-account-owned businesses and services! Let's see how well they survive without us!

Be careful what you wish for. Be REALLY careful!


Thank you for pointing this out Katheryne! :)
I completely agree with you there.

Coming back to the blog post, I have to thank the Lindens too, for giving us some information. :) Of course this doesnt mean they earned my trust back already, but its a good step to the right direction. Keep up the good work and lets pray that the next weekend will be ''normal''.
7mm Vyper
Registered User
Join date: 11 Jan 2009
Posts: 5
01-20-2009 01:55
kick my balls for it bull shit need make sl pay to play ppl need pay log on or pay 1 time to make a acc say you make sl pay to play cut cost on sims so do not make up ppl will give up on sl you do it butt i not see a fix good winer make ppl pay to make a acc so ppl love sl go on playing all newer acc pay to join will all attacks on sl i will pay 10us cash make new acc
Deeso Saeed
Registered User
Join date: 21 Dec 2007
Posts: 9
mysql ...blaeh
01-20-2009 03:01
I still don't know why SL has stressed so much the use of mySQL. "Stable" MySQL doesn't has support for partitioning which is critical for high volume databases. It also hasn't scalabe clustering. Partitioning and disk based clustering only have been recently introduced in the still crashy version 5.1. I guess that LL had to craft some custom solution based in hardware load balancers attacking several mySQL replicas.

In my humble opinion. LL should use a proper enterprise database at least for central services. Login, profile data, financials all transactions what considered really critical. One like Oracle DB RAC, Sybase or IBM DB2, which are well proven.

Damn... Drop a 16 node Oracle RAC database with a 10GbE for Cache Fusion interconnect. Each node with two quad cores (or 4 quad core if you fancy). I guarantee that can handle ANYTHING. Of course that's expensive. :p

In how many of TPC.org charts you can find a mysql Database?

Postgresql is really fine for a single node database. Sadly, as far I know it doesn't support clustering for load balancing which is very needed.

And of course. We are still still waiting for a solution for the bots problem which is related to the traffic measurement.
say Moo
.......
Join date: 14 Mar 2007
Posts: 284
01-20-2009 03:21
Decentralize the grid inmediately FFS! :)

It was announced by former ceo Phillip, that LL would openup their server codebase.. and guess what.. another lie..
Now their busy with a instable reversed enginered (based on .net :() project instead of opening their own stable server code.

LL wakeup, SL is on it's return, i see the rats moving on the SL ship..
Use the lifeboats while you still can... hurry!

Goofing aside...

i really think LL should quickly release the code of their own serverbase, asap. This way the grid is decentralized, and if LL has a problem, still people would be able to use the decentralized grid, impacting less customers/users.
And the database servers has less overload, since it's spread globaly. (pick trusting partners to replicate some services data, like IBM you work already close with)
load balance it, throughout the partners... so users are spread among serveral clusters, instead of ONE owned by LL.. if that one fails.. all fail..

See it as the current ISP market.. in older days, everything was centralized.. meaning if a major isp had a problem, allmost the whole world (online that is), had a problem. Now everything is decentralized.. and the NET cannot be brought down anymore. it simply cannot.. there are too many isp's too many servers/routers/darkfiber/peers/transits etc etc that is simply impossible to bring down that easy.

Think about it, LL.. but hurry.. i URGE YOU!...
Better be safe than sorry, especially as SL is still hyped (a bit), don't loose the battle with other parties emerging... (sony's home, etc etc)

Stay future minded, let your dominant monopolistic arrogant feel go (no offence here, but it's common with monopolistic players in a certain segment.. success makes more hunger for success and money), and openup.. share.. commit.. you will be rewarded massivly, i promise you that.. (their are numerous projects to back that up)
say Moo
.......
Join date: 14 Mar 2007
Posts: 284
01-20-2009 03:29
From: Deeso Saeed
I still don't know why SL has stressed so much the use of mySQL. "Stable" MySQL doesn't has support for partitioning which is critical for high volume databases.

In my humble opinion. LL should use a proper enterprise database at least for central services.
One like Oracle DB RAC, Sybase or IBM DB2, which are well proven.



For your information, i know from first hand (seeing and reading internally) that Google uses MySQL as their main database system.
(in huge clusters ofcourse)
And they are, infact never down, and have massive concurrent queries each millisecond, 24/7. Maybe the most queries in the world, in that perspective..
And MySQL is infact scaling very well, on their systems.. so conclusion MySQL is stable/reliable/fast if tuned correctly.
Anny Helsinki
Registered User
Join date: 23 May 2008
Posts: 50
01-20-2009 04:43
From: Proxima Saenz
Keep up the good work and lets pray that the next weekend will be ''normal''.


Please not normal, maybe working :)
_____________________
http://www.youtube.com/watch?v=OM0g-7sZeSo
Katheryne Helendale
(loading...)
Join date: 5 Jun 2008
Posts: 2,187
01-20-2009 04:48
From: 7mm Vyper
kick my balls for it bull shit need make sl pay to play ppl need pay log on or pay 1 time to make a acc say you make sl pay to play cut cost on sims so do not make up ppl will give up on sl you do it butt i not see a fix good winer make ppl pay to make a acc so ppl love sl go on playing all newer acc pay to join will all attacks on sl i will pay 10us cash make new acc

*scratches her head, trying to figure out what this just said*

Did he just offer to pay everyone's premium account fees? I sure hope so, because what little money I have left after paying rent and utilities is just barely enough to keep food on the table for my family...
Argent Stonecutter
Emergency Mustelid
Join date: 20 Sep 2005
Posts: 20,263
01-20-2009 05:00
From: Tsuitengu Kuramoto
Have you given thought to the pains with replication of a PostgreSQL vs. a MySQL implementation?
No, I've suffered the pains of replication with an Oracle implementation, though. Or are you talking about something unrelated to clustering (a word I will never again be able to think of without appending the great Anglo-Saxon Adjective that means "replication" to)?
_____________________
Argent Stonecutter - http://globalcausalityviolation.blogspot.com/

"And now I'm going to show you something really cool."

Skyhook Station - http://xrl.us/skyhook23
Coonspiracy Store - http://xrl.us/coonstore
Linda Brynner
Premium Member
Join date: 9 Jan 2007
Posts: 187
01-20-2009 05:22
Certainly very good info F.J.
My experience in 2008 was a bit better over 2007 i agree and you still have lots of work to do.
At this very moment i had to login 3x to to see my parcel info ( it stayed blanc ) and being able to push the music button. After 2 teleports i keep on seeing parcel info from the one i just left and also hear the old music too. I know from experience that there is a DNS thingy going on; i just have to wait a few hours lol

That said.

On Zee's latest forum about the stats i pointed out my statistical review concerning Bot and Alt Accounts. Now, i do not hope i am overshouting myself which certainly is not my intention, however it would be really nice to get feedback from LL what their planning is concerning that.

For my details: see that post.
Quick: ~15% Bot/Alts are online at the same time.
On many Islands which are high in the search massively used Robo alts are hidden with various methods ( not only high in the sky in a box ), or not even hidden but done in such a way you can hardly tell at first seight.

That also said,

What are you plans regarding this as they clutter logins too and effect performance.
To my best guess ~100 regions distroy the fun over the many thousends.
Give them a warning or something on short notice.
Let SL not be able to connect 3rd party thin roboclients.
Let SL not be able to login 3x from the same real user.

Just some proposals, but i would like to here LL's plans.
_____________________
Love, Linda

Land Store • Freebies • women Fashion
http://slurl.com/secondlife/Rundlelawn/14/58/30
http://AboutLand.wordpress.com

Beaches Mainland Protected, the best remaining in SL

http://slbotblacklist.wordpress.com/

CNN iReports http://www.ireport.com/docs/DOC-205129
Linnrenate Crosby
Registered User
Join date: 5 Jun 2007
Posts: 49
01-20-2009 05:42
Hiya Catherine,

I do work in the IT industry myself so I know how vulnerable databases can be, however saying that I have tried several times finding a way to get my complains about the lack of grid wide warnings thought to you the Lindens.

When I joined second life 2 years ago blue dialogue messages was normal to see when it was a problem on the grid... even the small problems. Through 2008 and continuing into 2009 (it seams) your grid team hardly take the afford of sending this out anymore, if your regular resident experience lag or slow inventory loading the first step is to relog.

So when relogging and seeing it fails and be prompted to check http://status.secondlifegrid.net for the current status of the grid (This has happen a lot of times to me) and most of the time that url has not been updated :( So of course people get pissy about it, we need information.

So please I think it's better that you send out a dialogue message to much rather than not sending out one at all, we your residents need to know the health of the grid too
Ivana Pawlowski
Registered User
Join date: 4 Dec 2006
Posts: 26
Outages affecting the SL economy
01-20-2009 06:12
While the current outage problems are indeed a problem for all they are a particularly bad problem for those of us that run businesses in SL. Our best time for sales etc should be the weekends and peek times as of course that is when the most people are on SL. However, the times we would expect to get the majority of our business are the times when SL fails. So in addition to loss of income we we also find ourselves spending lots of times correcting failed deliveries and payments etc.

Anyone can see that SL is overloaded as the pattern of failures so obviously follows the number of concurrent users. I appreciate that Linden Labs are looking at the problem but it would seem that some measures to relieve the stress on SL in the meantime would be in order. I won't say what as there are probably thousands of posts about it on these forums already
Maggie Darwin
Matrisync Engineering
Join date: 2 Nov 2007
Posts: 186
01-20-2009 06:33
From: Katheryne Helendale
So many of these "paying" members of our society have taken on such elitist attitudes about the fact that they "pay" that it seriously sickens and infuriates me whenever one of these egotists suggest that non-paying members be shoved to the back burner!

"elitist"..."egoist"...my, my, my....
From: Katheryne Helendale
BUT I am still expected to pay my tiers for land I have to buy from someone else (and will never fully own).
There's a simple cure for that
From: Katheryne Helendale

Further, I am expected to make my money through hard work...
Why, how unfair!
From: Katheryne Helendale

I am bold enough to say that I believe the non-paying members are the backbone of the SL economy.
"Bold" may be one word for it. Another word might be "wrong".

http://gwynethllewelyn.net/2008/10/13/the-hard-facts-about-the-second-life®-economy/
Cincia Singh
Registered User
Join date: 26 Jun 2007
Posts: 79
01-20-2009 06:48
From: Maggie Darwin
"elitist"..."egoist"...my, my, my....
There's a simple cure for that
Why, how unfair!
"Bold" may be one word for it. Another word might be "wrong".

http://gwynethllewelyn.net/2008/10/13/the-hard-facts-about-the-second-life®-economy/

Give up Maggie, they don't read, they don't listen, they just whine. It was a rocky weekend for sure and it's likely that it'll stay rocky for a while until LL gets their db's fixed. In the mean time we hang in there, live with the hiccups, and press on!
1 2 3 4 5 6 7 8