FJ Linden (Frank Ambrose) November Grid Update

Can Frank give us an indication why performance has been particularly bad more consistently over the past 2 weeks?

Uh.. What you had said was:

It's not one big asset server. It's split up between hardware and functionality.

Next time, I'll try to put my psychic interpreter on to respond to what you mean instead of what you say.

I'd love to hear FJ's take on the opening of the architecture and source code. It was my understanding that part of Philip's dream was to make everything as open as possible, and eventually put the server source code into our hands.

So what's FJ's vision for the future of SecondLife? Is it a service-centric set up where only LL controls the server and thus the service? Or is it more of a web model where others can run a server of their own for whatever reason? How do the current changes that FJ is making play into that?

Ah, I guess that is what I said. Apologies. But it does need to be setup in such a way that a server holds the data for a subset of users only.

I enter a sim, and I see 20 textures, say. the viewer knows the UUID and the Owner for each texture. Today, all 20 requests go to the Asset cluster, which is also handling the 60,000 requests from other clients. So you have a single point of failure and a bottle neck combined. What should happen is seperate smaller asset DBs, possibly organized by where the avatar sets home. When I enter the sim and request 20 textures, I should get back responses from 20 servers in parallel, and each responds quickly since they are holding a fraction of the data and doing a fraction of the work of the overall asset system. Since the requests for data are spread out over a larger group of DBs, and each database is only holding data for a fraction of users, response is lightning fast. By basing one DB server per rack, the system scales linearly with the growth of the grid - more users, more land, more DBs to handle it all no matter how massive it becomes.

I'm not sure you get to use both the word 'cluster' and 'single' in the same context..

I think you have way oversimplified what is necessary for an asset system for SL. The major flaw in the system that you have described is that you then need an additional db to know which server has which texture.

LL's asset system is currently tracking TERABYTEs of data. It is amazing it is working at all, none the less, actually reasonably well. And as per the topic it is being worked on and tweaked constantly. Some rather brilliant minds went into designing the system and over the years they have made leaps in it's performance. We used to have problems whenever the concurrency high mark jumped by just a 1,000. It worked really well for a long time as concurrency rose from 30k to 60k, which is quite an accomplishment. 60k+ has been rough in spots for a couple of months. For the 1st time ever, I noticed problems I had never seen in MG, only in Aditi. But the last month has been somewhat smoother and the performance will continue to rise.

I would be interested how the free 500,000 US$ of hardware which got freed from abandoned Open Space sims due to the new Price Policies get used now...
2000 islands were abandoned in November where i guess most were Open Space so for each of them 250 US$ were paid and the servers used to support these are free now.

To expect is that another 2000 or more follow in December which gives nothing less than 1,000,000 US$ free again...

The servers should still have access to the existing providers as the users still have to connect to the simulator. The new fiber should just take priority for the specific routes between facilities (and if it gets filled up, dallied out to other providers). Of course I don't have any specifics on how they have configured things, but that is generally what you do.

I'd like to see a reworking of the cache system. I don't think the current system is nearly big/clever enough to cope with the amount of data that's needs to be displayed.

Surely if there's more data in the cache then less data has to be transferred over the net, resulting in a more responsive experience and less chance of bottlenecks and other network related issues.

I am going to say one thing today.
I am realising that SL is asking more and more from my laptop and computer.
Some years ago, My computer could handle Sl without troubles or crashes or lag.
Then after many upates by Linden Lab.. my computer started to lag and crash, untill it couldn't handle sl anymore!

So I bought a laptop. Thank god the laptop was able to handle the latest version of Sl, with high settings. But now.. after another update.. and another one.. I have to run my vieuwer with Medium settings, and I am STILL lagging and crashing many times.

So I hereby ask Linden Lab to understand what it means, if they releease a new vieuwer.

Geeez I Can't Understand People Staying Positive About Sl.
I Came Online In Dec 2006.
We Are Now At Dec. 2008.

Conclusion:
Go Home !!!!
I Have Never Seen Ever Such A Bad Performance.
It Takes 60 Minutes To Render 90% Of Theobjects No Matter Where I Go.

Fix It !!!!!!!!!!!

I'd like to say that Frank's work and newsletters represent for me the most substantial pieces of good news from LL during my entire time in Second Life.

/esc

Dear Linden's

I don't know what you are all doing, but at this moment i get confirmation after confirmation inworld that SL has gotten super slow after the last updates.
And i have the same experiences.
I will spare you my entire Chat and IM log, but if you want i can send it to you.
All the people i talk to inworld experience the same thing.
SL comes to a point that it becomes unplayable.
Wether you are on a Mac, XP or Vista, duo core, quad core, 256 video memory or 512.
Since the last moments, SL totally sucks.
Wether 45000 or 70000 people online, wether, you have the cache cleared,
Or if you have your hard disk cleaned, or even all graphics settings set to low...
Avatar imposters, windlight shaders off, multiple threads on or not, fast Alpha,
Nothing helps ! Texture rendering is a huge drama since the last updates.

So... server side or a huge problem in the viewer.

FIX IT OR LOOSE IT !

Frank : Finally, I have made some internal organizational changes over the past month, that I hope will begin to drive more specialization in some key areas. This included adding a new network director, and more focused team leads managing databases, asset management, and data services. My belief is that, in addition to sound technical strategy, we need the right organizational alignment and specialized technical skills to achieve long term stability and scalability on the grid.


This is very un-linden like , Sir. Have you forgotten your tao? Thank god. If you could have one of your helpers burn that missguided hippy fluff you may yet get an actual plan and vision.

If all us "strawmen" dont tell you to get bent first.

I'm refusing to do any more public work here after the latest personal way I was shafted for my efforts by the lab.

...and btw, the grid has been unstable junk all day. Carry on, I'm sure the next patch fixes everything

and damn. i just find out this second my latest project of 6 months has been copybotted and made free. You guys rock LL. Keep up the good work! Epic fail.

who are you to suggest anyone should leave?
ppl are going to drum on the OS scam for ages to come, and ll deserve every last bit of it.

yes, 'single' isnt the word i would have chosen!

Hi Linda.
I read your post with interest.



I had lag for a while and I had, like many, to work out 'what kind of lag', as "Lag" is a general term covering many things?

1) Server lag: The time it takes for a server to complete instructions and tasks.

2) Network Lag: Could be the ISP, The link between countries or the satellite connection.

3) PC Lag: How good or bad your processor and memory can run the program, and how fast your hard drive can deliver the cache.

4) Graphics Interface: How fast can your card draw the changes, and then display them?
We know with assurance the servers are delivering virtually no lag, the lag they deliver, between network related requests such as within the message system, or the inventory system are minimal.

Network lag, well, that is up to your Internet Service Providers (ISP), I have seen people struggle over a network where each change needs a full 1.2 seconds to do a single path over a satellite connection (Makes it unworkable in my book). You can do ‘whois’ and ‘netconnect’ tests, even broadband’ speed tests’ to isolate your ISP lag. If they are causing the lag, change ISP, simple.

To lose PC lag requires some investment, the PC needs to do what you want it to do, if it doesn’t, upgrade or change it. The Linden machine has no influence over ISP lag or your PC lag. They are not ‘creating’ lag so you are unfairly hitting at them, only making yourself look silly.

The biggest cause of ‘lag’ for many people Linda, is the speed at which their video or graphic card ‘draws’ what it see’s. Increasing performance here gets proportionally better and better results. If you for example, use the latest 9800 cards, a good digital Monitor recognised for the card, you will find you can draw at full range pretty quickly, I for example run with everything at ‘max’ with no sign of slowdown. If, however, if I choose to visit the mainland sims, where every prim possible is being used, every surface is different and animated, even I struggle! Again Linda, it is not a Lindens Labs issue when you chose what equipment you want to use, your location or the amount of work you need your machine to do.

I feel we are getting a very good report from our techies in Linden Labs, they are building a heck of a big boys toy before the whole thing collapses, and best of luck to them, I feel Linda, your anger should be directed at those in the system that are contributing to your angst! In the interim Linda, why not check out your ISP and your PC, or even get your daddy to do it while your mommy packs your school things...

This has no foundation whatsoever. I have three Mac computers all with different specs and different graphics cards and I use different broadband connections regularly. The fastest, the NVDIA 8800, is just as laggy as the slowest graphics card - the 6600. In the last month the performance in SL has declined dramatically. It doesn't matter where you go. Grey screens and impossible to build or do anything.

Can Frank give us an indication why performance has been particularly bad more consistently over the past 2 weeks?

YOu have to see, that the latest updates are asking MORE of our Graphic cards than any other game in the world! My card in my pc could handle the simple vieuwer easily, untill LL started updating.. fixing.. killing bugs and stuff... now my Pc cant handle the stupid new vieuwer anymore, forcing me to give up on it. Thank god I still have my laptop with a good card, but I am affraid it soon cant handle Second Life anymore.

All these updates, these extra's... are asking TOO much of our graphic cards, and soon NOBODY will be able to run SL anymore. Only the top people, that can afford the latest computers. The people that are buying new stuff every 6 months!

Linden lab is like a train, that passed many of its residents, nobody can catch up.
All these updates, and fixes... result in ONE thing - Bad performance.

TY, but this is all well known to me and already tested. Otherwise i would not have made the open letter. When i say all is tried, ALL is tried.
I get a heck of a lot of reports from people inworld so it is very true !
1.20 has been tried and draws faster.
There is Jira about that issue by many, but Linden Lab has hardly adressed it !!
It is well known that with every new viewer Rendering slows down.
Good news: The latest Release candidate is even massively worse.
After standing 20 minutes in a place, 50% was still grey; rendering stalls.

So, having tested iterations of 3 viewers after another on the same afternoon,
1.20, 1.21, 1.22 RC and seeing drops of performance from 1.20 upto 1.22 RC i am very sure that LL has an issue. And i have met many inworld experiencing the same thing.
I have invested real money inworld and would like to get it out.
I'm not going to buy a new PC for SL ( which makes no difference anyway ).

Am i angry? Very !! What is wrong of being angry ? As i get no anwsers from LL at all.
My tickets have been very polite, but the answer was... "We'll monitor performance as always". What kind of answer is that !

SOLVE IT !

try some custom made viewers..

QFT!!!!!!!!

http://wiki.secondlife.com/wiki/Alternate_viewers

Plus I am sure there are more. One of my favs is MeerKat.

That alternative vieuwer is excellent!

I use the Nicholaz Edition at work, because I have a junk graphics card there and the Nicholaz OS (Old School) works just fine ... no pretty WindLight rendering. With the LL viewer, I get lots of vector errors that simply make the screen unviewable. I haven't had an error with the Nicholaz OS client since I started using it over 6 months ago.