Welcome to the Second Life Forums Archive

These forums are CLOSED. Please visit the new forums HERE

ass set server

Julianna Pennyfeather
Registered User
Join date: 19 Aug 2004
Posts: 136
11-05-2005 22:21
Last night the asset servers began returning "disk full" errors when they were not, in fact, full. Unfortunately SL doesn't handle this condition well, so many new assets failed to upload. This behavior took everyone by surprise - we've rebalanced the load and are adding more capacity to the system, and haven't seen the problem since last night. We're working with the vendor right now to figure out why his happened and how to avoid it in the future, but I don't expect it to recurr.

*************************
this message was posted by jesika linden who obviously is out of touch as it is no surprise that the asset server is bad. this has been a problem since day one of the creation of SL jesika wake up and smell the roses. it is the most over worked machine in the SL system and understandably so. it is amazing it even works with what it has to deal with.
Enabran Templar
Capitalist Pig
Join date: 26 Aug 2004
Posts: 4,506
11-05-2005 22:22
Oh, hohohoho.

This is going to be excellent.
_____________________
From: Hiro Pendragon
Furthermore, as Second Life goes to the Metaverse, and this becomes an open platform, Linden Lab risks lawsuit in court and [attachment culling] will, I repeat WILL be reverse in court.


Second Life Forums: Who needs Reason when you can use bold tags?
Ice Brodie
Head of Neo Mobius
Join date: 28 May 2004
Posts: 434
11-05-2005 22:56
Jeska, our beloved community services/forum management Linden honestly isn't on the hardware side and most likely is just relaying what Ops, the Lindens in charge of the LL hardware told her about.

Operations... was faced with a problem that is most likely operating system or RAID controller specific, then told Jeska the details, which she simply relayed.

There's a specific phrase that kind of applies... "don't shoot the messenger." It's generally seen as impolite.

Now for discussion over actual hardware issues:
Login Server: Handles the processing of user login, storage of your account information, your profile information, classifieds, and your instant messages. Also known as Database server, MySQL Server and I believe Toad...

Space Server: This server handles assigning which sims go where on the grid, you may recall that the day before 1.7 became live, this server crashed, due to a physical, self replicating object that was released. At this time, I believe Space handles inbound script e-mail and XML-RPC requests, though I personally haven't done the checks recently.

Asset server: indeed a workhorse, this server hosts a custom designed high capacity asset distribution system, it sends your prims, clothing, avatar shapes, notecards, and several other inventory related needs, as well as textures... textures being it's biggest load. Historically I believe it was called Gnome... though that may have changed.

Sims: These servers are Linux 1U systems that also house a part of the asset management system, a Squid based proxy that is your link to the asset server, they cache any content locally, and anything that's high demand gets stored on the sim server, rather than the main asset server (textures, are a good example), These servers also handle outbound e-mail. (Sim names are on the scheme of sim###.agni.lindenlab.com

The asset server is only really over-worked shortly after a shutdown of the grid, when the proxies are purged of data, thus meaning that everything is cached on the root server.

Our asset server is a 25K USD investment in hardware specifically designed for high volume file and database distribution, thus it's designed purpose is being served.
It however, is a machine, and like all machines, it has possible problems related to the XFS RAID file system, hardware controllers, and the RAID array it houses to store the mass of data (most likely exceeding a Terrabyte at this time). Hardware, like anything, does decay with use, and while our asset server is a top of the line piece of equipment, designed for the type of work it's doing, it is still a machine and thus goes through normal wear in the course of it's operational life resulting in things like disk full errors.
_____________________
Moopf Murray
Moopfmerising
Join date: 7 Jan 2004
Posts: 2,448
11-06-2005 00:10
From: Ice Brodie
The asset server is only really over-worked shortly after a shutdown of the grid, when the proxies are purged of data, thus meaning that everything is cached on the root server.


If only that were true, then we'd have no asset server problems in-between patches which...erm...well...we do. Quite a lot.
_____________________
Ice Brodie
Head of Neo Mobius
Join date: 28 May 2004
Posts: 434
11-06-2005 00:16
Sims get patched a lot more often than you think, the client and server versions aren't specifically tied, though major releases do signify new branches of both server and client at this time, the server side code for several systems have included no less than 5 server side patches issued during rolling updates.

A rolling update is where an update is issued that is still compatable with the older code enough that the grid does not have to be shut down in one fell swoop, generally the result is that each sim is exited on the server, updates are downloaded from the Space server, and the server then re-boots it's hardware.
_____________________
Moopf Murray
Moopfmerising
Join date: 7 Jan 2004
Posts: 2,448
11-06-2005 00:20
From: Ice Brodie
Sims get patched a lot more often than you think, the client and server versions aren't specifically tied, though major releases do signify new branches of both server and client at this time, the server side code for several systems have included no less than 5 server side patches issued during rolling updates.

A rolling update is where an update is issued that is still compatable with the older code enough that the grid does not have to be shut down in one fell swoop, generally the result is that each sim is exited on the server, updates are downloaded from the Space server, and the server then re-boots it's hardware.


Yes, I know. And I'm saying that the asset server has problems outside of those situations and has for a long time.
_____________________
Ice Brodie
Head of Neo Mobius
Join date: 28 May 2004
Posts: 434
11-06-2005 00:22
Can you please be more specific? Afraid the way you're putting it is so generic that no one would be able to properly address it.
_____________________
Moopf Murray
Moopfmerising
Join date: 7 Jan 2004
Posts: 2,448
11-06-2005 00:35
Sure, here's a selection for you:

/3/53/69740/1.html
(and before you question it, that's 5 days after the rolling update)
/3/51/54068/1.html
/3/52/37941/1.html
/3/42/33127/1.html

Those are just picked from the announcements forum from this year and not every time it's happened has an announcement been made in that forum. There have been plenty more times when the asset server has been extremely slow in-between any patching, rolling updates, sticky tape application etc. where an announcement isn't needed, but it's been obvious that the asset server is under great strain - you can see it in-world when it happens, it's fairly obvious.
_____________________
Ice Brodie
Head of Neo Mobius
Join date: 28 May 2004
Posts: 434
11-06-2005 00:50
If it's announced, it's probobly either a notice that it's happening or has been fixed to the best of the Ops team's knoweldge.
Rolling server updates also have been applied as recently as Thursday, extending into Friday so 5 days after update really isn't specifically relevent, as there have been updates since the 1.7.0 and 1.7.1 client updates.
_____________________
Moopf Murray
Moopfmerising
Join date: 7 Jan 2004
Posts: 2,448
11-06-2005 00:59
From: Ice Brodie
If it's announced, it's probobly either a notice that it's happening or has been fixed to the best of the Ops team's knoweldge.
Rolling server updates also have been applied as recently as Thursday, extending into Friday so 5 days after update really isn't specifically relevent, as there have been updates since the 1.7.0 and 1.7.1 client updates.


Where are you getting this information on rolling updates being applied Thursday into Friday?
_____________________
Ice Brodie
Head of Neo Mobius
Join date: 28 May 2004
Posts: 434
11-06-2005 01:00
Ben Linden, Kelly Linden, Phoenix Linden... the fact that the sims do shut down for the rolling updates... etc.
_____________________
Moopf Murray
Moopfmerising
Join date: 7 Jan 2004
Posts: 2,448
11-06-2005 01:11
From: Ice Brodie
Ben Linden, Kelly Linden, Phoenix Linden... the fact that the sims do shut down for the rolling updates... etc.


So no announcements then? They announced a rolling update here on the 31st that was going to happen on the 1st November:

/3/34/69156/1.html

But from what you're saying announcing these is unusual? So unless you speak to these Lindens, or see them mention it somewhere other than announcements or happen to be online and in a sim when it goes down (presumably for a rolling update, although how you'd know this is open to debate) you're not going to know that they're happening.

Must be nice to be kept informed :rolleyes:
_____________________
Ice Brodie
Head of Neo Mobius
Join date: 28 May 2004
Posts: 434
11-06-2005 01:27
Generally a rolling update isn't typically announced unless it's expected to significantly improve performance (last Thursday's was presented on the MOTD among other times) And I've been in Lusk during 3 of these updates, other sims during at liest 1 other update, along with friends who happen to be in sims that have gone down for the update.

As my work in SL happens to be exploration of the SL grid, part of that exploration includes talking with the development staff. Either in a public forum, or over instant message, they are limited in what they can tell users, but they do try to help me out on my project... and have been very helpful towards my work for the past year.

Personally, I prefere to know what goes on, so I can help identify a specific problem, rather than pointing at a generalized fault.

Second Life is the largest persistant world simulation on Earth, it's server cluster, known as Agni, is over 1200 servers strong, and growing, while I do not know if they have, or are willing to... run the benchmarks for supercluster status (as the supercluster benchmark takes hours LL doesn't want to shut SL down for) they are a supercluster on the top 500 supercluster list.

The world simulation is also the only persistant world simulation offering building tools that are able to apply world changes live... and one of less than 30 virtual worlds willing to give the public access. (they are also the only one of these virtual worlds to not refere to itself as a game)
_____________________
Moopf Murray
Moopfmerising
Join date: 7 Jan 2004
Posts: 2,448
11-06-2005 01:39
From: Ice Brodie
Personally, I prefere to know what goes on, so I can help identify a specific problem, rather than pointing at a generalized fault.


Erm yes, as do I. But that's the problem - if everybody who felt the same chased the Lindens, well, that's not feasible is it, so why don't they just announce them? There's a lack of transparency here and, if what you're saying is correct, and every time there's been asset server problems is because of a rolling update etc. then surely it's in their interests to announce them, so we know it might happen. Or am I just being stupid there, expecting to be kept informed of actions that may lead to performance issues?

From: Ice Brodie
Second Life is the largest persistant world simulation on Earth, it's server cluster, known as Agni, is over 1200 servers strong, and growing, while I do not know if they have, or are willing to... run the benchmarks for supercluster status (as the supercluster benchmark takes hours LL doesn't want to shut SL down for) they are a supercluster on the top 500 supercluster list.

The world simulation is also the only persistant world simulation offering building tools that are able to apply world changes live... and one of less than 30 virtual worlds willing to give the public access. (they are also the only one of these virtual worlds to not refere to itself as a game)


Sorry, not quite sure what the point of those last 2 paragraphs are, apart from "ooh, wow, isn't it great"??!?!? What bearing do these have on what we've been discussing?
_____________________
squall Murakami
Burning SOMETHING
Join date: 5 Sep 2005
Posts: 84
11-06-2005 01:43
asset server = kick butt server
-therefore-
Gnomes = Awesome

plus the 1200 server thing, how many servers do you think theyre'll be if they ever incorperate a visitable full scale moon... poor land server thingy

P.S. hmmmmm a terrabyte of servers... i wonder how many rooms that is??
Ice Brodie
Head of Neo Mobius
Join date: 28 May 2004
Posts: 434
11-06-2005 01:45
Another aspect, population, there are roughly 1,000 to 3,000 users online at any one time... of those... about 1/4 are complaining without specifying what is wrong. The QA team generally can't use bug reports that don't actually describe a problem in a way they can diagnose and reproduce. They are also a staff of 4 people. The total ammount of Linden Lab employees who are directly related with SecondLife's technical function are about 60, including Liasions, who serve as first contact for problems.

Personally I'd rather have LL employees fixing the problems than saying "Server patch" every other day until the system reaches 1.x.7 or 1.x.8.

the last 2 paragraphs are relevent to the scale of the environment we're talking about.
_____________________
Moopf Murray
Moopfmerising
Join date: 7 Jan 2004
Posts: 2,448
11-06-2005 01:50
From: Ice Brodie
Another aspect, population, there are roughly 1,000 to 3,000 users online at any one time... of those... about 1/4 are complaining without specifying what is wrong. The QA team generally can't use bug reports that don't actually describe a problem in a way they can diagnose and reproduce. They are also a staff of 4 people. The total ammount of Linden Lab employees who are directly related with SecondLife's technical function are about 60, including Liasions, who serve as first contact for problems.

Personally I'd rather have LL employees fixing the problems than saying "Server patch" every other day until the system reaches 1.x.7 or 1.x.8.

the last 2 paragraphs are relevent to the scale of the environment we're talking about.


Ah OK, I'm just going to leave this here as I'm starting to realise that you're just making excuses for LL now.

I mean, like announcing a server patch actually takes that much time. Sheesh. :rolleyes:

We can agree on one thing: I'd rather see Linden Lab fix problems. Does your inside information give us any idea of when that might start to happen?
_____________________
Ice Brodie
Head of Neo Mobius
Join date: 28 May 2004
Posts: 434
11-06-2005 02:28
Well, they've fixed a few thousand bugs already, the changelog doesn't even record them all.

I should know... I'm listed in the About Second Life dialog for my help with hunting 1.7 (and 1.6 before it) bugs in preview.

And I did bug report the fact that my name is listed twice.
_____________________
Moopf Murray
Moopfmerising
Join date: 7 Jan 2004
Posts: 2,448
11-06-2005 03:49
From: Ice Brodie
Well, they've fixed a few thousand bugs already, the changelog doesn't even record them all.

I should know... I'm listed in the About Second Life dialog for my help with hunting 1.7 (and 1.6 before it) bugs in preview.

And I did bug report the fact that my name is listed twice.


If they're doing such a great job on fixing bugs then why does the quality of my Second Life experience continually degrade over time? Something can't be correct there. Are they fixing the wrong (or easy) bugs, and leaving the more difficult, or right ones? That might explain why my overall experience in-world degrades.
_____________________
Eggy Lippmann
Wiktator
Join date: 1 May 2003
Posts: 7,939
11-06-2005 04:01
We do not have to report bugs in a way that is easy to diagnose or reproduce. Most of us are non technical people. If LL can't accept the fact that they are working with non technical people then they should remove the bug reporting tool :P
crucial Armitage
Clothing Designer
Join date: 30 Aug 2004
Posts: 838
11-06-2005 05:05
From: Eggy Lippmann
We do not have to report bugs in a way that is easy to diagnose or reproduce. Most of us are non technical people. If LL can't accept the fact that they are working with non technical people then they should remove the bug reporting tool :P


iggy it is not necessary to provide technical data for your bug report. the best thing to do with a bug report is give clear and concise detail of what your experiencing.

the problem with some bug reports i would imagin is that people are reporting things as

i cant move or i cant do this that or the other thing with out going any further.

so please bug report but give as much info in the report as you can as it pertains to said bug.
Khamon Fate
fategardens.net
Join date: 21 Nov 2003
Posts: 4,177
11-06-2005 06:55
From: Eggy Lippmann
We do not have to report bugs in a way that is easy to diagnose or reproduce. Most of us are non technical people. If LL can't accept the fact that they are working with non technical people then they should remove the bug reporting tool :P

It's a self-imposed hell. I phrased it somewhat differently in the middle of this page yesterday.
_____________________
Visit the Fate Gardens Website @ fategardens.net
DogSpot Boxer
vortex thruster
Join date: 23 Aug 2005
Posts: 671
11-06-2005 07:02
From: Ice Brodie
Well, they've fixed a few thousand bugs already, the changelog doesn't even record them all.


Which means they really don't know what they fixed or what sources were changed.


From: someone
I should know... I'm listed in the About Second Life dialog for my help with hunting 1.7 (and 1.6 before it) bugs in preview.

And I did bug report the fact that my name is listed twice.


I don't doubt that you know a lot about what's happening inside SL, but in reality, the only people who can speak with authority are LL employees.
_____________________
Dogspot Boxer
Charter Member Of The Socially Inept Club

Our Motto:

We may be inept, but at least we're social
Enabran Templar
Capitalist Pig
Join date: 26 Aug 2004
Posts: 4,506
11-06-2005 08:05
Well, this thread was a big disappointment. Not at all what I was hoping for. :(
_____________________
From: Hiro Pendragon
Furthermore, as Second Life goes to the Metaverse, and this becomes an open platform, Linden Lab risks lawsuit in court and [attachment culling] will, I repeat WILL be reverse in court.


Second Life Forums: Who needs Reason when you can use bold tags?
Enabran Templar
Capitalist Pig
Join date: 26 Aug 2004
Posts: 4,506
11-06-2005 09:52
From: Julianna Pennyfeather
*************************
this message was posted by jesika linden who obviously is out of touch as it is no surprise that the asset server is bad. this has been a problem since day one of the creation of SL jesika wake up and smell the roses. it is the most over worked machine in the SL system and understandably so. it is amazing it even works with what it has to deal with.


By the way,

Why is it you dislike Jeska so much? I can't remember.
_____________________
From: Hiro Pendragon
Furthermore, as Second Life goes to the Metaverse, and this becomes an open platform, Linden Lab risks lawsuit in court and [attachment culling] will, I repeat WILL be reverse in court.


Second Life Forums: Who needs Reason when you can use bold tags?
1 2