Town Hall Transcripts with cross talk, as many out loud comments were answered due to a low attendence allowing much less formality. It will also provide insight to some wierd statements that were not contexualized by the comments/questions that inspired them.
Kudos for one of the best town halls I have attended (note bolded and underlined points are relevant to this thread, some believe)
Jeska Linden: Hello everyone! Welcome to the Technical Town Hall with Ian Linden, director of operations. Thanks for coming out and Happy Halloween!

Jeska Linden: First some guidelines for the discussion. We will be focusing this Town Hall on questions about technical issues, please don't submit off-topic questions, as they will be skipped over.
Nexus Nash: hey! no pking me while i'm afk!
Jeska Linden: Ian is going to open with a brief introduction and then we will go into an open QA. Questions can be sent to me (Jeska Linden) in the form of an IM.
Ian Linden: I am?
David Valentino: lol
Jeska Linden: Also, for those who have joined us here, I'd like to ask you to please hold down the chat and offers of friendship until after the meeting as it can be distracting.
Cutter Rubio: lol
Jeska Linden: Ian, take it away.
Ian Linden: uh.. all you concerns will be fixed in, uh, 1.8, I, uh, promise
Ian Linden:

David Valentino: hehe
Ian Linden: actaully, since I know there are alot of performance questions
Cutter Rubio: He means 1.8.1(53)
Ian Linden: I think everyone should read the docs we've posted
Ian Linden: link
Ian Linden:
http://secondlife.com/tiki/tiki-index.php?page=How+do+I+read+the+sim+performance+stats%3FIan Linden: not very in-depth as yet but they'll grow
Ian Linden: and it's a good baseline
Ian Linden: err, that's all I've got, I'd rather just spend the time on questions
Jeska Linden: Excellent, send them on over to me in IM

Jeska Linden: Nexus Nash: What's the status on estate tools v2 ? (more stats (scripts, objects etc), more control to deed, etc)
Fizik Baskerville: /tickle anshe chung
Ian Linden: forthcoming, but not immediately
Ian Linden: I can't give a date, unfortunately
Ian Linden: I think really these features will come in several small bits
Ian Linden: instead of one big overhaul
Ian Linden: we're focusing right now on fixing the existing bugs
Ian Linden: and lowering the crash rate
Ian Linden: after that I think we'll be able to start improving those tools
Ian Linden: it's clear that they are key in the long term
Ian Linden: sorry if that sounds like a cop-out, the schedule just isn't firm
Anshe Chung:

Ian Linden: next q?
Jeska Linden: Surreal Farber: Since a private sim represents substantial investment, when will be able to register with LL things such as survivorship and/or designated representative.
Ian Linden: good question, I have no idea
Daniel Linden: I'll take it!
Nexus Nash: LOL
Ian Linden: daniel?
Daniel Linden: We're working on plans as we speak to give you all a MUCH higher level of control...
Daniel Linden: ...administratively, over your estates.
Ferran Brodsky hopes Surreal didn't take out an absurdly large policy
Daniel Linden: Cyn and I will be making the rounds to personally speak with you about these plans...
Daniel Linden: ...and make sure we moving in the direction you want.
Ian Linden: another "coming soon" I guess
Daniel Linden: Once we've finalized the details...we'll move pretty quickly.
paulie Femto: hello, evyone!
Daniel Linden: Stay tuned!
Jeska Linden: Cutter Rubio: What provisions, if any, is LL making for keeping the private sims up to date in terms of hardware performance?
Ian Linden: that I can answer
Dave Eisenberg: oh noes lindunz
Ian Linden: right now we replace old machines on a rough schedule
Ian Linden: the basic plan is three years although we've been shorter than that so far
Ian Linden: ultimately we won't keep class 2 hardware in service forever
Ian Linden: if you're on a class that gets upgraded/replaced
Ian Linden: then you'll be moved up the latest stuff
Ian Linden: this makes the most sense for us from a deployment perspective
Bub Linden: Script run-time error
Bub Linden: Too Many Listens
100: say help for a notecard
Ian Linden: I can't give a timeline for the death of class 2, but hopefully it will be sooner than the 3-year anniversary for those machines
Ian Linden: we've considered offering a way to pay to move up ahead of schedule
Ian Linden: but it doesn't seem like that would make alot of sense
Cutter Rubio: My next question

Ian Linden: given the high cost and relatively modest performance gains
Ian Linden: but feel free to disagree with that

Ian Linden: next?
Jeska Linden: David Valentino: With the new and more accurate script count/readings, is there a target for max number of scripts we should be looking at for optimal sim performance? Say in a 75% prim usage sim?
Ian Linden: hmm
Ian Linden: unfortunately that depends more on what the scripts are doing than how many there are
Ian Linden: clearly whenever we get above 2500 or so things start to slow down
Ian Linden: but nominally when you run a ton of scripts all you'll do is slow down the other scripts
Ian Linden: not the sim
David Valentino: well..we know listeners and such cause lag..but a number for other type scripts that used to be tossed around was 400 on the old counting
Ian Linden: that's the goal anyway
paulie Femto: the script counts seem high. is the reported number reflecting all scripts running on a server (stacked estates) or just the script running on our sim?
Nexus Nash: I've gotten 2500, with good sim prefomance.. it's the type of scripts. not just the number.
Ian Linden: it's just the number for the sim
Ian Linden: the old number, it turns out, failed to count scripts on link children
Ian Linden: oops
Nexus Nash: really?
Dave Eisenberg: oopsie daisy
Nexus Nash: damn.
Ian Linden: so they tell me
Katt Kongo: ao off
Ian Linden: so, clearly we need better accounting/scheduling
Ian Linden: as mentioned in the first question
Ian Linden: next?
Jeska Linden: Nexus Nash: Any reason why TD gets hammered post 1.7 whenever there is the slightless load on the server?
Ian Linden: yeah I'd like to know too
Ian Linden: a
Ian Linden: actually, we're not running havok in lockstep w/ the rest of the sim
Ian Linden: nominally this is more efficient
Ian Linden: but if the sim drops below 45 fps then havok suffers as well
Ian Linden: and vice versa
Ian Linden: I think we're going to find there are efficiency bugs in the current code
Nexus Nash: it seems to get smashed at the smallest load.. lots of spikes
Ian Linden: yeah, it's not as good as it should be
Nexus Nash: and sometimes psikes down for no reason at all
Nexus Nash: as long as you guys know
Nexus Nash:

paulie Femto: I know LL claims no performance degradation between sims on the same server, but can we find out what estates we are stacked with so that we could arrange large events so as not to conflict with stacked neihgbors events?
Ian Linden: I would say, give us some time to patch this release
Ian Linden: paulie: I'd like to put that Q in the queue, jeska ok?
Jeska Linden: Sure
paulie Femto: kk
Jeska Linden: Please send questions in IM form to me

paulie Femto: sorry
Ian Linden: anway, we've done some grid-wide sim stats analysis
Ian Linden: and I think we're zeroing in on some problems
Ian Linden: I'd have to go offline to get more detail
Ian Linden: next q?
Jeska Linden: Mike Getting: Do you allocate faster servers to islands with a higher average load, or do all islands get whatever's current when they're first created?
Ian Linden: no, they get what they got when they were created
Ian Linden: this seems to be the most fair
Ian Linden: esp. since we can't move them between servers while they're running
Ian Linden: yet
Ian Linden: next q?
Jeska Linden: Anshe Chung: In Dreamland we have more than 300 people who own land deeds. People buy and sell those every day, but the process is manual due to lack of tools....
Jeska Linden: When will you recognize this existing market and provide tools that allow us to automate the trading ofdeeds in zoned sims?
Ian Linden: daniel, maybe you can comment on that?
Nexus Nash: go anshe!
Daniel Linden: Yes!
Daniel Linden: Anshe, this issue is in our plans for THIS quarter.
Jeska Linden: When will you recognize this existing market and provide tools that allow us to automate the trading ofdeeds in zoned sims?
Jeska Linden: opps
Anshe Chung: Alrighty

Ian Linden: I think this sort of thing is a clear goal for us
Anshe Chung: Just note: trading. Not just rent.
Daniel Linden: We plan to start creating and rollilng out these tools (and more) as they're complete.
Anshe Chung smiles
Ian Linden: the mainland can't be the only estate where land deeds are automated
Ian Linden: again, look for these in a series of small steps
Ian Linden: I think the "one giant release" model has been shown to have some problems

Jeska Linden: Mike Getting: How do we find out what class we're in?
Ian Linden: yeah, that'd be handy, eh
Ian Linden: tell you what, after this is over, I'll post that to the sim stats wiki page
Daniel Linden: We're going to solidify these type of arrangements and put the tools into Second Life -- it'll be better for owners, renters, vistors....this is a big goal for us NOW.
Ian Linden: right now you have to look at the hostname of the server in the About SL window
Ian Linden: I suppose it would be a good idea to report it directly
Ian Linden: next?
Jeska Linden: paulie Femto: I know LL claims no performance degradation between sims on the same server, but can we find out what estates we are stacked with so that we could arrange large events so as not to conflict with stacked neihgbors events?
Ian Linden: interesting
Ian Linden: you can find out, again, by matching the hostname in About SL
Ian Linden: that tells you, at least, if two are on the same machine
Ian Linden: although it's a bit of a pain
Wayfinder Wishbringer: To match hostname... we'd have to check every sim on the grid. ; )
Ian Linden: yeah, but right now that's pretty much all you've got, unfortunately
Ian Linden: the number of sims stacked on a machine is generally fixed for each class
Ian Linden: at least
Ian Linden: anyway, sorry I can't offer a better method
paulie Femto: thanks.

Ian Linden: I will say, however, that generaly speaking you shouldn't find persistent issues w/ stacked sims
Ian Linden: keeping in mind that ALL class 3s are stacked and everyone knows they're a bit faster than class 2s
Ian Linden: which are NEVER stacked
Ian Linden: however, we do run into trouble with the memory usage sometimes
Ian Linden: and that is a bug
Ian Linden: which we will very much try to fix
Ian Linden: which should reduce the problems that do occur
Ian Linden: next q?
Jeska Linden: Wayfinder Wishbringer: Of similar interest... for months we have asked for multiple-manager access of estate tools in private sims. I LL recognizing this need and planning to provide such?
Ian Linden: yes, see the first question
Wayfinder Wishbringer: K thanks
Ian Linden: I don't think the form these tools will take are yet set in stone
Ian Linden: so we'd love to hear your wishes
Ian Linden: next q?>
Jeska Linden: Vx Shaw: Regarding performance, is there a benefit to replicating the same script, as opposed to similar scripts performing similar functions - for example, if an automated vendor was standardized within my sim?
Ian Linden: not really
Ian Linden: however, inasmuch as vendors are standardize, if you can update them in an automated way
Ian Linden: there are alot of benefits to that
Ian Linden: especially if you are able to optimize the script, etc
Ian Linden: but no implicit benefit to uniformity
Nexus Nash: i'm working on that with ADam

Ian Linden: next?
Jeska Linden: Cutter Rubio: Right now, if we want a new height map (RAW file) applied to the sim, LL has to do it - any plans to allow estate owners to do that themselves? Would this be one of the future tool possibilities?
Ian Linden: daniel, got anything on that one?
Ian Linden: I believe that this is a step we'll *have* to take eventually
Ian Linden: manual processes just don't work as we scale out
Ian Linden: but I'm not sure if that's near or long term
Daniel Linden: We want to automated as many steps of the Estate purchase process as possible...
Nexus Nash: *raises hand* i've done it.... just get the basic template and mod it and send it to support....
Cutter Rubio: I'd like to do it on the fly Nexus

Daniel Linden: ....and uploading land form is also something our internal content team wants to make easier...
Nexus Nash: FYI it doesn't work that well and you'll still need to smooth stuff
Nexus Nash: ya auto would OWN
Daniel Linden: ...at this point, our internal tools aren't good enough to allow what you're asking, but everyone wants to ease this part of the process!
Ian Linden: I can certainly imagine uploading a raw file to a sim and having it load it in on the fly
Cutter Rubio: Great
Ian Linden: but imagination and reality take a while to get together
Ian Linden: next?
Jeska Linden: cua Curie: What is the current timeline for SL upgrades? What is the target timeframe for adding HTML on a prim? Havok2? mono and the new rendering system?
Nexus Nash: I have to get to class, later guys, will someone send me or post this somewhere?
Cyn Linden: Sure
Ian Linden: woah
David Valentino: hehe
Wayfinder Wishbringer:

good questions.

Ian Linden: havok2 is one of our big projects right now
Ian Linden: since we wanted it out for 1.7
Ian Linden: it's still coming along nively
Ian Linden: nicely
Ian Linden: HTML on a prim is into the "next thing" category, so a little farther out
Ian Linden: new rendering system is still way out there, as we've slowed down on that
Ian Linden: to fix problems w/ the current version
Ian Linden: mono is anybody's guess
Ian Linden: we have a working prototype
Ian Linden: but it's going to take a while to hammer out the bugs and get the performance where we want it
Ian Linden: all of these thing, ultimately, we come out when they're done
Ian Linden: we've had poor luck giving out release dates in the past
Ian Linden: so I won't embarass myself by repeating past mistakes
Ian Linden: next?
Jeska Linden: Mulch Ennui: regarding the " memory usage" bug you just mentioned, what are the symptoms of this bug in a stacked server situation?
Ian Linden: I haven't studied it in 1.7 enough to give a good breakdown
Ian Linden: but the main symptom will be a rapid drop in performance for the same number of avs
Ian Linden: and the same general load
Ian Linden: the first thing that happens is, the autosave process (which eats extra memory)
Ian Linden: will start impacting the sim
Ian Linden: so you'll see these hourly drops in framerate which don't last very long
David Valentino: sounds very familiar
Ian Linden: the next thing that happens is, the autosaves from the OTHER sims start doing it
Ian Linden: so then you have multiple drops/hour, on a regular schedule
Ian Linden: the end stage is, all sims on that machine slow down because they're swapping
paulie Femto: wow
Ian Linden: it's not pretty, but it doesn't often get that bad
Ian Linden: I won't deny that there are memory leaks in the sim codepaulie Femto: well, we did ask for realtime weather.

Wayfinder Wishbringer: Isn't that what we were asking in that huge forum war? And LL denied it. Big area of contention there (no sour grapes, but...)... ; )
David Valentino: sounds exactly like what we were talking about
Ian Linden: well, here's the thing
Ian Linden: along with the legitimate cases of this occurence
Anshe Chung: "all sims on that machine".... are there multiple sim per server now?
Mulch Ennui: where have u been anshe
Hiro Queso: lol
Ian Linden: there have been alot of problem blamed on "stacking" that really aren't a result of stacking
Surreal Farber laughs
Ian Linden: so it probably sounds a bit contradictory in the end
Wayfinder Wishbringer: Agreed Ian, but there *are* problems... yes?
Ian Linden: since we're respoding to different complains
Ian Linden: and the day when there are no problems, I can retire
Wayfinder Wishbringer: LOL
Vague Speculaas: is this a problem that griefers could utilize?
Ian Linden: I doubt it
Hiro Queso: lol they wont be even able to move in the sims

Ian Linden: I don't think people know the best ways to cause a sim to bloat
Ian Linden: I know I don't
paulie Femto: would increasing memory per sim help any?
Vague Speculaas: we don't need another one of those grid wide take down
Ian Linden: yes it would but there are logistical problems with upgrading 900 machines
Ian Linden: and it wouldn't realy solve the problem
Ian Linden: I'd rather fix the leaks
Ian Linden: honestly if a griefer could consistently cause a sim to bloat up to a huge size quickly
Ian Linden: it would really help our debugging efforts
Cutter Rubio: lol
Ian Linden: as it is, most sims takes a long time to grow, if they do at all
Vague Speculaas: is that a challenge to the griefers?
Vague Speculaas: geez
Wayfinder Wishbringer: LOL
Mulch Ennui: lol
Ian Linden: which is why the problems intensify which we haven't done a patch in several weeks
Ian Linden: with 1.7 there seems little risk of that for a while though
Ian Linden: next?
Wayfinder Wishbringer: ruh roh. shouldn't have said that. LOL
Jeska Linden: Fizik Baskerville: the current island layouts are a total mess, is there any plans to create a more ordered and navigatable system ?
Ian Linden: well we've certainly talked about ways to do that
paulie Femto: floating islands?

Ian Linden: and I think most of that got drowned in more immediate problems like fixing the sim list in the map
Ian Linden: I don't think we arrived at anything really solid
paulie Femto: or flyin islands? woot!
Ian Linden: so we'd like to hear your ideas
paulie Femto: imagin THAT crossin a sim? lol
Ian Linden: but yeah it's not an easy problem
Ian Linden: since geographic location has alot to do with identity
Ian Linden: next?
Jeska Linden: paulie Femto: How is the program for providing offshore "void" sims, attached to estates, coming along?
Surreal Farber raises hand with subquestion
Ian Linden: daniel?
Cyn Linden: I can take that
Cyn Linden: We are ready to do some more beta testing with void sims
Cyn Linden: You can email me for details
Jeska Linden: Nexus Nash: what % of load does the SL software take on the servers?
Ian Linden: all of it
Cutter Rubio: hehehe
Ian Linden: all of the time
Ian Linden: which is why SL uses 220kW

Ian Linden: next?
Jeska Linden: Surreal Farber: heh.. in regards to previous question on sim location... aren't sim adresses dynamic now, so moving doesn't change your LMs?
Ian Linden: if we do it right

paulie Femto: cool!
Ian Linden: but that's not a very smooth process
Ian Linden: if it actually works at all, I'm not really up on that one
Ian Linden: because landmarks are actually still referencing a location
Ian Linden: when a sim moves we have to update them
Ian Linden: which isn't very efficient
Ian Linden: so yes/no
Ian Linden: nexT?
Jeska Linden: We've got time for one or two more questions from the queue.
Jeska Linden: Nyteshade Vesperia: Are there plans to allow people to run sims using their own hardware?
Ian Linden: not in the near future
Fly Speed Booster whispers: Fly Speed Boost Activated!
Ian Linden: long term there are various business scenarios in which that could happen
Ian Linden: but there are a TON of issues with that
paulie Femto: long term it woul dhave to happen wouldnt it?
Ian Linden: not least of which is the implicit DRM embodied by the permissions system
Ian Linden: which wouldn't work in that scenario
Ian Linden: paulie: I dunno, our server building has 50,000 square feet of empty space
Ian Linden: I can fit alot of sims in there
paulie Femto: doh!
Ian Linden: next?
Jeska Linden: Anshe Chung: When you "sell" us so-called "land" or "sims", do we obtain something that maps to one certain value of computing resource that improve as technology improves and hardware price drop...
Jeska Linden: ...or did we buy into something that continuously devalues through technology improvement. Will you pass on hardware improvement to land owners? Or will our "land" be devalued? Or will you use advances to just decrease your costs and improve profi
Jeska Linden: margins.... I paid 1250$ for one sim. Does this mean that in 2 years I still own 1250$ worth of simulation power?
Ian Linden: as stated before, we will upgrade old server, so in theory if you keep a sim for 10 years
Ian Linden: you won't end up w/ a 10-year-old sim
Ferran Brodsky: very kurzwiel question

Ian Linden: I'm not eager to comment on what our pricing will be in the future
Ian Linden: since we don't have any real plans
Anshe Chung: Yes, but will that sim be as powerful as what you can buyq for 1250$ in 10 years?
paulie Femto: in 2012, SL will merge with "reality" anyway, so the q will be moot.

Ian Linden: but i'm sure things will change
Ian Linden: I like paulie's answer
Ferran Brodsky: Im hoping some of our 200 a month is going towards future upgrades
Anshe Chung: Or will in 10 years every SL user have 100 sims?
Ian Linden: I think the question is, are paying less for our servers while charging residents the same amount
paulie Femto: itll be the SLingularity
Surreal Farber reserves her vat of goo
Ian Linden: our per-cpu purchase cost goes up and down
Ian Linden: but stayes pretty stable
Ian Linden: note that each class of sim is a bit faster than the last
Ian Linden: 5 years from now, who knows?
Ian Linden: but note that mainland sims start at $1000
Anshe Chung: kk, so maybe one day sims sell for 12.5$?
Ian Linden: I would expect to continue to pay a premium for private islands
Ian Linden: we could sell for $1 if actually putting the thing online were free
Ian Linden: which it isn't right now
Ian Linden: even if the hardware were free
Ian Linden: I guess, long term, expect that we'll try to keep the prices fair
Ian Linden: we're not interested in gouging our customers
Mulch Ennui: i recently read an article stating hte processor power had reached the cieling. that now dual processors were the only way to improve performance, which makes it difficult to share servers as the demands of the platform raise
Anshe Chung: Right, but hardware become cheaper. So maybe in 10 years you could sell sims with 15000 prims for 12.5 US$. Or you could upgrade all sims to 1.5 mio prims and sell sims for 1250$
Anshe Chung: If you upgrade that way we win
Ian Linden: it could happen
Anshe Chung: If you just let price drop we loose
Ian Linden: that latter is probably more likely
Ian Linden: ultimately I personally think we're all better off if owning land in SL is a real investment
Ian Linden: and if the price remains stable
Anshe Chung nods
Wayfinder Wishbringer: Agreed
Ian Linden: next?
Jeska Linden: Ok, last question -- Wayfinder Wishbringer: Jeska, I think a major question that is underlying here is why did LL decide to stack servers without informing/asking the clients who pay for those servers? Wouldn't an inexpensive dual-core 1 sim/1 server ha
Wayfinder Wishbringer: ... have been a wiser investment. LOL
Jeska Linden: (thanks)
Ian Linden: well the main reason we did that was to save space and electricity
Wayfinder Wishbringer: At the potential expense of sim functionality? (with all respect)
Ian Linden: well, again, we've been stacking for a long time
Ian Linden: and I think generally the results are good
Ian Linden: because no one thinks about the other case
Ian Linden: when you have 4 sims one big machine, if 2 sims are small and light, there's more free room for 2 heavy ones before there are any problems
Ian Linden: likewise they all can share their various caches
Ian Linden: and are quicker to deploy updates to
Ian Linden: so I think that stacking really is a net positive
Ian Linden: and we're quite sure that the performance is good, as long as the sims aren't too bloated
Ian Linden: and they have to get pretty darn bloated before things start slowing down
Ian Linden: so I think what we all really want is for us to fix our memory leaks
Ian Linden: I will say, BTW, that this wasn't an obvious outcome
Ian Linden: when we first tested 4 sims on 2 dual-core machines it was a disaster
Ian Linden: the performance was terrible and we never would have put that in production
Ian Linden: but we were able to resolve the issue by moving to a 64-bit kernel
Ian Linden: and enabling NUMA
Ian Linden: which brought the performance back
Ian Linden: so we did consider the results carefully
paulie Femto: yay for the National Underwater and Marine Agency!
Ian Linden: ok, I guess that's all the time we have but I can hang around here for a few more minutes..
Mulch Ennui: lmao
David Valentino: lol
(continued next post, informal chat very informative)