Rolling Restart : server version 1.25.3 / 1.25.4

I feel like we are back to last year at this time.... why so many problems with this database... why still after years of database issues have you not fixed this......please shake heads and get NEW database..and stop breaking everyones scripts ......gives you very dirty look......... why do we only get excuses for this database and oh I'm sorry we are working on it, over 2 years this has been a major issue why ohhhhhhh why is it still an issue it should not take 2 years to fix this.

I can't login either...

Hey Prospero,

Would you enlighten us a little on why we are having an increase in grid instability? I wanna blame it 1.24.10 server deploy but there has to be more to it than that.

Thanx

Today we hit the highest concurrancy I've ever seen....just before the poo really hit the fan. I know FJs people are working on improving internal network and database performance.

"Software is inherently complex. Many of the most complicated artifacts ever created by humans consist largely of computer software. Taming inherent complexity, and avoiding needless complexity, is the software engineer's most important technical task." --Steve McConnell, author of "Code Complete"

I wish they'd sync up the status page to itself.

There's a post at 3:09 saying logins are *restricted*.

With nothing saying they've been opened up again.

And one at 3:51 saying logins are *intermittent*.

I'm still trying to get to the bottom of what my problem is.

It appears that my client is being told to connect to the sims on the wrong port.

My log for example:

2009-01-19T06:22:01Z INFO: addRegion: Adding new region (1105:101
2009-01-19T06:22:01Z INFO: addRegion: Host: 216.82.18.222:13002
2009-01-19T06:22:01Z INFO: process_enable_simulator: simulator_enable() Enabling 216.82.18.222:13002 with code 581218717
2009-01-19T06:22:01Z WARNING: addRegion: LLWorld::addRegion exists, but isn't alive


Now, if you do a port scan on 216.82.18.222 on port 13002, you will see that the port is closed and no possible connection can be made.

BUT, port 12043 IS open on that server.

So, to me anyway, it appears that my client is being told to connect on port 13002 when it should be told to connect on port 12043.

Any ideas on this?

I'll supply more logs and information if anyone can help.

zg

Hmmm..... I wonder if there is any performance issues when my client is hammering on the system like this:

1 second of what my log file looks like when I switch sims, teleport, or log in:

2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 216.82.18.222:13002 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 8.2.34.42:12035 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 216.82.18.222:13002 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 8.2.34.42:12035 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 216.82.18.222:13002 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 8.2.34.42:12035 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 216.82.18.222:13002 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 8.2.34.42:12035 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 216.82.18.222:13002 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 8.2.34.42:12035 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 216.82.18.222:13002 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 8.2.34.42:12035 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 216.82.18.222:13002 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 8.2.34.42:12035 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 216.82.18.222:13002 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 8.2.34.42:12035 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 216.82.18.222:13002 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 8.2.34.42:12035 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 216.82.18.222:13002 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 8.2.34.42:12035 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 216.82.18.222:13002 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 8.2.34.42:12035 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 216.82.18.222:13002 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 8.2.34.42:12035 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 216.82.18.222:13002 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 8.2.34.42:12035 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 216.82.18.222:13002 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 8.2.34.42:12035 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 216.82.18.222:13002 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 8.2.34.42:12035 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 216.82.18.222:13002 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 8.2.34.42:12035 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 216.82.18.222:13002 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 8.2.34.42:12035 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 216.82.18.222:13002 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 8.2.34.42:12035 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 216.82.18.222:13002 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 8.2.34.42:12035 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 216.82.18.222:13002 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 8.2.34.42:12035 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 216.82.18.222:13002 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 8.2.34.42:12035 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 216.82.18.222:13002 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 8.2.34.42:12035 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 216.82.18.222:13002 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 8.2.34.42:12035 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 216.82.18.222:13002 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 8.2.34.42:12035 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 216.82.18.222:13002 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 8.2.34.42:12035 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 216.82.18.222:13002 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 8.2.34.42:12035 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 216.82.18.222:13002 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 8.2.34.42:12035 with code 581218717
2009-01-19T06:22:27Z INFO: process_enable_simulator: simulator_enable() Enabling 216.82.18.222:13002 with code 581218717
2009-01-19T06:22:27Z WARNING: setSeedCapability: Ignoring duplicate seed capability
2009-01-19T06:22:27Z WARNING: setSeedCapability: Ignoring duplicate seed capability


LOL... you got to see what the next second looks like!

009-01-19T06:22:51Z INFO: dumpResendCountAndReset: Circuit: 8.2.34.42:12035 resent 11 packets
2009-01-19T06:22:51Z INFO: dumpResendCountAndReset: Circuit: 216.82.18.222:13002 resent 10 packets
2009-01-19T06:22:53Z INFO: idle: Transmitting sessions stats
2009-01-19T06:22:53Z INFO: addToMessage: STAT: Version: 0
2009-01-19T06:22:53Z INFO: addToMessage: STAT: Vertex Buffers Enabled: 1
2009-01-19T06:22:58Z INFO: result: ViewerStatsResponder::result
2009-01-19T06:22:59Z INFO: display_stats: MEMORY: 9 MB
2009-01-19T06:23:12Z INFO: display_stats: FPS: 15.43
2009-01-19T06:23:28Z WARNING: updateMeshTextures: updateMeshTextures: invalid host for object: 2bcad221-7b44-47f9-964f-b04a6c42f69e
2009-01-19T06:23:46Z WARNING: checkCircuitTimeout: LLCircuitData::checkCircuitTimeout for 216.82.17.87:13002 last ping 100.655 seconds ago.
2009-01-19T06:23:46Z WARNING: checkCircuitTimeout: LLCircuitData::checkCircuitTimeout for 216.82.17.87:13002 still dead, dropping.
2009-01-19T06:23:46Z INFO: removeCircuitData: LLCircuit::removeCircuitData for 216.82.17.87:13002
2009-01-19T06:23:47Z WARNING: checkCircuitTimeout: LLCircuitData::checkCircuitTimeout for 63.210.158.4:13000 last ping 100.726 seconds ago.
2009-01-19T06:23:47Z WARNING: checkCircuitTimeout: LLCircuitData::checkCircuitTimeout for 63.210.158.4:13000 still dead, dropping.
2009-01-19T06:23:47Z INFO: removeCircuitData: LLCircuit::removeCircuitData for 63.210.158.4:13000
2009-01-19T06:23:47Z WARNING: checkCircuitTimeout: LLCircuitData::checkCircuitTimeout for 63.210.157.98:13000 last ping 101.053 seconds ago.
2009-01-19T06:23:47Z WARNING: checkCircuitTimeout: LLCircuitData::checkCircuitTimeout for 63.210.157.98:13000 still dead, dropping.
2009-01-19T06:23:47Z INFO: removeCircuitData: LLCircuit::removeCircuitData for 63.210.157.98:13000
2009-01-19T06:23:49Z WARNING: checkCircuitTimeout: LLCircuitData::checkCircuitTimeout for 8.2.35.229:13000 last ping 101.571 seconds ago.
2009-01-19T06:23:49Z WARNING: checkCircuitTimeout: LLCircuitData::checkCircuitTimeout for 8.2.35.229:13000 still dead, dropping.
2009-01-19T06:23:49Z INFO: removeCircuitData: LLCircuit::removeCircuitData for 8.2.35.229:13000
2009-01-19T06:23:50Z WARNING: checkCircuitTimeout: LLCircuitData::checkCircuitTimeout for 63.210.156.84:13001 last ping 100.619 seconds ago.
2009-01-19T06:23:50Z WARNING: checkCircuitTimeout: LLCircuitData::checkCircuitTimeout for 63.210.156.84:13001 still dead, dropping.
2009-01-19T06:23:50Z INFO: removeCircuitData: LLCircuit::removeCircuitData for 63.210.156.84:13001
2009-01-19T06:23:50Z WARNING: checkCircuitTimeout: LLCircuitData::checkCircuitTimeout for 8.2.32.40:13001 last ping 101.066 seconds ago.
2009-01-19T06:23:50Z WARNING: checkCircuitTimeout: LLCircuitData::checkCircuitTimeout for 8.2.32.40:13001 still dead, dropping.
2009-01-19T06:23:50Z INFO: removeCircuitData: LLCircuit::removeCircuitData for 8.2.32.40:13001
2009-01-19T06:23:51Z WARNING: checkCircuitTimeout: LLCircuitData::checkCircuitTimeout for 216.82.22.205:13001 last ping 101.523 seconds ago.
2009-01-19T06:23:51Z WARNING: checkCircuitTimeout: LLCircuitData::checkCircuitTimeout for 216.82.22.205:13001 still dead, dropping.
2009-01-19T06:23:51Z INFO: removeCircuitData: LLCircuit::removeCircuitData for 216.82.22.205:13001
2009-01-19T06:23:54Z WARNING: checkCircuitTimeout: LLCircuitData::checkCircuitTimeout for 216.82.17.90:13001 last ping 104.675 seconds ago.
2009-01-19T06:23:54Z WARNING: checkCircuitTimeout: LLCircuitData::checkCircuitTimeout for 216.82.17.90:13001 still dead, dropping.
2009-01-19T06:23:54Z INFO: removeCircuitData: LLCircuit::removeCircuitData for 216.82.17.90:13001
2009-01-19T06:23:56Z WARNING: updateMeshTextures: updateMeshTextures: invalid host for object: 0145bb3a-f131-4f70-8606-418309dd9f09
2009-01-19T06:24:10Z INFO: parse: LLSDXMLParser::Impl:arse: XML_STATUS_ERROR parsing:
2009-01-19T06:24:10Z WARNING: error: Unexpected HTTP error. status: 499, reason: STATUS_EXPIRED


HELP!!!!

zg

REMINDER EVERYBODY : this thread is about the rolling restart and the new server version. It's not a general support forum, nor is it a general forum for griping about instability. I've just deleted a bunch of messages that are general support related, plus the usual flood we get whenever there are system problems.

Zolan : please contact support with your issues.

I did leave a couple of the instability posts in. One question was, why are rollouts always connected to system instability? The answer is, that's just perception. We're having a lot of system instability right now. It's been bad for the last 6 weeks. It's not caused by the rollout.

As for why : the central database is overloading again. It was bad last April, and we made some changes, and things got better for the rest of the year. (Yes, May was also bad, but that was the asset server, and we also fixed that up.) The central database, however, has come to the limit again. And, yes, we're working on it, and I'm sorry that we can't get it fixed in a day.

The Grid Status page lists a tentative 1.25.4 pilot roil scheduled for Jan 22nd. Is 1.25.3 being rolled back, or is this simply an update to fix a few of the bugs found in 1.25.3 before proceeding with the full roll?

To be honest, none of the bugs in 1.25.3 seemed like complete showstoppers to me. The animation issue was irritating but easily fixable (which I assume is in 1.25.4). Please tell me that this is going to be a direct roll from 1.25.3 to 1.25.4 on those pilot regions, without having to roll back in the interim?

Yes, just an update. We'll redo the same pilot regions, but there will be no backwards in between. (If there had been a backwards, we would probably have done it late last week.)

Who do we complain to?

Who is the linden in charge of network and why are they still employeed? Is it so hard to predict the grid will grow? Do you really need to be taking by suprise every time 500 more bots log onto the system?

It's a been a few weeks of daily fails. Daily messages telling people not to actually do anything or buy anything. Yet, your CFO makes a big happy post about how profits are up and user hours are up....do you SEE THE ISSUE? I'm sure he meant USELESS hours are up.

Go ahead and delete this. You don't seem to have any other skills.

Um, wait until after the 1.25.4 rollout...

Um, because they just hired him?

No, but knowing by how much and in what dimensions is non-trivial. It's not like they can go out and buy another ten crates of "database performance".

lol, yeah they can! XD

Don't say that too loudly, people get really defensive about their showstopper issues.

There used to be an easy page to find the server versions known bugs and updates.... why don't you make this easily accessible from the main page of the secondlifegrid or secondlife site. Why is that? I can't find it anymore....

Here ya go

Because there is code to help reduce the load on the database in 1.25.4, we're going to do the rolling restart this week, before the weekend-- in hopes of making things better over the weekend.

A blog post with the schedule is here:

http://status.secondlifegrid.net/2009/01/20/post454/

We are going backwards? If so, then why?

-Liny

Cool, and if RC6 of the viewer comes out before the weekend and solves the redownload problem, this weekend should be MUCH better than last weekend.

Edit: whoa, a plague of typos. I'm sure that means "1.25.4". Or is this really intended to be a "rofl-out"?

Fixed -- that meant to say 1.25.4.

Considering how much trouble you seem to have been having with the 1.25 release how much faith can we have that this will not present more problems? Is it really a good idea to roll out a release with this kind of track record so quickly?

Prospero dude...
We love ya, guy. Vektor too. And Andrew. and all the folks making this happen.

Hang in there. Almost there...

${random sincere Stuart Smalley affirmation}

That depends...how much testing did you do on the beta grid?

Seems to me 1.25 has been banged around pretty good.
It's only taken so long because Mono got to go first.

I think the trouble has been grid-wide and not specific to 1.25 but rather database related. Roll it out! Let's get on with it! Let's make progress! Good job Prospero and the team!