11-18-2003 15:02
So, I'm sure you've all noticed that we're still offline. In short, what was originally a seemingly routine maintanence issue has transformed into a long, drawn out, fairly painful experience.

Effectively, the server that stores all of the user content (things like textures, sounds, etc), lost a disk Friday. This, by itself, is not a big deal. However, the replacement disk was also faulty, and caused the asset server to crash. This happened on Friday morning, and was the cause of the emergency restart that some of you may have experienced.

On Monday, the asset server started logging small filesystem errors. The reason for the extended downtime this morning was to allow time to both repair the asset server hardware, and to run filesystem repairs. The server hardware repair went smoothly; the filesystem repair is simply taking much longer than expected. You guys make a lot of content. :)

The server is currently running at full tilt through the final filesystem pass; this is going to take a while, but it's well on it's way.

Sorry about this.