Bringing reliability to the item purchases & the backend database

Clearly, we are having realibility issues with the backend database (not the asset server).

The problem being that in order to have a high read / write transactional database you need to have either have Oracle on a mainframe or a clustered multi tier application server, the likes of which BEA and IBM have developed.

If there is another way, I'd like to hear it.

Here is my proposal for clustering the login / main database:

Basically, replace the current application server, whatever it might be, with a clustered EJB Weblogic (or websphere) 9.0 application server.

Advantages

- State replication via replica aware stubs
- Failover when one ejb server goes down
- Load balancing (remove load from database to EJB servers)

I've done this before and it works well. I have done it with 2-3K people all logging in at the same time, however it was a mostly read scenario. I've run a production environment with about 1000 peak usage in high update and pf course I have load tested 10s of thousands doing high updates as well.

Weblogic 9.0 has come a long way. I recommend checking out http://e-docs.bea.com/wls/docs90/cluster/overview.html. At the very least you'll get some good insights on how to tackle your database problems.

And yes, transactions have ACID properties across cluster

I have a diagram below. I'd like to hear from anyone who is familar with how this may or may not work. Please comment!

From: http://e-docs.bea.com/wls/docs90/cluster/overview.html

--------

What Are the Benefits of Clustering?

A WebLogic Server cluster provides these benefits:

* Scalability

The capacity of an application deployed on a WebLogic Server cluster can be increased dynamically to meet demand. You can add server instances to a cluster without interruption of service—the application continues to run without impact to clients and end users.

* High-Availability

In a WebLogic Server cluster, application processing can continue when a server instance fails. You "cluster" application components by deploying them on multiple server instances in the cluster—so, if a server instance on which a component is running fails, another server instance on which that component is deployed can continue application processing.

The choice to cluster WebLogic Server instances is transparent to application developers and clients. However, understanding the technical infrastructure that enables clustering will help programmers and administrators maximize the scalability and availability of their applications.


What Are the Key Capabilities of a Cluster?

This section defines, in non-technical terms, the key clustering capabilities that enable scalability and high availability.

* Application Failover

Simply put, failover means that when an application component (typically referred to as an "object" in the following sections) doing a particular "job"—some set of processing tasks—becomes unavailable for any reason, a copy of the failed object finishes the job.

For the new object to be able to take over for the failed object:

o There must be a copy of the failed object available to take over the job.
o There must be information, available to other objects and the program that manages failover, defining the location and operational status of all objects—so that it can be determined that the first object failed before finishing its job.
o There must be information, available to other objects and the program that manages failover, about the progress of jobs in process—so that an object taking over an interrupted job knows how much of the job was completed before the first object failed, for example, what data has been changed, and what steps in the process were completed.

WebLogic Server uses standards-based communication techniques and facilities—multicast, IP sockets, and the Java Naming and Directory Interface (JNDI)—to share and maintain information about the availability of objects in a cluster. These techniques allow WebLogic Server to determine that an object stopped before finishing its job, and where there is a copy of the object to complete the job that was interrupted.

Information about what has been done on a job is called state. WebLogic Server maintains information about state using techniques called session replication and replica-aware stubs. When a particular object unexpectedly stops doing its job, replication techniques enable a copy of the object pick up where the failed object stopped, and finish the job.

* Site Failover
* Server Migration
* WebLogic Server 9.0 supports automatic and manual migration of a clustered server instance from one machine to another. A Managed Server that can be migrated is referred to as a migratable server. This feature is designed for environments with requirements for high availability. The server migration capability is useful for
o Ensuring uninterrupted availability of singleton services—services that must run on only a single server instance at any given time, such as JMS and the JTA transaction recovery system, when the hosting server instance fails. A Managed Server configured for automatic migration will be automatically migrated to another machine in the even of failure.
o Easing the process of relocating a Managed Server, and all the services it hosts, as part of a planned system administration process. An administrator can initiate the migration of a Managed Server from the Administration Console or command line.

The server migration process relocates a Managed Server in its entirety—including IP addresses and hosted applications—to on of a predefined set of available host machines.

* Load Balancing

Load balancing is the even distribution of jobs and associated communications across the computing and networking resources in your environment. For load balancing to occur:

o There must be multiple copies of an object that can do a particular job.
o Information about the location and operational status of all objects must be available.

WebLogic Server allows objects to be clustered—deployed on multiple server instances—so that there are alternative objects to do the same job. WebLogic Server shares and maintains the availability and location of deployed objects using multicast, IP sockets, and JNDI.

A detailed discussion of how communications and replication techniques are employed by WebLogic Server is provided in Communications in a Cluster.

LL already researched these a while ago (they posted this into the ops blog) LL has a better solution planned apparently which is going to be more distrobuted.

Edit: I dont know what these plans are exactly, but they are more true to what LL originally planned for the DB server, than the centralised system they ended up with today.

-Adam

Do we read the same blog? Where do they talk about this in the ops blog?

The only thing I've seen is this, which is from Philip:



However, simply chunking into seperate DBs will not solve the problem.

What if i go to buy something and then the buy action fails but the inventory transfer doesn't? Or vice versa? (kinda like what's happening now, folks!)

We need ACID transactions in order to have a reliable ecommerce system.

Hmm, I stand corrected, wasnt posted to the ops blog *trying to think where he read it*, I believe it was hungry linden who made the comment, so, it could have been on IRC.

-Adam

Another possibility and probably one they are looking into and maybe using already...

http://dev.mysql.com/doc/mysql/en/replication.html

Yikes!

As Adam once said, "decentralization of read-write databases is tough" or something like that.

This is the kind of thing you get companies like BEA or IBM in combination with Oracle to do.

Open source mysql is not (and it's unlikely will ever be) ready for this type of challenge.

The complicated works in migrating ACID transactions across a cluster and the logging required for realibility is hugely complex and requires millions of man hours of development and testing.

Maybe you're right and maybe they should go with Big Iron.

I think it would be dope if we could get someone from ops to post in this thread.

Personally, I'm in favour of a new solution which doesnt involve any centalisation at all (even if it's just a centralised cluster).

The big reason for this is: SL needs to be able to grow. Big iron; Oracle while going to scale better than MySQL, wont be able to go an order of magnitude or-two larger than that. If SL is ever to have 30,000 simulataneous users, it's going to need a decentralised solution to this problem.

An example of a potential solution would be following the same path as email: Fix all assets, users, etc with a hostname, point the asset to a server which can deal with it, as part of the name.

For instance, we could have a 'adamzaius@agni', which tells anyone dealing with my avatar to seek agni for any transactions regarding it. This also opens the door for the longer term goal of SL being non-wholly owned by LL, and would bring it a step closer to a true metaverse application.

I suspect it's something like this that LL is starting to implement at the moment; a solution that doesnt need to be revisited every 9 months because SL keeps growing.

-Adam

I've found MySQL replication to be a great solution for my company. Distributed load (if you wish), but more importantly, instant back up.

-Flip

Uh, Blaze, some reading for you:

http://www.mysql.com/customers/

Google? Yahoo? NASA? Those seem like slightly larger challenges than our lil' grid over here from a data standpoint.

Also, an independent benchmarking studying putting MySQL 4 and Oracle 9i fairly close in performance:

http://www.eweek.com/article2/0,3959,293,00.asp

MySQL can be pretty damn high powered, despite the silly name!

-Flip

Yahoo, Google and Nasa use software that I've developed. So?

The question isn't who's using, but how it is being used.

However, I do agree with Adam's point. An architecture more similar to that of the darpa style internet itself would be wise.