Broken money balances

Some residents have had L$ incorrectly added or deducted from their accounts. I have a list of those affected and I'll be fixing the balances accounts tonight. The few people that were inadvertantly pushed into a negative balance will get a bonus for their trouble.

If by 8:00 tomorrow morning (Pacific) you think your account has not been corrected please send an email to [email]andrew@lindenlab.com[/email] and I will investigate.

Sorry about the inconvenience.

Since I'm still in the process of fixing things I thought I should provide an update.

Last night I repaired the handful of accounts that had accidentally been pushed into negative balances. I executed this slow manual endeavor because I wanted to be careful and I wanted to get a better feel for the damage.

When I started to get bleary eyed in the wee hours of the morning I figured no one wanted me to be messing with their balances while I was half awake -- so I took a nap (the data would all still be there in the morning since it had been backed up off to the side). Now I'm back on the job -- onward into the next year!

Here are the details:

As most of you know from looking at your accounting records through the SL user interface, we log information about monetary transactions including: time, to, from, amount, and both ending balances. We also keep each resident's current balance in another field of the database.

What we discovered was that some of the ending balances of transactions didn't agree with the expected value based on the transacted amount and the ending balance of the previous transaction. That is, either (a) some real transactions were applied to balances but were not logged, or (b) some bogus transactions were applied to the balance and not logged. This may have been related to database woes, it isn't clear yet. Right now only the timing suggests some correlation.

So, this turns out to be a good opportunity for me to bring in the new year by beefing up on some python and mysql skills. I've got a book for each on my desk and am whipping up some scripts to compute all of the missing transactions so I can look for matched pairs and some kind of pattern.

If they aren't in the database, they never happened right? Don't worry, if I'm in doubt, I'll pick the generous option and let you keep the transaction or reimburse you what you lost -- the average value of the missing transactions are small: about L$20, and there are only about 450 of them.

I'm not quite done. I got sidetracked for a few hours helping to get 1.5.12 built, tested and released and then celebrated in world for another two hours.

I've finished the part that crawls the a list of transactions and identifes the discrepancies, but now I need to write the framework that builds the lists and then summarizes the results. I'll complete that over the weekend from home.

Happy New Year!

I've finally been able to compute what everyone's account balances are supposed to be.

Here is an outline of what happened. As I mentioned earlier we keep a list of transactions in the database with ending balances for both parties. By checking the differences in ending balances for each resident's list of transactions I was able to find places where there was evidence of missing transactions.

The ending balance of each transactions is obtained by querying a totally separate table of balances in the database. It turns out that when several transactions are recorded nearly simultaneously it is possible for two or more transactions to "collide" in the sense that their timestamp, amount, and ending balances don't line up, however by the time all of the transactions complete the net discrepancy between the total amount and last ending balance is zero.

So why is that important? Because when we realized that we had lost some transactions we tried to fix them by doing a quick analysis of a period of transactions that were in a mirrored copy of the database. Since the database was a copy, and was lagging behind the real database... and the cutoff time in the mirrored copy was at a moment when the original database had been under heavy load which means that there was a larger than usual number of temporary transaction discrepancies that had not yet been "completed".

So the quick fix actually made things worse -- it reimbursed some people who didn't need to be paid and debited some who didn't need to be fined.

So over the weekend I wrote some code to scan everybody's transactions from the 24th to the 31st and genereate a list of corrections. The results I split up into two categories: give_money and take_money.

I ran the give_money transactions this morning so some of you may have noticed some adjustments arrive in your balance.

Since most of the take_money corrections were necessary because of our mistakes I decided to let everybody on that list to keep what they've got.

I haven't yet handed out compensation for those we pushed into the negative but I'll be doing that later this week.

If you think I totally botched your balance you can send me email at [email]andrew@lindenlab.com[/email], however I'm pretty confident that I got things right this time. Also, I'm going to run another scan later this week to sweep for anything I missed.

Sorry this took so long.