Improvements to mapping and upgrade to SLurl.com

(reprinted from main SL blog post)

Hi all!

For those wondering generally what I've been up to, I said in earlier transition announcements from CEO to Chairman that I wanted to get back into engineering and the design of Second Life in a more hands-on way. So I am now working with two other great engineers as one of Linden Lab's technical teams. In early December we decided that a good first project to work together on would be improving the performance and experience of using the web-based SL map, and also changing the architecture to be able to handle a much larger number of sims/servers. And, I have to say I'm having a great time getting back into development.

Hopefully, the general direction we are going with the map can help to make place and proximity more meaningful in SL. It should be fun to explore the map: to look at what is near your home, to drill down for more detail, to randomly surf around and jump to places that look interesting from the air. We always felt that this would be a big part of the appeal of SL, but the practical challenge of rendering things far away combined with the weaknesses of the existing web and in-world maps have made this more an aspiration than a reality.

More specifically, we've just released some improvements to the <a href="http://slurl.com/">SL web-based map (http://www.slurl.com)</a> that should be immediately noticeable. First, the map should load faster and be smoother when you use it. Secondly, you can now single-click on any location on the map and teleport directly there. The map is also accurately updated much more rapidly then the former map - so changes to the Second Life Grid or content should now be visible within about 2 days. Finally, the map images themselves look smoother and are more accurate when compared to the actual in-world content, and you can now zoom out and see how big SL has really become!

We're not done with this project! We also intend to update the SL viewer to read its map tiles in the same way as the web map, meaning that you will be able to zoom and move around on the in-world map in a manner similar to the webmap, with much better performance than before. Look for a viewer release candidate next with those changes.

For SL developers, and the more technically interested: We are now serving map tiles directly from <a href="http://aws.amazon.com/s3/">Amazon S3</a>, and have changed the file naming convention to hopefully make it much easier to develop other 3rd party maps based on the tiles we are generating. More documentation on using the webmap API is available <a href="http://wiki.secondlife.com/wiki/Webmap_API">on our wiki</a>. The map generation process is much more scalable than before, and is able to image the entire SL grid of about 30K regions in less than 2 days with just 4 dedicated machines. In general, the architecture approach of generating tiled images on sims into a public repository like S3 seems like the right approach for both performance and openness - it anticipates the ability for individual sim owners/operators in the future to create their own maps (or not make them at all!) and upload them to a common shared filespace on which multiple mapping systems can then depend.

Note: If you have built a map that depends on the old mapping API, <strong>we will leave the old tiles accessible for another few weeks</strong>, but will not be updating them in the future. You will need to switch to reading tiles directly from S3. We will post more information about this changeover <a href="http://wiki.secondlife.com/wiki/Webmap_API">on the wiki</a>.

Excellent news! I did notice the map was upgrading more often, after the nearly two-year freeze. Glad things are going to improve even more!

The map looks great and being able to single click and teleport by scrolling the map is a wonderful development.

Hopefully you'll soon have it updating within two days, my old openspace is still on it and that went before Christmas....

Also minor quibble, please please please get the forums fixed so we can all utilise this feature here.

Hi Philip,

It's interesting to hear what you've been up to and it will be good to have a more up-to-date map.

I attended one of the Metanomics viewings of your Interview the other day where this was mentioned.

There has been a lot of discussion and speculation - in in-world groups, blogs and the forums - since then about your comments on merging the Grids (teen and adult). Whilst I think some residents understand where you were coming from, as you might imagine there is a lot of rumour and misunderstanding as well.

If you have time, perhaps you could expand on your ideas sometime soon?

In the meantime I'll include a link to the Transcript of your interview for anyone who hasn't seen it:
http://www.metanomics.net/transcript011909

Best wishes,

Sylvie

Really impressive speed !

Thank you Philip

OMG, I knew SL had around 30k sims, but today was the first time I zoomed out slurl and could actually see it all!!! Its huge! idk why I never tried it before, just moved a bit and seemed so slowly, but yes now its really smooth and loading faster!

And great news on the mapping timeline too!!! WTG Philip

This is an improvement that I was waiting for! Especially the single-click for any location. Very handy, thank you!

And... now I can finally visit my sim neighbours lol.

I really like the new SLurl maps! Hopefully later on the push-pins will not disappear completely once you close the title box Also it would be great if a SLurl with a photo would display properly - usually the title box does not increase in size to accommodate the photo and any associated message (Firefox, Safari on Mac).

The sim tiles looks great! Aloha

Oh nice! So fast and detailed - I can see my house from there

Will be great when this get into sl - the map as it is now, is way to slow for just crusing around

Great progress so far Phillip. Maybe sometime in the future street view type functionality will be possible but I'm not holding my breath on that. I guess until then I will just have to go in world and stare through the windows if I want to spy on the secret Linden parties.

Hello Philip!

Nice going. I am glad to hear you're getting into the engineering side of things. I wanted to make a post on the threads about the weekend downtime and grid emergency maintenance issues posted by Catherine and Frank, but got too busy at work and did not get a chance to post a few thoughts before the topics closed. Maybe this is a good time to do it. There was an active discussion about MySQL as the principle database technology for the asset cluster, etc., and some forum participants mentioned using PostGres, Oracle or MSSQL as database alternatives. Someone also mentioned switching to shards, but to do that would mean segmenting off sections of the grid, so anyone who is in one grid can't interact directly, except through IMs perhaps, with someone in another shard of the grid, right? There really is no perfect solution out there off-the-shelf, and what is MySQL-based now at Linden Labs is not MySQL the ordinary person can download off the web, I am sure

I came across a blog post by a guy who is really familiar with some other opensource database technologies such as Amazon S3 which you used for the map improvements; but also along with that there are things like Google's BigTable and derivations such as HBase, if hash keys are an option, which it seems that since everything is mapped into UUIDs, perhaps it could be. The Anti-RDMS article talks about a couple other database technologies that seem to have promise as a longer-term upgrade from MySQL, in particular Project Voldemort and Scalaris. Scalaris looks really hot, except right now it's all memory resident. But Scalaris is highly scalable, fault-tolerant, and has extremely low latency for database transactions. Another issue might be that the language sets used for these two are Java and Erlang, and if you don't have those type of specialists on board, that increases the challenge.

I'll post a couple of these URLs I came across, perhaps for your interest, members of your team, and the SL community at large.

http://www.metabrew.com/article/anti-rdbms-a-list-of-distributed-key-value-stores
http://www.highscalability.com

Take Care,
Laska

WOW! Thats great and unbelievable fast, the map looks so crisp, that me wants TP to explore even more places as i could.
/me scrolls since hours over this nice map, so many well landscaped sims. This new map shows it much better as the old, where everything looked a bit blurry.
Thanks cool work!

Maybe a location search line would be cool to have in this new map.
Jira ticket for the searchy line

Ciao
Dil

Good stuff! Can't wait to see more improvements! Especially to inworld maps, as I use them when flying aircraft - or try to. Excellent

The new map is great, I agree.

But like mentioned above, Phillip, we're very much wondering about your comments in the interview from Monday about grids merging.

More info on this would be fabulous. Real info, besides just "it's in discussion."

Please. I know that's off-topic, but the residents really want to know, as evidenced by the thread in Resident Answers.

Philip, you are hot. Period.

Is my picture "Surfer Girl", the one you bought by Arlin, still on your wall?

If I fly to San Francisco, can i have a kiss?

Thanks for this. I recently replaced the city map prims at the Nova Albion Infohub with new prims that automatically update the top face with a map of each city region. Originally they didn't load very well, but recently there's been a HUGE improvement. Now I know why. Very nice.

Does this mean that texture based tiles of the map will no longer be updated once the client is setup to pull from amazon directly?

If so, there are many commercial products inworld that depend on these textures being available inworld without using parcel media. I run a free service that gives the current texture uuid for a region by name, that many products use.
If the textures are no longer updated by LL automatically, this will cause all of those products to become useless. Attempting to duplicate the functionality of the existing map service would mean an operating cost of over L$300,000 every time the world map images are updated... which according to the estimate right now, would be every 2 days.

I hope that the current world map UUID functionality is maintained, using the new image generator system.

Hi! I'm one of the engineers working on this with Philip.

@Ciaran Laval:
Your OpenSpace is still showing because I haven't finished the "garbage collector" that takes down images for regions that have gone offline or moved. It's not hard, I just haven't gotten to it yet.

Once the code is finalized, there will be a "decay period" for offline regions. If a region is disabled, the tile will stay up for several days (in case it comes back or something else has gone wrong) before the tile disappears. That several-day period will be longer than the time it takes to update existing regions or put new regions on the map.

James Linden

WOW this is really impressive and a fast load.

Phil, i have a new challenge for you, faster texture load since viewer 1.21 and 1.22 RC's

VWR-8503

I can even dream that number now, however it's more of a nightmare
Hope to wake up from that one soon since emmm only last Oct

Thanks for all the kind words, folks! Very motivating. We'll work as quickly as we can toward a viewer that can perform as well as the webmap, we are partway there already.

Laskaliscious - thx for the links and thoughts on scaling.

Briana - you are more than welcome to visit! Would be an honor to meet a famous 'old' SL'er.

Lex - not sure I understand your app well enough to know whether it would be affected. Can you send me email with more details? [email]philip@lindenlab.com[/email]

Thanks for the reply James.

Thanks Philip, that is also great to hear about viewer speed

Heya Philip! & James!

TY!!

SLurl Link Says:
January 22nd, 2009 at 6:42 PM
Looks good! Nice terrain textures.

There is however an odd behavior in the way the map renders prim texture colors at the moment. The same texture being rendered to a different color tone (probably based on the Z position shift) rather than a unified average tint like before.

e.g. http://slurl.com/secondlife/Hyboria/
The logo colors should likely match the outside border instead of lighter and darker prims in the middle.

It seems as if the “depth” behavior used for terrain and water should not apply to prim colors/textures somehow…

One thing I'd like to see brought back to the in-world Map is classified ad markers. I manage a large number of covenant controlled regions, and being able to see when classified ads pop up was a great way to understand traffic patterns better, as well as catch illicit behavior.