The new All search in NOT broken

In various threads, some people have observed that the new All search doesn't work as well as they think it should. I've just read one person even saying that it's broken. It's not broken, but it isn't yet working as well as it should, and it's not LL's fault - we are resonsible to make it work well. So let me explain how it works, and why WE need to make it work well. (I'll just talk about places in the results.)

The engine ranks places according to relevancy; i.e. how relevant a place is to the phrase that the searcher typed into the searchbox. A place's relevancy is determined from text sources about the place.

LL creates a webpage (HTML) for each parcel of land. That's the main text source. That's the page that is parsed by the search system, and it is used to determine relevancy to the searchterm.

Different parts of the page score differently for rankings. THE most important part of the page is the Title tag (the content of that tag isn't isn't visible in the viewable part of the browser, but it exists in the HTML). The words in the Title tag score heavily for ranking purposes. The words in the Title tag are the words of a parcel's name.

The Title is repeated in the viewable part of the page in a heading tag - more weight than just ordinary text on the page - so the ranking weight that the parcel's name carries is increased.

The page's Description tag is also not seen in the browser, but it counts as well. The words in the Description tag are the words in a parcel's description - which are also repeated in the viewable part.

Also on the page is a list of objects on the parcel that are set to show in search. This is body text, and carries less weight than the Title, but it does score ranking points.

The Title tag (parcel name) is THE most important part of the page for rankings, but there is another very important ranking factor - inbound links (IBLs).

IBLs are counted from Picks in people's profiles (any number of links), the group (1 link), the Popular Places pages (1 - 12 links), and maybe one or two other things that escape me for the moment. IBLs are THE most important ranking factor of all. In Google they are on a par with page Titles (and the All search is a Google engine), but there can be many IBLs and only 1 page Title, so IBLs far outweigh anything else.

The text of links, e.g. people's Picks, is attributed to the page that they point to - the parcel's page - and they weigh more than mere text on the page. Where does the Picks text usually come from? The land's name and description again! You see how powerful the land's name is? And the description is pretty darned powerful too if plenty of people have the place in their Picks.

That's more or less how the new All search ranks places, but how many places have optimised their land's name and descriptions for the All search. It's the lack of doing that that makes the results not as good as they could be, but it will improve as more and more people realise that they need to do it - that WE need to do it.

If the land name says "Phil's shop - terrific furniture at low prices", it will be very low down in the 'low prim furniture' rankings because so many other shops will have "low prim furniture" in the land's name. It doesn't matter if Phil's Shop contains the best furniture is SL (which it does ) because, unless the land's name and description is optimised for 'low prim furniture', it will languish low down in the rankings, and Phil may be tempted to think that the All search doesn't work well, or is broken, because his shop isn't in sight and it should be.

The All search is new, and for it to work to its potential, it needs people to optimise for it. It isn't like the Web - we have very little control over the page content - we can't organise it for rankings as we can with webpages. But we do have control of the most important parts, and we need to use it to make the All search work well for users. All it has to work with is what we give it. It isn't broken - it just hasn't matured yet.

Because your post was so informative, I'm going to check out your furniture store today.

This is a great message with tons of cool info, Phil.

For those who need to know the ways in which new search (not just All, but other tabs as well) IS broken, you might start with the Jira meta-issue on new search, which points to the specific bugs:

https://jira.secondlife.com/browse/SVC-1011

Once again Phil - many thanks for sharing this info so openly!

Please do Raymond - and don't miss the Chesterfields

Thank you for that Nika. I've had a look and the problems there are more of technical nature. I was trying to address something that, on the whole, I don't think people have realised yet. One of the comments in that jira page:-

That's the sort of thing I was addressing, and it was written by someone who posts here. When people learn that they have to optimise their land's name and description, then difficulties like that will gradually disppear. It will take time, of course, but the relvancy side of the All search isn't broken - it's just doing the best it can with the limited amount of data, and the data itself isn't yet helping.

And this is where the system breaks - the *default* text for the Pick item comes from the lands name and description *but* this is editable by the end user. Sometimes you want to add another person as a pick - you do this by adding a pick for the land you happen to be on, then change the picture, name and text for the person you are interested in.

Result - the new search picks up totally irrelevant results since it picks up relevance from a description someone has added about a person (or some other topic) and then just happened to be on a particular plot of land.

What is more disturbing is if as you claim the text in Picks has a greater relevance that the text in the actual description.

It also opens up mining and griefing opportunities - mining the system is simple - create 100 free alts (which you can now do without any cost whatsoever), add a pick for your land to each alt and then never use them again (unlike camping this doesn't have any impact on the number of avatars online just the number of registered avatars). I suspect we will find people selling such a service.

To grief, just create 100 alts, and for each alt add a pick for the land you want to throw off the search engine and change the decription to totally irrelevant search terms.

Matthew

I think when someone says something 'does not work' doesnt mean exactly that it is 'broken'. It is doing exactly what IT is designed to do, but that doesnt mean it is the most affective way to do it. I understand 'how' it works, but I dont think it is an affective way for people to find things. Not everyone is going to understand how a complicated search system works. Newbies are not going to know this. People that dont visit the forums to read posts on how it works are not going to know this, even though this is very educational, its not widely known. You bet your booty that people that will game the system will know how it works, thus taking clear advantage. Until it is refind and more affective, to me personally the new search doesnt work. Again, its not broken, its just unaffective. I will stick with the old one and pay, of course, close attention to the listings as I scoot past the camping, botting, alt-havens, false listings on and on, and try to find the places with what I really want. Accepting the fact that it is never going to be easy to find the true popular places (by individual real people) or the real top designers (loved and purchased from my real individual players). Its just what we have been forced to deal with while people out their play the business game.

eta;
Matthew this is a good point, and why I went and made my picks at specific places that I like. I plan to change them around until I have included all of my real favorite places even if it isnt a pick about them. Most people have no clue that picks now are connected to search and how to work that yet. To create a million alts to fill their picks is just ridiculous, but it sits well with alts and bots n the like to build traffic. By business people, its a good way to twiddle with the system.

Good point about the Picks and the person, Matthew. For a search on the person's name, it will cause a place to be ranked as well. I don't really see that as being a problem though. It can't be used in the way you described - creating alts to outweigh the real Title. All they will do is get the place ranked for other things; they won't cause it's rankings to suffer for the real things. It's known as Googlebombing on the Web.

The thing about link text is that it is a Google engine, and Google have always weighed link text and Titles heavier than page text. They did it to try and avoid the on-page search engine spam that the existing engines of the time had to deal with. They use a particular way (2 seperate indexes) that may not be included in the appliance, but it may be. Either way, it is sure to weigh link text heavier than page body text.

Does anyone know if people TP in from their pick, or from finding something in search does that have any impact at all on relevance? For example, if I search "Low Prim furniture," see Phil's store, and then TP there from the search does that add to relevance for "low prim furniture" at that location?

Not that I've heard Nimue. I would imagine that LL would consider the 12 Popular Places pages as taking it into account - if it ever crossed their mind.

Thanks Phil.

A group only links to the founder, not to a parcel. Every search listed parcel does get 1 link from the region page of the sim it's located in.

I would hazard a guess that about 50% of peoples picks were for people, 30% for places and 20% for other information.

Whilst we could argue of those figures - the fact remains that many picked will add irrelevant search terms to the location rather than improving its ranking for relevant terms.

To improve this LL would need to seperate out favourite places from picks.




We could argue the semantics of whether the new search is broken or just not working but this is the crux of the matter - a Google appliance is designed an optimised for searching the *web* for *web pages*. Unfortunately despite analogies of SL with the "3D Web" - it is not the web: a location plus objects is not the same as an HTML web page (which would typically have a lot more text and context to analyse), picks are not the same as inbound links from HTML web pages (again an inbound webpage has much more text, and there is a lot more context - for instance you can weight the significance of an inbound link on its own inbound links etc.).

Google has spent a lot of time fine-tuning and continuing to fine tune the Google engine for handling web data. It is not, there, surprising that it does not perform well on a very different datasets.

As regards googlebombing - again the nature of the web and SL are different. An attempt to googlebomb from a single webserver is trivial to detect (and hence ignore in the rating) - to successfully googlebomb you either need to buy a lot of webservers (distributed across the globe) or convince a large community to help you. The time and effort involved offsets the value of doing it in the first place (especially given Google are continually tuning the system to detect such Googlebombing).

Googlebombing the SL search engine is a case of spending the time to create avatars and add a pick to them - slightly time consuming but beyond that there is no cost (particularly as LL no longer charge $10 for an alt). Also Google isn't tuning the engine to detect SL related googlebombing.

Bottomline is trying to shoe horn a search engine tuning for searching a particular set of data, into a different context is not going to yield particularly good results. LL needs a search engine tuned to its data - not one tuned to the web.

Matthew

An informative post. But Phil, the fact remains that the new Search is pretty useless in actually performing its function, whether you put the blame on LL or on the land owners themselves.

Weighting the name of the parcel heavily may work...IF you happen to own the land. What about a renter on my property? The name of the parcel is "The Villas at Lebettu". This isn't going to help someone who owns, say, "Phil's Low Prim Furniture Outlet". I've got six shops for rent. Should I re-name the place "Phil's Low Prim Furniture Tom's Sex Toys Oki's Bondage Gear Brenda's Pie and Panty Shop Trout's Fishing Gear Miles' Christmas Decorations"? If I do, what about the people who are looking for a house to rent? We have those too!

Are you sure about that, Kitty. I spoke with someone at LL beause I assumed that the group that a piece of land is deeded to counts as a link for the land. I thought it was unfair because many lands are not deeded. The person I spoke with consulted the search specialist there (maybe just the search specialist in the help dept) who said that it won't make much difference to the rankings. They didn't say that the group doesn't count as a link for the land, even though that was my concern.

That's the way it is, Lindal. I'm not suggesting that the search appliance is the bees knees for SL - it isn't. It's designed for internal use - the indexing and ranking of one's own website or system, where the pages are created by internal people with search in mind. LL is using it in an external way, and not creating the pages to suit the search. That bit is left for us to do. All I'm trying to do here is inform people why their places may not show when they ought to, and that the All search in not "broken" in that respect.

There are 2 ways of dealing with your situation that readily come to mind. One is to use the land's name and description to further your own rentals business on it, and leave the renters to promote their own stuffs. I think that's the norm. The other is to split the land into parcels and optimise each parcel accordingly.

I've always thought that LL should give us a separate tab for people picks so we could use the "picks" tab the way it was intended and still be able pay tribute to our good friends.

This is making me think about "New Search Makeovers" for stores and such.

Someone could post there parcel name, description, etc. and folks would respond with their suggestions for more effective ones.

That's not the way that the algorithm works, Matthew. When a page is parsed, each word on it, together some info about the word, is stored seperately - even sentences don't remain intact. A page doesn't have any searchterms associated with it at all. The index contains individual words, together with some information about each individual word.

So it doesn't matter how many Picks have their text changed, all it does is cause the page to rank for other things, such as names. That's in addition to it ranking for its normal words and phrases. The normal rankings aren't affected at all.

You are mistaken about that. The Google search appliance is designed for internal use, and it is capable of handling dozens of document formats - not just HTML pages. It isn't a Web search engine like Google is.

What you described earlier about getting alts and changing the Names in their Picks is the same as Googlebombing and, like the Web Google, there's nothing to be gained by it other than getting people and places ranked highly for silly searchterms. None of it affects any other rankings. It's not detrimental to them.

It isn't tuned to the Web. It's tuned for internal use. What's lacking for SL is that the operators don't write the HTML pages with the search in mind. Well they do, but they can't tailor the most important parts to reflect a place's content. That's something that we have to do, which is why I thought it is worthwhile starting a thread about it. I was prompted by a post in which someone said that the All search is no good, and stated that she can't find her place in it. I started a reply, but I thought it was better to have it's own thread.

I agree with you that the new search is not "broken", but optimizing one's use of it can be challenging.

What if the parcel owner likes to change the name of the parcel periodically to promote different events or new items? Because the parcel names do not appear to auto-update in people's profile picks (unless something has recently changed), lots of people will end up with old parcel names in their picks.

Here is my question - since profile picks are treated as inbound links, do avatars have to be listed to show in search in order to have their inbound links 'counted'? Presumably, these links are being grabbed from webpages for each individual avatar in order to be counted. If one opts out of search, do the profile picks not count?

I am going somewhere with this. If this is the case, people who use 1,000 alts to game the profile picks will be very transparent - all you will have to do is search the place name in the new search and all of those alts will show up as having that place in their picks.

Incidentally, my friend did create a couple of alts for this purpose, to see how it would work. However, these alts, though listed to show in search, do not appear in the search all results when the place name is searched, though older avatars who have selected the store in their picks (including me) do appear. If you search directly for the names of the new alts, you can pull up their profiles in the new search, but the text in their profiles is not indexed so they don't show up in search when places listed in their profiles are searched. I wonder why this is?

I hope I'm making sense. I'm running late and am headed out the door, but had to ask.

The appliance cannot do everything. As James Linden noted in the classifieds thread, they are trying to help newbies explore. But exploring and shopping are two different things. Neither search engine is equipped to adequately shop OR explore because the features ideally used for shopping weigh against non-profit organizations that may not have objects for sale.

The only things that DO do everything are titles and descriptions, and as in all things, they are subject to abuse. There really is no way to get around abuse.

I agree with Phil, you need to break those stores out onto their own individual parcels, otherwise they will never be found. Rentals should also have individual parcels, that way you can control music streams individually.

The new search is broken if only for the fact that it returns ONLY FIVE PAGES FOR ANY GIVEN SEARCH.

This is like having a yellow pages that cuts off arbitrarily after the B's.

The Lindens said they would fix this in a few months maybe.

If anyone has heard anything about this beyond what I've reported here, let me know, cause I haven't tested it lately.

coco

Hi Xi.

I've noticed that too - I mentioned in a thread a couple of weeks ago. New alts with Picks were not listed as links to the place, whereas an old alt was. It could be to do with age, or it could be something. I've no idea.

That could be advantageous. Picks text isn't changed when the parcel text changes, so you could promote something new by using the land's Title and Description, but still keep all those IBLs in Picks, which could possibly maintain the rankings for the older searchterms.

The index is updated every 12 hours, so changes are quite quick if things like special events need to be targeted briefly. It may work out that a place could change the Title to target other things when the Picks are able to maintain the older ranking - and keep on doing it. Just a thought.

I've seen it limited to 5 pages and since then I've seen it keep on going. Some time later it was back to 5 pages again. I suspect that there are 2 datacenters, with one set to 50 results and the other set to a lot more, but I don't know of course.

Web search engines cut off at 1000 results - Google started that. 1000 is fine because the relevancy starts to deteriorate after the first 2 or 3 pages, or even sooner.