Search Engine Effectiveness - Practical Test [Digg.com This!]

I've been wondering recently how effective alternate search engines are compared to Google. As a small test I decided to take 25 searches I had performed in the last week and search on Google, Yahoo and MSN Live Search to compare results.

The following table shows the results. A score of zero means the target info was found on the search page itself (I didn't have to click through to the site). A number 1-10 is the position of the search result with the target info. A blank score means the target info was not found on the first page of links.

The score is a sum of each search position negated from 11. As so:

Target info on search page: 11-0 = 11
Target info on page result 4: 11-4 = 7
Target info on page result 10: 11-10 = 1
Target info not found: 0



Conclusions:

I was pretty suprised Google was so far ahead compared to Yahoo and MSN. Google won because it was better at finding more niche searches, whereas Yahoo and MSN failed completely. For more obvious searches all three performed flawlessly and may do for day-to-day searches.

Google tended to either have the result first, or not at all. Yahoo had a few useful results in the top five. MSN had some results nearer the bottom of the top 10. This is significant for MSN because much lower results seriously increases the time taken to find what you want.

Some oddities included Yahoo having numerous entries that were highly irrelevant and MSN having a curious slant to open source results, particularly source code.