UK Web Site Design and Search Engine Optimisation
What are the most used Search Engines?
Despite seeming simple, this question can be a tough one to answer. If you are looking for overall stats, then comScore Media Metrix release periodic results which currently show Google, Yahoo, Aol and MSN as the top search portals, attracting around 94% of searchers between them.
There are a couple of complications with the comScore measurements though. First they tend to group properties wherever they can, therefore the Aol results include searches performed at their Netscape portal. However, the Yahoo results on the other hand do not include the searches that are performed at Overture, Altavista or AllTheWeb despite the fact that they own those properties.
Search Engine Results
This is where we require some little known (or rarely talked about) piece of information. There are only a few actual databases from which the search engines tend to derive their results.
Currently if you went to Google, Aol or Netscape amongst others, you would find yourself looking at results taken from the Google database. The various search engines may make a few minor changes either via their search algorithms or by mixing in results from other databases, but the main source of their results remains Google.
Of the top 4 sites, Yahoo and MSN are the only ones that use something other than the Google database. At this time MSN uses results provided by the Inktomi database. However Inktomi was bought by Yahoo at the start of 2003, and so Microsoft have since looked into building their own independent search engine with rumours of release sometime in 2004.
Yahoo themselves have recently switched their results, and since buying Inktomi and Overture (including Altavista and AllTheWeb) in early 2003 they only now seem to be combining their resources. As it stands Altavista and Alltheweb still currently use their own original databases, but it will only be a matter of time before they switch to the new Yahoo index.
Smaller Search Engines
Of the remaining search engines of note, Lycos has results provided by AllTheWeb, AskJeeves has results provided by Teoma and Overture provides results from its own Pay Per Click database followed with backfill by Inktomi. Hotbot these days is a meta search engine where you can see results from several other search engines, although the default search is Inktomi results.
The actual figures for people using search engines to find your website may be very different to those found by these reports. Overall figures may suggest that no-one is using regional ISPs to find websites. However, you may find that in your experience you have a high proportion of visitors using Freeserve to find your website.
If you wish to gauge more accurate figures you will need to define your market segment and then analyse their use. Sometimes traffic figures broken down by market are published, but mostly you will need to find other ways of ascertaining this information. You could ask your competitors but these kinds of statistics are usually closely guarded.
In order to keep up with the relationships between the various search engines and their databases, Bruce Clay keeps an updated relationship chart in pdf form available. Another chart specifically for UK based search engines can be found at Alphaquad.
Online Directories
On top of the search engines, there are a few directories (peer reviewed hierarchical databases) that are worth noting. The first is one of the most important starting points on the web, the Open Directory Project (also known as the ODP or DMOZ). Most crawler based search engines start their indexes from here, so it is important for websites to be listed in the ODP.
The next two are diminishing in importance, but can still be useful. Yahoo was perhaps the original (certainly most talked about) directory, before it started to focus on its search results. Looksmart has been a one trick pony for some time, albeit the son of Shergar, as it provided the directory results for MSN. However, MSN has now dropped them and Looksmart are struggling to find new partners. The main shame is that Zeal, Looksmart’s volunteer based directory now appears to be facing Internet oblivion alongside it.
Other Web Resources
There are many other search engines, meta searches and directories which haven’t been talked about here. Some are enterprise packages such as Northern Light, others are new like Gigablast or Skaffe, while a few simply aren’t advertised overtly (Amazons A9 search solution).
Many other resources exist, but are not influential enough to mention in a generic overview. On a final note, it is always worth keeping an eye out for industry specific search engines as these can be invaluable for targeted lead acquisition sources.
Hopefully with this knowledge safe at hand, the next time some person suggests that they will submit your website to 100,000 search engines you can just turn around and ignore their woefully inadequate sales pitch!