Every one has an idea of working of search engines like google, yahoo, MSN, AOL, etc.
In simple words it follows four steps:
1. Crawling-In this step spiders or robots or MSN bots which are actually S/W program, crawls the entire site related to particular keyword or search query.
2. Indexing-Then it collects all sites related to that topic or word and stores them in search engines database.
3. Processing- As results are in lakhs, it should be sorted according to the relevancy for the search topic.So, it creates another list for specific answers of the search topic.
4. Result page- Ultimately what ever is the out come of the processing will become the search page result.
According to me this is the process followed by the all the search engines . Although different search engines applies different algorithm for crawling S/W program.