Ever wanted to see things on your Web site in the same context as the Google Bot does as it is crawling your Web pages? Well, thanks to this article and a little user agent switching – you can do just that.
The search engines index your website by using their bots which are also called crawlers or spiders. And these bots do nothing but visits the URLs and check your links and visits those links too. The keywords on the pages are indexed and recording all keywords it can find* and including it in the indexed information.