An excellent article on how spyders work - including a link to the first article.

I'm sure you will enjoy it and learn something (just make sure to explore all the links).

http://searchenginewatch.com/searchday/article.php/3406891

Tim