Also known as a spider, this combs the web downloading pages and content to add to its index. For example "the updated version of a page won't be recognized by Google until it is crawled by GoogleBot."
