I do not know about crawler in magento.I search but do not know exactly what it do ? there are lots of crawler extension about cache crawler but I want to know about crawler.Anyone can understand about crawler in cache in magento ?
As I know crawling is just process to save pages in cache by visit all pages. Module used for crawling will visit pages programmatically and stored data in cache so another time when user try to visit that page data will come from cache.
You can see cache data in var/cache and if you have selected stored cache in database you can see stored data in core_cache table. There can be another table for custom cache modules.
Hope it will help you to understand basic crawling.