Web crawler thread details
You can monitor the Web crawler to see how many threads are actively crawling Web sites and how many are in an inactive state.
When you view details about
a Web crawler while monitoring a collection,
you can view the status of the crawler threads. The states that you
are most likely to see include:
- Waiting
- Indicates that the thread does not have a URL to crawl. This condition can occur when a thread finishes a crawl and the crawler cannot find more URLs to crawl fast enough. For example, if the crawler property that controls how long the crawler must wait before it can retrieve another page from same site is too high, it can prevent URLs from being supplied fast enough.
- Fetching
- Indicates that the thread is downloading a page from a Web site.
- Completed
- Indicates that the thread is sending the pages that it crawled to the rest of the crawler, but is not yet ready to crawl another URL.
- Suspended
- Indicates that the crawler is paused
Ideally, all threads are fetching pages all of the time. If threads are often in a completed state, then the database might be having throughput problems.
If threads are often in a waiting state, review the value specified for the Maximum number of active hosts field in the crawler properties. If the value is low, there might not be enough sites in the crawl space to keep the threads busy, or there might not be enough URLs eligible to be crawled. Conditions that can cause low activity include DNS lookup failures and robot lookup failures.