The following table summarizes the performance and storage requirements associated with crawling different numbers of documents and URLs.
| # of Docs | # of URLs | Time | Crawled size | Cached size | Avg Page Size | Avg Pages/Sec | Avg Mbit/Sec |
|---|---|---|---|---|---|---|---|
| 50,000 | 50,241 | 15:44 | 1.47 GB | 0.90 GB | 30.74 KB | 53.22 | 12.8 |
| 150,000 | 149,201 | 45:51 | 4.97 GB | 2.75 GB | 34.96 KB | 54.24 | 14.8 |
| 250,000 | 247,161 | 1:15:41 | 9.25 GB | 4.76 GB | 39.25 KB | 54.43 | 16.7 |
| 500,000 | 491,545 | 2:42:31 | 18.33 GB | 9.53 GB | 39.11 KB | 50.41 | 15.4 |