Price £400 = $787
Total Size: 135 gB on 3.5" hard drive=$70 for hard drive
Number of HTTP requests: 324,880 homepages + 3,215,171 permalinks + 100,649 feeds * 11 weeks = 4,647,190
This number of requests in 11 weeks results from about 41 requests per minute or less than one request per second. I am averaging 20-50 requests per second on on my office box. My home Comcast cable service allows only 2 simultaneous requests and a page load every 2 seconds is not a wild assumption. The service cost for 11 weeks is about $100. So, you cane save around $600 minus unknown amount of dev and ops cost if you choose to run your own crawler. However, you won't be able to submit a paper to TREC.
0 comments:
Post a Comment