spider with 100 threads that can run for several weeks withotu crashing or slowing down(repost)
$30-100 USD
Cancelado
Publicado há quase 12 anos
$30-100 USD
Pago na entrega
Need a 100-threaded spider that can spider sites for several weeks at a time without slowing down or crashing (its ok if it restarts itself whenever it crashes or starts to slow down). Must support resume/pause. Must be able to divide these 100 threads among multiple sites.
It must be able to spider an average of 100 urls minimum per minjute over a 60 minute period.
Must be based on eiother one of these libraries below:
<[login to view URL]>
<[login to view URL]>
it can ALSO be your own design but then please give reasons why yoru own design is better than these liraries (I wil bu the librarieis t teh qualified coder i choose).
Please write what experience you have with makign spiders that use 100 threads or more and how long time you have had them running and what the usual problems are with it. Also write why you are the right coder for this project.
Also write if you have a demo already to show/test.