Need a php based spider that will act "exactly" as a normal browser, accepting cookies and downloading js.
It needs to spider all the links in a given domain and follow the links and request every page in that domain. Does not follow external links.
It must have configurability.
1. user agents.
2. proxy list.
3. random units of time between requests and switching proxy/ua's.
4. Must be able to run from local Windows XP/ Foxserv setup or a webserver under Linux/freebsd. Any module dependancies must keep this in mind. No access to httpdconf or even telnet access.
The script will run a random number of requests with one user agent and proxy with random times between requests. It will then switch proxy and user agents randomly and run random number of links with random times between requests, and so on.
2) Installation package that will install the software (in ready-to-run condition) on the platform(s) specified in this bid request.