Need a php based spider that will act "exactly" as a normal browser, accepting cookies and downloading js.
It needs to spider all the links in a given domain and follow the links and request every page in that domain. Does not follow external links.
It must have configurability.
1. user agents.
2. proxy list.
3. random units of time between requests and switching proxy/ua's.
4. Must be able to run from local Windows XP/ Foxserv setup or a webserver under Linux/freebsd. Any module dependancies must keep this in mind. No access to httpdconf or even telnet access.
The script will run a random number of requests with one user agent and proxy with random times between requests. It will then switch proxy and user agents randomly and run random number of links with random times between requests, and so on.
1) Complete and fully-functional working program(s) in executable form as well as complete source code of all work done.
2) Installation package that will install the software (in ready-to-run condition) on the platform(s) specified in this bid request.