Need a php based spider that will act "exactly" as a normal browser, accepting cookies and downloading js.

It needs to spider all the links in a given domain and follow the links and request every page in that domain. Does not follow external links.

It must have configurability.

1. user agents.

2. proxy list.

3. random units of time between requests and switching proxy/ua's.

4. Must be able to run from local Windows XP/ Foxserv setup or a webserver under Linux/freebsd. Any module dependancies must keep this in mind. No access to httpdconf or even telnet access.

The script will run a random number of requests with one user agent and proxy with random times between requests. It will then switch proxy and user agents randomly and run random number of links with random times between requests, and so on.

It must accept and retain cookies in a natural browser behavior for each proxy/ua session. After each transition to a new proxy/ua cookie resets and accepts new one. It must also download javascripts and execute the js to obtain the cookies.

## Deliverables

1) Complete and fully-functional working program(s) in executable form as well as complete source code of all work done.

2) Installation package that will install the software (in ready-to-run condition) on the platform(s) specified in this bid request.

## Platform

windows/foxserv linux/freebsd

Habilidades: Engenharia, MySQL, PHP, Arquitetura de software, Teste de Software

Veja mais: work ua, agent module, does not follow, spider, linux run external program, freebsd source, switch engineering, linux proxy setup, session cookies, windows program switching, telnet proxy, spider proxy, proxy spider software, cookie links, local domain time, spider external links, proxy spider, windows telnet, page setup browser, user agents list, php transition page, freebsd source code, browser user agent, page transition php, php page transition

Acerca do Empregador:
( 14 comentários ) United States

ID do Projeto: #3060745

Concedido a:


See private message.

$85 USD em 3 dias
(55 Comentários)

3 freelancers estão ofertando em média $283 para esse trabalho


See private message.

$722.5 USD in 3 dias
(11 Comentários)

See private message.

$42.5 USD in 3 dias
(3 Comentários)