Encerrado

High speed multithreaded url collector

High speed webcrawler for collecting unique fully qualified domain names from the internet. A list of fqdns must be saved in a text file, on fqdn on each line. Configurable options: - Number of threads - Maximum number of fqdns in one file, before starting a new one - Line template (say: "INSERT INFTO results_table ('fqdncolumn') VALUES ('[fqdn]');" - Command when starting a new file (say: mysql < [filename]) Requirements: - Check [url removed, login to view] - Don't get stuck, go on till I kill! - Collect thousands of urls in one hour

## Deliverables

Running, stable & documented application that do all things that are listed above. The program must use efficient memeory so it can run for months without problems. Split the application in classes or functions so it can easy extended with new functionalities. (like grabbing DNS info, content etc.)

## Platform

Linux 2.6

Habilidades: Programação C, Engenharia, Linux, MySQL, Perl, PHP, Python, Arquitetura de software, Teste de Software

Ver mais: Webcrawler, speed\, linux check domain, run linux command php, kill linux, insert txt, line check template, perl split, linux insert text file, url check, grabbing url, content threads, dns program, php split file, split text file content, dns list, collect domain names, php mysql dns, url internet, domain info perl, internet url list, mysql multithreaded, internet url, speed internet, unique text file php

Acerca do Empregador:
( 3 comentários ) Netherlands

ID do Projeto: #2961826

9 freelancers estão ofertando em média $253 para este trabalho

macFR

See private message.

$63.75 USD in 7 dias
(37 Comentários)
5.5
esixoft

See private message.

$170 USD in 7 dias
(29 Comentários)
5.0
truekonrads

See private message.

$1275 USD in 7 dias
(5 Comentários)
4.9
andr233sborja

See private message.

$170 USD in 7 dias
(7 Comentários)
3.7
stonewell

See private message.

$170 USD in 7 dias
(5 Comentários)
3.2
emonkey

See private message.

$212.5 USD in 7 dias
(3 Comentários)
1.1
randle00

See private message.

$42.5 USD in 7 dias
(0 Comentários)
0.0
danny81vw

See private message.

$42.5 USD in 7 dias
(0 Comentários)
0.0
pdsm

See private message.

$127.5 USD in 7 dias
(1 Comentário)
0.0