Encerrado

Expired Domain Web Crawler Needed

I am looking for a software application that can crawl a list of websites for expired domains (any domains with a status of 404). The crawler should crawl a list of inputted names via a source text file. The source text file would look like this, 1 domain name per line:

[login to view URL]

[login to view URL]

[login to view URL]

[login to view URL]

Once software grabs the source domains, it should fully crawl each site and report to a csv file any expired domains it finds. The format for the csv should be:

domain where found, URL where found, expired domain

An example line would be:

[login to view URL],[login to view URL],[login to view URL]

The crawler should use just one thread, and for the user agent, should just be chrome if that is possible.

I plan to do a phase 2 later where the crawler will check an API to see if the expired domains are available for purchase, and also to add threads, proxies, and other features. This will all be phase 2 and 3.

Right now, I just want phase 1 completed where it takes an input of "[login to view URL]" and then for domains it finds in 404 status, it should output "[login to view URL]"

This needs to be desktop based running on Windows 10 Pro. It can run in Powershell or Python, and other languages are fine as well. For continued development, it would be beneficial to have it in Python, but any language you prefer is fine.

Please include these with your bid:

1. Project Price in Full to do phase 1

2. Time to complete, since I am really needing this soon

3. What language you will be using

4. Any relevant experience you have with crawlers

Habilidades: Captura de dados na web

Veja mais: java web crawler image capture, web bloggers needed, web crawler cpu usage, write web crawler, expired domain checker, web crawler names, web sites needed, web designers needed, web crawler system requirement, web crawler demo gathering information function, java spring web crawler, web page sql database input, good sites web projects needed, simple input file php script web crawler, php web crawler input, expired domain crawler

Acerca do Empregador:
( 67 comentários ) Loveland, United States

ID do Projeto: #20031872

27 freelancers estão ofertando em média $167 para esse trabalho

e3d

Hi, I can make this crawler for you, but I actually already providing my clients with expired domains with backlinks found on sites of their choosing. also for the architecture that you've outlined you'll need lots of Mais

$250 USD in 3 dias
(248 Comentários)
8.7
zekovicm

Hi there,I am Web Scraping expert from Bosnia & Herzegovina,Europe. I have created similar crawler in the past for checking expired domains . I can do it again for you ! Check out my profile and former clients feedba Mais

$220 USD in 5 dias
(93 Comentários)
7.2
devi222

Hi Sir, I have created same crawler 2 years ago for some employer and a lot of crawlers for other types, I can complete in 2 days. Im a Python Programmer. I have created many Bots to scrape the websites like Google, Al Mais

$150 USD in 2 dias
(125 Comentários)
6.8
schoudhary1553

Hello, I have gone through your job posting and become very much interested to work with you. I am an expert in this field. I have already completed several projects like this. For evidence you can see my profile. Pl Mais

$250 USD in 4 dias
(16 Comentários)
5.8
Bluesky122

Hello. I have read your project description carefully and bid here. I am a senior python developer and have expertise in webscraping. I can complete your project in a day. I will use [login to view URL] for GUI. I can pro Mais

$200 USD em 1 dia
(29 Comentários)
5.9
yongjin818

Dear As an expert in web scraping, I developed 500+ spiders using python, scrapy, php,nodejs,c# and proxies. And I scraped data and download images, files for CEOs and output result as csv, json, xml(yellow, ebay, wi Mais

$140 USD in 7 dias
(63 Comentários)
6.0
brianconey

Hello How are you I have read your job description in detail . I am Xu It is a piece of cake i have done it before Please contact me and Let's discuss more thanks for your posting . hope to hear from you .

$140 USD in 7 dias
(17 Comentários)
5.6
mtanveer2012

Hi, 1. Project Price in Full to do phase 1 Ans: 150 USD. 2. Time to complete, since I am really needing this soon Ans: 1-2 Days maximum. 3. What language you will be using Ans: Python. 4. Any relevant experience you ha Mais

$150 USD em 1 dia
(54 Comentários)
5.7
smsaurabhv

‌Hi, I have gone through your requirement to scrape lots of websites. I am EXPERT in building scraping tools /scripts. Hence, I can SURELY work on your project. I am having 4 YEARS of EXPERIENCE in developing PHP-PYTHO Mais

$111 USD in 3 dias
(57 Comentários)
5.2
shiyusun180409

I checked your description. As I am a software and scrap expert , I have a wealth of experience and skills in these projects. I prefer python language and i will complete it within 1~2 days. You will get good results. Mais

$150 USD in 2 dias
(16 Comentários)
4.5
alexwmsoft

Dear Client Thank you for taking for your precious time to looking for my bid. I have fully read the job description and it is very clear to me. I am a senior web scraper who is good at python. I am using pytho Mais

$155 USD in 3 dias
(5 Comentários)
4.5
sky930320

Dear Sir. Glad to meet you. I'm Web developer specializing in web scraping crawling and indexing web pages. Skills: python, scrapy, selenium, requests, beautifulsoup, mechanize, lxml, urllib2, automation, bots, spide Mais

$250 USD in 7 dias
(9 Comentários)
4.2
writingapp

Hi. Great app writer for your projects. I have writen scraping app for many years. I am ready to write your project. Thank you for visiting my profile

$140 USD in 5 dias
(3 Comentários)
3.9
shaktistanwar

Hi I am a technical evangelist, developer and can do this task for you. I have more than 15 years of experience in software development and am a microsoft certified professional.I have strong work experience in Nopcom Mais

$250 USD in 3 dias
(1 Comentário)
3.4
PiushGoutam2018

Hi, I am interested in taking up your project. I have done 10+ projects in python software development,website development,desktop automation,website automation and web scraping,.I deliver fast,clean and quality work Mais

$35 USD em 1 dia
(19 Comentários)
3.5
drohanyadav

Hi, I will be using python language to do this project. I work fast, and I will probably finish it in few hours, but there are always some small complications, so it will be take me maximum of 2 days. I have been web s Mais

$60 USD in 2 dias
(9 Comentários)
3.4
NonikNone

Hello how are you? My name is Ugur Catak. I am a software developer. About your project: Since we don't need to download entire page, we can request for the status code only to speed the bulk query cycle. 1. Project Mais

$110 USD in 2 dias
(2 Comentários)
2.4
Xiong402Jin

1. Project Price in Full to do phase 1 $200 2. Time to complete, since I am really needing this soon two days 3. What language you will be using python 4. Any relevant experience you have with crawl Mais

$200 USD in 3 dias
(3 Comentários)
2.3
dhirajbaheti

Hi, My name is Dhiraj and I am a professional web scrapper and data mining expert in python with more than 6 years of experience. I have expertise in scraping complex/dynamic websites using requests, BeautifulSou Mais

$150 USD em 1 dia
(6 Comentários)
2.4
khoadeveloper

Hi, Dear client. I am a web scraping/automation/bot/chrome extension expert with 6 years of experience in NodeJS, Puppeteer, Chromy, Selenium, Chrome-extension, Chrome-headless, Firefox-headless, SilmerJS, PhantomJS Mais

$255 USD in 3 dias
(1 Comentário)
2.2