Encerrado

Node app - website crawler

I need a node application which crawls few websites (pagination) and in each page fetch all the links that starts with a specific string and then check if this url (substring of the url) exist in DB table, if not, insert it.

the website url structure needs to be broken down is as follows:

https://domain/***CountryCode***/genre/id***ID***?letter=***Letter***&page=***Numerator***#page

All the fields with *** are changing variables.

in some cases for a specific letter the numerator goes up to 150 and for some till 30.

Habilidades: node.js, Javascript, MySQL

Veja mais: web crawler, cheerio npm, typescript web crawler, node js web scraping login, nodejs phantomjs crawler, nodejs web scraping, node crawler, node crawler queue, website crawler script, 2008 website crawler, user app website, website crawler software, django website crawler, website crawler retrieve url, php website crawler, check code website crawler, website crawler downloads, java code website crawler, website crawler downloader, asp website crawler

Acerca do Empregador:
( 0 comentários ) Tel Aviv, Israel

ID do Projeto: #22496662

2 freelancers estão ofertando em média $40 para esse trabalho

billinhit

Hello, I am strong on nodejs and also use nodejs to scrape website. This project is easy for me. Please contact me!

$40 USD em 1 dia
(15 Comentários)
4.5
Vikasv8

Hello I went through with your requirement, I've 5+yrs of experience as a MEAN stack developer/ having scrapping project experience as well. I worked on a similar project before as well, I can develop the site like thi Mais

$40 USD in 2 dias
(15 Comentários)
3.6