We've built out a well-tested, docker swarm + typescript + timescale + bull-queue based infrastructure for crawling a variety of food ratings sites. It has full CI/CD to production servers and is ready for crawling the USA. It was fully working a few months ago on runs in SF, so now the job is just fixing up the tests that are breaking in the last couple months, getting proxy rotation working nicely, and scaling it up to run continuous crawling on the US.
15 freelancers estão ofertando em média $30/hora nesse trabalho
Hello, dear! I am very interested in your project. I have over 10 years of experience with web scraping and crawling. I can use Python andTypeScript for crawling. I can scrape any sites. Please contact me. Best wishes.