Real-time web scraping multiple websites

I need a solution for extraction data in "near real time" from different websites but with the same general topic.

I don't know the numbers of starting pages (maybe 500.000)

The software should do:

- Autodiscovery new URLs (based on specific rules)

- Extract and save some particular data (object name, description, price, ...) and general data (HTML title, author, ...)

- These data must be updated continuously

- These data should be saved in a database

- I need a JSON API that should inquiry that database with these functionalities

- parametric search

- aggregate search

- full-text search

- In case of error or new needs, It should be reprocessed all or some pages.

- Easy to add new sites.

- Enable to scrape ajax based websites.

I'd like to have a solution that can be easy to install, configure and most of all easy to update in case of errors.

Habilidades: AJAX, JSON, Captura de dados na web

Veja mais: web scraper chrome, how to scrape data from a website, real time web scraping python, web scraping api python, web scraping applications, visual web scraper, web scraping api free, web scraping tutorial

Acerca do Empregador:
( 2 comentários ) varese, Italy

ID do Projeto: #19710624

57 freelancers estão ofertando em média €3756 para esse trabalho


Hi, Can you share more details about your project? I think Python, exactly scrapy is the best for your project. As a professional full stack developer who has rich experience of Web scraping, I'm very interested in Mais

€4000 EUR in 7 dias
(29 Comentários)

Hi. I did read the project description and have a few questions. 1. Do you need the script as well or data only? 2. What is the format of the output data? CSV is OK? We can do other formats as well. 3. Which fields do Mais

€4000 EUR in 5 dias
(39 Comentários)

Hello? Nice to meet you. I am excited to work with you on this project. I am ready to start work immediately. Here is on of my scrap with python working result. [login to view URL] I can do python with seleniu Mais

€4000 EUR in 7 dias
(44 Comentários)

Greetings. I have extensive knowledge and expertise in web scrapping/ crawling software. Would you please come over chat so that we can discuss this project further? Willing to showcase my work over chat. It seems like Mais

€4000 EUR in 7 dias
(11 Comentários)

Hi, I am Fatima a full time PREFERRED FREELANCER and Python certified developer with 100% guaranteed delivery. Before we start the project would like to have a detailed conversation with you regarding this project to Mais

€4444 EUR in 30 dias
(88 Comentários)

Hello there! I would like to be considered for your project. We can build a script to scrape multiple websites according to your requirement.

€5000 EUR in 8 dias
(95 Comentários)

Hello! Collecting data from various places is my profession. I am expert in Data extraction, validation, mapping, Data Analysis & reporting scrapping using C# and Python. I develop my own scrapers and Bots. I have expe Mais

€4000 EUR in 7 dias
(27 Comentários)

Greetings, I am an experienced professional scrapper and have done similar projects in the past. Same can be verified from my profile. Let me allow to assist you with your requirements. Thanks

€3000 EUR in 8 dias
(45 Comentários)

Hi I am a technical evangelist, developer and can do this task for you. I have more than 12 years of experience in software development and am a Microsoft certified professional.I have strong work experience in Nopcom Mais

€4444 EUR in 30 dias
(4 Comentários)

Hi there, I have read initial brief & wondering if you can please provide me URL of those websites, from where you want to scrap the data? Please click on the chat button for further discussion about your project Mais

€5555 EUR in 70 dias
(5 Comentários)

Hi there how are you I am a senior python developer and i have used python and selenium . Please contact me and tell me about your scraping project . thanks for your posting

€5360 EUR in 2 dias
(19 Comentários)

Hello, and thanks for the opportunity to bid on your project. Please check my portfolio and reviews from other customers https://www.freelancer.com/u/TenStar718.html I am an expert in many different area’s of web Mais

€3000 EUR in 30 dias
(10 Comentários)

Hello, I am Python expert and expereienced on scraping web sites. I can build a nice app for your usage. Can we discuss details over chat please? Best Regards

€4000 EUR in 7 dias
(37 Comentários)

** WE ARE SCRAPING SPECIALIZED ** Dear Client, I would like to inform you that We have already worked on similar kind of requirements as yours and We are confident that our skills and experience will definitely help Mais

€3000 EUR in 30 dias
(5 Comentários)

Greatings, I would love to have the opportunity to discuss your project. Web scrapping is based on RegExp tech. I have full experience with Scrapping... real-time. I would like to work with you. Please message so Mais

€4000 EUR in 30 dias
(3 Comentários)

Hello, Dear How are you? I have read your project description and am ready for discussing with you about project. I have experienced in developing AJAX, JSON, Web Scraping. I will work very hard and best for you. Mais

€3000 EUR in 30 dias
(11 Comentários)

Hi, I can design and develop a solution for data scrapping of differetn websites let's talk to decide further It will have a catchy and interactive landing page My 9 years of development experience in SAAS an Mais

€4000 EUR in 45 dias
(1 Comentário)

Hi, As per understand your project requirement we found that you are looking real time data extraction for multiple website in same time. You want three type of search parametric search, aggregate search, full-text s Mais

€3627 EUR in 30 dias
(3 Comentários)

Nice to meet you I have more than 15 years development experience and a great team, we can complete your project with clean code and best practices To be able to give clients what exactly they want, we are workin Mais

€4444 EUR in 5 dias
(11 Comentários)

Hello, I have gone through your Project requirements and Yes I can do it and also assure you that I can do this job perfectly. Hope to hear you soon. Thank you.

€3000 EUR in 30 dias
(8 Comentários)