Currently I am building a medium sized scrape project (with Python/Scrapy) where different websites with different formats needs to be scraped. I am looking for a co-worker/ expert in web scraping (e.g. json, api). The websites/sources will have (many) dynamic parameters/variables.
Challenges with for example large tables with dynamic headers are things you have to like in this job:) The job contains optimise existing codes or build complete new codes /files to extract certain websites(all on GitHub). The data will be stored in a online cloud storage. (SQL, AWS)