i need a bot that will scrape odds on various sporting events from a number of bookie websites. some of these have real-time XML feeds which can be used, others, the actual odds need to be scraped. the odds are then to be stored into a SQL database for analysis. provisions need to be made to ensure that the pages that are being scraped are not done too regularly in order to avoid IP being blocked. i can provide guidance on this area. some formatting needs to be done between some sites, as different bookies have different ways of representing data.
i would like the application ideally written in either C sharp or .net (c sharp).
there will be a further phase of this project which will involve automating logon to the bookie sites and placing bets. my aim is to award this project to whoever successfully completes this phase, so when bidding please bear this in mind.
i can give you a couple of examples of sites that need to be scraped.
and http://www.skybet.com/skybet?action=GoEvClass&id=35 is another.
the main challenge is that the structure of every site is different. for example the first one nicely has all the tennis events listed on one page, but the second one does not, they are in sub sections. we need them all married together and represented as one single event. i would also recommend storing the path in the DB as it will make phase two easier.
i think there will be a few elements to this, one we need to create a common db structure, then we need to write individually customised scrapers for each bookie page.
initially i will be looking for 5 bookies, but eventually i will have upto 50 of them, so the system needs to be scalable, and you need to be able to commit to being able to convert them all.
are you comfortable with being able to automate logging into sites, filling out forms, and posting them?
Good day! We are the software development team from Eastern Europe. We are good experienced in C#, .NET and MS SQL. It would be great to working on your Web scraping bots project. Thank you very much!