Rewrite a crawler for 10 sites.
We have a small crawler application for which we don't have the source code that crawls a number of websites and populates a database. Because the author is no longer available, we need this crawler rewritten.
- We will provide you with the DB schema
- We will provide you the current application (binary only) as is so you can check it out
- We will provide you a list of sites to crawl and some notes on how to do it
- The sites are in Spanish. Your crawler must work with all Spanish characters
- This is for a book store, and the sites are universities and book publishers sites.
- Your application must be able to start from a previously populated database, doing the required updates. You can't just start from scratch each time the app is run.
- You can write the application in any language you want, but it needs to have a decent interface. If you want to use PHP then you'll have to provide a small website to control it, for example.
- The target database is MySQL.