I need an experienced developer that can write an php application that has crawler type of functionality that will do the following things:
1. Have the crawler go to a number of specified websites every day at a time that I specify (automated time) or when activated manually, login with a username and password that I will also specify (I should be able to type in this information into the application’s interface).
2. Retrieve product information form my suppliers website and then update my website, This may include images. (if the images already exist on my site then the application should not download the images again)
3. It also then needs to be able to apply a margin (the margin will be based on several variables) and courier costs (the courier cost is a flat rate) to the pricing information retrieved from my suppliers website. (I should be able to type in this information into the application’s interface)
4. I want to be able to configure the crawler for different sites. Ease-of-use in this process is important.
5. Must be able to deal with any site and extract any type of information.
6. The crawler should be very fast, not slow due to bad programming.
7. The web crawler must be able to put all of the products in to categories such as Video cards, CPU’s and so on.
8. Different Websites may use different categories - I will need to be able to merge categories from my suppliers website with categories on my website or create new ones with in the applications interface.
All providers must have your own test server so that I can preview a working model of the crawler before it is implemented into my live site
You must be available in the future for paid maintenance and further development.