We require a Drupal module that adds crawling functionality to our web site.
The solution must: allow us to create scripts for crawling and extracting data from arbitrary (number of) web sites, import the extracted data into Drupal for review and publication, and provide scheduling of the crawlers from within Drupal.
The solution must be based on Selenium RC, PHP and MySQL and work on Ubuntu Server.
Please see attachment for full requirements spec.
Drupal 6.x Selenium RC 1.x (with Firefox as browser) PHP 5.x MySQL 5.x Ubuntu Server 9.04