A prior vworker project substantially completed this product, which is a streaming server, handling database uploads and downloads of database dumps. It has a few bugs to be fixed, and a few features to be added.
The server streams SQL data dumps to and from a PostgreSQL backend server. The data goes to or from the client, or to or from Amazon S3. It uses [url removed, login to view] , so familiarity with that is a plus.
+ When failure exceptions are raised, they get swallowed somewhere, and the page returned is returned by a a success handler (a callBack), rather than an error page (an errback), sometimes with the error message.
+ Errors from the backend (pg_restore) do not get presented in the result page.
+ An extended S3 copy would cause the browser to timeout, given that the page is sent last. A needed improvement is to send the page early and add whitespace intermittently to keep the connection alive.
+ Send multiple S3 blocks at once (up to 3) for faster transfer. Now they are sent sequentially.
+ Have a status page that just displays the number of current connections, optionally with a time remaining estimate.
+ Add a graceful-shutdown feature that starts rejecting page requests, but runs until all current transfers complete.
+ Add a 'get-url' feature that will stream input from an arbitrary http or https url.
+ ensure that each S3 block was sent without error, and resend the fails
+ add a test suite that demonstrates the proper functionality of each feature, including failure handlers.
Provide your own server, (a small VPS will suffice), with Postgresql 8.3, and Python 2.5 for development. I will provide the current server and python mock modules that return Postgresql and S3 account identifiers and authentication. The current server is on a public subversion host right now: see [url removed, login to view] .
It relies on an S3 library hosted at: [url removed, login to view]~dkeeney/txaws/add-multipart-upload
I will need to verify functionality on your server before completion. I will handle deployment on my development server, but if I need assistance, we can arrange a follow-up hourly contract for support. In your delivery package, indicate clearly all dependencies (if any) other than the current twisted subversion checkout, python's standard lib, and the mock modules. Keep all modified or new modules in their own namespace.
The current server is running on my development site, [url removed, login to view], and accounts are free, so if you wish you can check the current version out now (linked as 'Bulk Data Transfer') from the profile page.