I've got a bunch of mysql tables with the same structure and a hex md5hash unhexed stored as binary(16) in my mysql tables. However, when I add data randomly into my different mysql tables (to split write performance on different servers and tables), I can't enforce uniqueness without a centralized table with a UNIQUE/PRIMARY index or an external key-value optimized storage system. Mysql is TOO SLOW to enforce uniqueness because the time to insert decreases exponentially as you add more and more records to the unique table. I need a sample and working model with an external system like mongodb, couchdb, tokyocabinet, tokyotyrant, voldemort, or cassandra where before the md5hash is inserted into a mysql table, it is checked in batch against any of these systems. My requirements is that you can check and insert 1 Million unique hashes into the key-value store in under 5 seconds, when the key count in less than 100M unique keys, and the value are nulls.
PHP, perl, or c++ implementation demonstrating a centralized uniqueness enforcing architecture, so that all unique hashes can be inserted or read from the centralized place in a timely fashion - 5 seconds per million in or out (excluding data tranfer time) 1) Complete and fully-functional working program(s) in executable form as well as complete source code of all work done.
2) Deliverables must be in ready-to-run condition, as follows? (depending on the nature? of the deliverables):
a)? For web sites or? other server-side deliverables intended to only ever exist in one place in the Buyer's environment--Deliverables must be installed by the Seller in ready-to-run condition in the Buyer's environment.
b) For all others including desktop software or software the buyer intends to distribute: A software? installation package that will install the software in ready-to-run condition on the platform(s) specified in this bid request.
3) All deliverables will be considered "work made for hire" under U.S. Copyright law. Buyer will receive exclusive and complete copyrights to all work purchased. (No GPL, GNU, 3rd party components, etc. unless all copyright ramifications are explained AND AGREED TO by the buyer on the site per the coder's Seller Legal Agreement).
linux centos 64bit using php or perl or c++ on mongodb or couchdb or tokyocabinet or tokyotyrant or voldemort, or cassandra