**Create or find a compression algorithm which has a good compression ratio for small packets.
We have a large Database with millions of? log records.
The log files are small (10-5000 bytes).
When using standard compression the compression header (which is stored together with the compressed packet) occupies a significant portion of the entire packet.
This ratio is more efficient when the data is large but becomes significantly less desirable when the data is small.
In other words, when dealing with small packets the compression header is often too taxing in relation to the entire packet size.
**The algorithm must be build to be used as a function, SP or assembly with SQL 2005.
The server Side must be:
·? ? ? ? ? ? ? ? MS SQL 2005
·? ? ? ? ? ? ? ? Any applications or libraries must be C#
·? ? ? ? ? ? ? ? All source code must be available and provided upon completion.
·? ? ? ? ? ? ? ? Third party modules if required will be purchased following the approved recommendation of the developer.
A successful solution should be able to deliver a compressoin ratio of at least 30%. Lower levels of success may be a 40-50% ratio of conversion.
**See attached File for Real Data Sample Sheet and further details.
****A. Row by Row Compression
**Each record would be compressed in its Data column.
? The table would exist in exactly the same way but instead of the records in the Data column
a reference would exist to a table which would store the compressed file.
Or alternatively the compressed data will be stored in the same table. Upon extraction it will be decompressed using the respective module as discussed herein.
**B. Session by Session Compression
**The table would exist in exactly the same way but instead of the records in the Data column
a reference would exist to a table which would store the entire session.
The downside of this is that although a greater compression ratio would be obtained, the compression would not be a real time.
The system would have to wait for an entire session to finish or to run a scheduled task to convert the row data into session data.
**C. Universal Header Compression
**Instead of a header created for each row or each session. A universal header (or more general headers) could be used (stored in another table)
which could be used as references accessed by the Data records in determining the rules for compression and decompression.
This could be achieved programmatically as well using a global algorithim.
**D. OTHER IDEAS
**We would welcome any other efficient solutions.