Incremental Compression

[Eyal Lotem]

Adam DePrince wrote:

> On Sat, 2006-03-25 at 03:08 +0200, Eyal Lotem wrote:
>> Hey.
>> 
>> I have a problem in some network code. I want to send my packets
>> compressed, but I don't want to compress each packet separately (via
>> .encode('zlib') or such) but rather I'd like to compress it with regard
>> to the history of the
>> compression stream.  If I use zlib.compressobj and flush it to get the
>> packet data, I cannot continue to compress with that stream.
> 
> Yes, you can.
> 
> Help on built-in function flush:
> 
> flush(...)
>     flush( [mode] ) -- Return a string containing any remaining
> compressed data.
>     mode can be one of the constants Z_SYNC_FLUSH, Z_FULL_FLUSH,
> Z_FINISH; the
>     default value used when mode is not specified is Z_FINISH.
>     If mode == Z_FINISH, the compressor object can no longer be used
> after
>     calling the flush() method.  Otherwise, more data can still be
> compressed.
> 
> you want to call
> 
> mycompressor.flush( zlib.Z_SYNC_FLUSH )
> 
> The difference between the flushes is this:
> 
> 1. Z_SYNC_FLUSH.  This basically send enough data so that the receiver
> will get everything you put in.  This does decerase your compression
> ratio (however, in weird case when I last played with it, it helped.)
> 
> 2. Z_FULL_FLUSH.  This sends enough data so that the receiver will get
> everything you put in.  This also wipes the compressors statistics, so
> the when you pick up where you left of, the compressor will compress
> about as well as if you had just started, you are wiping its memory of
> what it saw in the past.
> 
> 3. Z_FINISH.  This is the default action, this is what is killing you.
> 
> Good luck - Adam DePrince

Thanks! That really helps.

> 
>> 
>> I cannot wait until the end of the stream and then flush, because I need
>> to flush after every packet.
>> 
>> Another capability I require is to be able to copy the compression
>> stream. i.e: To be able to create multiple continuations of the same
>> compression stream. Something like:
>> 
>> a = compressobj()
>> pre_x = a.copy()
>> x = a.compress('my_packet1')
>> # send x
>> # x was not acked yet, so we must send another packet via the pre_x
>> compressor
>> y = pre_x.compress('my_packet2')
>> 
>> Is there a compression object that can do all this?
> 
> 
> Ahh, you are trying to "pretune" the compressor before sending a little
> bit ... I think C-zlib does this, but I don't know for sure.

Yeah, but I don't need a powerful tuning, just a means to copy the
compressor's state.  I guess I'll need to write some C for this.

Thanks again!

> 
> - Adam DePrince