zlib, gzip and HTTP compression.

[Alan Kennedy]

Andrew, 

Yes, that was exactly the problem. I am running on win2k and winnt4.
By adding the lines

import msvcrt
msvcrt.setmode(sys.stdout.fileno(), os.O_BINARY)

The entire problem went away!

I never realised that stdout was doing character translation. So using

sys.stdout.write(compressedBuf)

thinking that this was giving me a raw binary write, and bypassing
possible "print" related character translation, was the wrong
approach.

So the character translation was happening on the file descriptor
itself? Is that standard C behaviour? Has it been that long since I
coded in C? :-) Even then, I never did any stdio coding on dos/win32.

After many hours of struggling with this, I hadn't really gotten any
further. I had discovered that some files compressed and transmitted
correctly with the code that I had, whereas other did not. I'm sure
now that if I go back, I'll find that the gzipped files that didn't
work contained 0x0A.

Whew! That's a relief!

Thanks to all who helped out with hints, tips, suggestions and
solutions.

Sincerely,

Alan.


Andrew MacIntyre <andymac at bullseye.apana.org.au> wrote in message news:<mailman.1010859382.21963.python-list at python.org>...
> On 11 Jan 2002, Alan Kennedy wrote:
> 
> > There is obviously some small detail that I am missing, such as
> > character translation during the print statement(?), one extra byte
> > need somewhere, etc?
> 
> ...
> 
> > ------------------------------------------
> > #! C:/python21/python.exe
> 
> ...
> 
> From the above I'd guess that you're using Cygwin. On Windows (MSVC or
> Cygwin) or OS/2 (VACPP or EMX/gcc), stdout (which is what print uses) will
> be in text mode - ie newline characters get translated to CRLF.  If your
> compressed data has a newline character (LF) in it, this will result in a
> corrupted output stream.
> 
> You'll need to find a way to change stdout to binary mode ('wb').  At the
> moment I don't know of a practical way to do this with either OS/2 port,
> and have no information about the Win32 ports.