[Python-Dev] Fw: SocketServer and makefile() [from comp.lang.python]

[Greg Stein]

I ran into this same problem on the client side.

The server does a makefile() so that it can do readline() to fetch the HTTP
request line and then the MIME headers. The *problem* is that if you do
something like:

    f = sock.makefile()
    line = f.readline()
    data = sock.recv(1000)

You're screwed if you have buffering enabled. "f" will read in a bunch of
data -- past the end of the line. That data now sits inside f's buffer and
is not available to the sock.recv() call.

If you forget about sock and just stick to f, then you'd be okay. But
SocketServer and/or BaseHTTPServer doesn't -- it uses both objects to do the
reading.

Solution? Don't use rfile for reading, but go for the socket itself. Or
revamp the two classes to forget about the socket once the files (wfile and
rfile) are created. The latter might not be possible, tho.

Dunno why the unbuffered reading would be slow. I'd think it would still
read large chunks at a time when you request it.

Cheers,
-g

On Thu, Aug 31, 2000 at 07:47:45PM +0200, Fredrik Lundh wrote:
> iirc, I've been bitten by this a couple of times too
> (before I switched to asyncore...)
> 
> any special reason why the input socket is unbuffered
> by default?
> 
> </F>
> 
> ----- Original Message ----- 
> From: "Andy Bond" <bond@dstc.edu.au>
> Newsgroups: comp.lang.python
> Sent: Thursday, August 31, 2000 8:41 AM
> Subject: SocketServer and makefile()
> 
> 
> > I've been working with BaseHTTPServer which in turn uses SocketServer to
> > write a little web server.  It is used to accept PUT requests of 30MB chunks
> > of data.  I was having a problem where data was flowing at the rate of
> > something like 64K per second over a 100MB network.  Weird.  Further tracing
> > showed that the rfile variable from SocketServer (used to suck in data to
> > the http server) was created using makefile on the original socket
> > descriptor.  It was created with an option of zero for buffering (see
> > SocketServer.py) which means unbuffered.
> > 
> > Now some separate testing with socket.py showed that I could whip a 30MB
> > file across using plain sockets and send/recv but if I made the receivor use
> > makefile on the socket and then read, it slowed down to my 1 sec per 64K.
> > If I specify a buffer (something big but less than 64K ... IP packet size?)
> > then I am back in speedy territory.  The unbuffered mode seems almost like
> > it is sending the data 1 char at a time AND this is the default mode used in
> > SocketServer and subsequently BaseHTTPServer ...
> > 
> > This is on solaris 7, python 1.5.2.  Anyone else found this to be a problem
> > or am I doing something wrong?
> > 
> > andy
> 
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev@python.org
> http://www.python.org/mailman/listinfo/python-dev

-- 
Greg Stein, http://www.lyra.org/