urllib slow on FreeBSD 4.7? sockets too

[Skip Montanaro]

    Mike> As was pointed out, all the delay is in the reading, and it really
    Mike> only happens when dealing with these file-wrapped sockets as
    Mike> returned by urlopen(). They can be on the localhost, even.

What happens if you reach into the file wrapper and grab the raw socket?

    Mike> Result on Mandrake 8.1, Python 2.1.1:
    Mike> bytes: 4343332; time: 0.141s (29992 KB/s)
    Mike> ...and subsequent runs are about DOUBLE that speed.

    Mike> Result on FreeBSD 4.7, Python 2.2.1 (on comparable hardware):
    Mike> bytes: 4343332; time: 9.404s (451 KB/s)
    Mike> ...and subsequent runs are the same speed.

This suggests it's a FreeBSD problem more than a Python problem.

    Mike> def listen():
    Mike>     try:
    Mike>         (conn, addr) = sock.accept()
    Mike>         starttime = time.time()
    Mike>         fd = conn.makefile('r')
    Mike>         bytes = fd.read()
    Mike>         elapsed = time.time() - starttime
    Mike>         l = len(bytes)
    Mike>         print "read %d bytes in %0.3fs (%d KB/s)" % (l, elapsed,
    Mike> l/1024./elapsed)
    Mike>         conn.close()
    Mike>     except:
    Mike>         raise

Again, what happens if you read from the socket instead of the file object
returned by makefile()?  Also, what happens if you break the read up in to a
loop, reading smaller chunks?

Did you build Python with or without pymalloc?  Can you try the opposite?

-- 
Skip Montanaro - skip at pobox.com
http://www.mojam.com/
http://www.musi-cal.com/