urllib slow on FreeBSD 4.7? sockets too

[Mike Brown]

"Skip Montanaro" <skip at pobox.com> wrote:
> What happens if you reach into the file wrapper and grab the raw socket?

How do I do that? u.fp is as close as I seem to be able to get.

u = urllib.urlopen('http://localhost/4MBfile')
print u
print
print dir(u)
print
print u.fp
print
print dir(u.fp)
u.close()

<addinfourl at 135862508 whose fp = <open file '<socket>', mode 'rb' at
0x8191900>>

['__doc__', '__init__', '__module__', '__repr__', 'close', 'fileno', 'fp',
'geturl', 'headers', 'info', 'read', 'readline', 'readlines', 'url']

<open file '<socket>', mode 'rb' at 0x8191900>

['__class__', '__delattr__', '__doc__', '__getattribute__', '__hash__',
'__init__', '__iter__', '__new__', '__reduce__', '__repr__', '__setattr__',
'__str__', 'close', 'closed', 'fileno', 'flush', 'isatty', 'mode', 'name',
'read', 'readinto', 'readline', 'readlines', 'seek', 'softspace', 'tell',
'truncate', 'write', 'writelines', 'xreadlines']


> Again, what happens if you read from the socket instead of the file object
> returned by makefile()?

Maybe 2 KB/s improvement:

read 501760 bytes in 0.041s (11923 KB/s)
read 512000 bytes in 0.043s (11671 KB/s)
read 522240 bytes in 0.045s (11214 KB/s)
read 532480 bytes in 0.093s (5609 KB/s)
read 542720 bytes in 0.046s (11633 KB/s)
read 552960 bytes in 0.047s (11459 KB/s)
read 563200 bytes in 0.048s (11392 KB/s)
read 573440 bytes in 0.072s (7725 KB/s)
read 583680 bytes in 0.090s (6304 KB/s)
read 593920 bytes in 0.052s (11241 KB/s)
read 604160 bytes in 0.054s (11022 KB/s)

> Also, what happens if you break the read up in to a
> loop, reading smaller chunks?

No change.

bytes: 4343332; time: 9.064s (467 KB/s) using u.read(1024)
bytes: 4343332; time: 9.050s (468 KB/s) using u.read(2048)
bytes: 4343332; time: 9.050s (468 KB/s) using u.read(4096)

> Did you build Python with or without pymalloc?

I installed it from the FreeBSD packages collection. If pymalloc is not
normally built-in, then it's probably not in the package.

> Can you try the opposite?

I'd rather not obliterate my current installation, no.