Python 2.6.4 - Urllib2 - Windows XP - Reading streaming HTTP source kills network card ... (believe it or not)

[MRAB]

Sandy Walsh wrote:
> This is very odd. Hopefully someone can shed some insight. I've tried 
> this with Python 2.5.2 and recently upgraded to 2.6.4 and see the same 
> problem.
> 
> I'm running on Windows XP sp3. I'm interfacing with an IP camera that 
> streamed jpeg frames at 10fps over HTTP.
> 
> The format of the stream is:
> 
> 4 bytes - size of the frame N
> N bytes - the jpeg frame
> 
> I have the following program to read the data. It works fine for about 
> 30-40 iterations and then my NIC dies. All connectivity to the outside 
> world goes away until I need to reboot. So, first I thought it was a 
> driver problem. I've replaced the NIC, tried other drivers, you name it 
> ... same problem.
> 
> I've tried it on another machine and while it doesn't take down the NIC, 
> all communications to the camera after a while fail with "connection 
> reset by peer" exception.
> 
> The camera attempts to keep sending frame after frame (one every 100ms), 
> but I'm only interested in the first frame. After I grab it I kill the 
> connection to the camera. There is still data coming in. I assume it's 
> in HTTP Chunked format, but have not put Wireshark on it yet. I suspect 
> Python doesn't like me killing the connection when there is still data 
> coming down ... but why would it take down my NIC too?
> 
> The code is very simple:
> 
> #-------------------------------------
> import urllib2
> import struct
> import time
> import datetime
> 
> ip='192.168.1.189'
> username='user'
> password='password'
> 
> password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
> top_level_url = "http://%s" % ip
> password_mgr.add_password(None, top_level_url, username, password)
> 
> handler = urllib2.HTTPBasicAuthHandler(password_mgr)
> 
> opener = urllib2.build_opener(handler)
> urllib2.install_opener(opener)
> 
> url = "http://%s/user/img_stream0.cgi" % (ip, )
> 
> while 1:
>     response = urllib2.urlopen(url)
> 
>     size_bytes = response.read(4)
>     size, = struct.unpack("i", size_bytes)
> 
>     frame = response.read(size)
>     response.close()
> 
>     print "Got ", datetime.datetime.now()
>     time.sleep(1)
> #-------------------------------------
> 
> Which gives the following output:
> 
> testpull.py
> Got  2010-01-12 15:30:08.125000
> Got  2010-01-12 15:30:09.453000
> Got  2010-01-12 15:30:10.812000
> Got  2010-01-12 15:30:12.156000
> Got  2010-01-12 15:30:13.515000
> Got  2010-01-12 15:30:14.890000
> Got  2010-01-12 15:30:16.265000
> Got  2010-01-12 15:30:17.625000
> Got  2010-01-12 15:30:19.031000
> Got  2010-01-12 15:30:20.390000
> Got  2010-01-12 15:30:21.765000
> Got  2010-01-12 15:30:23.093000
> Got  2010-01-12 15:30:24.437000
> Got  2010-01-12 15:30:25.765000
> Got  2010-01-12 15:30:27.109000
> Got  2010-01-12 15:30:28.750000
> Got  2010-01-12 15:30:30.078000
> Got  2010-01-12 15:30:31.437000
> Got  2010-01-12 15:30:32.781000
> Got  2010-01-12 15:30:34.546000
> Got  2010-01-12 15:30:35.906000
> Got  2010-01-12 15:30:37.250000
> Got  2010-01-12 15:30:38.609000
> Got  2010-01-12 15:30:39.953000
> Got  2010-01-12 15:30:41.281000
> Got  2010-01-12 15:30:42.578000
> Got  2010-01-12 15:30:43.921000
> Got  2010-01-12 15:30:45.250000
> Got  2010-01-12 15:30:46.562000
> Got  2010-01-12 15:30:47.890000
> Got  2010-01-12 15:30:49.265000
> Got  2010-01-12 15:30:50.625000
> Got  2010-01-12 15:30:51.968000
> Got  2010-01-12 15:30:53.328000
> Got  2010-01-12 15:30:54.734000
> Traceback (most recent call last):
>   File "C:\dev\5110Snapshot\testpull.py", line 22, in <module>
>     response = urllib2.urlopen(url)
>   File "C:\Python26\lib\urllib2.py", line 124, in urlopen
>     return _opener.open(url, data, timeout)
>   File "C:\Python26\lib\urllib2.py", line 389, in open
>     response = self._open(req, data)
>   File "C:\Python26\lib\urllib2.py", line 407, in _open
>     '_open', req)
>   File "C:\Python26\lib\urllib2.py", line 367, in _call_chain
>     result = func(*args)
>   File "C:\Python26\lib\urllib2.py", line 1146, in http_open
>     return self.do_open(httplib.HTTPConnection, req)
>   File "C:\Python26\lib\urllib2.py", line 1121, in do_open
>     raise URLError(err)
> urllib2.URLError: <urlopen error [Errno 10060] A connection attempt 
> failed because the connected party did not properly
> respond after a period of time, or established connection failed because 
> connected host has failed to respond>
> 
> At which point all connectivity on the PC is dead.
> 
> This is truly bizarre.
> 
> Anyone have any insights as to what might be happening. Or is there 
> something blatantly wrong with my code?
> 
> Help!
> 
Have you tried keeping the connection open and just reading and
discarding the frames you don't want?