Python3 "pickle" vs. stdin/stdout - unable to get clean byte streams in Python 3

[Steven D'Aprano]

John Nagle wrote:

>    I'm starting to think that the "cpickle" module, which Python 3
> uses by default, has a problem. After the program has been
> running for a while, I start seeing errors such as
> 
>   File "C:\projects\sitetruth\InfoSiteRating.py", line 200, in scansite
>     if len(self.badbusinessinfo) > 0 :                  # if bad stuff
> NameError: name 'len' is not defined
> 
> which ought to be impossible in Python, and

"Impossible"?

py> len
<built-in function len>
py> import __builtin__  # use builtins in Python 3
py> del __builtin__.len
py> len
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'len' is not defined


Why something is deleting builtins len is a mystery. Sounds to me that your
Python installation is borked.


>   File "C:\projects\sitetruth\subprocesscall.py", line 129, in send
>     self.writer.dump(args)                          # send data
> OSError: [Errno 22] Invalid argument
> 
> from somewhere deep inside CPickle.

Why do you say "deep inside CPickle"? The traceback says 
C:\projects\sitetruth\subprocesscall.py

Is it possible you have accidentally shadowed the CPickle module with
something? What does this say?

import cPickle
print cPickle.__file__

Use _pickle in Python 3.


> I got
> 
>   File "C:\projects\sitetruth\InfoSiteRating.py", line 223, in
> get_rating_text
>     (ratingsmalliconurl, ratinglargiconurl, ratingalttext) =
> DetailsPageBuilder.getratingiconinfo(rating)
> NameError: name 'DetailsPageBuilder' is not defined
> (That's an imported module.  It worked earlier in the run.)
> 
> and finally, even after I deleted all .pyc files and all Python
> cache directories:
> 
> Fatal Python error: GC object already tracked
> 
> Current thread 0x00001a14 (most recent call first):
>   File "C:\python34\lib\site-packages\pymysql\connections.py", line 411
> in description
>   File "C:\python34\lib\site-packages\pymysql\connections.py", line 1248
> in _get_descriptions
>   File "C:\python34\lib\site-packages\pymysql\connections.py", line 1182
> in _read_result_packet
>   File "C:\python34\lib\site-packages\pymysql\connections.py", line 1132
> in read
>   File "C:\python34\lib\site-packages\pymysql\connections.py", line 929
> in _read_query_result
>   File "C:\python34\lib\site-packages\pymysql\connections.py", line 768
> in query
>   File "C:\python34\lib\site-packages\pymysql\cursors.py", line 282 in
> _query
>   File "C:\python34\lib\site-packages\pymysql\cursors.py", line 134 in
> execute
>   File "C:\projects\sitetruth\domaincacheitem.py", line 128 in select
>   File "C:\projects\sitetruth\domaincache.py", line 30 in search
>   File "C:\projects\sitetruth\ratesite.py", line 31 in ratedomain
>   File "C:\projects\sitetruth\RatingProcess.py", line 68 in call
>   File "C:\projects\sitetruth\subprocesscall.py", line 140 in docall
>   File "C:\projects\sitetruth\subprocesscall.py", line 158 in run
>   File "C:\projects\sitetruth\RatingProcess.py", line 89 in main
>   File "C:\projects\sitetruth\RatingProcess.py", line 95 in <module>
> 
> That's a definite memory error.
>
> So something is corrupting memory.  Probably CPickle.


> All my code is in Python. Every library module came in via "pip", into a
> clean Python 3.4.3 (32 bit) installation on Win7/x86-64.
> Currently installed packages:
> 
> beautifulsoup4 (4.3.2)
> dnspython3 (1.12.0)
> html5lib (0.999)
> pip (6.0.8)
> PyMySQL (0.6.6)
> pyparsing (2.0.3)
> setuptools (12.0.5)
> six (1.9.0)
> 
> And it works fine with Python 2.7.9.
> 
> Is there some way to force the use of the pure Python pickle module?

Try renaming the _pickle module. This works on Linux:

mv /usr/local/lib/python3.3/lib-dynload/_pickle.cpython-33m.so /usr/local/lib/python3.3/lib-dynload/_pickle.cpython-33m.so~


> My guess is that there's something about reusing "pickle" instances
> that botches memory uses in CPython 3's C code for "cpickle".

How are you reusing instances?


-- 
Steven