Python3 "pickle" vs. stdin/stdout - unable to get clean byte streams in Python 3

[John Nagle]

On 3/12/2015 5:18 PM, John Nagle wrote:
> On 3/12/2015 2:56 PM, Cameron Simpson wrote:
>> On 12Mar2015 12:55, John Nagle <nagle at animats.com> wrote:
>>> I have working code from Python 2 which uses "pickle" to talk to a
>>> subprocess via stdin/stdio.  I'm trying to make that work in Python
>>> 3.

   I'm starting to think that the "cpickle" module, which Python 3
uses by default, has a problem. After the program has been
running for a while, I start seeing errors such as

  File "C:\projects\sitetruth\InfoSiteRating.py", line 200, in scansite
    if len(self.badbusinessinfo) > 0 :                  # if bad stuff
NameError: name 'len' is not defined

which ought to be impossible in Python, and

  File "C:\projects\sitetruth\subprocesscall.py", line 129, in send
    self.writer.dump(args)                          # send data
OSError: [Errno 22] Invalid argument

from somewhere deep inside CPickle.

I got

  File "C:\projects\sitetruth\InfoSiteRating.py", line 223, in
get_rating_text
    (ratingsmalliconurl, ratinglargiconurl, ratingalttext) =
DetailsPageBuilder.getratingiconinfo(rating)
NameError: name 'DetailsPageBuilder' is not defined
(That's an imported module.  It worked earlier in the run.)

and finally, even after I deleted all .pyc files and all Python
cache directories: 	

Fatal Python error: GC object already tracked

Current thread 0x00001a14 (most recent call first):
  File "C:\python34\lib\site-packages\pymysql\connections.py", line 411
in description
  File "C:\python34\lib\site-packages\pymysql\connections.py", line 1248
in _get_descriptions
  File "C:\python34\lib\site-packages\pymysql\connections.py", line 1182
in _read_result_packet
  File "C:\python34\lib\site-packages\pymysql\connections.py", line 1132
in read
  File "C:\python34\lib\site-packages\pymysql\connections.py", line 929
in _read_query_result
  File "C:\python34\lib\site-packages\pymysql\connections.py", line 768
in query
  File "C:\python34\lib\site-packages\pymysql\cursors.py", line 282 in
_query
  File "C:\python34\lib\site-packages\pymysql\cursors.py", line 134 in
execute
  File "C:\projects\sitetruth\domaincacheitem.py", line 128 in select
  File "C:\projects\sitetruth\domaincache.py", line 30 in search
  File "C:\projects\sitetruth\ratesite.py", line 31 in ratedomain
  File "C:\projects\sitetruth\RatingProcess.py", line 68 in call
  File "C:\projects\sitetruth\subprocesscall.py", line 140 in docall
  File "C:\projects\sitetruth\subprocesscall.py", line 158 in run
  File "C:\projects\sitetruth\RatingProcess.py", line 89 in main
  File "C:\projects\sitetruth\RatingProcess.py", line 95 in <module>

That's a definite memory error.

So something is corrupting memory.  Probably CPickle.

All my code is in Python. Every library module came in via "pip", into a
clean Python 3.4.3 (32 bit) installation on Win7/x86-64.
Currently installed packages:

beautifulsoup4 (4.3.2)
dnspython3 (1.12.0)
html5lib (0.999)
pip (6.0.8)
PyMySQL (0.6.6)
pyparsing (2.0.3)
setuptools (12.0.5)
six (1.9.0)

And it works fine with Python 2.7.9.

Is there some way to force the use of the pure Python pickle module?
My guess is that there's something about reusing "pickle" instances
that botches memory uses in CPython 3's C code for "cpickle".

				John Nagle