MacOS 10.9.2: threading error using python.org 2.7.6 distribution

[Chris Angelico]

On Mon, Apr 28, 2014 at 12:16 AM, Matthew Pounsett
<matt.pounsett at gmail.com> wrote:
> On Friday, 25 April 2014 10:05:03 UTC-4, Chris Angelico  wrote:
>> First culprit I'd look at is the mixing of subprocess and threading.
>> It's entirely possible that something goes messy when you fork from a
>> thread.
>
> I liked the theory, but I've run some tests and can't reproduce the error that way.  I'm using all the elements in my test code that the real code runs, and I can't get the same error.  Even when I deliberately break things I'm getting a proper exception with stack trace.
>

In most contexts, "thread unsafe" simply means that you can't use the
same facilities simultaneously from two threads (eg a lot of database
connection libraries are thread unsafe with regard to a single
connection, as they'll simply write to a pipe or socket and then read
a response from it). But processes and threads are, on many systems,
linked. Just the act of spinning off a new thread and then forking can
potentially cause problems. Those are the exact sorts of issues that
you'll see when you switch OSes, as it's the underlying thread/process
model that's significant. (Particularly of note is that Windows is
*very* different from Unix-based systems, in that subprocess
management is not done by forking. But not applicable here.)

You may want to have a look at subprocess32, which Ned pointed out. I
haven't checked, but I would guess that its API is identical to
subprocess's, so it should be a drop-in replacement ("import
subprocess32 as subprocess"). If that produces the exact same results,
then it's (probably) not thread-safety that's the problem.

>> Separately: You're attempting a very messy charset decode there. You
>> attempt to decode as UTF-8, errors ignored, and if that fails, you log
>> an error... and continue on with the original bytes. You're risking
>> shooting yourself in the foot there; I would recommend you have an
>> explicit fall-back (maybe re-decode as Latin-1??), so the next code is
>> guaranteed to be working with Unicode. Currently, it might get a
>> unicode or a str.
>
> Yeah, that was a logic error on my part that I hadn't got around to noticing, since I'd been concentrating on the stuff that was actively breaking.  That should have been in an else: block on the end of the try.
>

Ah good. Keeping bytes versus text separate is something that becomes
particularly important in Python 3, so I always like to encourage
people to get them straight even in Py2. It'll save you some hassle
later on.

ChrisA