basic python threading question

[David Bolen]

"Tom" <nospam at nospam.com> writes:

> Thanks for the answer.
> 
> But, even an immutable object has its reference count modified by a binding
> operation.  And this sounds like a problem to me, since (quoting from
> section 8.1 of  API Ref):
> 
> "when two threads simultaneously increment the reference count of the same
> object, the reference count could end up being incremented only once instead
> of twice. "

Well, this is at the C level if you're doing an extension or embedding
Python.  So at that level, if you're using native C threads, you need
to pay more attention and at a lower level than within the Python code
itself.  But even that is pretty straight forward.

You excluded the most important part of the above quote - the sentence
starts off with:

    "Without the [global] lock, even the simplest operations could cause
    problems in a multi-threaded program: for example..."

That is, the portion you quote cites what could be a problem if the
lock wasn't present - it's what the global interpreter lock protects
against.

> I'm finding threading on Python to be more complicated than threading in C.
> It also seems that threading in any interpreted environment must have a
> fairly heavy overhead, and so is best avoided.

In many respects, threading on Python inside of C extension code is
much simpler than C - at least as far as interacting with the Python
core - because your code is effectively single threaded when it comes
to dealing with Python.

>From a C extension perspective, your C code is only called from within
a single Python interpreter thread at a time (and is governed by the
global interpreter lock), so while your C code runs on behalf of a
Python request, there's only one native thread running against that
interpreter.  That's also why you want to release the global lock if
you enter into a blocking operation, or else all Python threads are
blocked while the one thread calling your function waits for you to
return.

If you're embedding Python, you need to acquire the lock before
calling into Python - the same requirements, just oriented slightly
differently than an embedded situation.

Within Python code itself, the only real concerns with threading is
that you lock access to state changes in objects or mutable data types
that might be touched from multiple threads.  So that's a pretty high
level concern and reasonably easy to ensure, and certainly as easy or
easier than native C threading.

As for overhead, achieving any sort of parallel operation is going to
have some form of overhead, since state much be maintained at some
level among the various operations.  Threading is one such solution,
but certainly not the only one, and depending on the sort of
operations you are attempting to put in parallel, may or may not be
the most appropriate.  Personally, I like threading a lot, especially
when the tasks to be performed are hard to divide into repeatable
units (that could be interwoven by a single thread doing the
processing), or difficult to schedule in discrete units of operation
(in which case processing in an idle-time function might work well).
But it's really a question of a good design pattern for the job, and
differs by job.

--
-- David
-- 
/-----------------------------------------------------------------------\
 \               David Bolen            \   E-mail: db3l at fitlinxx.com  /
  |             FitLinxx, Inc.            \  Phone: (203) 708-5192    |
 /  860 Canal Street, Stamford, CT  06902   \  Fax: (203) 316-5150     \
\-----------------------------------------------------------------------/