[Python-ideas] Thread IPC idea: Quipe? Sockqueue? (Re: Python and Concurrency)

[Ron Adam]

Jim Jewett wrote:
> On 3/28/07, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
>> I was thinking about thread killing, and why we
>> think it's okay to kill OS processes but not
>> threads.
> 
> [Suggestion for a Queue variant that knows when one end is dead]
> 
> I think the bigger problem is when threads don't restrict themselves
> to queues, but just use the same memory directly.  If a thread dies in
> the middle of an "atomic" action, other threads will see corrupted
> memory.  If a process dies then, nobody else would be able to see the
> memory anyhow.
> 
> What we really need is a Task object that treats shared memory
> (perhaps with a small list of specified exceptions) as immutable.

I think a set of all of these as tools would be good.  They are all 
different parts of the same elephant.  And I don't see why it needs to be a 
single unified thing.

* A 'better' task object for easily creating tasks.
      + We have a threading object now. (Needs improving.)

* A message mechanism (Que-pipe) for getting the status of a task.
      + In and out message ques for communicating with a thread.
      + A way to wait on task events (messages) nicely.
      + A way for exceptions to propagate out of task objects.

* Shared memory -
      + Prevent names from being rebound
      + Prevent objects from being altered

(It isn't just about objects being immutable, but also about name's not 
being rebound to other objects. Unless you want to pass objects references 
for every object to every task?  Or you trust that any imported code will 
play nice.)


Would the following be a good summery of the issues?

(Where the terms used mean)

     frozen: object can't be altered while frozen
     locked: name can't be rebound to another object


Threading issues:  (In order of difficulty)

     1. Pass immutable objects back and forth.
           + Works now.

     2. Share immutable objects by-ref.
           + Works now.

     3. Pass mutable "deep" copies back and forth.
           ? Works now. (but not for all objects?)

     4. Pass frozen mutable objects.
           - Needs freezable/unfreezable mutable objects.
             (Not the same as making an immutable copy.)

     5. Share immutable object in a shared name space.
           - Needs name locking.

     6. Share "frozen" mutable objects in shared name space.
           - Needs name locking
           - Needs freezable mutable objects

     7. pass mutable objects.
           - Has conflicts with shared access.

     8. Share mutable object by-ref.
           - Has conflicts. (same as #7.)

     9. Share mutable object in shared name space.
           - Needs name locking.
           - Has conflicts.


If we can make the first 6 of these work, that may be enough. 7,8 and 9 
have to do with race conditions and other simultaneous data access issues.

Name locking might work like this: Doing 'lock <name>' and 'unlock <name>' 
could just move the name to and from a locked name space in the same scope. 
Attempting to rebind a name while it's in a locked name space would raise 
an exception.

The point is, rather than attach a lock to each individual name, it may be 
much easier, faster, and save memory by having a new name spaces for this. 
You could also pass a locked-name-space reference to a task all at once 
like we pass locals or globals now.

It doesn't identify who locked what, so unlocking a name used by a thread 
would be in the "if it hurts, don't do that" category.  If locking or 
unlocking a name in outside scopes is disallowed, then knowing who locked 
what won't be a problem.


Freezing would be some way of preventing an object from being changed.  It 
isn't concerned with the name it is referenced with.  And it should not 
require copying the object.  Part of that may be made easier by locking 
names.(?)

Frozen objects may be useful in other ways besides threading.(?)

Names locked to immutable objects act like constants so they may have other 
uses as well.


Cheers,
    Ron




> If you're willing to rely on style guidelines, then you can already
> get this today.
> 
> If you want safety, and efficiency ... that may be harder to do as an addon.