dual processor

[Michael Sparks]

Jeremy Jones wrote:

> Michael Sparks wrote:
>>Steve Jorgensen wrote:
>>>On 05 Sep 2005 10:29:48 GMT, Nick Craig-Wood <nick at craig-wood.com> wrote:
>>>
>>>    
>>>
>>>>Jeremy Jones <zanesdad at bellsouth.net> wrote:
>>>>> One Python process will only saturate one CPU (at a time) because
>>>>> of the GIL (global interpreter lock).
>>>>I'm hoping python won't always be like this.

>>>I don't get that.  Python was never designed to be a high performance
>>>language, so why add complexity to its implementation by giving it
>>>high-performance capabilities like SMP?

>>It depends on personal perspective.
>>
> Ummmm....not totally.  It depends on what you're doing.  

Yes, it does. Hence why I said personal perspective. 

>  Sort of.  Steve
> brings up an interesting argument of making the language do some of your
> thinking for you.  Maybe I'll address that momentarily....

Personally I think that the language and tools will have to help. I'm
working on the latter, I hope the GIL goes away to help with the former, 
but does so in an intelligent manner. Why am I working on the latter? 

I work with naturally concurrent systems all the time, and they're
concurrent not for performance reasons, but simply because that's what they
are. And as a result I want concurrency easy to deal with, and efficiency as
a secondary concern. However in that scenario, having multiple CPU's not
being utilised sensibly *is* a concern to me.

> I'm not saying I wish the GIL would stay around.  I wish it would go.
> As the price of computers goes down, the number of CPUs per computer
> goes up, and the price per CPU in a single system goes down, the ability
> to utilize a bunch of CPUs is going to become more important.  

> And maybe  
> Steve's magical thinking programming language will have a ton of merit.

I see no reason to use such derisory tones, though I'm sure you didn't mean
it that way. (I can see you mean it as extreme skepticism though :-)

>>If in a few years time we all have
>>machines with multiple cores (eg the CELL with effective 9 CPUs on a chip,
>>albeit 8 more specialised ones), would you prefer that your code *could*
>>utilise your hardware sensibly rather than not.
>>  
>>

> But let me play devil's advocate for
> a sec.  Let's say we *could* fully utilize a multi CPU today with
> Python. 
...
> I would almost bet money that the majority of code would 
> not be helped by that at all.  

Are you so sure? I suspect this is due to you being used to writing code
that is designed for a single CPU system. What if you're basic model of
system creation changed to include system composition as well as
function calls? Then each part of the system you compose can potentially
run on a different CPU. Take the following for example:

(sorry for the length, I prefer real examples :-)

Graphline(
     EXIT = ExceptionRaiser("FORCED SYSTEM QUIT"),
     MOUSE = Multiclick(caption="",
                        position=(0,0),
                        transparent=True,
                        msgs = [ "", "NEXT", "FIRST", "PREV", "PREV","NEXT" ],
                        size=(1024,768)),
     KEYS = KeyEvent(outboxes = { "slidecontrol" : "Normal place for message",
                                  "shutdown" : "Place to send some shutdown messages",
                                  "trace" : "Place for trace messages to go",
                                },
                     key_events = {112: ("PREV", "slidecontrol"),
                                   110: ("NEXT","slidecontrol"),
                                   113: ("QUIT", "shutdown"),
                                  }),
     SPLITTER = Splitter(outboxes = {"totimer" : "For sending copies of key events to the timer",
                                     "tochooser" : "This is the primary location for key events",
                                    }),
     TIMER = TimeRepeatMessage("NEXT",3),
     FILES = Chooser(items = files, loop=True),
     DISPLAY = Image(size=(1024,768),
                     position=(0,0),
                     maxpect=(1024,768) ),
     linkages = {
        ("TIMER", "outbox") : ("FILES", "inbox"),

        ("MOUSE", "outbox") : ("SPLITTER", "inbox"),
        ("KEYS", "slidecontrol") : ("SPLITTER", "inbox"),
        ("SPLITTER", "tochooser") : ("FILES", "inbox"),
        ("SPLITTER", "totimer") : ("TIMER", "reset"),

        ("KEYS", "shutdown") : ("EXIT", "inbox"),
        ("FILES", "outbox") : ("DISPLAY", "inbox"),
     }
).run()

What does that do? Its a slideshow program for display pictures. There's a
small amount of setup before this (identifying files for display, imports, etc), 
but that's by far the bulk of the system.

That's pure python code (aside from pygame),  and the majority of the code
is written single threaded, with very little concern for concurrency. However
the code above will naturally sit on a 7 CPU system and use all 7 CPUs (when
we're done). Currently however we use generators to limit the overhead in
a single CPU system, though if the GIL was eliminated sensibly, using threads
would allow the same code above to run on a multi-CPU system efficientally.

It probably looks strange, but it's really just a logical extension of the
Unix command line's pipelines to allow multiple pipelines. Similarly, from
a unix command line perspective, the following will automatically take
advantage of all the CPU's I have available:

   (find |while read i; do md5sum $i; done|cut -b-32) 2>/dev/null |sort

And a) most unix sys admins I know find that easy (probably the above
laughable) b) given a multiprocessor system will probably try to maximise
pipelining c) I see no reason why sys admins should be the only people
writing programs who use concurrency without thinking about it :-)

I *do* agree it takes a little getting used to, but then I'm sure the same
was true for many people who learnt OOP after learning to program. Unlike
people who learnt OO at the time they started learning programming and just
see it as a natural part of a language.

> There are benefits to writing code in C and Java apart from
> concurrency.  Most of them are massochistic, but there are benefits
> nonetheless.  For my programming buck, Python wins hands down.
> 
> But I agree with you.  Python really should start addressing solutions
> for concurrent tasks that will benefit from simultaneously utilizing
> multiple CPUs.

That's my point too. I don't think our opinions really diverge that far :)

Best Regards,


Michael.