[Python-ideas] solving multi-core Python

[M.-A. Lemburg]

On 25.06.2015 11:35, Sturla Molden wrote:
> Nathaniel Smith <njs at pobox.com> wrote:
> 
>> Continuing my vague and uninformed impressions, I suspect that this
>> would actually be relatively easy to fix by hooking the import system
>> to do something more intelligent, like nominate one node as the leader
>> and have it do the file lookups and then tell everyone else what it
>> found (via the existing message-passing systems). 
> 
> There are two known solutions. One is basically what you describe. The
> other, which at least works on IBM blue brain, is to import modules from a
> ramdisk. It seems to be sufficient to make sure whatever is serving the
> shared disk can deal with the 100k client DDoS.

Another way to solve this problem may be to use our eGenix PyRun
which embeds modules right in the binary. As a result, all reading
is done from the mmap'ed binary and automatically shared between
processes by the OS:

http://www.egenix.com/products/python/PyRun/

I don't know whether this actually works on an IBM Blue Brain with
100k clients - we are not fortunate enough to have access to one of
those machines :-)

Note: Even though the data reading is shared, the resulting
code and modules objects are, of course, not shared, so you still
have the overhead of using up memory for this, unless you init
your process cluster using fork() after you've imported all
necessary modules (then you benefit from the copy-on-write
provided by the OS - code objects usually don't change after
they have been created).

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jun 25 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> mxODBC Plone/Zope Database Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2015-06-25: Released mxODBC 3.3.3 ...             http://egenix.com/go79
2015-06-16: Released eGenix pyOpenSSL 0.13.10 ... http://egenix.com/go78
2015-07-20: EuroPython 2015, Bilbao, Spain ...             25 days to go

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/