[Python-Dev] Yet another "A better story for multi-core Python" comment

[Gary Robinson]

> I guess a third possible solution, although it would probably have
> meant developing something for yourself which would have hit the same
> "programmer time is critical" issue that you noted originally, would
> be to create a module that managed the data structure in shared
> memory, and then use that to access the data from the multiple
> processes.

I think you mean, write a non-python data structure in shared memory, such as writing it in C? If so, you’re right, I want to avoid the time overhead for writing something like that. Although I have used C data in shared-memory in the past when the data structure was simple enough. It’s not a foreign concept to me — it just would have been a real nuisance in this case.

An in-memory SQLLite database would have been too slow, at least if I used any kind of ORM. Without an ORM it still would have slowed things down while making for code that’s harder to read  and write. While I have used in-memory SQLite code at times, I’m not sure how much slowdown it would have engendered in this case. 

> Your suggestion (2), of having a non-refcounted data structure is
> essentially this, doable as an extension module. The core data
> structures all use refcounting, and that's unlikely to change, but
> there's nothing to say that an extension module couldn't implement
> fast data structures with objects allocated from a pool of
> preallocated memory which is only freed as a complete block.

Again, I think you’re talking about non-Python data structures, for instance C structures, which could be written to be “fast”? Again, I want to avoid writing that kind of code. Sure, for a production project where I had more programmer time, that would be a solution, but that wasn’t my situation. And, ideally, even if I had more time, I would greatly prefer not to have to spend it on that kind of code. I like Python because it saves me time and eliminates potential bugs that are associated with language like C but not with Python (primarily memory management related). To the extent that I have to write and debug external modules in C or C++, it doesn’t.

But, my view is: I shouldn’t be forced to even think about that kind of thing. Python should simply provide a solution. The fact that the reference counters are mixed in with the data structure, so that copy-on-write causes copies to be made of the data structure shouldn’t be something I should have to discover by trial and error, or by having deep knowledge of language and OS internals before I start a project, and then have to try to find a way to work around.

Obviously, Python, like any language, will always have limitations, and therefore it’s arguable that no one should say that any language “should” do anything it doesn’t do; if I don’t like it, I can use a more appropriate language. 

But these limitations aren’t obvious up-front. They make the language less predictable to people who don’t have a deep knowledge and just want to get something done and think Python (especially combined with things like SciPy) looks like a great choice to do them. And that confusion and uncertainty has to be bad for general language acceptance. I don’t see it as  “PR issue” — I see it as a practical issue having to do with the cost of knowledge acquisition. Indeed, I personally lost a lot of time because I didn’t understand them upfront!

Solving the problem I mention here would provide real benefits even with the current multiprocessing module. But it would also make the “A better story” subinterpreter idea a better solution than it would be without it. The subinterpreter multi-core solution is a major project — it seems like it would be a shame to create that solution and still have it not solve the problem discussed here.

Anyway, too much of this post is probably spent proseletyzing for my point of view. Members of python-dev can judge it as they think fit — I don’t have much more to say unless anyone has questions.

But if I’m missing something about the solutions mentioned by Paul, and they can be implemented in pure Python, I would be much appreciative if that could be explained!

Thanks,
Gary




-- 

Gary Robinson
garyrob at me.com
http://www.garyrobinson.net

> On Sep 8, 2015, at 11:44 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> 
> On 8 September 2015 at 15:12, Gary Robinson <garyrob at me.com> wrote:
>> So, one thing I am hoping comes out of any effort in the “A better story” direction would be a way to share large data structures between processes. Two possible solutions:
>> 
>> 1) More the reference counts away from data structures, so copy-on-write isn’t an issue. That sounds like a lot of work — I have no idea whether it’s practical. It has been mentioned in the “A better story” discussion, but I wanted to bring it up again in the context of my specific use-case. Also, it seems worth reiterating that even though copy-on-write forking is a Unix thing, the midipix project appears to bring it to Windows as well. (http://midipix.org)
>> 
>> 2) Have a mode where a particular data structure is not reference counted or garbage collected. The programmer would be entirely responsible for manually calling del on the structure if he wants to free that memory. I would imagine this would be controversial because Python is currently designed in a very different way. However, I see no actual risk if one were to use an @manual_memory_management decorator or some technique like that to make it very clear that the programmer is taking responsibility. I.e., in general, information sharing between subinterpreters would occur through message passing. But there would be the option of the programmer taking responsibility of memory management for a particular structure. In my case, the amount of work required for this would have been approximately zero — once the structure was created, it was needed for the lifetime of the process.
> 
> I guess a third possible solution, although it would probably have
> meant developing something for yourself which would have hit the same
> "programmer time is critical" issue that you noted originally, would
> be to create a module that managed the data structure in shared
> memory, and then use that to access the data from the multiple
> processes. If your data structure is generic enough, you could make
> such a module generally usable - or there may even be something
> available already... I know you said that putting the data into a
> database would be too slow, but how about an in-memory Sqlite database
> (using shared memory so that there was only one copy for all
> processes)?
> 
> Your suggestion (2), of having a non-refcounted data structure is
> essentially this, doable as an extension module. The core data
> structures all use refcounting, and that's unlikely to change, but
> there's nothing to say that an extension module couldn't implement
> fast data structures with objects allocated from a pool of
> preallocated memory which is only freed as a complete block.
> 
> These suggestions are probably more suitable for python-list, though,
> as (unlike your comment on non-refcounted core data structures) they
> are things you can do in current versions of Python.
> 
> Paul