Embedding Python, threading and scalability

[Jimmy Retzlaff]

Aahz (aahz at pythoncraft.com) wrote:
>Wenning Qiu <wenning_qiu at csgsystems.com> wrote:
>>
>>Has anyone on this list run into the same problem that I have, or does
>>anyone know of any plan of totally insulating multiple embedded Python
>>interpreters?
>
>Sure!  Use multiple processes.

This can have further scalability benefits. The first time I ran up
against the GIL, I grumbled my way through tweaking my code to use
multiple processes and an IPC mechanism (which was surprisingly easy).
After I finished, I realized that nothing about the IPC mechanism I
happened to choose limited my solution to one machine and suddenly my
app was running on 7 CPUs instead of the 2 CPUs I had originally hoped
for. On just 2 CPUs it was probably a little slower than it might have
been because of the IPC overhead, but 7 CPUs put my app in a totally
different league.

Jimmy