Questions about GIL and web services from a n00b

[Dan Stromberg]

On Fri, Apr 15, 2011 at 9:33 AM, Chris H
<chris.humphres at windsorcircle.com> wrote:

> 1. Are you sure you want to use python because threading is not good due to
> the Global Lock (GIL)?  Is this really an issue for multi-threaded web
> services as seems to be indicated by the articles from a Google search?  If
> not, how do you avoid this issue in a multi-threaded process to take
> advantage of all the CPU cores available?

Concurrency in Python is a largish topic.

It's true that CPython's multithreading is poor.  In fact, running a
multithreaded CPython application on n vs 2n cores can actually take
more time on the 2n cores.

However:
1) In Jython, and likely IronPython, threading is good.
2) In CPython, there's a module called "multiprocessing" that's a
little slower than a good threading implementation, but gives looser
coupling between the discrete components of your software.
Programming with multiprocessing feels similar to programming with
threads - you have safe queues, safe scalars or simple arrays in
shared memory, etc.
3) There's something called "stackless" and (similar to stackless)
"greenlets".  While stackless allows you to use thousands of threads
comfortably, it's still pretty single-core.  It's essentially a fork
of CPython, and is being made a part of PyPy.  I believe greenlets are
an attempt to bring what's good about stackless to CPython, in the
form of a C extension module.
4) I've heard that in CPython 3.2, the GIL is less troublesome, though
I've not yet heard in what way.
5) Even in CPython, I/O-bound processes are not slowed significantly
by the GIL.  It's really CPU-bound processes that are.
6) PyPy is quite a bit faster than CPython for non-concurrent
applications, but also has a GIL.
7) Cython, which compiles a dialect of Python into faster C
(especially if you give it a few type declarations), has
GIL-implications.  I've heard that Cython can selectively release the
GIL on request, but I've also heard that C extension modules always
release the GIL.  This seems contradictory, and merits further
investigation.

It's important to remember: Python != CPython.  CPython remains the
reference implementation, and runs the most Python code, but there are
other, significant implementations now.

This page looks like a summary of many (more) options:
http://wiki.python.org/moin/Concurrency/

> 2. Are there good web services frameworks available for building a REST
> based service?  I admit I have looked at web2py, Django, pyramid/pylons, and
> a few others.  SOAP seems to be pretty well supported but I'm not finding
> the same for quick development of REST based services for exchanging JSON or
> XML formatted data.  This is probably just my n00b status, but what tools
> are best for building a simple REST data exchange API?

I have no experience with REST, but Grig of SoCal Piggies did a
presentation on "restish" a while back.  You might see if you can find
something about that.