questions regarding stack size use for multi-threaded python programs

[Andrew MacIntyre]

Gabriel Genellina wrote:
> En Mon, 09 Nov 2009 16:05:31 -0300, Eyal Gordon <eyal.gordon at gmail.com>
> escribió:
> 
>> background:
>> we are using python 2.4.3 on CentOS 5.3 with many threads - and our 
>> shell's
>> default stack size limit is set to 10240KB (i.e. ~10MB).
>>
>> we noticed that python's Threading module appears to create threads with
>> this value as their stack size (we ran a sample program that creates 10
>> threads and measured its virtual memory size, then reduced the stack size
>> limit of the shell to 5120KB - and saw that the program's virtual memory
>> size was reduced by ~50MBs).
>>
>> the problem:
>> our program uses numerous threads, and thus the virtual memory size 
>> gets to
>> be very large. we would like to reduce the size of the stack to reduce 
>> this
>> size. we were looking for information about recommendation for the stack
>> size to use, but found none.
> 
> You can set the thread stack size (for threads that are going to be
> created, not existing threads) using threading.stack_size(SIZE)
> http://docs.python.org/library/threading.html#threading.stack_size

Sadly for the OP, that capability was introduced in Python 2.5.  Prior
to that, the only way to adjust thread stack size is via a compile time
option (THREAD_STACK_SIZE).  In the absence of that compile time option,
the platform default is used - on pthread systems (incl Linux) refer to
the manpage for pthread_attr_setstacksize() for more info, but I believe
what the OP reports could reasonably be expected for his platform.

The threading.stack_size() support would not be hard to backport to
Python 2.4.

>> questions:
>> 1. is there some rule-of-thumb for the recommended stack size for python
>> programs of various sorts?

There is no rule of thumb because there are too many variables amongst
the supported platforms.  Extensive testing would be required to
validate any selected size.

I would suggest starting with 1MB and working down.  On a 32bit platform
I would suggest that 64kb might be adequate for child threads but most
likely not for the primary thread (the thread that the Python
interpreter starts in).  If your app uses regexes, your stack space
requirements are likely to be larger.

FWIW the default thread stack size on Win32 is 1MB which anecdotally
seems sufficient for a wide variety of applications on that platform.

{...}

>> 4. would the size of the stacks (which are probably not really 
>> allocated by
>> the linux virtual memory sub-system, unless used) have a noticeable
>> performance effect on a python program? same question regarding the 
>> use of a
>> large number of threads?
> 
> I think it doesn't matter, unless you create so many threads as to exhaust
> the available addressing space (in 32bits, 4GB address space and 10MB per
> thread means 400 threads maximum).

The performance of course will degrade once the committed memory in the
active working set exceeds the available real memory, as the OS will
start swapping.  How the OS treats unused stack space will affect that
threshold.

-- 
-------------------------------------------------------------------------
Andrew I MacIntyre                     "These thoughts are mine alone..."
E-mail: andymac at bullseye.apana.org.au  (pref) | Snail: PO Box 370
        andymac at pcug.org.au             (alt) |        Belconnen ACT 2616
Web:    http://www.andymac.org/               |        Australia