Thread Locking issue - Can't allocate lock (sem_init fail)

[Philip Semanchuk]

On Dec 15, 2008, at 4:56 AM, jamskip at googlemail.com wrote:

> Hi all,
>
> I have a peculiar problem with a multithreaded program of mine
> (actually I've sort of inherited it). Before i show you the error,
> here's a litle background. Its a program to check email addresses are
> valid, and its main task is to verify the domain names.
>
> Here's the basic functionality:
>
> * The prog has a list of domains it has seen before which is read into
> memory (the 'rollover').
> * A new list of emails is read-in from a file (to a queue) and is
> checked against the rollover.
> * If we've seen the domain before then update the existing entry.
> * If we've not seen the domain before, add it.
>
> The program is multithreaded to speed up the processing...there are
> input and output Queues.
>
> Now, each domain entry is an class object containing various bits of
> info. Each domain class also has its own lock, so that only one thread
> can modify each domain at a time.
>
> I'm load-testing the program with a sample of 1 million email
> addresses and when i hit about the 500,000 mark i get a locking
> error...
>
>    sem_init: No space left on device
>    Exception in thread Thread-2:
>    Traceback (most recent call last):
>      File "/usr/local/lib/python2.4/threading.py", line 442, in
> __bootstrap
>        self.run()
>      File "/usr/local/lib/python2.4/threading.py", line 422, in run
>        self.__target(*self.__args, **self.__kwargs)
>      File "jess.py", line 250, in worker
>        record.result = function( id, record.value )
>      File "jess.py", line 291, in action
>        found_domain = domains.addNewDomain( domain_name )
>      File "jess.py", line 123, in addNewDomain
>        self.domain_store.append( self.Domain( name = name ) )
>      File "jess.py", line 46, in __init__
>        self.lock = Lock()
>    error: can't allocate lock
>
> Googling for this sort of error doesn't yield any results, and i can't
> find any information about limits to the number of locks you can have
> in Python. The 'No space left on device' message indicates a memory
> issue, however i doubt this since its running on a linux server with 4
> cores and 16GB ram. It seems more like an internal Python limit has
> been hit (sem_init - semaphore initialisation?). Does anyone know more
> about threading internals and any internal limits?

Hi Jamskip,
I don't work with threading code but I have been working with  
semaphores for my IPC extensions. sem_init() is a call to create a  
semaphore (http://linux.die.net/man/3/sem_init). If it is failing,  
then I'd guess you're trying to create an awful lot of semaphores  
(intentionally or otherwise) and that you're hitting some internal  
limit.

I would not be too quick to assume that the number of semaphores one  
can create is bounded by the amount of RAM in your system. I don't  
think they're simple chunks of malloc-ed memory. They're probably  
represented in a kernel data structure somewhere that's hardcoded to  
some generous but fixed value.

Please note that this is all speculation on my part. I think that the  
Python threading implementation would use the "local" (i.e. not  
process-shared) semaphores which can be allocated on the process'  
heap. This would seem only RAM-limited, but I'll bet it isn't.

You might want to start debugging by track exactly how many locks  
you're creating. If the number is really big, start investigating  
kernel semaphore limits and how they're set.


Good luck
Philip