Thread Locking issue - Can't allocate lock (sem_init fail)

[jamskip at googlemail.com]

Hi all,

I have a peculiar problem with a multithreaded program of mine
(actually I've sort of inherited it). Before i show you the error,
here's a litle background. Its a program to check email addresses are
valid, and its main task is to verify the domain names.

Here's the basic functionality:

* The prog has a list of domains it has seen before which is read into
memory (the 'rollover').
* A new list of emails is read-in from a file (to a queue) and is
checked against the rollover.
* If we've seen the domain before then update the existing entry.
* If we've not seen the domain before, add it.

The program is multithreaded to speed up the processing...there are
input and output Queues.

Now, each domain entry is an class object containing various bits of
info. Each domain class also has its own lock, so that only one thread
can modify each domain at a time.

I'm load-testing the program with a sample of 1 million email
addresses and when i hit about the 500,000 mark i get a locking
error...

    sem_init: No space left on device
    Exception in thread Thread-2:
    Traceback (most recent call last):
      File "/usr/local/lib/python2.4/threading.py", line 442, in
__bootstrap
        self.run()
      File "/usr/local/lib/python2.4/threading.py", line 422, in run
        self.__target(*self.__args, **self.__kwargs)
      File "jess.py", line 250, in worker
        record.result = function( id, record.value )
      File "jess.py", line 291, in action
        found_domain = domains.addNewDomain( domain_name )
      File "jess.py", line 123, in addNewDomain
        self.domain_store.append( self.Domain( name = name ) )
      File "jess.py", line 46, in __init__
        self.lock = Lock()
    error: can't allocate lock

Googling for this sort of error doesn't yield any results, and i can't
find any information about limits to the number of locks you can have
in Python. The 'No space left on device' message indicates a memory
issue, however i doubt this since its running on a linux server with 4
cores and 16GB ram. It seems more like an internal Python limit has
been hit (sem_init - semaphore initialisation?). Does anyone know more
about threading internals and any internal limits?

For the timebeing, I've implemented locking at a higher level which
will reduce performance but keep the number of lock objects to a
minumum.

Thanks