[Moin-user] Lupy indexer errors with "too many open files"

[Rettig, Lane]

I'm not sure if this is the best place to ask questions about Lupy, but
I'll give it a shot for lack of a more appropriate list.

I'm attempting to do scalability testing of the MoinMoin software to
benchmark its performance with large numbers of pages, users, etc..  As
part of this testing, I created a 5,000 page wiki (the contents of each
page is identical and around 10kb).  Performing searches on the wiki was
painfully slow, taking on the order of minutes.  So I decided to try
building a Lupy index of the contents to see if this improved the search
performance.

However, whenever I try to build the index, Lupy gets through maybe a
couple thousand pages before dying with the following error:

Traceback (most recent call last):
  File "/var/wiki/lib/MoinMoin/script/moin.py", line 22, in ?
  File "/var/wiki/lib/MoinMoin/script/moin.py", line 15, in run
  File "/var/wiki/lib/MoinMoin/script/../../MoinMoin/script/_util.py",
line 76, in run
  File "/var/wiki/lib/MoinMoin/script/../../MoinMoin/script/_util.py",
line 176, in mainloop
  File "/var/wiki/lib/MoinMoin/script/../../MoinMoin/script/_util.py",
line 76, in run
  File
"/var/wiki/lib/MoinMoin/script/../../MoinMoin/script/lupy/build.py",
line 41, in mainloop
  File
"/var/wiki/lib/MoinMoin/script/../../MoinMoin/script/lupy/build.py",
line 47, in command
  File "/var/wiki/lib/MoinMoin/script/../../MoinMoin/lupy.py", line 300,
in indexPages
  File "/var/wiki/lib/MoinMoin/script/../../MoinMoin/lupy.py", line 530,
in _index_pages
  File "/var/wiki/lib/MoinMoin/script/../../MoinMoin/lupy.py", line 480,
in _index_page
  File
"/var/wiki/lib/MoinMoin/script/../../MoinMoin/support/lupy/index/indexwr
iter.py", line 56, in addDocument
  File
"/var/wiki/lib/MoinMoin/script/../../MoinMoin/support/lupy/index/indexwr
iter.py", line 133, in maybeMergeSegments
  File
"/var/wiki/lib/MoinMoin/script/../../MoinMoin/support/lupy/index/indexwr
iter.py", line 151, in mergeSegments
  File
"/var/wiki/lib/MoinMoin/script/../../MoinMoin/support/lupy/index/segment
merger.py", line 517, in __init__
  File
"/var/wiki/lib/MoinMoin/script/../../MoinMoin/support/lupy/index/field.p
y", line 140, in __init__
  File
"/var/wiki/lib/MoinMoin/script/../../MoinMoin/support/lupy/store.py",
line 71, in openFile
  File
"/var/wiki/lib/MoinMoin/script/../../MoinMoin/support/lupy/store.py",
line 236, in __init__
IOError: [Errno 24] Too many open files:
'/var/wiki/staging/wikis/test/cache/lupy/index/_1121.fdx'

I tried reducing the mergeFactor in support/lupy/index/indexwriter.py
from 20 to 10, as recommended elsewhere, but this didn't help.  Nor did
upping the limit on the max no. of file descriptors for the process from
256 to 1024.

Any advice would be greatly appreciated.  Thanks!

Lane Rettig