[Moin-user] Lupy indexer errors with "too many open files"

[Thomas Waldmann]

> I'm not sure if this is the best place to ask questions about Lupy, but
> I'll give it a shot for lack of a more appropriate list.

Lupy is dead. Lupy development was discontinued quite some time ago. We 
tried to fix it on our own, but we couldn't find the source of misc. bugs.

Thus, we replaced Lupy by Xapian in 1.6.

Xapian is the search engine used by gmane.org (see 
http://search.gmane.org/ ) and obviously scales very well.

> I'm attempting to do scalability testing of the MoinMoin software to
> benchmark its performance with large numbers of pages, users, etc..  As
> part of this testing, I created a 5,000 page wiki (the contents of each
> page is identical and around 10kb).

Of course I don't know your application, but I guess the typical wiki 
page size is below 10k.

> Performing searches on the wiki was
> painfully slow, taking on the order of minutes.

And of course this also depends on the machine (CPU, RAM, DISK speed).

> However, whenever I try to build the index, Lupy gets through maybe a
> couple thousand pages before dying with the following error:

If searching vast amounts is important in your evaluation, you should 
maybe try the (not yet released, but available) 1.6 code.

See http://hg.thinkmo.de/moin/1.6

> I tried reducing the mergeFactor in support/lupy/index/indexwriter.py
> from 20 to 10, as recommended elsewhere, but this didn't help.  Nor did
> upping the limit on the max no. of file descriptors for the process from
> 256 to 1024.

Maybe try some other values. But please understand that we won't invest 
much time into debugging this as the lupy code doesn't exist any more in 
1.6 (but much better xapian code).

Concerning production use, you could maybe just start with latest 1.5.x 
release (and the non-lupy slow search, which usually works quite well 
for small wikis) and build up your content (will take some time, I 
guess) and later switch to 1.6 when it is released and the slow search 
is getting a real problem for you.