[Mailman-Developers] Google Summer of Code: Integration of Search Code

[Shayan Md]

On Fri, Mar 30, 2012 at 5:05 AM, Stephen J. Turnbull <stephen at xemacs.org>wrote:

> On Fri, Mar 30, 2012 at 3:55 AM, Shayan Md <mdoshayan at gmail.com> wrote:
>
> > Assuming that we have something like this(object-ID-addressable, If I am
> > not wrong, mailman3 made it possible but not yet implemented as it's part
> > of archiver), is it over ambitious to plan to implement indexer/searcher
> > for mailman3 and a REST API to use this searcher, extend client to use
> this
> > api,
> > and django search form along with this client api? All this independent
> of
> > archiver. Because the only part common with archiver is message retrieval
> > part,
> > If we implement whole searcher, and rest of the client code, later when
> > archiver is implemented message retrieval code can used in searcher. When
> > archiver is completely mature may we can even merge them together. Is it
> > possible? Or this plan has any 'non-sense' parts?
>
> Hi, Shayan.  It's not nonsense, but (1) how do you propose to test if
> you have no archives to run it on?  And (2) search and retrieval may
> do a *lot* of message access, for example if you want to do data
> mining (see Ana from Spain's thread).  In that case, you may find that
> maildir imposes too many stats, which are very expensive on some
> platforms (and not cheap on any that I know of) and mbox requires too
> large memory.  So for some purposes a summary/index/search engine may
> want an optimized message store.
>
Isn't it the purpose of index? I mean, when we search for something we
don't have stat all the messages, search the index return the best matching
message IDs. If you wish to see the message then retrieve it(may be through
archiver). Probably we can index messages through cron every night.

>
> Those may not be your purposes, of course, in which case a simple mbox
> accessor and a download of a couple of mboxes from any public Mailman
> list will give you test fodder.
>
> Testing itself is not really a matter of personal preference.
> Although Mailman is not a 100% test-driven shop, Mailman 3 already has
> a *lot* of tests and Barry would like to see any new modules covered,
> I'm sure.  Also, this area is fraught with pitfalls for the developer.
>  See this thread, for example:
> http://thread.gmane.org/gmane.comp.python.devel/131395/focus=131406.
> _______________________________________________
> Mailman-Developers mailing list
> Mailman-Developers at python.org
> http://mail.python.org/mailman/listinfo/mailman-developers
> Mailman FAQ: http://wiki.list.org/x/AgA3
> Searchable Archives:
> http://www.mail-archive.com/mailman-developers%40python.org/
> Unsubscribe:
> http://mail.python.org/mailman/options/mailman-developers/mdoshayan%40gmail.com
>
> Security Policy: http://wiki.list.org/x/QIA9
>