[mmgsoc] Pipermail Archive-UI

[Barry Warsaw]

Hi Dushyant,

On Jun 09, 2011, at 08:19 PM, Dushyant Bansal wrote:

>I would like to discuss some strategies that I have adopted for archiving a
>message using Pipermail in order to minimize the work involved.
>
>1. Generation of conversation page:
>
>    * Similar to {date, subject, article, author, thread}, keep a new
>      dumbBTree of conversations which stores the mapping:
>      conversation-id *-->* list of msgids present in that conversation.
>    * when a new message is archived, conversation file corresponding to
>      its conversation-id is built from the "list of msgids".

This is one area where you and Andrew will want to coordinate.  Remember
Andrew is working on converting Pipermail to use Storm/SQLite instead of
pickles.  ISTM that you could do the above with an additional table and ORM
object instead of a dumbBTree.

>2. Generation of conversations list pages:
>     Suppose at one point, we have 100 pages containing conversations
>     lists. When a new message arrives for archiving, we can just update one
>     or two pages by inserting new entry corresponding to this conversation
>     and removing old entry(only if it is present in those one or two
>     pages). This will eventually leave static files of conversation lists in
>     an inconsistent state. So, we have to rebuild more or all pages of
>     conversations list as a cron job.

I think we should explore again the performance of generating these pages on
the fly.  As I mentioned to Andrew, if doing so would cause problems meeting
the schedule, it shouldn't be the highest priority, but if you and he can
somehow share the load, then I think it's worth exploring.  You'll have to
figure out whether dynamic generation of the pages will be acceptable
performance wise, and you'll need some way to cache the results, but if it
works out, I think it could be quite powerful.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/mmgsoc/attachments/20110610/47c3ceec/attachment.pgp>