From cs5070214 at cse.iitd.ac.in Thu Jun 2 20:27:37 2011 From: cs5070214 at cse.iitd.ac.in (Dushyant Bansal) Date: Thu, 02 Jun 2011 23:57:37 +0530 Subject: [mmgsoc] Pipermail archive UI In-Reply-To: <20110531103425.6664e7f2@neurotica.wooz.org> References: <4DE4F511.8080301@cse.iitd.ac.in> <20110531103425.6664e7f2@neurotica.wooz.org> Message-ID: <4DE7D619.801@cse.iitd.ac.in> On Tuesday 31 May 2011 08:04 PM, Barry Warsaw wrote: > >> *archive-ui* -> Yian's work on archive-ui as part of last year's GSOC on mailman 2.1 >> *pipermail* -> Pipermail on mailman 3 >> >> 1. In pipermail, all the static htmls generated for each message(that goes >> into archives) are arranged in directories corresponding to different months. >> Whereas in archive-ui, all the conversations are stored in a single >> directory. The reason is that since a conversation can span across months, >> it is somewhat ambiguous where to keep that conversation. > No, in fact just about everybody finds Pipermail's forced divisions pretty > crappy exactly for the reason you state. Okay. I'll leave the current strategy of keeping all conversations in a single directory untouched. And, as pointed out by Terry, I'll implement "search by date" functionality. >> Also, in archive-ui, when a message is archived, all the conversations are >> generated again. Ideally, we should do minimum amount of work on archiving a >> message. I am trying to work this out too. > Agreed. This is one reason to think about doing less work to generate a > static page and doing the actual final rendering dynamically (with possibly a > cache). Actually, I was talking about the way more work done by archive-ui. As per its current implementation, when a message is archived, all the static html files corresponding to all conversations are generated again. Ideally, we should rewrite only that conversation to which this message belongs. (Example of a conversation: http://dev.systers.org/pipermail/testing/all/mentor.html#93f118f1003182226h10373c79y98d5a7fcb5fe150a at mail.gmail.com ) But your point of serving pages dynamically is also something that we can think about. Should I look more in this direction? Regards, Dushyant From barry at list.org Tue Jun 7 22:31:45 2011 From: barry at list.org (Barry Warsaw) Date: Tue, 7 Jun 2011 16:31:45 -0400 Subject: [mmgsoc] Pipermail archive UI In-Reply-To: <4DE7D619.801@cse.iitd.ac.in> References: <4DE4F511.8080301@cse.iitd.ac.in> <20110531103425.6664e7f2@neurotica.wooz.org> <4DE7D619.801@cse.iitd.ac.in> Message-ID: <20110607163145.4d8e75c7@neurotica.wooz.org> On Jun 02, 2011, at 11:57 PM, Dushyant Bansal wrote: >But your point of serving pages dynamically is also something that we can >think about. Should I look more in this direction? Doing that would be a huge architectural change, so I wouldn't make it a priority. It's too easy to get lost in rewriting huge chunks of code. Do keep it in mind though as you work on it, and try to make decisions that allow for this change at some point down the road. If it makes your life much harder though, don't worry about it too much. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From cs5070214 at cse.iitd.ac.in Thu Jun 9 16:49:52 2011 From: cs5070214 at cse.iitd.ac.in (Dushyant Bansal) Date: Thu, 09 Jun 2011 20:19:52 +0530 Subject: [mmgsoc] Pipermail Archive-UI Message-ID: <4DF0DD90.6070403@cse.iitd.ac.in> Hi all, I would like to discuss some strategies that I have adopted for archiving a message using Pipermail in order to minimize the work involved. 1. Generation of conversation page: * Similar to {date, subject, article, author, thread}, keep a new dumbBTree of conversations which stores the mapping: conversation-id *-->* list of msgids present in that conversation. * when a new message is archived, conversation file corresponding to its conversation-id is built from the "list of msgids". [Example of conversation page: http://dev.systers.org/pipermail/testing/all/systersdev4.html] 2. Generation of conversations list pages: Suppose at one point, we have 100 pages containing conversations lists. When a new message arrives for archiving, we can just update one or two pages by inserting new entry corresponding to this conversation and removing old entry(only if it is present in those one or two pages). This will eventually leave static files of conversation lists in an inconsistent state. So, we have to rebuild more or all pages of conversations list as a cron job. [An example of conversation list: http://dev.systers.org/pipermail/testing/all/conversation.html] Please provide your feedback and suggestions. I have finalized and implemented these strategies just to get a better idea of how to do things. So, please don't hesitate to offer completely new ideas. Regards, Dushyant -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at list.org Fri Jun 10 22:10:27 2011 From: barry at list.org (Barry Warsaw) Date: Fri, 10 Jun 2011 16:10:27 -0400 Subject: [mmgsoc] Pipermail Archive-UI In-Reply-To: <4DF0DD90.6070403@cse.iitd.ac.in> References: <4DF0DD90.6070403@cse.iitd.ac.in> Message-ID: <20110610161027.568d7c4b@neurotica.wooz.org> Hi Dushyant, On Jun 09, 2011, at 08:19 PM, Dushyant Bansal wrote: >I would like to discuss some strategies that I have adopted for archiving a >message using Pipermail in order to minimize the work involved. > >1. Generation of conversation page: > > * Similar to {date, subject, article, author, thread}, keep a new > dumbBTree of conversations which stores the mapping: > conversation-id *-->* list of msgids present in that conversation. > * when a new message is archived, conversation file corresponding to > its conversation-id is built from the "list of msgids". This is one area where you and Andrew will want to coordinate. Remember Andrew is working on converting Pipermail to use Storm/SQLite instead of pickles. ISTM that you could do the above with an additional table and ORM object instead of a dumbBTree. >2. Generation of conversations list pages: > Suppose at one point, we have 100 pages containing conversations > lists. When a new message arrives for archiving, we can just update one > or two pages by inserting new entry corresponding to this conversation > and removing old entry(only if it is present in those one or two > pages). This will eventually leave static files of conversation lists in > an inconsistent state. So, we have to rebuild more or all pages of > conversations list as a cron job. I think we should explore again the performance of generating these pages on the fly. As I mentioned to Andrew, if doing so would cause problems meeting the schedule, it shouldn't be the highest priority, but if you and he can somehow share the load, then I think it's worth exploring. You'll have to figure out whether dynamic generation of the pages will be acceptable performance wise, and you'll need some way to cache the results, but if it works out, I think it could be quite powerful. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From terri at zone12.com Sat Jun 11 00:25:00 2011 From: terri at zone12.com (Terri Oda) Date: Fri, 10 Jun 2011 18:25:00 -0400 Subject: [mmgsoc] Pipermail Archive-UI In-Reply-To: <20110610161027.568d7c4b@neurotica.wooz.org> References: <4DF0DD90.6070403@cse.iitd.ac.in> <20110610161027.568d7c4b@neurotica.wooz.org> Message-ID: <4DF299BC.6040903@zone12.com> Barry Warsaw wrote: >> 2. Generation of conversations list pages: >> Suppose at one point, we have 100 pages containing conversations >> lists. When a new message arrives for archiving, we can just update one >> or two pages by inserting new entry corresponding to this conversation >> and removing old entry(only if it is present in those one or two >> pages). This will eventually leave static files of conversation lists in >> an inconsistent state. So, we have to rebuild more or all pages of >> conversations list as a cron job. > > I think we should explore again the performance of generating these pages on > the fly. As I mentioned to Andrew, if doing so would cause problems meeting > the schedule, it shouldn't be the highest priority, but if you and he can > somehow share the load, then I think it's worth exploring. You'll have to > figure out whether dynamic generation of the pages will be acceptable > performance wise, and you'll need some way to cache the results, but if it > works out, I think it could be quite powerful. As Barry says... basically do what allows you to meet your schedule. I'm wondering if it might not be easier to do this particular task dynamically, though: then you could just query the database and get the most up-to-date listing. Would it be easier to write the code for that and then for the static code, use that same script to regenerate the conversation when a new message comes in? Incidentally, how do you propose to do conversations in Mailman 3? Systers is using a modified codebase where conversations are separated using +conversation stuff stuck into the email address used for posting. (So for example, I might post a job interview to systers+jobs at ... if I were posting a job to the mailing list). You won't have that logic, so will you be using References:/In-Reply-To: combined with the subject line to approximate that? It's considered a bit of a technical problem in general, but probably any approximation we use will be "good enough" to start. I can't remember if we've talked about that before... Terri From dcrodman at gmail.com Sat Jun 11 02:51:02 2011 From: dcrodman at gmail.com (Andrew Rodman) Date: Fri, 10 Jun 2011 20:51:02 -0400 Subject: [mmgsoc] Pipermail Archive-UI In-Reply-To: <4DF299BC.6040903@zone12.com> References: <4DF0DD90.6070403@cse.iitd.ac.in> <20110610161027.568d7c4b@neurotica.wooz.org> <4DF299BC.6040903@zone12.com> Message-ID: Hey Dushyant, I'm not always on IRC so I figured I'd reply so you have my email if you want to reach me. As Barry mentioned, I've planned to retire dumbBTree in favor of a Storm ORM object with all of the Article types mapped to various properties (with the same naming scheme) since we won't be using pickles anymore. Mapping a list of message-ids to build conversations seems like a good idea to me and could easily be added to the ORM class I've written to represent Articles. Let me know if you'd like to discuss anything further, even if it is as simple as adding an additional list property. - Drew On Fri, Jun 10, 2011 at 6:25 PM, Terri Oda wrote: > Barry Warsaw wrote: > >> 2. Generation of conversations list pages: >>> Suppose at one point, we have 100 pages containing conversations >>> lists. When a new message arrives for archiving, we can just update >>> one >>> or two pages by inserting new entry corresponding to this conversation >>> and removing old entry(only if it is present in those one or two >>> pages). This will eventually leave static files of conversation lists >>> in >>> an inconsistent state. So, we have to rebuild more or all pages of >>> conversations list as a cron job. >>> >> >> I think we should explore again the performance of generating these pages >> on >> the fly. As I mentioned to Andrew, if doing so would cause problems >> meeting >> the schedule, it shouldn't be the highest priority, but if you and he can >> somehow share the load, then I think it's worth exploring. You'll have to >> figure out whether dynamic generation of the pages will be acceptable >> performance wise, and you'll need some way to cache the results, but if it >> works out, I think it could be quite powerful. >> > > As Barry says... basically do what allows you to meet your schedule. > > I'm wondering if it might not be easier to do this particular task > dynamically, though: then you could just query the database and get the most > up-to-date listing. Would it be easier to write the code for that and then > for the static code, use that same script to regenerate the conversation > when a new message comes in? > > Incidentally, how do you propose to do conversations in Mailman 3? Systers > is using a modified codebase where conversations are separated using > +conversation stuff stuck into the email address used for posting. (So for > example, I might post a job interview to systers+jobs at ... if I were > posting a job to the mailing list). You won't have that logic, so will you > be using References:/In-Reply-To: combined with the subject line to > approximate that? It's considered a bit of a technical problem in general, > but probably any approximation we use will be "good enough" to start. I > can't remember if we've talked about that before... > > Terri > > _______________________________________________ > MMGSOC mailing list > MMGSOC at python.org > http://mail.python.org/mailman/listinfo/mmgsoc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cs5070214 at cse.iitd.ac.in Sat Jun 11 08:46:23 2011 From: cs5070214 at cse.iitd.ac.in (Dushyant Bansal) Date: Sat, 11 Jun 2011 12:16:23 +0530 Subject: [mmgsoc] Pipermail Archive-UI In-Reply-To: <4DF299BC.6040903@zone12.com> References: <4DF0DD90.6070403@cse.iitd.ac.in> <20110610161027.568d7c4b@neurotica.wooz.org> <4DF299BC.6040903@zone12.com> Message-ID: <4DF30F3F.7080204@cse.iitd.ac.in> > > I'm wondering if it might not be easier to do this particular task > dynamically, though: then you could just query the database and get > the most up-to-date listing. Would it be easier to write the code for > that and then for the static code, use that same script to regenerate > the conversation when a new message comes in? Yes, definitely, I'd like to go ahead with the dynamic generation of pages. > > Incidentally, how do you propose to do conversations in Mailman 3? > Systers is using a modified codebase where conversations are separated > using +conversation stuff stuck into the email address used for > posting. (So for example, I might post a job interview to > systers+jobs at ... if I were posting a job to the mailing list). At one point of time, I explored this. Archive-ui was written on top of systers? implementation of mailman, which supports some extra mailman extension - DList and conversations view also shows email address corresponding to the conversation. But Dlist has not been ported to mm3. > You won't have that logic, so will you be using > References:/In-Reply-To: combined with the subject line to approximate > that? It's considered a bit of a technical problem in general, but > probably any approximation we use will be "good enough" to start. I > can't remember if we've talked about that before... Yes, currently I am using just subject line to approximate that. I thought about using 'In-Reply-To' to ensure uniqueness of conversation in future. Regards, Dushyant From cs5070214 at cse.iitd.ac.in Sat Jun 11 21:56:57 2011 From: cs5070214 at cse.iitd.ac.in (Dushyant Bansal) Date: Sun, 12 Jun 2011 01:26:57 +0530 Subject: [mmgsoc] Pipermail Archive-UI In-Reply-To: <20110610161027.568d7c4b@neurotica.wooz.org> References: <4DF0DD90.6070403@cse.iitd.ac.in> <20110610161027.568d7c4b@neurotica.wooz.org> Message-ID: <4DF3C889.7030008@cse.iitd.ac.in> On Saturday 11 June 2011 01:40 AM, Barry Warsaw wrote: > > I think we should explore again the performance of generating these pages on > the fly. As I mentioned to Andrew, if doing so would cause problems meeting > the schedule, it shouldn't be the highest priority, but if you and he can > somehow share the load, then I think it's worth exploring. You'll have to > figure out whether dynamic generation of the pages will be acceptable > performance wise, and you'll need some way to cache the results, but if it > works out, I think it could be quite powerful. Yes, we should definitely give it a try. As a first step, I explored the options for the setup. It seems that out of two options CGI and mod_wsgi, mod_wsgi is preferred due to efficiency and some other reasons. Also, I think that, for our work, we don't need to use any framework. What do you say? Regards, Dushyant From barry at list.org Mon Jun 13 17:43:35 2011 From: barry at list.org (Barry Warsaw) Date: Mon, 13 Jun 2011 11:43:35 -0400 Subject: [mmgsoc] Pipermail Archive-UI In-Reply-To: <4DF299BC.6040903@zone12.com> References: <4DF0DD90.6070403@cse.iitd.ac.in> <20110610161027.568d7c4b@neurotica.wooz.org> <4DF299BC.6040903@zone12.com> Message-ID: <20110613114335.4306ec61@neurotica.wooz.org> On Jun 10, 2011, at 06:25 PM, Terri Oda wrote: >Incidentally, how do you propose to do conversations in Mailman 3? Systers is >using a modified codebase where conversations are separated using >+conversation stuff stuck into the email address used for posting. (So for >example, I might post a job interview to systers+jobs at ... if I were posting a >job to the mailing list). You won't have that logic, so will you be using >References:/In-Reply-To: combined with the subject line to approximate that? >It's considered a bit of a technical problem in general, but probably any >approximation we use will be "good enough" to start. I can't remember if >we've talked about that before... If the References/In-Reply-To headers will do the trick, that's great. I'm not opposed to making some changes to the MM3 code base to better support this, if we find that +conversation is necessary. We'd have to figure out how to integrate this with the various MTAs and the LMTP server in Mailman. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at list.org Mon Jun 13 17:48:28 2011 From: barry at list.org (Barry Warsaw) Date: Mon, 13 Jun 2011 11:48:28 -0400 Subject: [mmgsoc] Pipermail Archive-UI In-Reply-To: <4DF5FE1B.3020605@cse.iitd.ac.in> References: <4DF5FE1B.3020605@cse.iitd.ac.in> Message-ID: <20110613114828.6f879f5c@neurotica.wooz.org> On Jun 13, 2011, at 05:40 PM, Dushyant Bansal wrote: >Resending the message as my inbox says >"Delivery to the following recipient failed permanently: barry at list.org" Please let me know if that continues. You can always reach me at barry at python.org which goes through a completely different server (though lands in the same final inbox). I don't control list.org and it does occasionally have problems. >Yes, we should definitely give it a try. As a first step, I explored the >options for the setup. It seems that out of two options CGI and mod_wsgi, >mod_wsgi is preferred due to efficiency and other reasons. Also, I think >that, for our work, we don't need to use any framework. What do you say? mod_wsgi does seem preferable, especially because of all the tools that are wsgi compatible these days. If you don't need a framework, don't use one! It's easy to create your own wsgi server (e.g. src/mailman/rest/wsgiapp.py). Of course, a framework is different than a templating system, which is something you might indeed want to look at. There are lots of choices here and I don't have any clear preferences, except it should be something that supports i18n well. Maybe Florian has some thoughts about that. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From terri at zone12.com Mon Jun 13 18:03:59 2011 From: terri at zone12.com (Terri Oda) Date: Mon, 13 Jun 2011 12:03:59 -0400 Subject: [mmgsoc] Pipermail Archive-UI In-Reply-To: <20110613114335.4306ec61@neurotica.wooz.org> References: <4DF0DD90.6070403@cse.iitd.ac.in> <20110610161027.568d7c4b@neurotica.wooz.org> <4DF299BC.6040903@zone12.com> <20110613114335.4306ec61@neurotica.wooz.org> Message-ID: <4DF634EF.5040207@zone12.com> Barry Warsaw wrote: > If the References/In-Reply-To headers will do the trick, that's great. I'm > not opposed to making some changes to the MM3 code base to better support > this, if we find that +conversation is necessary. We'd have to figure out how > to integrate this with the various MTAs and the LMTP server in Mailman. I don't think +conversation will be necessary because probably a few misplaced messages in a conversation thread isn't a showstopper (and it can happen with +conversation too, although it's likely more rare). But I *do* think +conversation is useful for other reasons that we might want to consider: the big one being that people can then unsubscribe from conversations they hate without having to leave the list or write their own filters. Systers has a clickable "leave this conversation" link and it's pretty darned slick. Having used our existing "topics" interface extensively on another list, I'd like to see +conversation used to augment our current topic structure. Terri From barry at list.org Mon Jun 13 18:28:36 2011 From: barry at list.org (Barry Warsaw) Date: Mon, 13 Jun 2011 12:28:36 -0400 Subject: [mmgsoc] Pipermail Archive-UI In-Reply-To: <4DF634EF.5040207@zone12.com> References: <4DF0DD90.6070403@cse.iitd.ac.in> <20110610161027.568d7c4b@neurotica.wooz.org> <4DF299BC.6040903@zone12.com> <20110613114335.4306ec61@neurotica.wooz.org> <4DF634EF.5040207@zone12.com> Message-ID: <20110613122836.4f355cb5@neurotica.wooz.org> On Jun 13, 2011, at 12:03 PM, Terri Oda wrote: >But I *do* think +conversation is useful for other reasons that we might want >to consider: the big one being that people can then unsubscribe from >conversations they hate without having to leave the list or write their own >filters. Systers has a clickable "leave this conversation" link and it's >pretty darned slick. Having used our existing "topics" interface extensively >on another list, I'd like to see +conversation used to augment our current >topic structure. Cool, I'm definitely open to discussions here. One of the things I've had in the back of my mind is to include a mailsieve implementation in Mailman. There's now a fairly decent (so I'm told) implementation in Python, so I think it would be feasible. It's probably way too complicated for the average user, but I *think* sieve might make a good technology for implementing topics-NG and other types of filters. And maybe make it available to advanced users to control the email they get from a Mailman site. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From cs5070214 at cse.iitd.ac.in Mon Jun 13 23:51:59 2011 From: cs5070214 at cse.iitd.ac.in (Dushyant Bansal) Date: Tue, 14 Jun 2011 03:21:59 +0530 Subject: [mmgsoc] Pipermail Archive-UI In-Reply-To: <20110613114828.6f879f5c@neurotica.wooz.org> References: <4DF5FE1B.3020605@cse.iitd.ac.in> <20110613114828.6f879f5c@neurotica.wooz.org> Message-ID: <4DF6867F.3040208@cse.iitd.ac.in> On Monday 13 June 2011 09:18 PM, Barry Warsaw wrote: > > mod_wsgi does seem preferable, especially because of all the tools that are > wsgi compatible these days. If you don't need a framework, don't use one! > It's easy to create your own wsgi server (e.g. src/mailman/rest/wsgiapp.py). > Of course, a framework is different than a templating system, which is > something you might indeed want to look at. There are lots of choices here > and I don't have any clear preferences, except it should be something that > supports i18n well. Maybe Florian has some thoughts about that. Okay. Two frameworks that I know about are 'web2py' and 'django' and both supports i18n. I am familiar with Django, so, I'll try to look at it first. -- Dushyant From cs5070214 at cse.iitd.ac.in Wed Jun 22 18:29:54 2011 From: cs5070214 at cse.iitd.ac.in (Dushyant Bansal) Date: Wed, 22 Jun 2011 21:59:54 +0530 Subject: [mmgsoc] Archive UI Web Framework Message-ID: <4E021882.6080406@cse.iitd.ac.in> Hi all, I was looking for a web framework for Pipermail UI. I have decided to use Pylons. Anna and benste wanted to know the reasons for choosing pylons over Django (As benste is already using Django for mailman WebUI). So, I thought I should discuss it with all of you. As you know, Andrew is working on converting Pipermail to use Storm/SQLite instead of pickles to save data. So, the framework needs to have support for Storm ORM. Django has its own ORM and Storm cannot be used with it natively (Though storm is said to have support for django, I could not find anything further to try/test Storm with Django). Pylons doesn't provide any ORM and I have tested a small application using Storm. Please provide your feedback. Regards, Dushyant -------------- next part -------------- An HTML attachment was scrubbed... URL: From benedict.stein at googlemail.com Wed Jun 22 19:30:44 2011 From: benedict.stein at googlemail.com (Benedict Stein) Date: Wed, 22 Jun 2011 19:30:44 +0200 Subject: [mmgsoc] Archive UI Web Framework In-Reply-To: <4E021882.6080406@cse.iitd.ac.in> References: <4E021882.6080406@cse.iitd.ac.in> Message-ID: <1308763844.2460.2.camel@vaio-fe31m> Dushyant - know that this is the mailman-gsoc only discussion list ? Am Mittwoch, den 22.06.2011, 21:59 +0530 schrieb Dushyant Bansal: > Hi all, > > I was looking for a web framework for Pipermail UI. I have decided to > use Pylons. > Anna and benste wanted to know the reasons for choosing pylons over > Django (As benste is already using Django for mailman WebUI). So, I > thought I should discuss it with all of you. > > As you know, Andrew is working on converting Pipermail to use > Storm/SQLite instead of pickles to save data. So, the framework needs > to have support for Storm ORM. Django has its own ORM and Storm cannot > be used with it natively (Though storm is said to have support for > django, I could not find anything further to try/test Storm with > Django). > Pylons doesn't provide any ORM and I have tested a small application > using Storm. > > Please provide your feedback. > > Regards, > Dushyant > > _______________________________________________ > MMGSOC mailing list > MMGSOC at python.org > http://mail.python.org/mailman/listinfo/mmgsoc -- Einen sch?nen Tag w?nscht: Benedict Stein -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part URL: