From rajeevs1992 at gmail.com Mon Aug 4 20:57:41 2014 From: rajeevs1992 at gmail.com (Rajeev S) Date: Tue, 05 Aug 2014 00:27:41 +0530 Subject: [Mailman-Developers] [CLI Project] Added Backup and Restore Tool Message-ID: <53DFD7A5.8010003@gmail.com> Hi, As the Summer of Code period comes to an end, I am one more step closer to completion of the features mentioned in my proposal. I have built the initial version of the backup and restore tool, that currently supports backup and restore of a sqlite Mailman Instance, by archiving the var_dir to a zip archive. Further, the CLI is now installable using `python setup.py install` and the `mmclient` command is added to the `bin`. So the commands can now begin as `mmclient ...` I have also replaced the `in list` filter present in the `show` commands, with a regular expression matcher for better usability of the command. I have begun code scrubbing and my primary aim will be improving the documentation and more importantly, continue writing unit tests from where I left off. I expect to complete the tasks by the suggested pencil down date [11/08/2014] and spend the last week on improving the code structure, if possible anywhere. From rajeevs1992 at gmail.com Mon Aug 4 21:00:09 2014 From: rajeevs1992 at gmail.com (Rajeev S) Date: Tue, 05 Aug 2014 00:30:09 +0530 Subject: [Mailman-Developers] [CLI Project] Added Backup and Restore Tool In-Reply-To: <53DFD7A5.8010003@gmail.com> References: <53DFD7A5.8010003@gmail.com> Message-ID: <53DFD839.5070903@gmail.com> On Tuesday 05 August 2014 12:27 AM, Rajeev S wrote: > Hi, > > As the Summer of Code period comes to an end, I am one more step > closer to completion of the > features mentioned in my proposal. > > I have built the initial version of the backup and restore tool, that > currently supports backup > and restore of a sqlite Mailman Instance, by archiving the var_dir to > a zip archive. > > Further, the CLI is now installable using `python setup.py install` > and the `mmclient` command is > added to the `bin`. So the commands can now begin as `mmclient ...` > > I have also replaced the `in list` filter present in the `show` > commands, with a regular expression > matcher for better usability of the command. > > I have begun code scrubbing and my primary aim will be improving the > documentation and > more importantly, continue writing unit tests from where I left off. > > I expect to complete the tasks by the suggested pencil down date > [11/08/2014] and spend the last week > on improving the code structure, if possible anywhere. > Sorry, I missed to mention that I have committed and pushed the r69 that carries the above changes. Here is a link to the r69 http://bazaar.launchpad.net/~rajeevs1992/mailman.client/mailmancli/revision/69 From stephen at xemacs.org Wed Aug 6 08:44:45 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 06 Aug 2014 15:44:45 +0900 Subject: [Mailman-Developers] [CLI Project] Added Backup and Restore Tool In-Reply-To: <53DFD839.5070903@gmail.com> References: <53DFD7A5.8010003@gmail.com> <53DFD839.5070903@gmail.com> Message-ID: <87y4v2jew2.fsf@uwakimon.sk.tsukuba.ac.jp> Hi, I have some comments on your CLI, r69. File is attached. Steve -------------- next part -------------- Comments on Rajeev's CLI ======================== ================== ===================================================== Comment Date: 2014-08-05 22:16 Referenced Commit: rajeevs1992 at gmail.com-20140804184004-kaan7063cifoasb1 Author: Rajeev S Branch Nick: mailmancli Commit Date: Tue 2014-08-05 00:10:04 +0530 ================== ===================================================== Tree organization ----------------- I don't think the CLI should be buried several levels deep in the source. The mailman.client package is really a collection of services, and the CLI should be at the same level in some sense. All file references below are relative to mailman.client/src/mailmanclient/cli. Is there any code outside of this directory that is yours? It's hard to recognize which files are part of this project. This should be reflected in header comments and/or module docstrings. I think that these files should also acknowledge support from Google, and provide contact addresses (yours and mentors'). There should be a README file of some kind at the top of your subtree, or perhaps a section in the mailman.client top level README. mmclient.py ----------- This file is part of your 2014 GSoC project, and not inherited from mailman.client, right? I think there should be some comments in this file, for example explaining why you have what seem to be two different REPLs (c.run() and s.cmdloop()) invoked. It's not clear to me why you define a separate main() function in this script -- it doesn't make sense to import it as a module. docs ---- Run a spellchecker on these files (I saw at least one typo). Generally they look very good: clear and complete. lib --- In lib/utils.py, I think there should be a more flexible way to specify the emphasis and so on. This is especially important for color-blind and partially-blind users. For now you don't need to provide a user interface, but I'd be happier if the control sequences were kept in module-level variables and interpolated into the messages using %-formatting or str.format. (Note that if you wanted to, you could use HTML tags in such variables. Eg, many Qt4 and Qt5 widgets that can display text respect HTML tags in display.) In lib/utils.py class Filter, what are the class attributes for? They are not used or useful. core ---- In core/domains.py, you instantiate the Domain list then delete it, commenting it as testing the connection. Why does this need to be done in the constructor? Although Domain doesn't currently provide any services that can be accomplished without a database query that I noticed, so the check doesn't really hurt, it could. (For example, validating domain names against the syntax recommended by RFCs 822 and 1034 for mail domains.) The same kind of service would apply to list names. It's also possible that there is a transient interruption that is resolved before any actual work is done, but this check would make the script fail. BTW, I believe there is no need to assign self.client.domains to a variable (the Python optimizer isn't that smart AFAIK). Even if you do assign, there's no need to delete the variable, as the value will be GC'd after exiting the constructor. IIRC, del doesn't guarantee that the object will be GC'd any time soon, just DECREF's it. Assuming that an HTTPError means that the operation failed in an expected way (eg, deleting a nonexistent domain) is unacceptable. In particular, the transaction may fail if Mailman is stopped. The code must check the information provided by the HTTPError object, at least for errors that can be expected from invalid input. The rest can be "Unknown HTTPError was raised" for now. The various classes have a lot of code in common (eg, in the __init__() and get_listing() methods). I don't know if it's worth refactoring because there is a lot of class-specific code in these "lookalike" methods, but there does seem to be some potential for it. client ------ I don't understand the parser yet, but the rest of the code looks clean. tests ----- The tests look complete and properly set up. From rajeevs1992 at gmail.com Wed Aug 6 20:32:06 2014 From: rajeevs1992 at gmail.com (Rajeev S) Date: Thu, 07 Aug 2014 00:02:06 +0530 Subject: [Mailman-Developers] [CLI Project] Added Backup and Restore Tool In-Reply-To: <87y4v2jew2.fsf@uwakimon.sk.tsukuba.ac.jp> References: <53DFD7A5.8010003@gmail.com> <53DFD839.5070903@gmail.com> <87y4v2jew2.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <53E274A6.2020205@gmail.com> Hi Steve, On Wednesday 06 August 2014 12:14 PM, Stephen J. Turnbull wrote: > Tree organization > ----------------- > > I don't think the CLI should be buried several levels deep in the > source. The mailman.client package is really a collection of > services, and the CLI should be at the same level in some sense. I feel that it will make the code cluttered. Since the CLI code is independent of the rest of mailman client, won't it be better to maintain the CLI code in a separate folder, as it is now? > All file references below are relative to > mailman.client/src/mailmanclient/cli. > > Is there any code outside of this directory that is yours? I have made modifications to setup.py, other than that, no. > It's hard to recognize which files are part of this project. This > should be reflected in header comments and/or module docstrings. I > think that these files should also acknowledge support from Google, > and provide contact addresses (yours and mentors'). > > There should be a README file of some kind at the top of your subtree, > or perhaps a section in the mailman.client top level README. > Will add those information in the headers by next revision. > mmclient.py > ----------- > > This file is part of your 2014 GSoC project, and not inherited from > mailman.client, right? Yes. I created this file. > I think there should be some comments in this file, for example > explaining why you have what seem to be two different REPLs (c.run() > and s.cmdloop()) invoked. > > It's not clear to me why you define a separate main() function in this > script -- it doesn't make sense to import it as a module. I was not quite familiar with using the python setuptools and I assumed that such a format was necessary to specify module entry points in setup.py. I have now removed the main function from mmclient.py > > docs > ---- > > Run a spellchecker on these files (I saw at least one typo). Yikes! Sorry about that. Fixed at least ten of them :) > > Generally they look very good: clear and complete. > > > lib > --- > > In lib/utils.py, I think there should be a more flexible way to > specify the emphasis and so on. This is especially important for > color-blind and partially-blind users. For now you don't need to > provide a user interface, but I'd be happier if the control sequences > were kept in module-level variables and interpolated into the messages > using %-formatting or str.format. (Note that if you wanted to, you > could use HTML tags in such variables. Eg, many Qt4 and Qt5 widgets > that can display text respect HTML tags in display.) Sure. Will add that in the next revision. > In lib/utils.py class Filter, what are the class attributes for? They > are not used or useful. The Filter class is used to filter out objects meeting a certain condition specified by a key, op [aka operator] and value. The filtering process works by iterating through each element in the data_set and removing the objects that do not match the criterion, from a copy of the data_set. The data_set cannot be used alone, as removing entries from a running `for i in list` loop can cause the loop to misbehave. The filtered copy is finally returned. All the class attributes are being used in the process described above. > > core > ---- > > In core/domains.py, you instantiate the Domain list then delete it, > commenting it as testing the connection. Why does this need to be > done in the constructor? Although Domain doesn't currently provide > any services that can be accomplished without a database query that I > noticed, so the check doesn't really hurt, it could. (For example, > validating domain names against the syntax recommended by RFCs 822 and > 1034 for mail domains.) The same kind of service would apply to list > names. It's also possible that there is a transient interruption that > is resolved before any actual work is done, but this check would make > the script fail. Now that the connection part is managed by a single function (Utils.connect), I will move the connection checking part to that function. I will try to come up with a better way to check for the connection, so that the mentioned possible issues can be put away. > > BTW, I believe there is no need to assign self.client.domains to a > variable (the Python optimizer isn't that smart AFAIK). Even if you > do assign, there's no need to delete the variable, as the value will > be GC'd after exiting the constructor. IIRC, del doesn't guarantee > that the object will be GC'd any time soon, just DECREF's it. Those lists were not stored into variables until some revisions back. I got a random thought that it is a bad practice , and replaced that code with a variable assignment. It was then when my PEP8 checker returned an error stating the presence of an unused variable, which I solved by deleting the assigned variable! > Assuming that an HTTPError means that the operation failed in an > expected way (eg, deleting a nonexistent domain) is unacceptable. In > particular, the transaction may fail if Mailman is stopped. The code > must check the information provided by the HTTPError object, at least > for errors that can be expected from invalid input. The rest can be > "Unknown HTTPError was raised" for now. Will do. > The various classes have a lot of code in common (eg, in the > __init__() and get_listing() methods). I don't know if it's worth > refactoring because there is a lot of class-specific code in these > "lookalike" methods, but there does seem to be some potential for it. > I will look into the possibility of refactoring in those parts, I have some plans in mind. Hope that those work! > client > ------ > > I don't understand the parser yet, but the rest of the code looks > clean. > I will add the grammar for each command to a README file under parsers, so that it can be studied, modified or extended with ease. > tests > ----- > > The tests look complete and properly set up. Will be adding more tests shortly. All the mentioned changes (except the tests) would be completed at the latest, by 9th Aug 2014, Saturday. (r 70) Planning to spend the weekend on writing new unit tests. (r 71) Thank You! From stephen at xemacs.org Thu Aug 7 08:30:17 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 07 Aug 2014 15:30:17 +0900 Subject: [Mailman-Developers] [CLI Project] Added Backup and Restore Tool In-Reply-To: <53E274A6.2020205@gmail.com> References: <53DFD7A5.8010003@gmail.com> <53DFD839.5070903@gmail.com> <87y4v2jew2.fsf@uwakimon.sk.tsukuba.ac.jp> <53E274A6.2020205@gmail.com> Message-ID: <87oavwke12.fsf@uwakimon.sk.tsukuba.ac.jp> Rajeev S writes: > I feel that it will make the code cluttered. Since the CLI code > is independent of the rest of mailman client, won't it be > better to maintain the CLI code in a separate folder, as it is now? Of course. I didn't explain myself well. What you have now is mailman/ -+- client/ -+- _client.py +- docs/ +- tests/ +- cli/ -+- client.py +- core/ +- docs/ +- lib/ +- tests/ and what I would like to see is something like mailman/ -+- client/ -+- _client.py | +- docs/ | +- tests/ +- cli/ -+- client.py +- core/ +- docs/ +- lib/ +- tests/ It's only one level higher, but it makes the relationship (mailman.client provides services to mailman.cli) clearer, and emphasizes the user app (cli). > > Is there any code outside of this directory that is yours? > > I have made modifications to setup.py, other than that, no. OK, that's great! > > It's not clear to me why you define a separate main() function in > > this script -- it doesn't make sense to import it as a module. > > I was not quite familiar with using the python setuptools and I > assumed that such a format was necessary to specify module entry > points in setup.py. I have now removed the main function from > mmclient.py OK. It's up to you (and maybe Barry has a stylistic opinion on it). > Yikes! Sorry about that. Fixed at least ten [typos] :) And I thought I was being a cranky old man. Glad we caught those. :-) > > In lib/utils.py class Filter, what are the class attributes for? They > > are not used or useful. > > The Filter class is used to filter out objects [...] > > All the class attributes are being used in the process described above. I don't see where. In Filter, you have: class Filter(): key = None value = None operator = None data_set = [] utils = Utils() def __init__(self, key, value, operator, data): self.key = key self.value = value self.operator = operator self.data_set = data So as soon as you instance a Filter, __init__() creates instance attributes which shadow the class attributes (except utils). I don't see any references to Filter. or to super(), so I don't see how you can be using the class attributes. > Now that the connection part is managed by a single function > (Utils.connect), I will move the connection checking part to that > function. OK. > Those lists were not stored into variables until some revisions back. I got > a random thought that it is a bad practice , and replaced that code with > a variable assignment. It was then when my PEP8 checker returned an error > stating the presence of an unused variable, which I solved by deleting the > assigned variable! ;-) > I will look into the possibility of refactoring in those parts, I > have some plans in mind. Hope that those work! That would be very cool! As I say, I'm not confident that it's worth the work, so be careful about wasting your time just because *I* said it. (It's not a waste of time if you do it because you think it's an interesting problem.) > All the mentioned changes (except the tests) would be completed at > the latest, by 9th Aug 2014, Saturday. (r 70) Planning to spend the > weekend on writing new unit tests. (r 71) Sounds good! From rajeevs1992 at gmail.com Thu Aug 7 20:13:38 2014 From: rajeevs1992 at gmail.com (Rajeev S) Date: Thu, 07 Aug 2014 23:43:38 +0530 Subject: [Mailman-Developers] [CLI Project] Added Backup and Restore Tool In-Reply-To: <87oavwke12.fsf@uwakimon.sk.tsukuba.ac.jp> References: <53DFD7A5.8010003@gmail.com> <53DFD839.5070903@gmail.com> <87y4v2jew2.fsf@uwakimon.sk.tsukuba.ac.jp> <53E274A6.2020205@gmail.com> <87oavwke12.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <53E3C1D2.6070404@gmail.com> Hi Steve, On Thursday 07 August 2014 12:00 PM, Stephen J. Turnbull wrote: > Rajeev S writes: > > > I feel that it will make the code cluttered. Since the CLI code > > is independent of the rest of mailman client, won't it be > > better to maintain the CLI code in a separate folder, as it is now? > > Of course. I didn't explain myself well. What you have now is > > mailman/ -+- client/ -+- _client.py > +- docs/ > +- tests/ > +- cli/ -+- client.py > +- core/ > +- docs/ > +- lib/ > +- tests/ The above tree is wrong, this is the current directory structure mailmanclient/ -+- _client.py +- docs/ +- tests/ +- cli/ -+- mmclient.py +- core/ +- docs/ +- lib/ +- tests/ +- client/ That is, the cli/ directory lives directly inside the mailmanclient directory. > > > I don't see where. In Filter, you have: > > class Filter(): > key = None > value = None > operator = None > data_set = [] > utils = Utils() > > def __init__(self, key, value, operator, data): > self.key = key > self.value = value > self.operator = operator > self.data_set = data > > So as soon as you instance a Filter, __init__() creates instance > attributes which shadow the class attributes (except utils). I don't > see any references to Filter. or to super(), so I don't see how > you can be using the class attributes. Now I get it! Will fix that. > > Now that the connection part is managed by a single function > > (Utils.connect), I will move the connection checking part to that > > function. > > OK. > The connection can be checked without using a database query by trying to invoke the `client.system` , at a single place. From barry at list.org Thu Aug 7 21:07:43 2014 From: barry at list.org (Barry Warsaw) Date: Thu, 7 Aug 2014 15:07:43 -0400 Subject: [Mailman-Developers] [CLI Project] Added Backup and Restore Tool In-Reply-To: <87oavwke12.fsf@uwakimon.sk.tsukuba.ac.jp> References: <53DFD7A5.8010003@gmail.com> <53DFD839.5070903@gmail.com> <87y4v2jew2.fsf@uwakimon.sk.tsukuba.ac.jp> <53E274A6.2020205@gmail.com> <87oavwke12.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20140807150743.6a00a506@anarchist.wooz.org> On Aug 07, 2014, at 03:30 PM, Stephen J. Turnbull wrote: >Of course. I didn't explain myself well. What you have now is > >mailman/ -+- client/ -+- _client.py > +- docs/ > +- tests/ > +- cli/ -+- client.py > +- core/ > +- docs/ > +- lib/ > +- tests/ > >and what I would like to see is something like > >mailman/ -+- client/ -+- _client.py > | +- docs/ > | +- tests/ > +- cli/ -+- client.py > +- core/ > +- docs/ > +- lib/ > +- tests/ FWIW, my intention is to make the `mailman` package a pure namespace package[*] so potentially lots of sourcefully-separate subpackages could live under it. That can't happen until Mailman 3 is a Python 3 project, but that is still in the plans. Cheers, -Barry [*] http://legacy.python.org/dev/peps/pep-0420/ https://docs.python.org/3/reference/import.html#namespace-packages From advax at triumf.ca Fri Aug 8 03:53:09 2014 From: advax at triumf.ca (Andrew Daviel) Date: Thu, 7 Aug 2014 18:53:09 -0700 (PDT) Subject: [Mailman-Developers] Invalid HTML in mailman 2.1.18 Message-ID: I have wasted some time trying to get mailman 2.1.18 archives and webpages to pass the W3C online validator (validator.w3.org) - a probably pointless exercise if 2.1 is not under active development. However, there are a few pieces of low-hanging-fruit. - in the mailto URLs in templates, the &In-Reply-To parameter should be escaped to become &In-Reply-To (a browser will unescape it before passing the URL to a mail client) -