From skoranda at gmail.com Thu Mar 8 11:34:50 2018 From: skoranda at gmail.com (Scott Koranda) Date: Thu, 8 Mar 2018 10:34:50 -0600 Subject: [Mailman-Developers] 3.1.x REST API enhancements around preferred email address In-Reply-To: <1512108961.11845.12.camel@asynchronous.in> References: <20171120172013.GC33354@paprika.localdomain> <1512108961.11845.12.camel@asynchronous.in> Message-ID: <20180308163450.zjhget43ursc26im@paprika.local> Hello, My apologies for a late reply. The work to slightly extend the REST API has once again come to the top of my queue. Please see inline comments and a question below. > Hi Scott, > > On Mon, 2017-11-20 at 11:20 -0600, Scott Koranda wrote: > > Hello, > > > > I am interested in contributing enhancements around preferred address > > to the REST API for version 3.1.x. > > > > To prove out my use case(s) and familiarize myself with the code I have > > made changes already. Here are some example curl command line > > invocations and output to illustrate: > > > > 1) Create an address for an existing user and mark that address as the > > user's preferred address: > > > > curl -u restadmin:restpass \ > > -X POST \ > > -d email=skoranda at example.nil \ > > -d preferred=1 \ > > http://mytestbed.com:8001/3.1/users/e57ecfd0c0c74a319ac958b18882a6d8/addr > > esses > > This returns a '201 Created'. > > > > 2) Return the preferred address, if set, for a user: > > > > curl -u restadmin:restpass \ > > -X GET \ > > http://mystestbed.com:8001/3.1/users/e57ecfd0c0c74a319ac958b18882a6d8 | > > python -m json.tool > > > > { > > "created_on": "2017-11-11T22:20:45.950949", > > "display_name": "Scott Koranda", > > "http_etag": "\"26bc0d9f21eb145248884aedae4c1ffd00b608d2\"", > > "is_server_owner": false, > > "password": > > "$6$rounds=656000$gWGfwpajfZ7rVn5O$Sl559B2TgtpJWXA2i67G5ukjzkV6iTp4NgP.6FJpMFM > > UTDdDXULAmwdN8YW92w87EdctgqFqAUkUqS6.EOTCz/", > > "preferred_address": "skoranda at example.nil", > > "self_link": "http://mytestbed.com:8001/3.1/users/e57ecfd0c0c74a319ac958b1 > > 8882a6d8", > > "user_id": "e57ecfd0c0c74a319ac958b18882a6d8" > > } > > > > 3) Update/patch a user to set a preferred address: > > > > curl -u restadmin:restpass \ > > -X PATCH \ > > -d preferred_address=skoranda at example.nil \ > > http://mytestbed.com:8001/3.1/users/e57ecfd0c0c74a319ac958b18882a6d8 > > > > This returns a '204 No Content'. > > > > 4) Get all addresses for a user with the preferred address (if set) having > > the attribute 'preferred' set to 'true': > > > > curl -u restadmin:restpass \ > > -X GET \ > > http://mytestbed.com:8001/3.1/users/e57ecfd0c0c74a319ac958b18882a6d8/addr > > esses | python -m json.tool > > > > { > > "entries": [ > > { > > "email": "skoranda at example.nil", > > "http_etag": "\"492824d150f9f11e32d530d3cad0f76422bddb48\"", > > "original_email": "skoranda01 at example.nil", > > "preferred": true, > > "registered_on": "2017-11-13T22:35:17.232180", > > "self_link": "http://mytestbed.com:8001/3.1/addresses/skoranda at exa > > mple.nil", > > "user": "http://mytestbed.com:8001/3.1/users/e57ecfd0c0c74a319ac95 > > 8b18882a6d8", > > "verified_on": "2017-11-20T16:48:51.738145" > > }, > > { > > "email": "skoranda01 at example.nil", > > "http_etag": "\"f0f28c1c6a925a595269d3fe21ef7fc1515d7670\"", > > "original_email": "skoranda01 at example.nil", > > "registered_on": "2017-11-20T16:46:20.941262", > > "self_link": "http://mytestbed.com:8001/3.1/addresses/skoranda01 at e > > xample.nil", > > "user": "http://mytestbed.com:8001/3.1/users/e57ecfd0c0c74a319ac95 > > 8b18882a6d8", > > "verified_on": "2017-11-20T16:46:20.941595" > > }, > > ], > > "http_etag": "\"89c46bd312c6f1a344e6e6180aed6f11b62e6048\"", > > "start": 0, > > "total_size": 2 > > } > > > > Before I submit a pull request I would be grateful for any comments > > or feedback on the functionality as illustrated above, or any other comments > > or feedback. > > I think these all seem like reasonable additions. Thank you for looking at it in detail. > - an extra `preferred` boolean attribute when creating an address is good, I > couldn't find an endpoint to actually change the preferred_address of a user, so > this is good. > > - I think instead of embedding the `preferred_address` in user (/user/), we > could have a sub-resource that represents the preferred address, if any exists. > so something like /user/ - The rest of the operations about the preferred address would then be based on > GET/POST on this sub-resource. I don't think there is a need to do PATCH for > this sub-resource. > > I hope that it was helpful! It was, thank you. > How does that sound? Good. I would like to proceed in a way that makes it as easy as possible for you to consider the changes. > > > > I will also begin the "copyright assignment" paperwork as explained at > > > > https://wiki.list.org/DEV/Home > > Great! I have sent the first email/form to fsf-records at gnu.org and cc'd mailman-cabal at python.org. I will email again after I have sent the second form. My primary question is which branch at https://gitlab.com/mailman/mailman.git should I work from and then target a merge request at? I presume it would be the release-3.1 branch. Is that correct, or should it be against master? Thank you, Scott K From kamransoomro84 at gmail.com Thu Mar 8 09:32:09 2018 From: kamransoomro84 at gmail.com (Kamran Soomro) Date: Thu, 8 Mar 2018 14:32:09 +0000 Subject: [Mailman-Developers] mailman3_router router configuration for exim4 not working Message-ID: <9789D06D-3FEE-4D57-BA73-3F35DC0F15CA@gmail.com> Hi, I have followed the instructions at https://github.com/maxking/docker-mailman and copied the config files for exim4 from the core/assets/exim directory to /etc/exim4/conf.d/main. However, when I run update-exim4.conf I get the following error; 2018-03-08 14:28:59 Exim configuration error in line 238 of /var/lib/exim4/config.autogenerated.tmp: main option "mailman3_router" unknown Invalid new configfile /var/lib/exim4/config.autogenerated.tmp, not installing /var/lib/exim4/config.autogenerated.tmp to /var/lib/exim4/config.autogenerated After a bit of investigation I noticed that the mailman3 router and transport configurations appear before the begin router and begin transport sections in the autogenerate file. Is this expected? If not, how can I fix this? ?Kamran -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3870 bytes Desc: not available URL: From mark at msapiro.net Mon Mar 19 18:30:39 2018 From: mark at msapiro.net (Mark Sapiro) Date: Mon, 19 Mar 2018 15:30:39 -0700 Subject: [Mailman-Developers] Problem running tox on hyperkitty Message-ID: <5bda5980-379a-d3d0-e9e5-dbc95b04b4fe@msapiro.net> I am trying to run tox -e py35-django111 on HyperKitty. I run into an issue in that the tests run up to a point after which all but the last 2 fail with a "too many open files" exception. These files are logs created in directories /tmp/hyperkitty-testing-*, and over 100 /tmp/hyperkitty-testing-* directories are left behind. This is with an open file limit of 1024. If I raise the limit to 4096, everything runs and no /tmp/hyperkitty-testing-* directories are left behind, but it seems perhaps hyperkitty/tests/utils.TestCase._post_teardown() should have something added to undo what hyperkitty/tests/utils.setup_logging() sets up. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From maxking at asynchronous.in Mon Mar 19 18:57:40 2018 From: maxking at asynchronous.in (Abhilash Raj) Date: Mon, 19 Mar 2018 15:57:40 -0700 Subject: [Mailman-Developers] Problem running tox on hyperkitty In-Reply-To: <5bda5980-379a-d3d0-e9e5-dbc95b04b4fe@msapiro.net> References: <5bda5980-379a-d3d0-e9e5-dbc95b04b4fe@msapiro.net> Message-ID: <1521500260.2832734.1308902248.62943034@webmail.messagingengine.com> On Mon, Mar 19, 2018, at 3:30 PM, Mark Sapiro wrote: > I am trying to run > > tox -e py35-django111 > > on HyperKitty. I run into an issue in that the tests run up to a point > after which all but the last 2 fail with a "too many open files" > exception. These files are logs created in directories > /tmp/hyperkitty-testing-*, and over 100 /tmp/hyperkitty-testing-* > directories are left behind. This is with an open file limit of 1024. > > If I raise the limit to 4096, everything runs and no > /tmp/hyperkitty-testing-* directories are left behind, but it seems > perhaps hyperkitty/tests/utils.TestCase._post_teardown() should have > something added to undo what hyperkitty/tests/utils.setup_logging() sets up. Thanks a lot to find the real reason for this problem. I did find this problem when I was working on the Python 3 port but couldn't really pin-point the real reason for why is this happening. > > -- > Mark Sapiro The highway is for gamblers, > San Francisco Bay Area, California better use your sense - B. Dylan > _______________________________________________ > Mailman-Developers mailing list > Mailman-Developers at python.org > https://mail.python.org/mailman/listinfo/mailman-developers > Mailman FAQ: http://wiki.list.org/x/AgA3 > Searchable Archives: > http://www.mail-archive.com/mailman-developers%40python.org/ > Unsubscribe: > https://mail.python.org/mailman/options/mailman-developers/raj.abhilash1%40gmail.com > > Security Policy: http://wiki.list.org/x/QIA9 -- Abhilash Raj maxking at asynchronous.in From leon at green-side.de Tue Mar 20 08:57:34 2018 From: leon at green-side.de (Leon Merten Lohse) Date: Tue, 20 Mar 2018 13:57:34 +0100 Subject: [Mailman-Developers] Problem running tox on hyperkitty In-Reply-To: <5bda5980-379a-d3d0-e9e5-dbc95b04b4fe@msapiro.net> References: <5bda5980-379a-d3d0-e9e5-dbc95b04b4fe@msapiro.net> Message-ID: <01cef182-96fb-1b0d-de3d-95421ce7e439@green-side.de> On 19.03.2018 23:30, Mark Sapiro wrote: > I am trying to run > > tox -e py35-django111 > > on HyperKitty. I run into an issue in that the tests run up to a point > after which all but the last 2 fail with a "too many open files" > exception. These files are logs created in directories > /tmp/hyperkitty-testing-*, and over 100 /tmp/hyperkitty-testing-* > directories are left behind. This is with an open file limit of 1024. It might look like that but the logs are not the actual problem as those files are properly closed after the test finishes. I tried to disable logging to files all together but the problem persisted. The real cause for the excessive number of file descriptors is django-q, which creates opens lots and lots of files (for inter process communication?). Best Leon From maxking at asynchronous.in Tue Mar 20 12:03:44 2018 From: maxking at asynchronous.in (Abhilash Raj) Date: Tue, 20 Mar 2018 09:03:44 -0700 Subject: [Mailman-Developers] Problem running tox on hyperkitty In-Reply-To: <01cef182-96fb-1b0d-de3d-95421ce7e439@green-side.de> References: <5bda5980-379a-d3d0-e9e5-dbc95b04b4fe@msapiro.net> <01cef182-96fb-1b0d-de3d-95421ce7e439@green-side.de> Message-ID: <1521561824.713408.1309787656.25BBBDFE@webmail.messagingengine.com> On Tue, Mar 20, 2018, at 5:57 AM, Leon Merten Lohse wrote: > On 19.03.2018 23:30, Mark Sapiro wrote: > > I am trying to run > > > > tox -e py35-django111 > > > > on HyperKitty. I run into an issue in that the tests run up to a point > > after which all but the last 2 fail with a "too many open files" > > exception. These files are logs created in directories > > /tmp/hyperkitty-testing-*, and over 100 /tmp/hyperkitty-testing-* > > directories are left behind. This is with an open file limit of 1024. > > It might look like that but the logs are not the actual problem as those > files are properly closed after the test finishes. > > I tried to disable logging to files all together but the problem persisted. > > The real cause for the excessive number of file descriptors is django-q, > which creates opens lots and lots of files (for inter process > communication?). It was trying to create File Descriptors and failing every time, but I am not 100% sure if that itself is the reason for this. It could be, but I can't replicate this issue easily, maybe filing a bug with django-q and seeing what they think might be worthwhile? > > Best > Leon > _______________________________________________ > Mailman-Developers mailing list > Mailman-Developers at python.org > https://mail.python.org/mailman/listinfo/mailman-developers > Mailman FAQ: http://wiki.list.org/x/AgA3 > Searchable Archives: > http://www.mail-archive.com/mailman-developers%40python.org/ > Unsubscribe: > https://mail.python.org/mailman/options/mailman-developers/raj.abhilash1%40gmail.com > > Security Policy: http://wiki.list.org/x/QIA9 -- Abhilash Raj maxking at asynchronous.in From maxking at asynchronous.in Tue Mar 20 12:07:20 2018 From: maxking at asynchronous.in (Abhilash Raj) Date: Tue, 20 Mar 2018 09:07:20 -0700 Subject: [Mailman-Developers] Problem running tox on hyperkitty In-Reply-To: <1521561824.713408.1309787656.25BBBDFE@webmail.messagingengine.com> References: <5bda5980-379a-d3d0-e9e5-dbc95b04b4fe@msapiro.net> <01cef182-96fb-1b0d-de3d-95421ce7e439@green-side.de> <1521561824.713408.1309787656.25BBBDFE@webmail.messagingengine.com> Message-ID: <1521562040.715207.1309795472.69E82993@webmail.messagingengine.com> On Tue, Mar 20, 2018, at 9:03 AM, Abhilash Raj wrote: > > > On Tue, Mar 20, 2018, at 5:57 AM, Leon Merten Lohse wrote: > > On 19.03.2018 23:30, Mark Sapiro wrote: > > > I am trying to run > > > > > > tox -e py35-django111 > > > > > > on HyperKitty. I run into an issue in that the tests run up to a point > > > after which all but the last 2 fail with a "too many open files" > > > exception. These files are logs created in directories > > > /tmp/hyperkitty-testing-*, and over 100 /tmp/hyperkitty-testing-* > > > directories are left behind. This is with an open file limit of 1024. > > > > It might look like that but the logs are not the actual problem as those > > files are properly closed after the test finishes. > > > > I tried to disable logging to files all together but the problem persisted. > > > > The real cause for the excessive number of file descriptors is django-q, > > which creates opens lots and lots of files (for inter process > > communication?). > > It was trying to create File Descriptors and failing every time, but I > am not 100% sure if that itself is the reason for this. It could be, but > I can't replicate this issue easily, What I meant was, I can replicate the issue but haven't yet been able to find something that fixes this issue. I really shouldn't email before morning coffee. > maybe filing a bug with django-q > and seeing what they think might be worthwhile? > > > > > Best > > Leon > > _______________________________________________ > > Mailman-Developers mailing list > > Mailman-Developers at python.org > > https://mail.python.org/mailman/listinfo/mailman-developers > > Mailman FAQ: http://wiki.list.org/x/AgA3 > > Searchable Archives: > > http://www.mail-archive.com/mailman-developers%40python.org/ > > Unsubscribe: > > https://mail.python.org/mailman/options/mailman-developers/raj.abhilash1%40gmail.com > > > > Security Policy: http://wiki.list.org/x/QIA9 > > > -- > Abhilash Raj > maxking at asynchronous.in > _______________________________________________ > Mailman-Developers mailing list > Mailman-Developers at python.org > https://mail.python.org/mailman/listinfo/mailman-developers > Mailman FAQ: http://wiki.list.org/x/AgA3 > Searchable Archives: > http://www.mail-archive.com/mailman-developers%40python.org/ > Unsubscribe: > https://mail.python.org/mailman/options/mailman-developers/raj.abhilash1%40gmail.com > > Security Policy: http://wiki.list.org/x/QIA9 -- Abhilash Raj maxking at asynchronous.in From leon at green-side.de Tue Mar 20 13:37:13 2018 From: leon at green-side.de (Leon Merten Lohse) Date: Tue, 20 Mar 2018 18:37:13 +0100 Subject: [Mailman-Developers] Hyperkitty and Django Q Message-ID: <9fc83567-9609-9c24-a27f-0142f6618a3f@green-side.de> Startled by the observation that Hyperkitty's unit tests to fail because an excessive amount of file descriptors are opened, I began to dig a little. The majority of these file descriptors are opened by "Django Q" -- a django library for asynchronous tasks. I wondered what these asynchronous tasks might be in an e-mail archiver. After all, it has to perform 2 very specific jobs: 1. Receive e-mails via http requests, process them, and store them in a database. 2. Display the collected e-mails. On top of that, it generates some statistics, includes a voting system, and allows replying to e-mails in a forum-like fashion. Job 1 really requires very little processing and is mostly database IO as the message has to be sorted and its position in the thread has to be computed. None of this can be efficiently performed in the background/asynchronously because there is nothing to be parallelized. Performing the *whole* receive-and-store operation non-blocking also is not an option -- ultimately mailman needs to know if it succeeded. Job 2 is more or less what every web application does (reading from a database and rendering the result) and certainly does not require any asynchronous processing. So where is Django Q used in Hyperkitty? It all boils down to the file hyperkitty/tasks.py where a smaller number of asynchronous tasks are defined. They can be grouped into 3 classes: - query mailman core - repair the data structure (empty threads, orphans, ...) - rebuild query cache I would like to argue that none of these 3 groups of tasks need to be performed asynchronously. - Mailman-core only needs to be queried when mailing lists are changed. This is triggered by signals from postorius and in addition periodically by a cron job. - The data structure should not need to be repaired in the first place but the appropriate on_delete/on_save triggers should take care of this -- and I believe they do in recent versions. If for some reason the database becomes corrupted one can always start a repair operation. Nothing is gained however, from running this asynchronously. - Lastly the cache rebuild: Currently Hyperkitty rebuilds its cache (which caches the db queries, not the frontend) whenever an e-mail is received. Since it only involves *reading* from the db it actually is something that *can* be done asynchronously to reduce the time it takes to process an incoming e-mail. But is it really worth the tremendous additional complexity that is introduced by Django Q? - requires a "qcluster" to run in the background (see shipped unit file) - loss of determinism / debugging becomes much harder - enourmous amount of file descriptors are opened in testing - additional dependency A similar result can be obtained by simply scaling the wsgi application accordingly (if needed) and/or optimizing the db queries. Alternatively one could simply invalidate the affected caches instead of rebuild them every time an e-mail is received or don't trigger cache rebuilds on received e-mails at all... But maybe I overlooked something. I argue that we do not *need* Django Q. The question is: do we want it? What are your thoughts on this? Best Leon