From andrew.stuart at supercoders.com.au Sun Feb 1 01:31:10 2015 From: andrew.stuart at supercoders.com.au (Andrew Stuart) Date: Sun, 1 Feb 2015 11:31:10 +1100 Subject: [Mailman-Developers] User id - integer or UUID? Message-ID: I have read previous discussion here about format of user_id https://mail.python.org/pipermail/mailman-developers/2014-October/024024.html That?s fine, I understand the issues. The question is, I am storing user ids and it?s not clear to me if I should be storing them as INTs as returned by the REST API or converting back to UUID? It?s important to get right now because Im going to use the same approach everywhere. thanks From andrew.stuart at supercoders.com.au Sun Feb 1 01:54:41 2015 From: andrew.stuart at supercoders.com.au (Andrew Stuart) Date: Sun, 1 Feb 2015 11:54:41 +1100 Subject: [Mailman-Developers] User id - integer or UUID? In-Reply-To: References: Message-ID: <6E69AAA7-A83A-41C6-A742-9F32E414C28A@supercoders.com.au> I?m going to store as string representation of the integer as I don?t want to be dealing with databases or anything else that makes a distinction between different types and lengths of integer. Ideally the identifier would match the core internal identifier which is UUID but since userid-as-UUID is not seen from the REST API side of things then I?ll act like that concept does not exist and hope I?m not making things more confused around user id. On 1 Feb 2015, at 11:31 am, Andrew Stuart wrote: I have read previous discussion here about format of user_id https://mail.python.org/pipermail/mailman-developers/2014-October/024024.html That?s fine, I understand the issues. The question is, I am storing user ids and it?s not clear to me if I should be storing them as INTs as returned by the REST API or converting back to UUID? It?s important to get right now because Im going to use the same approach everywhere. thanks _______________________________________________ Mailman-Developers mailing list Mailman-Developers at python.org https://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/andrew.stuart%40supercoders.com.au Security Policy: http://wiki.list.org/x/QIA9 From andrew.stuart at supercoders.com.au Sun Feb 1 09:23:49 2015 From: andrew.stuart at supercoders.com.au (Andrew Stuart) Date: Sun, 1 Feb 2015 19:23:49 +1100 Subject: [Mailman-Developers] "Address already exists" when creating users, but it creates a new user anyway Message-ID: <5165C491-1D29-43D7-9A39-0AFF1A4DD77F@supercoders.com.au> In the commands below you can see that I start by getting a list of users, it returns "total_size": 1 I then execute a POST to the REST API multiple times. Each time it says ?Address already exists?, and then getting a list of users says total_size": 2 Is this the way its intended to work or a bug? thanks as ubuntu at server01:~$ ubuntu at server01:~$ curl --header "authorization: Basic cmVzdGFkbWluOnJlc3RwYXNz" http://localhost:8001/3.0/users {"entries": [{"password": "$6$rounds=109794$l/9MF2/dD7xSnoLD$6j3t0chh/UCnq61b3myleUaoHJEe4gNcidViWVR8kebKwU5oIVZ0ogsDiD7kgVr2V26zy8PaSfC5uQSY6rC0G1", "self_link": "http://localhost:8001/3.0/users/282607068158551958257628859360097200763", "user_id": 282607068158551958257628859360097200763, "http_etag": "\"7d81842c0a54679d42049c7eb25d64d2fd658b1a\"", "created_on": "2015-02-01T08:17:27.577221"}], "total_size": 1, "http_etag": "\"ae06e3745039c38b8a5c33d8bb468ba538fbb88a\"", "start": 0}ubuntu at server01:~$ ubuntu at server01:~$ ubuntu at server01:~$ curl -X POST --data "email=mailmanadmin at example.org" --header "authorization: Basic cmVzdGFkbWluOnJlc3RwYXNz" http://localhost:8001/3.0/users Address already exists: mailmanadmin at example.orgubuntu@server01:~$ ubuntu at server01:~$ ubuntu at server01:~$ curl --header "authorization: Basic cmVzdGFkbWluOnJlc3RwYXNz" http://localhost:8001/3.0/users {"entries": [{"password": "$6$rounds=109794$l/9MF2/dD7xSnoLD$6j3t0chh/UCnq61b3myleUaoHJEe4gNcidViWVR8kebKwU5oIVZ0ogsDiD7kgVr2V26zy8PaSfC5uQSY6rC0G1", "self_link": "http://localhost:8001/3.0/users/282607068158551958257628859360097200763", "user_id": 282607068158551958257628859360097200763, "http_etag": "\"7d81842c0a54679d42049c7eb25d64d2fd658b1a\"", "created_on": "2015-02-01T08:17:27.577221"}, {"self_link": "http://localhost:8001/3.0/users/92205393033160387796290186916038781151", "user_id": 92205393033160387796290186916038781151, "http_etag": "\"d44893d5fd8dccb797d19372c068d24b3d45dde9\"", "created_on": "2015-02-01T08:18:30.658821"}], "total_size": 2, "http_etag": "\"7060713ada4dbbbc97a8da2f1c3545f41c0878af\"", "start": 0}ubuntu at server01:~$ ubuntu at server01:~$ ubuntu at server01:~$ curl -X POST --data "email=mailmanadmin at example.org" --header "authorization: Basic cmVzdGFkbWluOnJlc3RwYXNz" http://localhost:8001/3.0/users Address already exists: mailmanadmin at example.orgubuntu@server01:~$ ubuntu at server01:~$ ubuntu at server01:~$ curl --header "authorization: Basic cmVzdGFkbWluOnJlc3RwYXNz" http://localhost:8001/3.0/users {"entries": [{"password": "$6$rounds=109794$l/9MF2/dD7xSnoLD$6j3t0chh/UCnq61b3myleUaoHJEe4gNcidViWVR8kebKwU5oIVZ0ogsDiD7kgVr2V26zy8PaSfC5uQSY6rC0G1", "self_link": "http://localhost:8001/3.0/users/282607068158551958257628859360097200763", "user_id": 282607068158551958257628859360097200763, "http_etag": "\"7d81842c0a54679d42049c7eb25d64d2fd658b1a\"", "created_on": "2015-02-01T08:17:27.577221"}, {"self_link": "http://localhost:8001/3.0/users/92205393033160387796290186916038781151", "user_id": 92205393033160387796290186916038781151, "http_etag": "\"d44893d5fd8dccb797d19372c068d24b3d45dde9\"", "created_on": "2015-02-01T08:18:30.658821"}, {"self_link": "http://localhost:8001/3.0/users/276766536585189861964818785630123864804", "user_id": 276766536585189861964818785630123864804, "http_etag": "\"edf93f04566715b1afaeaee0368bc2101aa0d8b3\"", "created_on": "2015-02-01T08:19:03.900670"}], "total_size": 3, "http_etag": "\"49ffe8ac58d624c250ec6af339f4cab609f4e138\"", "start": 0}ubuntu at server01:~$ ubuntu at server01:~$ From barry at list.org Sun Feb 1 23:28:11 2015 From: barry at list.org (Barry Warsaw) Date: Sun, 1 Feb 2015 17:28:11 -0500 Subject: [Mailman-Developers] "Address already exists" when creating users, but it creates a new user anyway In-Reply-To: <5165C491-1D29-43D7-9A39-0AFF1A4DD77F@supercoders.com.au> References: <5165C491-1D29-43D7-9A39-0AFF1A4DD77F@supercoders.com.au> Message-ID: <20150201172811.1179a544@limelight.wooz.org> On Feb 01, 2015, at 07:23 PM, Andrew Stuart wrote: >In the commands below you can see that I start by getting a list of users, it >returns "total_size": 1 How was this user created and with what email address? >I then execute a POST to the REST API multiple times. Each time it says >?Address already exists?, and then getting a list of users says total_size": >2 > >Is this the way its intended to work or a bug? Nope, and I can't reproduce it. Are you sure there are no other processes running, and nothing else hitting the REST API that could be racing with that POST? Try tailing the mailman.log and watching for http activity. Cheers, -Barry From barry at list.org Sun Feb 1 23:33:04 2015 From: barry at list.org (Barry Warsaw) Date: Sun, 1 Feb 2015 17:33:04 -0500 Subject: [Mailman-Developers] User id - integer or UUID? In-Reply-To: <6E69AAA7-A83A-41C6-A742-9F32E414C28A@supercoders.com.au> References: <6E69AAA7-A83A-41C6-A742-9F32E414C28A@supercoders.com.au> Message-ID: <20150201173304.114d794f@limelight.wooz.org> On Feb 01, 2015, at 11:54 AM, Andrew Stuart wrote: >I?m going to store as string representation of the integer as I don?t want to >be dealing with databases or anything else that makes a distinction between >different types and lengths of integer. That makes sense to me. Since Aurelien was okay with the status quo, I never did make the change described in that archive message. >Ideally the identifier would match the core internal identifier which is UUID >but since userid-as-UUID is not seen from the REST API side of things then >I?ll act like that concept does not exist and hope I?m not making things more >confused around user id. Yep, for now I think we'll keep the int-as-user-id concept, and it will be converted internally to a UUID. If/when we change that I'll have to keep backward compatibility with that API anyway. Cheers, -Barry From andrew.stuart at supercoders.com.au Mon Feb 2 00:04:36 2015 From: andrew.stuart at supercoders.com.au (Andrew Stuart) Date: Mon, 2 Feb 2015 10:04:36 +1100 Subject: [Mailman-Developers] User id - integer or UUID? In-Reply-To: <20150201173304.114d794f@limelight.wooz.org> References: <6E69AAA7-A83A-41C6-A742-9F32E414C28A@supercoders.com.au> <20150201173304.114d794f@limelight.wooz.org> Message-ID: <3F5EBA93-2AEF-4212-892B-B28FE9BB2C3C@supercoders.com.au> That?s fine I?m good with that. The other possibility, distasteful as it may be, is to make the changes to publish the UUID to the REST API and deprecate the use of the user_id as integer. I suggest this because others who come after me will have the same question and may run into the same issues as I have in trying to work out a ?broadly compatible? way of handling/storing integers. They?ll probably come up with their own solution - perhaps to store user ID?s as a variety of integer types/lengths - perhaps to convert to UUID - perhaps to store integer-as-string, and the question might be raised again. Short term pain now to move to UUID and deprecate user_id as integer, possibly longer term gain with clarity that a user_id is a UUID. Anyway I don?t mind much anyway I?ll just do as Barry says?.. as On 2 Feb 2015, at 9:33 am, Barry Warsaw wrote: On Feb 01, 2015, at 11:54 AM, Andrew Stuart wrote: > I?m going to store as string representation of the integer as I don?t want to > be dealing with databases or anything else that makes a distinction between > different types and lengths of integer. That makes sense to me. Since Aurelien was okay with the status quo, I never did make the change described in that archive message. > Ideally the identifier would match the core internal identifier which is UUID > but since userid-as-UUID is not seen from the REST API side of things then > I?ll act like that concept does not exist and hope I?m not making things more > confused around user id. Yep, for now I think we'll keep the int-as-user-id concept, and it will be converted internally to a UUID. If/when we change that I'll have to keep backward compatibility with that API anyway. Cheers, -Barry _______________________________________________ Mailman-Developers mailing list Mailman-Developers at python.org https://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/andrew.stuart%40supercoders.com.au Security Policy: http://wiki.list.org/x/QIA9 From andrew.stuart at supercoders.com.au Wed Feb 4 08:53:08 2015 From: andrew.stuart at supercoders.com.au (Andrew Stuart) Date: Wed, 4 Feb 2015 18:53:08 +1100 Subject: [Mailman-Developers] How to drop and recreate database using Alembic? Message-ID: <0406FF32-DAAF-4C73-A9D0-93C49FC6E7DD@supercoders.com.au> Anyone got quick tips for dropping the entiure database and recreating it with alembic? I can?t find any instructions - if there are some would you mind please pointing me to them? thanks as From raj.abhilash1 at gmail.com Wed Feb 4 09:04:58 2015 From: raj.abhilash1 at gmail.com (Abhilash Raj) Date: Wed, 04 Feb 2015 13:34:58 +0530 Subject: [Mailman-Developers] How to drop and recreate database using Alembic? In-Reply-To: <0406FF32-DAAF-4C73-A9D0-93C49FC6E7DD@supercoders.com.au> References: <0406FF32-DAAF-4C73-A9D0-93C49FC6E7DD@supercoders.com.au> Message-ID: <54D1D2AA.1060501@gmail.com> On Wednesday 04 February 2015 01:23 PM, Andrew Stuart wrote: > Anyone got quick tips for dropping the entiure database and recreating it with alembic? Just delete the database - `var/data/mailman.db`. It would automatically be (re)created the next time you start mailman. No need to alembic for this purpose. -- thanks, Abhilash Raj -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From andrew.stuart at supercoders.com.au Wed Feb 4 11:26:41 2015 From: andrew.stuart at supercoders.com.au (Andrew Stuart) Date: Wed, 4 Feb 2015 21:26:41 +1100 Subject: [Mailman-Developers] "Address already exists" when creating users, but it creates a new user anyway In-Reply-To: <20150201172811.1179a544@limelight.wooz.org> References: <5165C491-1D29-43D7-9A39-0AFF1A4DD77F@supercoders.com.au> <20150201172811.1179a544@limelight.wooz.org> Message-ID: It?s reproducible for me, I?m not sure what might be different in our configurations. For a clean start I deleted my Mailman 3 installation. I downloaded the latest version and installed it. I deleted and recreated my databases after first making sure they were empty. To see the problem I execute the following command, which instructs the REST API to create a new user with email address of mailmanadmin at example.org (you should be able to execute this too, it?s the default REST username and password in the authorization header) curl -X POST --data "email=mailmanadmin at example.org" --header "authorization: Basic cmVzdGFkbWluOnJlc3RwYXNz" http://localhost:8001/3.0/users Each time the command above is executed, an additional record is added to preferences, uid and user, despite reporting an ?address already exists? error. I?m definitely not running anything else that would be accessing the REST API. I tried it on both Postgres and sqlite and got the same problem. I had a look at src/mailman/rest/users.py to see if I could see a bug but there wasn?t anything obvious that would lead to this happening. Here is the output of a sqlite dump after running the above curl command three times. (venv3.4)ubuntu at server01:~/mailman/var/data$ sqlite3 sqlite.db .dump PRAGMA foreign_keys=OFF; BEGIN TRANSACTION; CREATE TABLE preferences ( id INTEGER NOT NULL, acknowledge_posts BOOLEAN, hide_address BOOLEAN, preferred_language VARCHAR, receive_list_copy BOOLEAN, receive_own_postings BOOLEAN, delivery_mode INTEGER, delivery_status INTEGER, PRIMARY KEY (id), CHECK (acknowledge_posts IN (0, 1)), CHECK (hide_address IN (0, 1)), CHECK (receive_list_copy IN (0, 1)), CHECK (receive_own_postings IN (0, 1)) ); INSERT INTO "preferences" VALUES(1,NULL,NULL,NULL,NULL,NULL,NULL,NULL); INSERT INTO "preferences" VALUES(2,NULL,NULL,NULL,NULL,NULL,NULL,NULL); INSERT INTO "preferences" VALUES(3,NULL,NULL,NULL,NULL,NULL,NULL,NULL); INSERT INTO "preferences" VALUES(4,NULL,NULL,NULL,NULL,NULL,NULL,NULL); CREATE TABLE domain ( id INTEGER NOT NULL, mail_host VARCHAR, base_url VARCHAR, description VARCHAR, contact_address VARCHAR, PRIMARY KEY (id) ); CREATE TABLE mailinglist ( id INTEGER NOT NULL, list_name VARCHAR, mail_host VARCHAR, list_id VARCHAR, allow_list_posts BOOLEAN, include_rfc2369_headers BOOLEAN, advertised BOOLEAN, anonymous_list BOOLEAN, created_at DATETIME, next_request_id INTEGER, next_digest_number INTEGER, digest_last_sent_at DATETIME, volume INTEGER, last_post_at DATETIME, accept_these_nonmembers BLOB, admin_immed_notify BOOLEAN, admin_notify_mchanges BOOLEAN, administrivia BOOLEAN, archive_policy INTEGER, autoresponse_grace_period DATETIME, autorespond_owner INTEGER, autoresponse_owner_text VARCHAR, autorespond_postings INTEGER, autoresponse_postings_text VARCHAR, autorespond_requests INTEGER, autoresponse_request_text VARCHAR, filter_action INTEGER, filter_content BOOLEAN, collapse_alternatives BOOLEAN, convert_html_to_plaintext BOOLEAN, bounce_info_stale_after DATETIME, bounce_matching_headers VARCHAR, bounce_notify_owner_on_disable BOOLEAN, bounce_notify_owner_on_removal BOOLEAN, bounce_score_threshold INTEGER, bounce_you_are_disabled_warnings INTEGER, bounce_you_are_disabled_warnings_interval DATETIME, forward_unrecognized_bounces_to INTEGER, process_bounces BOOLEAN, default_member_action INTEGER, default_nonmember_action INTEGER, description VARCHAR, digest_footer_uri VARCHAR, digest_header_uri VARCHAR, digest_is_default BOOLEAN, digest_send_periodic BOOLEAN, digest_size_threshold FLOAT, digest_volume_frequency INTEGER, digestable BOOLEAN, discard_these_nonmembers BLOB, emergency BOOLEAN, encode_ascii_prefixes BOOLEAN, first_strip_reply_to BOOLEAN, footer_uri VARCHAR, forward_auto_discards BOOLEAN, gateway_to_mail BOOLEAN, gateway_to_news BOOLEAN, goodbye_message_uri VARCHAR, header_matches BLOB, header_uri VARCHAR, hold_these_nonmembers BLOB, info VARCHAR, linked_newsgroup VARCHAR, max_days_to_hold INTEGER, max_message_size INTEGER, max_num_recipients INTEGER, member_moderation_notice VARCHAR, mime_is_default_digest BOOLEAN, moderator_password BLOB, newsgroup_moderation INTEGER, nntp_prefix_subject_too BOOLEAN, nondigestable BOOLEAN, nonmember_rejection_notice VARCHAR, obscure_addresses BOOLEAN, owner_chain VARCHAR, owner_pipeline VARCHAR, personalize INTEGER, post_id INTEGER, posting_chain VARCHAR, posting_pipeline VARCHAR, preferred_language VARCHAR, display_name VARCHAR, reject_these_nonmembers BLOB, reply_goes_to_list INTEGER, reply_to_address VARCHAR, require_explicit_destination BOOLEAN, respond_to_post_requests BOOLEAN, scrub_nondigest BOOLEAN, send_goodbye_message BOOLEAN, send_welcome_message BOOLEAN, subject_prefix VARCHAR, topics BLOB, topics_bodylines_limit INTEGER, topics_enabled BOOLEAN, welcome_message_uri VARCHAR, PRIMARY KEY (id), CHECK (allow_list_posts IN (0, 1)), CHECK (include_rfc2369_headers IN (0, 1)), CHECK (advertised IN (0, 1)), CHECK (anonymous_list IN (0, 1)), CHECK (admin_immed_notify IN (0, 1)), CHECK (admin_notify_mchanges IN (0, 1)), CHECK (administrivia IN (0, 1)), CHECK (filter_content IN (0, 1)), CHECK (collapse_alternatives IN (0, 1)), CHECK (convert_html_to_plaintext IN (0, 1)), CHECK (bounce_notify_owner_on_disable IN (0, 1)), CHECK (bounce_notify_owner_on_removal IN (0, 1)), CHECK (process_bounces IN (0, 1)), CHECK (digest_is_default IN (0, 1)), CHECK (digest_send_periodic IN (0, 1)), CHECK (digestable IN (0, 1)), CHECK (emergency IN (0, 1)), CHECK (encode_ascii_prefixes IN (0, 1)), CHECK (first_strip_reply_to IN (0, 1)), CHECK (forward_auto_discards IN (0, 1)), CHECK (gateway_to_mail IN (0, 1)), CHECK (gateway_to_news IN (0, 1)), CHECK (mime_is_default_digest IN (0, 1)), CHECK (nntp_prefix_subject_too IN (0, 1)), CHECK (nondigestable IN (0, 1)), CHECK (obscure_addresses IN (0, 1)), CHECK (require_explicit_destination IN (0, 1)), CHECK (respond_to_post_requests IN (0, 1)), CHECK (scrub_nondigest IN (0, 1)), CHECK (send_goodbye_message IN (0, 1)), CHECK (send_welcome_message IN (0, 1)), CHECK (topics_enabled IN (0, 1)) ); CREATE TABLE bounceevent ( id INTEGER NOT NULL, list_id VARCHAR, email VARCHAR, timestamp DATETIME, message_id VARCHAR, context INTEGER, processed BOOLEAN, PRIMARY KEY (id), CHECK (processed IN (0, 1)) ); CREATE TABLE ban ( id INTEGER NOT NULL, email VARCHAR, list_id VARCHAR, PRIMARY KEY (id) ); CREATE TABLE pended ( id INTEGER NOT NULL, token VARCHAR, expiration_date DATETIME, PRIMARY KEY (id) ); CREATE TABLE uid ( id INTEGER NOT NULL, uid CHAR(32), PRIMARY KEY (id) ); INSERT INTO "uid" VALUES(1,'aefa4caf73b64729a0245deb9f85f640'); INSERT INTO "uid" VALUES(2,'66a84f1623694e468d375e47d5cb5030'); INSERT INTO "uid" VALUES(3,'1fc2c84e096c41e2ac07a01024415759'); CREATE TABLE message ( id INTEGER NOT NULL, message_id VARCHAR, message_id_hash VARCHAR, path VARCHAR, PRIMARY KEY (id) ); CREATE TABLE user ( id INTEGER NOT NULL, display_name VARCHAR, password VARCHAR, _user_id CHAR(32), _created_on DATETIME, _preferred_address_id INTEGER, preferences_id INTEGER, PRIMARY KEY (id), FOREIGN KEY(preferences_id) REFERENCES preferences (id) ); INSERT INTO "user" VALUES(1,'','$6$rounds=98929$cAg4vQkCh8KYEQYM$sNVQnEWkQc9i3mZiTrafNk5DdEL1N0qD2B1.pV68g8oX5hDvNysiHMzUGTEAcvcmdi4GsQS1OjkmZ9ZjkPyyd/','aefa4caf73b64729a0245deb9f85f640','2015-02-04 10:11:03.080036',NULL,1); INSERT INTO "user" VALUES(2,'',NULL,'66a84f1623694e468d375e47d5cb5030','2015-02-04 10:12:14.014487',NULL,3); INSERT INTO "user" VALUES(3,'',NULL,'1fc2c84e096c41e2ac07a01024415759','2015-02-04 10:13:47.575524',NULL,4); CREATE TABLE pendedkeyvalue ( id INTEGER NOT NULL, "key" VARCHAR, value VARCHAR, pended_id INTEGER, PRIMARY KEY (id), FOREIGN KEY(pended_id) REFERENCES pended (id) ); CREATE TABLE listarchiver ( id INTEGER NOT NULL, mailing_list_id INTEGER NOT NULL, name VARCHAR NOT NULL, _is_enabled BOOLEAN, PRIMARY KEY (id), FOREIGN KEY(mailing_list_id) REFERENCES mailinglist (id), CHECK (_is_enabled IN (0, 1)) ); CREATE TABLE acceptablealias ( id INTEGER NOT NULL, mailing_list_id INTEGER NOT NULL, alias VARCHAR NOT NULL, PRIMARY KEY (id), FOREIGN KEY(mailing_list_id) REFERENCES mailinglist (id) ); CREATE TABLE contentfilter ( id INTEGER NOT NULL, mailing_list_id INTEGER, filter_type INTEGER, filter_pattern VARCHAR, PRIMARY KEY (id), FOREIGN KEY(mailing_list_id) REFERENCES mailinglist (id) ); CREATE TABLE _request ( id INTEGER NOT NULL, "key" VARCHAR, request_type INTEGER, data_hash VARCHAR, mailing_list_id INTEGER, PRIMARY KEY (id), FOREIGN KEY(mailing_list_id) REFERENCES mailinglist (id) ); CREATE TABLE address ( id INTEGER NOT NULL, email VARCHAR, _original VARCHAR, display_name VARCHAR, verified_on DATETIME, registered_on DATETIME, user_id INTEGER, preferences_id INTEGER, PRIMARY KEY (id), FOREIGN KEY(user_id) REFERENCES user (id), FOREIGN KEY(preferences_id) REFERENCES preferences (id) ); INSERT INTO "address" VALUES(1,'mailmanadmin at example.org',NULL,'',NULL,'2015-02-04 10:11:03.108943',1,2); CREATE TABLE member ( id INTEGER NOT NULL, _member_id CHAR(32), role INTEGER, list_id VARCHAR, moderation_action INTEGER, address_id INTEGER, preferences_id INTEGER, user_id INTEGER, PRIMARY KEY (id), FOREIGN KEY(address_id) REFERENCES address (id), FOREIGN KEY(preferences_id) REFERENCES preferences (id), FOREIGN KEY(user_id) REFERENCES user (id) ); CREATE TABLE autoresponserecord ( id INTEGER NOT NULL, address_id INTEGER, mailing_list_id INTEGER, response_type INTEGER, date_sent DATE, PRIMARY KEY (id), FOREIGN KEY(address_id) REFERENCES address (id), FOREIGN KEY(mailing_list_id) REFERENCES mailinglist (id) ); CREATE TABLE onelastdigest ( id INTEGER NOT NULL, mailing_list_id INTEGER, address_id INTEGER, delivery_mode INTEGER, PRIMARY KEY (id), FOREIGN KEY(mailing_list_id) REFERENCES mailinglist (id), FOREIGN KEY(address_id) REFERENCES address (id) ); CREATE TABLE alembic_version ( version_num VARCHAR(32) NOT NULL ); INSERT INTO "alembic_version" VALUES('33e1f5f6fa8'); CREATE INDEX ix_uid_uid ON uid (uid); CREATE INDEX ix_user__user_id ON user (_user_id); CREATE INDEX ix_user_preferences_id ON user (preferences_id); CREATE INDEX ix_pendedkeyvalue_pended_id ON pendedkeyvalue (pended_id); CREATE INDEX ix_listarchiver_mailing_list_id ON listarchiver (mailing_list_id); CREATE INDEX ix_acceptablealias_alias ON acceptablealias (alias); CREATE INDEX ix_acceptablealias_mailing_list_id ON acceptablealias (mailing_list_id); CREATE INDEX ix_contentfilter_mailing_list_id ON contentfilter (mailing_list_id); CREATE INDEX ix__request_mailing_list_id ON _request (mailing_list_id); CREATE INDEX ix_address_user_id ON address (user_id); CREATE INDEX ix_address_preferences_id ON address (preferences_id); CREATE INDEX ix_autoresponserecord_address_id ON autoresponserecord (address_id); CREATE INDEX ix_autoresponserecord_mailing_list_id ON autoresponserecord (mailing_list_id); COMMIT; (venv3.4)ubuntu at server01:~/mailman/var/data$ On 2 Feb 2015, at 9:28 am, Barry Warsaw wrote: On Feb 01, 2015, at 07:23 PM, Andrew Stuart wrote: > In the commands below you can see that I start by getting a list of users, it > returns "total_size": 1 How was this user created and with what email address? > I then execute a POST to the REST API multiple times. Each time it says > ?Address already exists?, and then getting a list of users says total_size": > 2 > > Is this the way its intended to work or a bug? Nope, and I can't reproduce it. Are you sure there are no other processes running, and nothing else hitting the REST API that could be racing with that POST? Try tailing the mailman.log and watching for http activity. Cheers, -Barry _______________________________________________ Mailman-Developers mailing list Mailman-Developers at python.org https://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/andrew.stuart%40supercoders.com.au Security Policy: http://wiki.list.org/x/QIA9 From andrew.stuart at supercoders.com.au Wed Feb 4 12:32:13 2015 From: andrew.stuart at supercoders.com.au (Andrew Stuart) Date: Wed, 4 Feb 2015 22:32:13 +1100 Subject: [Mailman-Developers] "Address already exists" when creating users, but it creates a new user anyway In-Reply-To: References: <5165C491-1D29-43D7-9A39-0AFF1A4DD77F@supercoders.com.au> <20150201172811.1179a544@limelight.wooz.org> Message-ID: <05CAA912-9D33-492B-B1CF-167E077D712B@supercoders.com.au> After executing the curl command: curl -X POST --data "email=mailmanadmin at example.org" --header "authorization: Basic cmVzdGFkbWluOnJlc3RwYXNz" http://localhost:8001/3.0/users The log file adds the following: Feb 04 11:30:05 2015 (17776) BEGIN (implicit) Feb 04 11:30:05 2015 (17776) SELECT count(*) AS count_1 FROM (SELECT uid.id AS uid_id, uid.uid AS uid_uid FROM uid WHERE uid.uid = ?) AS anon_1 Feb 04 11:30:05 2015 (17776) ('57fdf5a214334b388d6efc3c375dc392',) Feb 04 11:30:05 2015 (17776) INSERT INTO uid (uid) VALUES (?) Feb 04 11:30:05 2015 (17776) ('57fdf5a214334b388d6efc3c375dc392',) Feb 04 11:30:05 2015 (17776) SELECT count(*) AS count_1 FROM (SELECT user.password AS user_password, user.id AS user_id, user.display_name AS user_display_name, user._user_id AS user__user_id, user._created_on AS user__created_on, user._preferred_address_id AS user__preferred_address_id, user.preferences_id AS user_preferences_id FROM user WHERE user._user_id = ?) AS anon_1 Feb 04 11:30:05 2015 (17776) ('57fdf5a214334b388d6efc3c375dc392',) Feb 04 11:30:05 2015 (17776) INSERT INTO preferences (acknowledge_posts, hide_address, preferred_language, receive_list_copy, receive_own_postings, delivery_mode, delivery_status) VALUES (?, ?, ?, ?, ?, ?, ?) Feb 04 11:30:05 2015 (17776) (None, None, None, None, None, None, None) Feb 04 11:30:05 2015 (17776) INSERT INTO user (display_name, password, _user_id, _created_on, _preferred_address_id, preferences_id) VALUES (?, ?, ?, ?, ?, ?) Feb 04 11:30:05 2015 (17776) ('', None, '57fdf5a214334b388d6efc3c375dc392', '2015-02-04 11:30:05.112836', None, 7) Feb 04 11:30:05 2015 (17776) SELECT count(*) AS count_1 FROM (SELECT address.verified_on AS address_verified_on, address.id AS address_id, address.email AS address_email, address._original AS address__original, address.display_name AS address_display_name, address.registered_on AS address_registered_on, address.user_id AS address_user_id, address.preferences_id AS address_preferences_id FROM address WHERE address.email = ?) AS anon_1 Feb 04 11:30:05 2015 (17776) ('mailmanadmin at example.org',) Feb 04 11:30:05 2015 (17776) SELECT address.verified_on AS address_verified_on, address.id AS address_id, address.email AS address_email, address._original AS address__original, address.display_name AS address_display_name, address.registered_on AS address_registered_on, address.user_id AS address_user_id, address.preferences_id AS address_preferences_id FROM address WHERE address.email = ? LIMIT ? OFFSET ? Feb 04 11:30:05 2015 (17776) ('mailmanadmin at example.org', 1, 0) Feb 04 11:30:05 2015 (17776) COMMIT From barry at list.org Wed Feb 4 15:53:21 2015 From: barry at list.org (Barry Warsaw) Date: Wed, 4 Feb 2015 09:53:21 -0500 Subject: [Mailman-Developers] "Address already exists" when creating users, but it creates a new user anyway In-Reply-To: References: <5165C491-1D29-43D7-9A39-0AFF1A4DD77F@supercoders.com.au> <20150201172811.1179a544@limelight.wooz.org> Message-ID: <20150204095321.09eaec4e@anarchist.wooz.org> Hi Andrew, Thanks for the detailed response. I'll try to find some time tonight to go through your recipe step-by-step. Some questions in the meantime: On Feb 04, 2015, at 09:26 PM, Andrew Stuart wrote: >For a clean start I deleted my Mailman 3 installation. I downloaded the >latest version and installed it. Are you working from the 3.0b5 tarball or the bzr trunk? If the former, are you using the "A" release (Python 2.7) or the "B" release (Python 3.4)? If you're using the tarball, can you please try again with the head of the bzr trunk? Cheers, -Barry From andrew.stuart at supercoders.com.au Wed Feb 4 23:32:38 2015 From: andrew.stuart at supercoders.com.au (Andrew Stuart) Date: Thu, 5 Feb 2015 09:32:38 +1100 Subject: [Mailman-Developers] "Address already exists" when creating users, but it creates a new user anyway In-Reply-To: <20150204155947.30a46af6@limelight.wooz.org> References: <5165C491-1D29-43D7-9A39-0AFF1A4DD77F@supercoders.com.au> <20150201172811.1179a544@limelight.wooz.org> <20150204095321.09eaec4e@anarchist.wooz.org> <20150204155947.30a46af6@limelight.wooz.org> Message-ID: <7B6EBF97-C111-4CF8-A92F-F41768A74A48@supercoders.com.au> After a bit of deep cave diving into the code I suspect it?s this in model/usermanager.py This function appears to create a user before it checks to see if the address exists, with the result that multiple users are created when the address already exists. @implementer(IUserManager) class UserManager: """See `IUserManager`.""" def create_user(self, email=None, display_name=None): """See `IUserManager`.""" user = User(display_name, Preferences()) if email: address = self.create_address(email, display_name) user.link(address) return user this seems to fix the problem - but please don?t trust my solution - needs verification. @implementer(IUserManager) class UserManager: """See `IUserManager`.""" def create_user(self, email=None, display_name=None): """See `IUserManager`.""" if email: address = self.create_address(email, display_name) user = User(display_name, Preferences()) if email: user.link(address) return user On 5 Feb 2015, at 7:59 am, Barry Warsaw wrote: On Feb 05, 2015, at 07:23 AM, Andrew Stuart wrote: > To get the source I do: > > bzr branch lp:mailman > > Is this the correct way? > > I?m using Python 3.4 Yep, exactly so. Thanks. I'll give it a go tonight. -B From andrew.stuart at supercoders.com.au Thu Feb 5 00:53:31 2015 From: andrew.stuart at supercoders.com.au (Andrew Stuart) Date: Thu, 5 Feb 2015 10:53:31 +1100 Subject: [Mailman-Developers] When a user is deleted, is the uid record intentionally not deleted? Message-ID: I notice that when a user is deleted, the uid record remains. Is this because there is a bug in which the uid should be deleted but is not? or is this intentional behaviour to prevent the uid from being recreated as a duplicate? My testing code does large numbers of creations and deletions of users so the deleted uid records are building up. From andrew.stuart at supercoders.com.au Thu Feb 5 01:55:03 2015 From: andrew.stuart at supercoders.com.au (Andrew Stuart) Date: Thu, 5 Feb 2015 11:55:03 +1100 Subject: [Mailman-Developers] Is there a way to delete an unlinked/orphaned address via the REST API? Message-ID: <7FC20051-DCE1-4BF0-826C-68350517B96E@supercoders.com.au> After unlinking an address from a user, there is an address in the addresses table that is orphaned - i.e. it does not have a linked user. This is the expected behaviour (I think) - that?s fine, no problems at this point. However, I can?t see a way to delete these addresses via the REST API. Does one exist? Should it exist? Perhaps so - without it there?s no way to delete orphaned email addresses. thanks as From andrew.stuart at supercoders.com.au Thu Feb 5 23:22:52 2015 From: andrew.stuart at supercoders.com.au (Andrew Stuart) Date: Fri, 6 Feb 2015 09:22:52 +1100 Subject: [Mailman-Developers] Security - XSS JavaScript and SQL injection in Mailman, Postorius and Hyperkitty Message-ID: <5E64A2E3-7E19-4333-AEF0-EF069AE99FE9@supercoders.com.au> Does the code of Mailman 3, Hyperkitty and Postorius do anything to address concerns around SQL and JavaScript injections, either from inbound emails or via the fields coming in via web interface or REST API? thanks From barry at list.org Fri Feb 6 00:18:09 2015 From: barry at list.org (Barry Warsaw) Date: Thu, 5 Feb 2015 18:18:09 -0500 Subject: [Mailman-Developers] Security - XSS JavaScript and SQL injection in Mailman, Postorius and Hyperkitty In-Reply-To: <5E64A2E3-7E19-4333-AEF0-EF069AE99FE9@supercoders.com.au> References: <5E64A2E3-7E19-4333-AEF0-EF069AE99FE9@supercoders.com.au> Message-ID: <20150205181809.075ab0fe@anarchist.wooz.org> On Feb 06, 2015, at 09:22 AM, Andrew Stuart wrote: >Does the code of Mailman 3, Hyperkitty and Postorius do anything to address >concerns around SQL and JavaScript injections, either from inbound emails or >via the fields coming in via web interface or REST API? The core does not. It doesn't expose a public web or REST interface. I'm not aware of any email command vulnerabilities. Cheers, -Barry From barry at list.org Fri Feb 6 04:05:10 2015 From: barry at list.org (Barry Warsaw) Date: Thu, 5 Feb 2015 22:05:10 -0500 Subject: [Mailman-Developers] "Address already exists" when creating users, but it creates a new user anyway In-Reply-To: <7B6EBF97-C111-4CF8-A92F-F41768A74A48@supercoders.com.au> References: <5165C491-1D29-43D7-9A39-0AFF1A4DD77F@supercoders.com.au> <20150201172811.1179a544@limelight.wooz.org> <20150204095321.09eaec4e@anarchist.wooz.org> <20150204155947.30a46af6@limelight.wooz.org> <7B6EBF97-C111-4CF8-A92F-F41768A74A48@supercoders.com.au> Message-ID: <20150205220510.68a16cf6@anarchist.wooz.org> On Feb 05, 2015, at 09:32 AM, Andrew Stuart wrote: >After a bit of deep cave diving into the code I suspect it?s this in >model/usermanager.py You're exactly right. Thanks for filing LP: #1418280. I'll commit your fix along with some new tests. Cheers, -Barry From barry at list.org Fri Feb 6 04:45:34 2015 From: barry at list.org (Barry Warsaw) Date: Thu, 5 Feb 2015 22:45:34 -0500 Subject: [Mailman-Developers] Is there a way to delete an unlinked/orphaned address via the REST API? In-Reply-To: <7FC20051-DCE1-4BF0-826C-68350517B96E@supercoders.com.au> References: <7FC20051-DCE1-4BF0-826C-68350517B96E@supercoders.com.au> Message-ID: <20150205224534.65839ed1@anarchist.wooz.org> On Feb 05, 2015, at 11:55 AM, Andrew Stuart wrote: >After unlinking an address from a user, there is an address in the addresses >table that is orphaned - i.e. it does not have a linked user. This is the >expected behaviour (I think) - that?s fine, no problems at this point. Correct. >However, I can?t see a way to delete these addresses via the REST API. Does >one exist? Should it exist? Perhaps so - without it there?s no way to delete >orphaned email addresses. Right. I just added a DELETE method for addresses. Note that if the address is linked to a user, the user is not deleted, it's just unlinked. Cheers, -Barry From barry at list.org Fri Feb 6 05:04:37 2015 From: barry at list.org (Barry Warsaw) Date: Thu, 5 Feb 2015 23:04:37 -0500 Subject: [Mailman-Developers] When a user is deleted, is the uid record intentionally not deleted? In-Reply-To: References: Message-ID: <20150205230437.737185d6@anarchist.wooz.org> On Feb 05, 2015, at 10:53 AM, Andrew Stuart wrote: >I notice that when a user is deleted, the uid record remains. > >Is this because there is a bug in which the uid should be deleted but is not? > >or is this intentional behaviour to prevent the uid from being recreated as a >duplicate? > >My testing code does large numbers of creations and deletions of users so the >deleted uid records are building up. I think it's mostly that this is an unexpected user case. You've probably seen this, but the UniqueIDFactory behaves differently when the real system is running vs. when the test suite is running. In the latter case, we don't actually create new UID records in the database. Instead we use a file (protected by a lock) to monotonically increment a user-friendly UID. The testing machinery is hooked up to this so that when we reset the "database" between tests, the uid factory gets reset too. Under normal execution, we create a uuid4() instance, which generates a random UUID, which is stored in the database, exactly so we can guarantee no duplicates (they are unlikely to happen, but we have to make sure). What I didn't expect was for there to be a testing scenario which would hit the core while it was under normal operation mode, rather than testing mode. I think that's what you're doing, right? If so, then indeed, the UID table never gets cleared, and the UID for deleted users never gets deleted. I'm wary of deleting the UID when the user is deleted. It seems like this could leave us open to vulnerabilities where a UID gets reassigned to a new user. Your tests can't directly manipulate the database since you have to go through the REST API. If this is really a problem for you, then we'll have to expose something in the REST API for you to clear out the UID table, or at least specific UIDs. What I could imagine is that, under some situations (e.g. the presence of an environment variable perhaps?) a new top-level resource would get exposed called `/testing` and under that, would be resources you could touch (DELETE?) that would reset bits of the database for you. For example: DELETE /testing/uid_table would drop all UID rows. It would be up to you to call that at the appropriate time. Thoughts? Cheers, -Barry From andrew.stuart at supercoders.com.au Fri Feb 6 06:17:19 2015 From: andrew.stuart at supercoders.com.au (Andrew Stuart) Date: Fri, 6 Feb 2015 16:17:19 +1100 Subject: [Mailman-Developers] When a user is deleted, is the uid record intentionally not deleted? In-Reply-To: <20150205230437.737185d6@anarchist.wooz.org> References: <20150205230437.737185d6@anarchist.wooz.org> Message-ID: Yes it?s a tough question. I agree with leaving the uid records in the database to prevent reuse. You are correct my test code is exercising a full working system. It would be good if my tests could leave the database in the same state after completion of each test. The ideal would be a method that is exactly the same as the existing user delete code but also deletes the users uid record. maybe as you suggest, url route names like: DELETE `/testing/users/{user_identifier}' or DELETE `/reserved/users/{user_identifier}? Personally I?d not bother with exposing or hiding reserved methods based on environment variables or anything else - as you say, it?s an internal API not mean to used by the public so developers should know not to be using reserved or testing methods in their production code. The extra functionality might introduce bugs and needs writing and testing and documenting - not worth it. The fastest and simplest way ahead might be to duplicate the existing delete method under a new url route name (as above) and add ?delete uid? functionality on top of the existing ?delete user? code. All of this is up to you though - I don?t want to be making work for others. Just a heads up - there may be further similar questions coming up about the database because I?m only half way through writing my tests. as On 6 Feb 2015, at 3:04 pm, Barry Warsaw wrote: On Feb 05, 2015, at 10:53 AM, Andrew Stuart wrote: > I notice that when a user is deleted, the uid record remains. > > Is this because there is a bug in which the uid should be deleted but is not? > > or is this intentional behaviour to prevent the uid from being recreated as a > duplicate? > > My testing code does large numbers of creations and deletions of users so the > deleted uid records are building up. I think it's mostly that this is an unexpected user case. You've probably seen this, but the UniqueIDFactory behaves differently when the real system is running vs. when the test suite is running. In the latter case, we don't actually create new UID records in the database. Instead we use a file (protected by a lock) to monotonically increment a user-friendly UID. The testing machinery is hooked up to this so that when we reset the "database" between tests, the uid factory gets reset too. Under normal execution, we create a uuid4() instance, which generates a random UUID, which is stored in the database, exactly so we can guarantee no duplicates (they are unlikely to happen, but we have to make sure). What I didn't expect was for there to be a testing scenario which would hit the core while it was under normal operation mode, rather than testing mode. I think that's what you're doing, right? If so, then indeed, the UID table never gets cleared, and the UID for deleted users never gets deleted. I'm wary of deleting the UID when the user is deleted. It seems like this could leave us open to vulnerabilities where a UID gets reassigned to a new user. Your tests can't directly manipulate the database since you have to go through the REST API. If this is really a problem for you, then we'll have to expose something in the REST API for you to clear out the UID table, or at least specific UIDs. What I could imagine is that, under some situations (e.g. the presence of an environment variable perhaps?) a new top-level resource would get exposed called `/testing` and under that, would be resources you could touch (DELETE?) that would reset bits of the database for you. For example: DELETE /testing/uid_table would drop all UID rows. It would be up to you to call that at the appropriate time. Thoughts? Cheers, -Barry _______________________________________________ Mailman-Developers mailing list Mailman-Developers at python.org https://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/andrew.stuart%40supercoders.com.au Security Policy: http://wiki.list.org/x/QIA9 From mark at msapiro.net Sat Feb 7 06:05:05 2015 From: mark at msapiro.net (Mark Sapiro) Date: Fri, 06 Feb 2015 21:05:05 -0800 Subject: [Mailman-Developers] Mailman 2.1.19 release Message-ID: <54D59D01.6070209@msapiro.net> I am pleased to announce the second release candidate for Mailman 2.1.19. Python 2.4 is the minimum supported, but Python 2.7 is recommended. This release has several new features since 2.1.18-1, some newly implemented and some backported from the now truly defunct 2.2 branch - there is nothing left in that branch that isn't in this release. Associated with those changes are many things that have i18n impacts including new and modified strings in the message catalog, a new adminaddrchgack.txt template and a minor change to the admindbdetails.html template. I strongly encourage anyone with an interest in translations of Mailman to get this release and help with updating the translations. There are also bug fixes. See the attached README for more details. The differences between this and the first 2.1.19 candidate are the following. The subscribe_auto_approval feature in the first candidate has been augmented to accept @listname to auto approve members of another list. (LP: #1417093) Organization: headers are unconditionally removed from posts to anonymous lists and a bug involving ANONYMOUS_LIST_KEEP_HEADERS has been fixed. (LP: #1419132) The admindb interface display of a held message body has been fixed to not stop in the middle of a multi-byte character and not fail to convert to the display character set message bodies with invalid characters. (LP: #1415406) Additionally, and the main reason for this second candidate, Mailman's character set for both Romanian and Russian has been changed to utf-8. (LP: #1418735) and (LP: #1418448). This may be disruptive in that following this change, some archive pages and some list attributes for lists whose preferred_language is one of those two languages may not display correctly. Archives can be fixed by running 'bin/arch --wipe' for the affected lists, but list attributes will probably have to be fixed manually. I hope the benefits of this change will outweigh the pain, and I encourage anyone with Romanian or Russian Language lists to try this release and report your experience. It may be possible to fix at least some list attributes programmatically as part of the version update process. I plan to release the final 2.1.19 release at the end of February, and it would be good to have as many i18n updates as possible by then. Mailman is free software for managing email mailing lists and e-newsletters. Mailman is used for all the python.org and SourceForge.net mailing lists, as well as at hundreds of other sites. For more information, please see: http://www.list.org http://www.gnu.org/software/mailman http://mailman.sourceforge.net/ Mailman 2.1.19rc2 can be downloaded from https://launchpad.net/mailman/2.1/ http://ftp.gnu.org/gnu/mailman/ https://sourceforge.net/projects/mailman/ -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan -------------- next part -------------- 2.2 Branch Backports (released in conjunction with 2.1.19) The following New Features and Bug Fixes have been in an "unofficial, never to be released" Mailman 2.2 branch for several years. Until now, they were never implemented on the official 2.1 branch because of their i18n impacts. Given that there have been a number of i18n impacting changes due to DMARC mitigations in the last few releases, it has been decided to backport these as well. All of these changes have been running in production on several lists for years without problems other than untranslated strings, so they should be reasonably "bug free". New Features - There is a new list attribute 'subscribe_auto_approval' which is a list of email addresses and regular expressions matching email addresses whose subscriptions are exempt from admin approval. (LP: #266609) - Confirmed member change of address is logged in the 'subscribe' log, and if admin_notify_mchanges is true, a notice is sent to the list owner using a new adminaddrchgack.txt template. - Added an 'automate' option to bin/newlist to send the notice to the admin without the prompt. - The processing of Topics regular expressions has changed. Previously the Topics regexp was compiled in verbose mode but not documented as such which caused some confusion. Also, the documentation indicated that topic keywords could be entered one per line, but these entries were not handled properly. Topics regexps are now compiled in non-verbose mode and multi-line entries are 'ored'. Existing Topics regexps will be converted when the list is updated so they will continue to work. - Added real name display to the web roster. (LP: #266754) Bug fixes and other patches - Changed the response to an invalid confirmation to be more generic. Not all confirmations are subscription requests. - Changed the default nonmember_rejection_notice to be more user friendly. (LP: #418728) - Added "If you are a list member" qualification to some messages from the options login page. (LP: #266442) - Changed the 'Approve' wording in the admindbdetails.html template to 'Accept/Approve' for better agreement with the button labels. - Added '(by thread)' to the previous and next message links in the archive to emphasize that even if you got to the message from a subject, date or author index, previous and next are still by thread. 2.1.19rc2 (06-Feb-2015) New Features - The subscribe_auto_approval feature backported from the 2.2 branch and described above has been enhanced to accept entries of the form @listname to auto approve members of another list. (LP: #1417093) - There is a new list attribute dmarc_wrapped_message_text and a DEFAULT_DMARC_WRAPPED_MESSAGE_TEXT setting to set the default for new lists. This text is added to a message which is wrapped because of dmarc_moderation_action in a separate text/plain part that precedes the message/rfc822 part containing the original message. It can be used to provide an explanation of why the message was wrapped or similar info. - There is a new list attribute equivalent_domains and a DEFAULT_EQUIVALENT_DOMAINS setting to set the default for new lists which in turn defaults to the empty string. This provides a way to specify one or more groups of domains, e.g., mac.com, me.com, icloud.com, which are considered equivalent for validating list membership for posting and moderation purposes. - There is a new WEB_HEAD_ADD setting to specify text to be added to the section of Mailman's internally generated web pages. This doesn't apply to pages built from templates, but in those cases, custom templates can be created. (LP: #1409396) - There is a new DEFAULT_SUBSCRIBE_OR_INVITE setting. Set this to Yes to make the default selection on the admin Mass Subscriptions page Invite rather than Subscribe. (LP: #1404511) - There is a new list attribute in the Bounce processing section. bounce_notify_owner_on_bounce_increment if set to Yes will cause Mailman to notify the list owner on every bounce that increments a list member's score but doesn't result in a probe or disable. There is a new configuration setting setting DEFAULT_BOUNCE_NOTIFY_OWNER_ON_BOUNCE_INCREMENT to set the default for new lists. This in turn defaults to No. (LP: #1382150) Changed behavior - Mailman's log files, request.pck files and heldmsg-* files are no longer created world readable to protect against access by untrusted local users. Note that permissions on existing log files won't be changed so if you are concerned about this and don't rotate logs or have a logrotate process that creates new log files instead of letting Mailman create them, you will need to address that. (LP: #1327404) Other changes - The Python Powered logo image has been replaced in the misc/ directory in the source distribution. Depending on how you've installed these images, you may need to copy PythonPowered.png from the misc/ directory in the source or from the $prefix/icons/ installed directory to another location for your web server. (LP: #1408575) i18n - Mailman's character set for Romanian has been changed from iso-8859-2 to utf-8 and the templates and messages recoded. This change will require running 'bin/arch --wipe' on any existing Romanian language lists in order to recode the list's archives, and will require recoding any edited templates in lists/LISTNAME/ro/*, templates/DOMAIN/ro/* and templates/site/ro/*. It may also require recoding any existing iso-8859-2 text in list attributes. (LP: #1418735) - Mailman's character set for Russian has been changed from koi8-r to utf-8 and the templates and messages recoded. This change will require running 'bin/arch --wipe' on any existing Russian language lists in order to recode the list's archives, and will require recoding any edited templates in lists/LISTNAME/ru/*, templates/DOMAIN/ru/* and templates/site/ru/*. It may also require recoding any existing koi8-r text in list attributes. (LP: #1418448) - The Russian message catalog has been updated by Danil Smirnov. - The Romanian message catalog has been updated. (LP: #1415489) - The Russian templates have been updated by Danil Smirnov. (LP: #1403462) - The Japanese translation has been updated by SATOH Fumiyasu. (LP: #1402989) - A minor change in the French translation of a listinfo subscribe form message has been made. (LP: #1331194) Bug fixes and other patches - Organization: headers are now unconditionally removed from posts to anonymous lists. Regexps in ANONYMOUS_LIST_KEEP_HEADERS weren't kept if the regexp included the trailing ':'. This is fixed too. (LP: #1419132) - The admindb interface has been fixed so the the detail message body display doesn't lose part of a multi-byte character, and characters which are invalid in the message's charset are replaced rather than the whole body not being converted to the display charset. (LP: #1415406) - Fixed a bug in bin/rmlist that would throw an exception or just fail to remove held message files for a list with regexp special characters in its name. (LP:#1414864) - When applying DMARC mitigations, CookHeaders now adds the original From: to Cc: rather than Reply-To: in some cases to make MUA 'reply' and 'reply all' more consistent with the non-DMARC cases. (LP: #1407098) - The Subject: of the list welcome message wasn't always in the user's preferred language. Fixed. (LP: #1400988) - Accept email command in Subject: prefixed with Re: or similar with no intervening space. (LP: #1400200) - Fixed a UnicodeDecodeError that could occur in the web admin interface if 'text' valued attributes have unicode values. (LP: #1397170) - We now catch the NotAMemberError exception thrown if an authenticated unsubscribe is submitted from the user options page for a nonmember. (LP: #1390653) - Fixed an archiving bug that would cause messages with 'Subject: Re:' only to be indexed in the archives without a link to the message. (LP: #1388614) - The vette log entry for a message discarded by a handler now includes the list name and the name of the handler. (LP: #558096) - The options CGI now rejects all but HTTP GET and POST requests. (LP: #1372199) - A list's poster password will now be accepted on an Urgent: header. (LP: #1371678) - Fixed a bug which caused a setting of 2 for REMOVE_DKIM_HEADERS to be ignored. (LP: #1363278) - Renamed messages/sr/readme.sr to README.sr. (LP: #1360616) - Moved the dmarc_moderation_action checks from the Moderate handler to the SpamDetect handler so that the Reject and Discard actions will be done before the message might be held by header_filter_rules, and the Wrap Message and Munge From actions will be done on messages held by header_filter_rules if the message is approved. (LP: #1334450) -