From barry at python.org Thu Aug 26 22:13:23 2004 From: barry at python.org (Barry Warsaw) Date: Thu Aug 26 22:13:31 2004 Subject: [Mailman3-dev] Re: [Mailman-Developers] Debate about Mailman on BytesForAll In-Reply-To: References: <20040818100041.824A41E4026@bag.python.org> <412E2E37.4060202@bellanet.org> <1093546324.9126.31.camel@geddy.wooz.org> <412E335E.7000503@bellanet.org> <1093550497.9126.103.camel@geddy.wooz.org> Message-ID: <1093551202.9125.115.camel@geddy.wooz.org> (Redirecting to the Mailman 3 list) On Thu, 2004-08-26 at 16:10, Dale Newfield wrote: > On Thu, 26 Aug 2004, Barry Warsaw wrote: > > I've been thinking that, what with all the problems associated with > > BerkeleyDB, it might not make sense to switch to SQLite as the embedded, > > default database for MM3. > > The biggest problem with BerkeleyDB is that it REQUIRES that the file > system support memory mapping the files. This means that you cannot > guarantee correctness if these files are located on an NFS mount. > > http://www.sleepycat.com/docs/ref/env/remote.html Add that to the list of BDB problems, then. :/ -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/mailman3-dev/attachments/20040826/73c9fa87/attachment.pgp From barry at python.org Thu Aug 26 22:16:46 2004 From: barry at python.org (Barry Warsaw) Date: Thu Aug 26 22:16:49 2004 Subject: [Mailman3-dev] Re: [Mailman-Developers] Debate about Mailman on BytesForAll In-Reply-To: <412E446E.4090101@bellanet.org> References: <20040818100041.824A41E4026@bag.python.org> <412E2E37.4060202@bellanet.org> <1093546324.9126.31.camel@geddy.wooz.org> <412E335E.7000503@bellanet.org> <1093550497.9126.103.camel@geddy.wooz.org> <412E446E.4090101@bellanet.org> Message-ID: <1093551406.9126.120.camel@geddy.wooz.org> (Redirecting to Mailman3 list) On Thu, 2004-08-26 at 16:13, Kevin McCann wrote: > Might not make sense? Or might make sense? Might make sense to switch from BerkeleyDB to SQLite as the default embedded database for Mailman 3 (which BTW, I'm still happy to take primary responsibility for). > I have played with SQLite a > bit but not extensively, so I'm not aware of any major drawbacks. I do > recall at the sprint lunch that SQLite was suggested by that fellow who > was doing the email sprinting (can't remember his name, but he seemd to > be a sharp cookie). SQLite, if it's solid, is attractive to me because > it's built into PHP 5. Thinking down the road, it starts to look pretty > good. The biggest question in my mind is how SQLite handles concurrency issues. I know that it has some support for concurrency, through automatic locking and such, but I haven't looked into the details. This will be a requirement for MM3 because we'll have multiple processes reading and writing to the database all the time (web requests, qrunners, cron, command line, nntp, imap, etc., etc.). -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/mailman3-dev/attachments/20040826/0f4ba3b2/attachment.pgp From kmccann at bellanet.org Thu Aug 26 22:33:05 2004 From: kmccann at bellanet.org (Kevin McCann) Date: Thu Aug 26 22:30:50 2004 Subject: [Mailman3-dev] SQLite In-Reply-To: <1093551406.9126.120.camel@geddy.wooz.org> References: <20040818100041.824A41E4026@bag.python.org> <412E2E37.4060202@bellanet.org> <1093546324.9126.31.camel@geddy.wooz.org> <412E335E.7000503@bellanet.org> <1093550497.9126.103.camel@geddy.wooz.org> <412E446E.4090101@bellanet.org> <1093551406.9126.120.camel@geddy.wooz.org> Message-ID: <412E4901.2040006@bellanet.org> (also changing the subject line to something meaningful -- apologies if that messes anything up) Barry Warsaw wrote: >The biggest question in my mind is how SQLite handles concurrency >issues. I know that it has some support for concurrency, through >automatic locking and such, but I haven't looked into the details. This >will be a requirement for MM3 because we'll have multiple processes >reading and writing to the database all the time (web requests, >qrunners, cron, command line, nntp, imap, etc., etc.). > > Not sure if this'll help, but:: http://www.sqlite.org/lockingv3.html - Kevin From kmccann at bellanet.org Thu Aug 26 22:57:55 2004 From: kmccann at bellanet.org (Kevin McCann) Date: Thu Aug 26 22:55:38 2004 Subject: [Mailman3-dev] Re: [RETRANSMIT] Re: [Mailman-Developers] Debate about Mailman on BytesForAll In-Reply-To: References: <20040818100041.824A41E4026@bag.python.org> <412E2E37.4060202@bellanet.org> <1093546324.9126.31.camel@geddy.wooz.org> <412E335E.7000503@bellanet.org> <1093550497.9126.103.camel@geddy.wooz.org> Message-ID: <412E4ED3.5030903@bellanet.org> Brad Knowles wrote: > At 4:10 PM -0400 2004-08-26, Dale Newfield wrote: > >> The biggest problem with BerkeleyDB is that it REQUIRES that the file >> system support memory mapping the files. This means that you cannot >> guarantee correctness if these files are located on an NFS mount. > > > True enough. That's a known issue with BerkeleyDB. > > But why would you be putting any of this stuff on NFS anyway? And > how would you deal with all the file locking issues? And > cross-platform issues? I've been doing NFS for a very long time, and > I have yet to see a mail-related environment where NFS is a good > choice or works well. Given the sorts of things we're talking about > doing, I can't imagine that NFS could possibly be a good solution. > I agree, Brad. Barry has redirected some of the discussion over to mailman3-dev@python.org. I'll crosspost for this message, but maybe we should move over to mm3-dev? In any event, further to NFS, it seems SQLite has issues, too. From the concurrency article I pointed to: "SQLite uses POSIX advisory locks to implement locking on Unix. On windows it uses the LockFile(), LockFileEx(), and UnlockFile() system calls. SQLite assumes that these system calls all work as advertised. If that is not the case, then database corruption can result. One should note that POSIX advisory locking is known to be buggy or even unimplemented on many NFS implementations (including recent versions of Mac OS X) and that there are reports of locking problems for network filesystems under windows. Your best defense is to not use SQLite for files on a network filesystem." - Kevin From claw at kanga.nu Thu Aug 26 23:16:09 2004 From: claw at kanga.nu (J C Lawrence) Date: Thu Aug 26 23:16:14 2004 Subject: [Mailman3-dev] Re: [RETRANSMIT] Re: [Mailman-Developers] Debate about Mailman on BytesForAll In-Reply-To: Message from Kevin McCann of "Thu, 26 Aug 2004 16:57:55 EDT." <412E4ED3.5030903@bellanet.org> References: <20040818100041.824A41E4026@bag.python.org> <412E2E37.4060202@bellanet.org> <1093546324.9126.31.camel@geddy.wooz.org> <412E335E.7000503@bellanet.org> <1093550497.9126.103.camel@geddy.wooz.org> <412E4ED3.5030903@bellanet.org> Message-ID: <21267.1093554969@kanga.nu> On Thu, 26 Aug 2004 16:57:55 -0400 Kevin McCann wrote: > "SQLite uses POSIX advisory locks to implement locking on Unix. On > windows it uses the LockFile(), LockFileEx(), and UnlockFile() system > calls. SQLite assumes that these system calls all work as > advertised. If that is not the case, then database corruption can > result. One should note that POSIX advisory locking is known to be > buggy or even unimplemented on many NFS implementations (including > recent versions of Mac OS X) and that there are reports of locking > problems for network filesystems under windows. Your best defense is > to not use SQLite for files on a network filesystem." I've spent some time mucking about in this area and it is a minefield. Very simply, throwing out all of the messy details, there is only one operation which is atomic under the network filesystems: creat (2). The exceptions where creat(2) is not guaranteed atomic are for some of the more obscure cacheing network filesystes which attempt to be resilient across intermittent/bad connectivity and node failures. If you want to account for those filesystems as well then sorry, there's just no guaranteed correct locking methodology. Yes, it is that bad. -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw@kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live. From barry at python.org Thu Aug 26 23:18:10 2004 From: barry at python.org (Barry Warsaw) Date: Thu Aug 26 23:18:16 2004 Subject: [Mailman3-dev] SQLite In-Reply-To: <412E4901.2040006@bellanet.org> References: <20040818100041.824A41E4026@bag.python.org> <412E2E37.4060202@bellanet.org> <1093546324.9126.31.camel@geddy.wooz.org> <412E335E.7000503@bellanet.org> <1093550497.9126.103.camel@geddy.wooz.org> <412E446E.4090101@bellanet.org> <1093551406.9126.120.camel@geddy.wooz.org> <412E4901.2040006@bellanet.org> Message-ID: <1093555089.9126.127.camel@geddy.wooz.org> On Thu, 2004-08-26 at 16:33, Kevin McCann wrote: > (also changing the subject line to something meaningful -- apologies if > that messes anything up) Nope, that's fine! > Not sure if this'll help, but:: > > http://www.sqlite.org/lockingv3.html I did read that, but unfortunately I don't know if the pysqlite interface exposes the locking primitives to Python. It's worth extending it to do so if not (but that would break DBAPI2). Something to play with, for sure. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/mailman3-dev/attachments/20040826/0d4fd498/attachment.pgp From barry at python.org Fri Aug 27 00:11:28 2004 From: barry at python.org (Barry Warsaw) Date: Fri Aug 27 00:11:37 2004 Subject: [Mailman3-dev] Re: [RETRANSMIT] Re: [Mailman-Developers] Debate about Mailman on BytesForAll In-Reply-To: References: <20040818100041.824A41E4026@bag.python.org> <412E2E37.4060202@bellanet.org> <1093546324.9126.31.camel@geddy.wooz.org> <412E335E.7000503@bellanet.org> <1093550497.9126.103.camel@geddy.wooz.org> <412E4ED3.5030903@bellanet.org> Message-ID: <1093558288.9120.182.camel@geddy.wooz.org> (Removing mailman-developers) On Thu, 2004-08-26 at 17:02, Brad Knowles wrote: > > Your best defense is to not use SQLite for files on a network filesystem." > > Fair enough. Are we looking for something where use on a network > filesystem is an issue? My answer to that would be, if you want distributed operation for MM3, use a client/server database like MySQL or PostgreSQL. As far as the qrunners go, it shouldn't matter because of the way the hash space algorithm is designed. > Or do we care more about other issues, such as performance (where > I think BerkeleyDB would probably win), or interface methods/language > (where SQL might be preferred)? As I mentioned in my other email, ease of installation, use, and administration is very high on my list. I don't want performance to go through the toilet, but the default configuration doesn't have to be able to handle AOL-sized traffic. :) -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/mailman3-dev/attachments/20040826/b0e853e9/attachment.pgp From barry at python.org Fri Aug 27 00:13:50 2004 From: barry at python.org (Barry Warsaw) Date: Fri Aug 27 00:13:53 2004 Subject: [Mailman3-dev] Re: [Mailman-Developers] Debate about Mailman on BytesForAll: LDAP support? In-Reply-To: References: Message-ID: <1093558430.9125.184.camel@geddy.wooz.org> (Moved to mailman3-dev) On Thu, 2004-08-26 at 17:33, Michael Chang wrote: > > And don't forget LDAP support, either! > Volunteers will be welcomed! :) -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/mailman3-dev/attachments/20040826/b8dc3d56/attachment.pgp From darrell at grumblesmurf.net Fri Aug 27 00:20:06 2004 From: darrell at grumblesmurf.net (Darrell Fuhriman) Date: Fri Aug 27 01:29:46 2004 Subject: [Mailman3-dev] Re: [Mailman-Developers] Debate about Mailman on BytesForAll: LDAP support? In-Reply-To: <1093558430.9125.184.camel@geddy.wooz.org> (Barry Warsaw's message of "Thu, 26 Aug 2004 18:13:50 -0400") References: <1093558430.9125.184.camel@geddy.wooz.org> Message-ID: Barry Warsaw writes: > Volunteers will be welcomed! :) I'll be glad to code up some LDAP support, once the interface is finalized. d. From brad at stop.mail-abuse.org Fri Aug 27 00:12:38 2004 From: brad at stop.mail-abuse.org (Brad Knowles) Date: Fri Aug 27 01:29:46 2004 Subject: [Mailman3-dev] Re: [RETRANSMIT] Re: [Mailman-Developers] Debate about Mailman on BytesForAll In-Reply-To: <21267.1093554969@kanga.nu> References: <20040818100041.824A41E4026@bag.python.org> <412E2E37.4060202@bellanet.org> <1093546324.9126.31.camel@geddy.wooz.org> <412E335E.7000503@bellanet.org> <1093550497.9126.103.camel@geddy.wooz.org> <412E4ED3.5030903@bellanet.org> <21267.1093554969@kanga.nu> Message-ID: At 5:16 PM -0400 2004-08-26, J C Lawrence wrote: > I've spent some time mucking about in this area and it is a minefield. > Very simply, throwing out all of the messy details, there is only one > operation which is atomic under the network filesystems: creat (2). Which is why Nick Christensen built the Earthlink mail system around this feature. See . But I don't see how we can build all of Mailman around the same feature. -- Brad Knowles, "Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety." -- Benjamin Franklin (1706-1790), reply of the Pennsylvania Assembly to the Governor, November 11, 1755 SAGE member since 1995. See for more info. From r.barrett at openinfo.co.uk Fri Aug 27 12:10:43 2004 From: r.barrett at openinfo.co.uk (Richard Barrett) Date: Fri Aug 27 12:11:02 2004 Subject: [Mailman3-dev] Re: [Mailman-Developers] Debate about Mailman on BytesForAll In-Reply-To: <1093551406.9126.120.camel@geddy.wooz.org> References: <20040818100041.824A41E4026@bag.python.org> <412E2E37.4060202@bellanet.org> <1093546324.9126.31.camel@geddy.wooz.org> <412E335E.7000503@bellanet.org> <1093550497.9126.103.camel@geddy.wooz.org> <412E446E.4090101@bellanet.org> <1093551406.9126.120.camel@geddy.wooz.org> Message-ID: <5B57EB12-F811-11D8-A58E-000A957C9A50@openinfo.co.uk> On 26 Aug 2004, at 21:16, Barry Warsaw wrote: > (Redirecting to Mailman3 list) > > On Thu, 2004-08-26 at 16:13, Kevin McCann wrote: > >> Might not make sense? Or might make sense? > > Might make sense to switch from BerkeleyDB to SQLite as the default > embedded database for Mailman 3 (which BTW, I'm still happy to take > primary responsibility for). > >> I have played with SQLite a >> bit but not extensively, so I'm not aware of any major drawbacks. I do >> recall at the sprint lunch that SQLite was suggested by that fellow >> who >> was doing the email sprinting (can't remember his name, but he seemd >> to >> be a sharp cookie). SQLite, if it's solid, is attractive to me because >> it's built into PHP 5. Thinking down the road, it starts to look >> pretty >> good. > > The biggest question in my mind is how SQLite handles concurrency > issues. I know that it has some support for concurrency, through > automatic locking and such, but I haven't looked into the details. > This > will be a requirement for MM3 because we'll have multiple processes > reading and writing to the database all the time (web requests, > qrunners, cron, command line, nntp, imap, etc., etc.). > > -Barry > I have not been involved in the debate what the database this thread is referring to is being used for, but, from what has been said above, might not Postgresql be a sound choice. Robust, proper transaction handling/concurrency, can run from NFS mounted or local storage, proper client/server operation, reasonable security. It may not be the easiest to install but ... Another post to this list asked why NFS mounting of the database storage matters. Explanation for me: we have a large, high reliabilty, high availabilty, etc etc file server accessible over Gigabit via an Extreme switch; we have an effective backup strategy for the file systems which that file server publishes via NFS. I currently use NFS storage with MM 2.1.x and given a catastrophic hardware failure in my mailing list server I can switch to a backup server and have full service back within minutes without data loss. We are putting a large value of e-trading messages through the mailing list server each week and some of that incoming traffic has strict time limits on when responses are sent; we cannot accept loss of service. Why would I want to buy into the admin problems of being compelled to run a database out of storage local to my mail list server? Which is why BerkeleyDB does not look like the right solution for me. For the same reason I also cannot use Subversion for its purposes, even though I would rather like to. Just a thought. From brad at stop.mail-abuse.org Fri Aug 27 12:24:23 2004 From: brad at stop.mail-abuse.org (Brad Knowles) Date: Fri Aug 27 12:24:29 2004 Subject: [Mailman3-dev] Re: [Mailman-Developers] Debate about Mailman on BytesForAll In-Reply-To: <5B57EB12-F811-11D8-A58E-000A957C9A50@openinfo.co.uk> References: <20040818100041.824A41E4026@bag.python.org> <412E2E37.4060202@bellanet.org> <1093546324.9126.31.camel@geddy.wooz.org> <412E335E.7000503@bellanet.org> <1093550497.9126.103.camel@geddy.wooz.org> <412E446E.4090101@bellanet.org> <1093551406.9126.120.camel@geddy.wooz.org> <5B57EB12-F811-11D8-A58E-000A957C9A50@openinfo.co.uk> Message-ID: At 11:10 AM +0100 2004-08-27, Richard Barrett wrote: > Another post to this list asked why NFS mounting of the database > storage matters. Explanation for me: we have a large, high reliabilty, > high availabilty, etc etc file server accessible over Gigabit via an > Extreme switch; we have an effective backup strategy for the file > systems which that file server publishes via NFS. Not particularly unusual for large NFS server installations I've seen. > I currently use > NFS storage with MM 2.1.x And this works today? You don't have file locking problems, etc...? > and given a catastrophic hardware failure > in my mailing list server I can switch to a backup server and have > full service back within minutes without data loss. Fair enough. I wasn't aware that anyone was doing this sort of stuff today with NFS, and that this worked without all the traditional file locking problems, etc.... Certainly, I'd agree that we should support PostgreSQL and MySQL/MaxSQL, BerkeleyDB, etc... as alternative database methods that should be included with the base package. Now, whether one of them should be the primary database method, I'm not quite ready to agree to that, at least not yet. I tend to place pretty high value on the "Keep It Simple" style of solution, at least as far as the default stuff is concerned. If SQLite is easy to install, gives us decent performance, doesn't make it too difficult to replace with alternative database solutions, and satisfies our other criteria, then I'm all for that. In that case, I'd rather we spent more time providing hooks that would allow us to swap out the database back-end for something faster/more robust, as well as doing things like fixing the pipermail/archiving tool, archive searching tool(s), etc.... Out of curiosity, can someone remind me where the current mm3 "wish list" is? Are there mm3 specification and/or design documents yet? -- Brad Knowles, "Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety." -- Benjamin Franklin (1706-1790), reply of the Pennsylvania Assembly to the Governor, November 11, 1755 SAGE member since 1995. See for more info. From iane at sussex.ac.uk Fri Aug 27 14:09:57 2004 From: iane at sussex.ac.uk (Ian Eiloart) Date: Fri Aug 27 14:10:10 2004 Subject: [Mailman3-dev] Re: [RETRANSMIT] Re: [Mailman-Developers] Debate about Mailman on BytesForAll In-Reply-To: <412E4ED3.5030903@bellanet.org> References: <20040818100041.824A41E4026@bag.python.org> <412E2E37.4060202@bellanet.org> <1093546324.9126.31.camel@geddy.wooz.org> <412E335E.7000503@bellanet.org> <1093550497.9126.103.camel@geddy.wooz.org> <412E4ED3.5030903@bellanet.org> Message-ID: <0D8B622C717D83D3B1E2B881@lewes.staff.uscs.susx.ac.uk> --On Thursday, August 26, 2004 4:57 pm -0400 Kevin McCann wrote: > > Barry has redirected some of the discussion over to > mailman3-dev@python.org. I'll crosspost for this message, but maybe we > should move over to mm3-dev? In any event, further to NFS, it seems > SQLite has issues, too. From the concurrency article I pointed to: Good plan. I've been subscribed to mm3-dev because I want to keep track of progress. There have been three messages in three months up till now, so I've kind of assumed Mailman 3 development had ceased! -- Ian Eiloart Servers Team Sussex University ITS From iane at sussex.ac.uk Fri Aug 27 14:19:24 2004 From: iane at sussex.ac.uk (Ian Eiloart) Date: Fri Aug 27 14:19:25 2004 Subject: [Mailman3-dev] SQLite In-Reply-To: <412E4901.2040006@bellanet.org> References: <20040818100041.824A41E4026@bag.python.org> <412E2E37.4060202@bellanet.org> <1093546324.9126.31.camel@geddy.wooz.org> <412E335E.7000503@bellanet.org> <1093550497.9126.103.camel@geddy.wooz.org> <412E446E.4090101@bellanet.org> <1093551406.9126.120.camel@geddy.wooz.org> <412E4901.2040006@bellanet.org> Message-ID: <87DD50167CE3FCA6A787B853@lewes.staff.uscs.susx.ac.uk> --On Thursday, August 26, 2004 4:33 pm -0400 Kevin McCann wrote: > > Not sure if this'll help, but:: > > http://www.sqlite.org/lockingv3.html > > - Kevin To me, that looks like a pretty poor locking model for a large site. The problem is that the whole table (maybe even several tables) gets locked when a process wants to write to it. That means that changes to the database will temporarily prevent any READS from the database. It also means that you can't write to the database when any process is reading from the database. When a process wants to write, it has to take out a PENDING lock. Something like pgSQL, with transaction rollback and record level locking is going to have far better performance. Having said all that, for a site with low traffic, all of that isn't going to cause a problem. I reckon that an ideal installation process should ASK whether there is an existing database server, and if not, should offer to use SQLite after warning that performance problems will occur in high traffic sites. If a database server does exist which can be used, then the install process should (ideally) help with creating the necessary tables. -- Ian Eiloart Servers Team Sussex University ITS From iane at sussex.ac.uk Fri Aug 27 14:54:11 2004 From: iane at sussex.ac.uk (Ian Eiloart) Date: Fri Aug 27 14:54:13 2004 Subject: [Mailman3-dev] Re: [Mailman-Developers] Debate about Mailman on BytesForAll In-Reply-To: <1093551202.9125.115.camel@geddy.wooz.org> References: <20040818100041.824A41E4026@bag.python.org> <412E2E37.4060202@bellanet.org> <1093546324.9126.31.camel@geddy.wooz.org> <412E335E.7000503@bellanet.org> <1093550497.9126.103.camel@geddy.wooz.org> <1093551202.9125.115.camel@geddy.wooz.org> Message-ID: <7382D85C7A72313374B725F2@lewes.staff.uscs.susx.ac.uk> --On Thursday, August 26, 2004 4:13 pm -0400 Barry Warsaw wrote: > (Redirecting to the Mailman 3 list) > > On Thu, 2004-08-26 at 16:10, Dale Newfield wrote: >> On Thu, 26 Aug 2004, Barry Warsaw wrote: >> > I've been thinking that, what with all the problems associated with >> > BerkeleyDB, it might not make sense to switch to SQLite as the >> > embedded, default database for MM3. >> >> The biggest problem with BerkeleyDB is that it REQUIRES that the file >> system support memory mapping the files. This means that you cannot >> guarantee correctness if these files are located on an NFS mount. >> >> http://www.sleepycat.com/docs/ref/env/remote.html > > Add that to the list of BDB problems, then. :/ > > -Barry > I've been looking at the SQLite site here . They recommend that you don't use SQLite on NFS mounted file systems - for the same reasons. -- Ian Eiloart Servers Team Sussex University ITS From claw at kanga.nu Fri Aug 27 16:34:41 2004 From: claw at kanga.nu (J C Lawrence) Date: Fri Aug 27 16:34:47 2004 Subject: [Mailman3-dev] SQLite In-Reply-To: Message from Ian Eiloart of "Fri, 27 Aug 2004 13:19:24 BST." <87DD50167CE3FCA6A787B853@lewes.staff.uscs.susx.ac.uk> References: <20040818100041.824A41E4026@bag.python.org> <412E2E37.4060202@bellanet.org> <1093546324.9126.31.camel@geddy.wooz.org> <412E335E.7000503@bellanet.org> <1093550497.9126.103.camel@geddy.wooz.org> <412E446E.4090101@bellanet.org> <1093551406.9126.120.camel@geddy.wooz.org> <412E4901.2040006@bellanet.org> <87DD50167CE3FCA6A787B853@lewes.staff.uscs.susx.ac.uk> Message-ID: <2894.1093617281@kanga.nu> On Fri, 27 Aug 2004 13:19:24 +0100 Ian Eiloart wrote: > --On Thursday, August 26, 2004 4:33 pm -0400 Kevin McCann > wrote: > To me, that looks like a pretty poor locking model... Without disagreeing in the slightest I'd like to make a small point: Moving to a unified DB model for Mailman would be a large step forward no matter the distribution/locking problems. In fact the distribution problems would be no worse than they are now, but we'd have the added benefit of a unified DB. Sure, it would be better to have a better distribution/locking solution, but let's not fall back on getting a unified DB just because it doesn't fix other problems. > Something like pgSQL... Is a very large addition to the dependency graph and if required by a base install would effectively remove Mailman from consideration for most smaller sites, managed service providers etc. MySQL would do a bit better (and MySQL can be ACID these days), but remains a huge addition to the dependency graph. Ditto LDAP. They're great features to support as _plugins_. > I reckon that an ideal installation process should ASK whether there > is an existing database server, and if not, should offer to use SQLite > after warning that performance problems will occur in high traffic > sites. If a database server does exist which can be used, then the > install process should (ideally) help with creating the necessary > tables. +1 -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw@kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live. From r.barrett at openinfo.co.uk Fri Aug 27 16:46:05 2004 From: r.barrett at openinfo.co.uk (Richard Barrett) Date: Fri Aug 27 16:46:27 2004 Subject: [Mailman3-dev] Re: [Mailman-Developers] Debate about Mailman on BytesForAll In-Reply-To: References: <20040818100041.824A41E4026@bag.python.org> <412E2E37.4060202@bellanet.org> <1093546324.9126.31.camel@geddy.wooz.org> <412E335E.7000503@bellanet.org> <1093550497.9126.103.camel@geddy.wooz.org> <412E446E.4090101@bellanet.org> <1093551406.9126.120.camel@geddy.wooz.org> <5B57EB12-F811-11D8-A58E-000A957C9A50@openinfo.co.uk> Message-ID: On 27 Aug 2004, at 11:24, Brad Knowles wrote: > At 11:10 AM +0100 2004-08-27, Richard Barrett wrote: > >> Another post to this list asked why NFS mounting of the database >> storage matters. Explanation for me: we have a large, high >> reliabilty, >> high availabilty, etc etc file server accessible over Gigabit via an >> Extreme switch; we have an effective backup strategy for the file >> systems which that file server publishes via NFS. > > Not particularly unusual for large NFS server installations I've seen. > Didn't say it was. >> I currently use >> NFS storage with MM 2.1.x > > And this works today? You don't have file locking problems, etc...? > >> and given a catastrophic hardware failure >> in my mailing list server I can switch to a backup server and have >> full service back within minutes without data loss. > > Fair enough. I wasn't aware that anyone was doing this sort of stuff > today with NFS, and that this worked without all the traditional file > locking problems, etc.... > I said "switch to a backup server". Hot server, warm standby. When primary fails, then and only then, does the standby takes over so there is no contention in normal operation. Without NFS this approach cannot work easily because failure of the primary denies the backup server access to everything Mailman'ish. To be entirely accurate the only potentially delayed data is that in the current primary's local MTA queues (on local storage) which is undelivered to MM or not yet passed on to our outbound relay at the time the primary fails. But MM's qfiles directory is on NFS so the potential for delayed incoming mail is fairly small. I was not suggesting load sharing between the primary and secondary server, although I may go there someday. > Certainly, I'd agree that we should support PostgreSQL and > MySQL/MaxSQL, BerkeleyDB, etc... as alternative database methods that > should be included with the base package. > > Now, whether one of them should be the primary database method, I'm > not quite ready to agree to that, at least not yet. I tend to place > pretty high value on the "Keep It Simple" style of solution, at least > as far as the default stuff is concerned. > My concern is getting a heavy duty solution which will work with a database of my choice, i.e. one I have learned to love, trust and manage, which I can nominate at install time. > If SQLite is easy to install, gives us decent performance, doesn't > make it too difficult to replace with alternative database solutions, > and satisfies our other criteria, then I'm all for that. In that case, > I'd rather we spent more time providing hooks that would allow us to > swap out the database back-end for something faster/more robust, as > well as doing things like fixing the pipermail/archiving tool, archive > searching tool(s), etc.... > I am all for that, so that I can stop extending and maintaining my patches to integrate htdig and MHonArc with MM 2.1.x for these latter purposes. > Out of curiosity, can someone remind me where the current mm3 "wish > list" is? Are there mm3 specification and/or design documents yet? > > -- > Brad Knowles, > > "Those who would give up essential Liberty, to purchase a little > temporary Safety, deserve neither Liberty nor Safety." > > -- Benjamin Franklin (1706-1790), reply of the Pennsylvania > Assembly to the Governor, November 11, 1755 > > SAGE member since 1995. See for more info. From r.barrett at openinfo.co.uk Fri Aug 27 17:22:24 2004 From: r.barrett at openinfo.co.uk (Richard Barrett) Date: Fri Aug 27 17:22:38 2004 Subject: [Mailman3-dev] Re: [Mailman-Developers] Debate about Mailman on BytesForAll In-Reply-To: <7382D85C7A72313374B725F2@lewes.staff.uscs.susx.ac.uk> References: <20040818100041.824A41E4026@bag.python.org> <412E2E37.4060202@bellanet.org> <1093546324.9126.31.camel@geddy.wooz.org> <412E335E.7000503@bellanet.org> <1093550497.9126.103.camel@geddy.wooz.org> <1093551202.9125.115.camel@geddy.wooz.org> <7382D85C7A72313374B725F2@lewes.staff.uscs.susx.ac.uk> Message-ID: On 27 Aug 2004, at 13:54, Ian Eiloart wrote: > > > --On Thursday, August 26, 2004 4:13 pm -0400 Barry Warsaw > wrote: > >> (Redirecting to the Mailman 3 list) >> >> On Thu, 2004-08-26 at 16:10, Dale Newfield wrote: >>> On Thu, 26 Aug 2004, Barry Warsaw wrote: >>> > I've been thinking that, what with all the problems associated with >>> > BerkeleyDB, it might not make sense to switch to SQLite as the >>> > embedded, default database for MM3. >>> >>> The biggest problem with BerkeleyDB is that it REQUIRES that the file >>> system support memory mapping the files. This means that you cannot >>> guarantee correctness if these files are located on an NFS mount. >>> >>> http://www.sleepycat.com/docs/ref/env/remote.html >> >> Add that to the list of BDB problems, then. :/ >> >> -Barry >> > > I've been looking at the SQLite site here > . They recommend > that you don't use SQLite on NFS mounted file systems - for the same > reasons. Postgresql concurrency control is described here and so far as I m aware does not preclude the use NFS mounted storage as it does not depend on file system locking: http://www.postgresql.org/docs/7.4/static/mvcc.html > > -- > Ian Eiloart > Servers Team > Sussex University ITS From brad at stop.mail-abuse.org Sat Aug 28 02:32:14 2004 From: brad at stop.mail-abuse.org (Brad Knowles) Date: Sat Aug 28 03:09:42 2004 Subject: [Mailman3-dev] Re: [Mailman-Developers] Debate about Mailman on BytesForAll In-Reply-To: References: <20040818100041.824A41E4026@bag.python.org> <412E2E37.4060202@bellanet.org> <1093546324.9126.31.camel@geddy.wooz.org> <412E335E.7000503@bellanet.org> <1093550497.9126.103.camel@geddy.wooz.org> <412E446E.4090101@bellanet.org> <1093551406.9126.120.camel@geddy.wooz.org> <5B57EB12-F811-11D8-A58E-000A957C9A50@openinfo.co.uk> Message-ID: At 3:46 PM +0100 2004-08-27, Richard Barrett wrote: > I said "switch to a backup server". Hot server, warm standby. When > primary fails, then and only then, does the standby takes over so > there is no contention in normal operation. Speaking only for myself, even with just one machine on NFS, I've seen serious locking problems (among other things), just in terms of internal contention with itself and not having to do with any other machines trying to access the same data at the same time. NFS is fine for certain types of operations, but anything requiring real-time or near real-time access where multiple processes (either on the same machine or on different machines) may be trying to access the same data, is a real area of serious heartburn. Spools and queues just don't work well on NFS, although if you use various hashing techniques and the semi-atomic file creation trick, you can kinda-semi-sorta work around the worst of these issues, most of the time. Of course, there is a significant performance hit that you take for going this route. All IMO, of course. > My concern is getting a heavy duty solution which will work with a > database of my choice, i.e. one I have learned to love, trust and > manage, which I can nominate at install time. Agreed. > I am all for that, so that I can stop extending and maintaining my > patches to integrate htdig and MHonArc with MM 2.1.x for these > latter purposes. Understood. You've done a great deal of service to the community through your various patches, and your efforts are strongly appreciated! That said, I can certainly understand why you'd like to stop maintaining them. I've got a few programs myself that I am officially responsible for, but which I'd really kinda rather they come to a quick and violent death so that I didn't have to look at them or think about them any more. -- Brad Knowles, "Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety." -- Benjamin Franklin (1706-1790), reply of the Pennsylvania Assembly to the Governor, November 11, 1755 SAGE member since 1995. See for more info.