From rushing@n... Thu Jan 14 12:08:14 1999 From: rushing@n... (Samual M. Rushing) Date: 14 Jan 1999 12:08:14 -0000 Subject: [medusa] new code for CGI-ish thread pools Message-ID: <19990114120814.26127.qmail@g...> [ok, I'll make the first post] See the 'thread' subdirectory, you will find: thread_channel.py a good experiment, using pipes to bridge the sync/async gap. somewhat impractical because it wastes file descriptors select_trigger.py a special channel specifically designed for waking up the main select loop with output from other threads. a nice side-effect of this technique is that it uses a socket as a synchronization object; obviating the need for locks around medusa's i/o data structures. thread_handler.py a module for building cgi-like systems that use a pool of threads. uses select_trigger to create file-like objects that hide all the magic from you. -Sam p.s. a reminder, to access the read-only CVS repository: CVSROOT = :pserver:medusa@s...:/usr/local/cvsroot the password is 'medusa'. ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From rushing@n... Thu Jan 14 13:06:24 1999 From: rushing@n... (Samual M. Rushing) Date: 14 Jan 1999 13:06:24 -0000 Subject: [medusa] addition to the thread_handler demo Message-ID: <19990114130624.26323.qmail@g...> I've added a second test module, 'pi_module', that will compute a specified number of digits of pi. You can use it to test out the thread pool by taking up all the threads. If you have a request waiting for a thread, it will start up as soon as one of the others finishes. -Sam ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From rushing@n... Thu Jan 14 19:56:08 1999 From: rushing@n... (Samual M. Rushing) Date: 14 Jan 1999 19:56:08 -0000 Subject: [medusa] renaming of classes & methods Message-ID: <19990114195608.26512.qmail@g...> The names of the various classes, methods, variables, etc in the core library are mostly the result of accident; they were given quickly, and usually before their purpose was fully understood. For example, the 'dispatcher' class is hardly a dispatcher. If anything, it's a 'dispatchee'. Kinda screaming for a better name. I'm sure this doesn't help in trying to understand the code. I've thought for a while that the whole mess could use a renaming; if ever there were a time to do this it seems that *before* it went into the Python library would be it. Just wondering what others were thinking, along with any suggestions for more meaningful names, candidates for most-in-need-of-a-new-name, etc... -Sam ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From rushing@n... Sat Jan 16 16:19:21 1999 From: rushing@n... (Samual M. Rushing) Date: 16 Jan 1999 16:19:21 -0000 Subject: [medusa] rough draft of tutorial Message-ID: <19990116161921.16305.qmail@g...> I've filled in more of the tutorial. http://www.nightmare.com:8080/nm/apps/medusa/docs/programming.html Suggestions are welcome. The Reference section will come next. -Sam ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From rushing@n... Tue Jan 19 13:08:55 1999 From: rushing@n... (Samual M. Rushing) Date: 19 Jan 1999 13:08:55 -0000 Subject: [medusa] asyncore.py:compact_traceback(), handle_error() interfaces redesigned. Message-ID: <19990119130855.18284.qmail@g...> There is a warning about sys.exc_info() in the Python documentation. If only I'd thought to look there a long time ago, I would have saved myself much chasing of memory leaks. The basic problem is this: if you refer to a traceback from within an exception handler, you will automatically create a cycle (and thus leak every object in the entire stack frame) I've recently been peppering my code with try/finally/del sequences to avoid this, but a much easier solution is to move the call to sys.exc_info() out of the exception handler. It seems that in the old days (with sys.exc_type, sys.exc_value, sys.exc_traceback) you couldn't do this, which is why I've always passed the values around. This impacts not only asyncore.compact_traceback(), but also the handle_error() interface; which no longer takes any arguments. The necessary changes are fairly obvious, grep through the code for examples. -Sam ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From florent.heyworth@r... Fri Jan 22 01:47:27 1999 From: florent.heyworth@r... (florent.heyworth@r...) Date: Fri, 22 Jan 1999 01:47:27 -0000 Subject: [medusa] rough draft of tutorial Looks good. Looking forward to the reference section 8-) Cheers Florent Heyworth (keep up the good work) ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From carlosm@c... Wed Jan 27 18:28:47 1999 From: carlosm@c... (carlosm@c...) Date: Wed, 27 Jan 1999 18:28:47 -0000 Subject: [medusa] Medusa as web proxy server? Has anyone done any work on modifying Medusa so it can be used as a web proxy server? Thanks, Carlos ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From florent.heyworth@r... Thu Jan 21 08:02:15 1999 From: florent.heyworth@r... (florent.heyworth@r...) Date: Thu, 21 Jan 1999 08:02:15 -0000 Subject: [medusa] Re: rough draft of tutorial In-Reply-To: <19990116161921.16305.qmail@g...> <19990116161921.16305.qmai-@g...> wrote: Original Article: http://www.egroups.com/list/medusa/?start=4 > I've filled in more of the tutorial. > Looks good. Looking forward to seeing the reference section 8-) Cheers Florent Heyworth ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From florent.heyworth@r... Thu Jan 28 00:38:13 1999 From: florent.heyworth@r... (florent.heyworth@r...) Date: Thu, 28 Jan 1999 00:38:13 -0000 Subject: [medusa] Re: Medusa as web proxy server? Original Article: http://www.egroups.com/list/medusa/?start=7 > Has anyone done any work on modifying Medusa so it can be used as > a web proxy server? > > Thanks, > Carlos > Hi Carlos Sam Rushing has an example proxy server implementation in his Medusa tutorial. Check it out under: http://www.nightmare.com:8080/nm/apps/medusa/docs/programming.html Cheers Florent ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From robin@j... Thu Jan 28 05:45:23 1999 From: robin@j... (Robin Becker) Date: Thu, 28 Jan 1999 05:45:23 +0000 Subject: [medusa] How do I stop medusa gracefully Message-ID: <076c5GAzl$r2Ews4@j...> I'm just starting to play with medusa at home (Win95 OSR2) and find it does strange things to my dialup connection sometimes. I've just been killing the process; is there a better way? -- Robin Becker ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From florent.heyworth@r... Thu Jan 28 10:03:13 1999 From: florent.heyworth@r... (florent.heyworth@r...) Date: Thu, 28 Jan 1999 10:03:13 -0000 Subject: [medusa] Re: How do I stop medusa gracefully In-Reply-To: <076c5GAzl$r2Ews4@j...> <076c5gazl$r2ews-@j...> wrote: Original Article: http://www.egroups.com/list/medusa/?start=10 > I'm just starting to play with medusa at home (Win95 OSR2) and find it > does strange things to my dialup connection sometimes. I've just been > killing the process; is there a better way? > -- > Robin Becker > Hi I used to develop with Medusa on a Win95 machine. As the machine was a standalone it also had a dial-up connection to the Internet and never noticed any side effects. I'm now on NT and I've never noticed any side effects either. Could you be more specific as to what problems you've been encountering? Cheers Florent Heyworth ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From rushing@n... Thu Jan 28 18:21:54 1999 From: rushing@n... (rushing@n...) Date: Thu, 28 Jan 1999 10:21:54 -0800 (PST) Subject: [medusa] Re: How do I stop medusa gracefully In-Reply-To: References: <076c5GAzl$r2Ews4@j...> Message-ID: <14000.43652.959726.676173@g...> Robin Becker writes: > Occasionally my dialup works, but no connections can be established. I > suspected the pseudo DNS server, but can't think of any way it could do > this. When this has happened I later found that medusa was also a bit > strange; it would allow FTP, but not HTTP; a bit weird. by default start_medusa.py creates a DNS resolver pointing at the localhost. On a dialup connection you should change the '127.0.0.1' to point to your ISP's resolver. You can also just disable the resolving feature (you'll get IP numbers in your log). It should be possible to change the line rs = resolver.caching_resolver ('127.0.0.1') to read rs = None This affects the http and ftp servers, and is only used for logging, not for authentication. -Sam ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From florent.heyworth@r... Thu Jan 28 22:12:40 1999 From: florent.heyworth@r... (florent.heyworth@r...) Date: Thu, 28 Jan 1999 22:12:40 -0000 Subject: [medusa] Re: How do I stop medusa gracefully In-Reply-To: wrote: Original Article: http://www.egroups.com/list/medusa/?start=14 > thanks I tried this and it seemed to work fine; but what is the > preferred method of shutting medusa down? > -- > Robin Becker > If you're working with medusa by running it from Python.exe a Ctrl-C (Keyboard Interrupt) will attempt to shut down "gracefully" after a time out on the select loop. A more graceful, and fun, way of doing is by logging in to Medusa's monitor client. On Win32 you can call it as python monitor_client_win32.py [hostname] [port] on my machine I do: python monitor_client_win32.py 127.0.0.1 9999 at the password prompt enter the password (default is 'fnord') you can change it in start_medusa.py. on the interactive prompt enter: >>> import monitor >>> monitor.asyncore.close_all() That's it Cheers Florent ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From rushing@n... Fri Jan 29 02:57:51 1999 From: rushing@n... (rushing@n...) Date: Thu, 28 Jan 1999 18:57:51 -0800 (PST) Subject: [medusa] Re: How do I stop medusa gracefully In-Reply-To: References: Message-ID: <14001.8921.411033.535751@g...> Robin Becker writes: > I guessed close_all must be it; it doesn't seem to worry about timing > order of close at all though A more graceful close might shut down the servers only, then when the final client has exited (and thus there are no more sockets in the socket map) the loop should exit cleanly. This should also allow you to immediately restart a new set of servers, since the server ports will be available. -Sam ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From rushing@n... Sat Jan 30 01:43:22 1999 From: rushing@n... (rushing@n...) Date: Fri, 29 Jan 1999 17:43:22 -0800 (PST) Subject: [medusa] more uniform error handling in asyncore.dispatcher() ? Message-ID: <14002.25179.442430.872334@g...> Craig Silverstein has noticed that there's a discrepancy in the handling of errors in async_chat: calls to self.recv() are protected by a try/except clause, but calls to self.send() are not. While looking into this, I noticed that a possibly cleaner solution is to have the send() and recv() methods in asyncore.dispatcher call self.handle_error() directly, instead of re-raising unexpected errors that they see. I think this would relieve the responsibility to try/except from any derived class. Does anyone see a problem with this, or has anyone tried it already? -Sam -- rushing@n... rushing@e... ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From robin@j... Sat Jan 30 08:06:47 1999 From: robin@j... (Robin Becker) Date: Sat, 30 Jan 1999 08:06:47 +0000 Subject: [medusa] FTP permissions Message-ID: whilst playing with medusa's ftp server I noticed that even though I wasn't allowed to use the STOR command I was allowed to create remove/directories. Is this normal for FTP servers? -- Robin Becker ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From rushing@n... Sat Jan 30 20:28:37 1999 From: rushing@n... (rushing@n...) Date: Sat, 30 Jan 1999 12:28:37 -0800 (PST) Subject: [medusa] FTP permissions In-Reply-To: References: Message-ID: <14003.27219.235542.330350@g...> Robin Becker writes: > whilst playing with medusa's ftp server I noticed that even though I > wasn't allowed to use the STOR command I was allowed to create > remove/directories. Is this normal for FTP servers? No, this is major oops. There is a list of 'write' commands, and a comment to the affect that the list is checked against, but unfortunately the code that should reference it doesn't. It's a one-line fix, and I just checked it in: def check_command_authorization (self, command): if command in self.write_commands and self.read_only: return 0 else: return 1 Many thanks for pointing this out, Robin! -Sam ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From rushing@n... Sat Jan 30 21:32:13 1999 From: rushing@n... (rushing@n...) Date: Sat, 30 Jan 1999 13:32:13 -0800 (PST) Subject: [medusa] more uniform error handling in asyncore.dispatcher() ? In-Reply-To: <14002.25179.442430.872334@g...> References: <14002.25179.442430.872334@g...> Message-ID: <14003.31056.620896.457638@g...> rushing@n... writes: > While looking into this, I noticed that a possibly cleaner solution is > to have the send() and recv() methods in asyncore.dispatcher call > self.handle_error() directly, instead of re-raising unexpected errors > that they see. I think this would relieve the responsibility to > try/except from any derived class. On second thought, I don't think this is a good idea: it throws off the ability to exit correctly from a sequence of code: def method (self): n = self.send (data) [...other code...] If handle_error() is called automatically from send(), then there's no way for [...other code...] to know about the error condition so that it might exit the method (or choose a different branch of an if, etc...) -Sam ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From robin@j... Mon Feb 1 14:00:07 1999 From: robin@j... (Robin Becker) Date: Mon, 1 Feb 1999 14:00:07 +0000 Subject: [medusa] http error? Message-ID: I get the following error when requesting the same file repeatedly using a torture test. I don't get this error when re-loading as rapidly as possible with netscape. uncaptured python exception, closing channel (exceptions.AttributeError:'int' object has no attribute 'send' [c:\python\medusa\asyncore.py|poll|69] [c:\python\medusa\asyncore.py|handle_write_event|297] [c:\python\medusa\asynchat.py|handle_write|133] [c:\python\medusa\asynchat.py|initiate_send|185] [c:\python\medusa\http_server.py|send|351] [c:\python\medusa\asyncore.py|send|235] [|send|1]) -- Robin Becker ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From amos@a... Mon Feb 1 23:04:21 1999 From: amos@a... (Amos Latteier) Date: Mon, 01 Feb 1999 15:04:21 -0800 Subject: [medusa] FTP Limits Message-ID: <3.0.5.32.19990201150421.00b19c70@m...> One thing that's been discussed lately on the Zope list is the need for FTP limits to prevent denial of service attacks. The idea is that the server could be flooded with connections until it could no longer accept new legitimate connections. Right now it seems that Medusa's FTP server supports an authorizer object, but there do not seem to be any pre-built limit checking authorizers. I am looking for ideas on what sort of limits should be set, and how they could be enforced. It was suggested that limits could be set for different classes of users, for example, 'anonymous' users. This is slightly difficult in Zope's case, since authentication information is distributed, so the validity of a login cannot be verified without a directory in which to verify it. Thanks in advance for any pointers or experiences designing and implementing FTP limits. -Amos P.S. For those of you not familiar with Zope, it is an Open Source web application server which is using Medusa for HTTP and FTP services. http://www.zope.org/ ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From rushing@n... Mon Feb 8 22:24:02 1999 From: rushing@n... (rushing@n...) Date: Mon, 8 Feb 1999 14:24:02 -0800 (PST) Subject: [medusa] warning: server accept() threw an exception In-Reply-To: <88256712.00605040.00@b...> References: <88256712.00605040.00@b...> Message-ID: <14015.25153.666732.707340@g...> ray@u... writes: > Any idea what causes this this error to occur dozens of times in a row? > When it happens, the server will stop serving process and only print this > message.. sometimes hundreds of times and stopping http access for 2-30 > seconds. I have seen this sort of thing usually when bumping up against a file-descriptor limit. You may be able to verify this by looking at the number of open descriptors at the time the error is being thrown (sometimes you can get this from a /proc filesystem, otherwise you could add a bit of code to asyncore.loop() to see how many descriptors are being fed to select()). > I have noticed that the problem may be load related; it often begins when > the server is processing over 3k hits per hour, but does not incease in > frequency above that. But, it also occurs when hits are below 2k/hour. You might still be hitting descriptor limits under a lighter load if too many connections or files are staying open... > Using Medusa server (v3.19) serving only Bobo applications via the > bobo_handler (v1.03) on Solaris 2.6 Note also that if bobo is opening many files this will also take away from your budget. I've been told that Solaris is one of the better OS' for dynamically configuring around such limits; but I don't currently have a copy running to look into it myself. -Sam ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From ray@u... Wed Feb 24 00:05:12 1999 From: ray@u... (ray@u...) Date: Tue, 23 Feb 1999 16:05:12 -0800 Subject: [medusa] how to turn off dns resolving Message-ID: <88256722.000036D8.00@b...> We only want IP address in the logs. Thanks for you help. Ray ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From jam@n... Wed Feb 24 02:33:54 1999 From: jam@n... (Jonez) Date: Wed, 24 Feb 1999 02:33:54 +0000 Subject: [medusa] Re: how to turn off dns resolving References: <88256722.000036D8.00@b...> Message-ID: <36D36512.FB69E179@n...> ray@u... wrote: > > We only want IP address in the logs. > > Thanks for you help. > > Ray > greetings, it may take some small modifications to your application, but there is an 'unresolving_logger' object in logger.py.. you would probably need to derive from it to get the functionality you want, but it should be straight forward. perhaps you could post your startup script, and we can work on it from there? that will show how the logger object is being instantiated, and the modifications necessary will be clearer. hope that helps. regards, J ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From rushing@n... Wed Feb 24 03:13:12 1999 From: rushing@n... (rushing@n...) Date: Tue, 23 Feb 1999 19:13:12 -0800 (PST) Subject: [medusa] how to turn off dns resolving In-Reply-To: <88256722.000036D8.00@b...> References: <88256722.000036D8.00@b...> Message-ID: <14035.28151.461814.609649@g...> ray@u... writes: > > We only want IP address in the logs. The easiest way should be to change rs = resolver.caching_resolver ('127.0.0.1') to rs = None And then be sure to comment out the 'rs' in the list of status objects: status_objects = [ hs, ftp, ms, cs, # rs, lg ] -Sam ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From petrilli@a... Wed Feb 24 18:45:46 1999 From: petrilli@a... (Christopher G. Petrilli) Date: Wed, 24 Feb 1999 13:45:46 -0500 Subject: [medusa] asyncore.dispatcher methods Message-ID: <19990224134546.63759@a...> So can someone explain what the difference is between: handle_expt and handle_oob It seems only handle_expt is actually used, but maybe I missed something? THis is in asyncore.dispather Thanks! Chris -- | Christopher Petrilli ``Television is bubble-gum for | petrilli@a... the mind.''-Frank Lloyd Wright ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From rushing@n... Mon Mar 1 22:09:49 1999 From: rushing@n... (rushing@n...) Date: Mon, 1 Mar 1999 14:09:49 -0800 (PST) Subject: [medusa] asyncore.dispatcher methods In-Reply-To: <19990224134546.63759@a...> References: <19990224134546.63759@a...> Message-ID: <14043.4022.980510.779002@g...> Christopher G. Petrilli writes: > So can someone explain what the difference is between: > > handle_expt > > and > > handle_oob > > It seems only handle_expt is actually used, but maybe I missed > something? THis is in asyncore.dispather You're right. I think this is old code, probably there from the early days when I supported WSAAsyncSelect() on Win32 (which I think had an FD_OOB message). I'll take it out right now, thanks for the info! -Sam ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From amos@a... Tue Mar 16 19:12:41 1999 From: amos@a... (Amos Latteier) Date: Tue, 16 Mar 1999 11:12:41 -0800 Subject: [medusa] select_trigger.trigger requires posix Message-ID: <3.0.5.32.19990316111241.011b94f0@m...> I just noticed that the latest version of select_trigger.trigger inherits from asyncore.file_dispatcher, and thus is only available under posix. However, I would like to use select_trigger.trigger under NT, as well. Is there any way to allow select_trigger.trigger use under NT? Thanks. -Amos ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From rushing@n... Tue Mar 16 23:25:43 1999 From: rushing@n... (rushing@n...) Date: Tue, 16 Mar 1999 15:25:43 -0800 (PST) Subject: [medusa] select_trigger.trigger requires posix In-Reply-To: <3.0.5.32.19990316111241.011b94f0@m...> References: <3.0.5.32.19990316111241.011b94f0@m...> Message-ID: <14062.59402.933828.877673@s...> Amos Latteier writes: > I just noticed that the latest version of select_trigger.trigger inherits > from asyncore.file_dispatcher, and thus is only available under posix. > However, I would like to use select_trigger.trigger under NT, as well. > > Is there any way to allow select_trigger.trigger use under NT? This was an oversight... select_trigger should be able to use a loopback socket on win32. I will get this to you today. -Sam ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From ray@u... Fri Apr 2 18:34:28 1999 From: ray@u... (ray@u...) Date: Fri, 02 Apr 1999 18:34:28 -0000 Subject: [medusa] Changing logger.file_logger output How would I trim down the ipaddress in the log to eliminate the numbers after the ':'? Currently the first column in each row has the ipaddress formatted like this: 206.58.165.2:1206 and we would like it to be entered like this: 206.58.165.2 The curent format of the log files is causing our log analysis software to choke on the ipaddress calculations. Ray Price ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From rushing@n... Fri Apr 2 18:42:09 1999 From: rushing@n... (rushing@n...) Date: Fri, 2 Apr 1999 10:42:09 -0800 (PST) Subject: [medusa] Changing logger.file_logger output In-Reply-To: <199904021834.NAA03733@s...> References: <199904021834.NAA03733@s...> Message-ID: <14085.3886.43829.133624@s...> ray@u... writes: > How would I trim down the ipaddress in the log to eliminate the numbers after the ':'? > Currently the first column in each row has the ipaddress formatted like this: > 206.58.165.2:1206 > and we would like it to be entered like this: > 206.58.165.2 > > The curent format of the log files is causing our log analysis software to choke on the ipaddress calculations. This should do the trick: def log (self, bytes): self.channel.server.logger.log ( self.channel.addr[0], '%d - - [%s] "%s" %d %d\n' % ( self.channel.addr[1], self.log_date_string (time.time()), self.request, self.reply_code, bytes ) ) ==> def log (self, bytes): self.channel.server.logger.log ( self.channel.addr[0], ' - - [%s] "%s" %d %d\n' % ( self.log_date_string (time.time()), self.request, self.reply_code, bytes ) ) -Sam ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From skip@c... Thu Apr 22 00:57:11 1999 From: skip@c... (skip@c...) Date: Wed, 21 Apr 1999 23:57:11 -0000 Subject: [medusa] Re: How do I stop medusa gracefully In-Reply-To: <14001.8921.411033.535751@g...> Message-ID: I'm just starting to use Medusa via the ZServer stuff with Zope. I'd like to be able to identify and kill my servers by process id. Accordingly, my ZServer/start.py looks like open(PID_FILE, "w").write("%d\n" % os.getpid()) ... yada yada yada ... asyncore.loop() try: os.unlink(PID_FILE) except os.error: pass The only problem is when I kill the process, asyncore.loop appears not to return in the usual fashion (calls sys.exit somewhere I suppose). I wind up with a lot of pid files laying about (for various reasons they currently have the pid in the file name). Is there some way I can get asyncore.loop to return in the usual manner? Thanks, Skip Montanaro ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/group/medusa Free Web-based e-mail groups by eGroups.com From rushing@n... Fri Apr 23 03:20:33 1999 From: rushing@n... (rushing@n...) Date: Thu, 22 Apr 1999 19:20:33 -0700 (PDT) Subject: [medusa] Re: How do I stop medusa gracefully In-Reply-To: References: <14001.8921.411033.535751@g...> Message-ID: <14111.55211.647805.600070@s...> skip@c... writes: > The only problem is when I kill the process, asyncore.loop appears > not to return in the usual fashion (calls sys.exit somewhere I > suppose). I wind up with a lot of pid files laying about (for > various reasons they currently have the pid in the file name). > Is there some way I can get asyncore.loop to return in the usual manner? It seems to me the only 'normal' way to avoid getting to your unlink would be if asyncore.loop() was exited via an exception... it may be that Zope is using an exception to signal exit in some way. If that's the case, you *should* be able to work around it by using a try/finally clause... -Sam ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/group/medusa Free Web-based e-mail groups by eGroups.com From pwelliso@i... Tue Apr 27 19:10:30 1999 From: pwelliso@i... (pwelliso@i...) Date: Tue, 27 Apr 1999 18:10:30 -0000 Subject: [medusa] HTTPS available ? Message-ID: Is there a https module available ? One that makes calls to SSLeay would be great ? I would be willing to help debug/test/write one if there is not one available. Peter ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/group/medusa Free Web-based e-mail groups by eGroups.com From rushing@n... Wed Apr 28 00:12:25 1999 From: rushing@n... (rushing@n...) Date: Tue, 27 Apr 1999 16:12:25 -0700 (PDT) Subject: [medusa] HTTPS available ? In-Reply-To: References: Message-ID: <14118.17195.189469.46089@s...> pwelliso@i... writes: > Is there a https module available ? > One that makes calls to SSLeay would be great ? I haven't had time to look into this seriously. If SSLeay doesn't support non-blocking i/o, then it would be difficult. One way it might work would be to have medusa's select() loop handle the descriptors for sockets that have SSL code wrapped around them. I.e., medusa sees that a socket is ready for reading, and calls the SSL version of recv() to get the data. We may have to add another layer in Python that handles the case of this recv() failing (say, because the SSL layer needs the rest of a packet of some kind). > I would be willing to help debug/test/write one if there is not one > available. That would be great! I'll help in any way I can. -Sam ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/group/medusa Free Web-based e-mail groups by eGroups.com From amos@a... Thu May 6 17:48:58 1999 From: amos@a... (Amos Latteier) Date: Thu, 06 May 1999 09:48:58 -0700 Subject: [medusa] ftp server on win95 Message-ID: <3.0.5.32.19990506094858.0121a050@m...> Hello, I've gotten several reports that Medusa's ftp server fails to run under win95 when not attached to a network. Here's a sample traceback: log: adding channel Traceback (innermost last): File "D:\WebSite\ZServer\start.py", line 96, in ? from FTPServer import FTPServer File "D:\WebSite\ZServer\FTPServer.py", line 140, in ? from medusa.ftp server import ftp_channel, ftp_server, recv_channel File "D:\WebSite\ZServer\medusa\ftp_server.py", line 58, in ? HOSTNAME = socket.gethostbyaddr (IP_ADDRESS)[0] socket.error: host not found (Note this user is using ZServer.) Apparently IP_ADDRESS is set to "127.0.0.1" and they have an entry for this IP in their hosts file. Any suggestions on how to keep socket.gethostbyaddr from failing? Alternately, any suggestions on avoiding this call to begin with--perhaps we could avoid lines 57, and 58 of ftp_server.py, since they seem to really be useful to calculate the HOSTNAME, which can be passed to the ftp_server upon initialization... Thanks. -Amos ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/group/medusa http://www.eGroups.com - Simplifying group communications From rushing@n... Fri May 7 01:54:13 1999 From: rushing@n... (rushing@n...) Date: Thu, 6 May 1999 17:54:13 -0700 (PDT) Subject: [medusa] ftp server on win95 In-Reply-To: <3.0.5.32.19990506094858.0121a050@m...> References: <3.0.5.32.19990506094858.0121a050@m...> Message-ID: <14130.14503.627295.836062@s...> Amos Latteier writes: > Any suggestions on how to keep socket.gethostbyaddr from failing? > Alternately, any suggestions on avoiding this call to begin > with--perhaps we could avoid lines 57, and 58 of ftp_server.py, > since they seem to really be useful to calculate the HOSTNAME, > which can be passed to the ftp_server upon initialization... The IP_ADDRESS setting used to be in there for figuring out what address to bind the PASV port to. That code was changed a while ago to use getsockname() instead, so it's not needed any more. I removed the IP_ADDRESS and HOSTNAME globals, and now defaults its hostname to socket.gethostname(), which shouldn't fail in any case - but even if it does it can be avoided by passing in the hostname parameter. There's actually no reason to even have the hostname (other than reporting status), but I'm loath to remove it completely... -Sam ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/group/medusa http://www.eGroups.com - Simplifying group communications From rushing@n... Mon May 10 18:28:12 1999 From: rushing@n... (rushing@n...) Date: Mon, 10 May 1999 10:28:12 -0700 (PDT) Subject: [medusa] cvs pserver back up Message-ID: <14135.5683.133110.390488@s...> seattle.nightmare.com suffered a Nasty Disk Failure on Friday. Nothing was lost, but I've had to build a whole new system over the past couple of days. I just brought the cvs read-only pserver back online, if anyone has any trouble with it please let me know. As a reminder: CVSROOT=:pserver:medusa@s...:/usr/local/cvsroot The password is 'medusa'. Thanks, -Sam ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/group/medusa http://www.eGroups.com - Simplifying group communications From ray@u... Wed May 12 18:54:00 1999 From: ray@u... (ray@u...) Date: Wed, 12 May 1999 17:54:00 -0000 Subject: [medusa] using auth_handler and bobo_handler Message-ID: Has anybody successfully used auth_handler.py and bobo_handler together? If so, any tips on what's involved. Ray ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/group/medusa http://www.eGroups.com - Simplifying group communications From rushing@n... Thu May 13 01:43:20 1999 From: rushing@n... (rushing@n...) Date: Wed, 12 May 1999 17:43:20 -0700 (PDT) Subject: [medusa] using auth_handler and bobo_handler In-Reply-To: References: Message-ID: <14138.7928.949989.373497@s...> ray@u... writes: > Has anybody successfully used auth_handler.py and bobo_handler > together? If so, any tips on what's involved. It shouldn't be any more difficult than wrapping the one with the other. There's an example in demo/publish.py, where an auth handler is wrapped around a put_handler: | # Supports the HTTP PUT method... | ph = put_handler.put_handler (fs, '/.*') | | # ... but be sure to wrap it with an auth handler: | ah = auth_handler.auth_handler (users, ph) | | [...] | | hs.install_handler (ah) # for PUT If this doesn't work, please let me know... -Sam ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/group/medusa http://www.eGroups.com - Simplifying group communications From ita.mis@e... Thu May 13 13:47:17 1999 From: ita.mis@e... (ita.mis@e...) Date: Thu, 13 May 1999 12:47:17 -0000 Subject: [medusa] non-threaded CGI apps Message-ID: Hi, is there an example of a script which does CGI parameter processing on a non-threaded python installation? is this at all possible right now? best regard Marco Tasselli ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/group/medusa http://www.eGroups.com - Simplifying group communications From rushing@n... Fri May 14 02:05:03 1999 From: rushing@n... (rushing@n...) Date: Thu, 13 May 1999 18:05:03 -0700 (PDT) Subject: [medusa] non-threaded CGI apps In-Reply-To: References: Message-ID: <14139.30319.442867.851030@s...> ita.mis@e... writes: > is there an example of a script which does CGI parameter processing > on a non-threaded python installation? by this do you mean parsing form data? If so, I believe I have used one of the Python library modules to handle it... -Sam ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/group/medusa http://www.eGroups.com - Simplifying group communications From rushing@n... Sat May 15 03:13:37 1999 From: rushing@n... (rushing@n...) Date: Fri, 14 May 1999 19:13:37 -0700 (PDT) Subject: [medusa] non-threaded CGI apps In-Reply-To: References: Message-ID: <14140.55038.246703.135936@s...> ita.mis@e... writes: > is there an example of a script which does CGI parameter processing > on a non-threaded python installation? > > is this at all possible right now? Well, *now* it is. 8^) I spruced up the script_handler a bit. New features: 1) restricted execution is a flag (off by default, so scripts will use execfile) 2) support for PUT/POST requests, data is handed directly to the script via stdin as a StringIO object. 3) persistent script handler. this is a bit like the bobo/zope system. modules can be added and deleted from the handler. the modules are persistent. the 'main' procedure of the module is handed the http_request object as its only parameter (rather than a bunch of CGI environment variable madness). A 'script_handler_demo' directory is also included, with lots of comments. It should run out-of-the-box. Note: restricted execution is off by default because apparently the cgi module cannot be imported? But many folks will want to use it for forms. -Sam ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/group/medusa http://www.eGroups.com - Simplifying group communications From amos@a... Tue May 25 19:50:48 1999 From: amos@a... (Amos Latteier) Date: Tue, 25 May 1999 11:50:48 -0700 Subject: [medusa] NT socket error, asyncore.py Message-ID: <3.0.5.32.19990525115048.00c7d250@m...> A non-text attachment was scrubbed... Name: not available Type: text/enriched Size: 1070 bytes Desc: not available Url : http://mail.python.org/pipermail-21/medusa-dev/attachments/3ae3ce41/attachment.bin From pwelliso@i... Wed May 26 00:01:38 1999 From: pwelliso@i... (pwelliso@i...) Date: Tue, 25 May 1999 23:01:38 -0000 Subject: [medusa] Re: HTTPS available ? In-Reply-To: Message-ID: <7ifa4i$j9kc@e...> wrote: Original Article: http://www.egroups.com/group/medusa/?start=36 > Is there a https module available ? > > One that makes calls to SSLeay would be great ? > > I would be willing to help debug/test/write one if there is not one > available. A quick solution to this is STunnel: http://mike.daewoo.com.pl/computer/stunnel/ It took about 2 hours and it was up and running. The software acts as a wrapper, you hit port 443 with a https connection, the software handles the SSL part then passes off the plain text to the normal web server running on port 80. The only problem is that it passes the local host IP address to the web server so logging is not very useful. ------------------------------------------------------------------------ eGroups.com home: http://www.egroups.com/group/medusa http://www.egroups.com - Simplifying group communications From skip@m... Wed May 26 02:56:52 1999 From: skip@m... (Skip Montanaro) Date: Tue, 25 May 1999 21:56:52 -0400 Subject: [medusa] Strange exception in ZServer/medusa/asynchat.py Message-ID: <199905260156.VAA29018@c...> I'm using Medusa by way of Zope's ZServer. I have ZServer sitting behind an Apache proxy and am bombing it from a threaded client script that reads a web server log file as input and throws requests at the Apache server. I can vary the number of threads to adjust the load. I normally run it with five simultaneous threads. by default, the system runs briefly then (as they say), "Whammo! Blammo!", I get the following error: Unhandled exception in thread: Traceback (innermost last): File "/home/dolphin/skip/src/Zope/ZServer/PubCore/ZServerPublisher.py", line 97, in __init__ response._finish() File "/home/dolphin/skip/src/Zope/ZServer/HTTPResponse.py", line 209, in _finish self.stdout.close() File "/home/dolphin/skip/src/Zope/ZServer/HTTPResponse.py", line 235, in close self._channel.push(CallbackProducer(self._channel.done)) File "/home/dolphin/skip/src/Zope/ZServer/HTTPServer.py", line 307, in push if send: self.initiate_send() File "/home/dolphin/skip/src/Zope/ZServer/medusa/asynchat.py", line 200, in initiate_send self.refill_buffer() File "/home/dolphin/skip/src/Zope/ZServer/medusa/asynchat.py", line 184, in refill_buffer self.producer_fifo.pop() File "/home/dolphin/skip/src/Zope/ZServer/medusa/asynchat.py", line 255, in pop del self.list[0] IndexError: list assignment index out of range If you look at the code in asynchat.fifo.pop, the error can only be explained by the fact that two threads are modifying the fifo simultaneously, since access to self.list[0] succeeded in the statement before the statement that throws the IndexError: def pop (self): if self.list: result = (1, self.list[0]) # non-null list here del self.list[0] # empty list here else: result = (0, None) return result I modified the fifo class to lock access to its list: class fifo: def __init__ (self, list=None): self.lock = thread.allocate_lock() self.lock.acquire() if not list: self.list = [] else: self.list = list self.lock.release() def __len__ (self): self.lock.acquire() l = len(self.list) self.lock.release() return l def first (self): self.lock.acquire() item = self.list[0] self.lock.release() return item def push (self, data): self.lock.acquire() self.list.append (data) self.lock.release() def pop (self): self.lock.acquire() if self.list: result = (1, self.list[0]) del self.list[0] else: result = (0, None) self.lock.release() return result but this didn't help: Unhandled exception in thread: Traceback (innermost last): File "/home/dolphin/skip/src/Zope/ZServer/PubCore/ZServerPublisher.py", line 97, in __init__ response._finish() File "/home/dolphin/skip/src/Zope/ZServer/HTTPResponse.py", line 209, in _finish self.stdout.close() File "/home/dolphin/skip/src/Zope/ZServer/HTTPResponse.py", line 235, in close self._channel.push(CallbackProducer(self._channel.done)) File "/home/dolphin/skip/src/Zope/ZServer/HTTPServer.py", line 307, in push if send: self.initiate_send() File "/home/dolphin/skip/src/Zope/ZServer/medusa/asynchat.py", line 200, in initiate_send self.refill_buffer() File "/home/dolphin/skip/src/Zope/ZServer/medusa/asynchat.py", line 175, in refill_buffer p = self.producer_fifo.first() File "/home/dolphin/skip/src/Zope/ZServer/medusa/asynchat.py", line 254, in first item = self.list[0] IndexError: list index out of range This error also looks like a threading problem. When refill_buffer tests self.producer.fifo, it sees something, but by the time it calls fifo.first, the list has evaporated: while 1: if len(self.producer_fifo): # non-null list here p = self.producer_fifo.first() # null list here Any ideas? Is there a way to easily turn off threading? It seems that the data structures are not properly protected. Skip Montanaro | Mojam: "Uniting the World of Music" http://www.mojam.com/ skip@m... | Musi-Cal: http://www.musi-cal.com/ 518-372-5583 ------------------------------------------------------------------------ eGroups.com home: http://www.egroups.com/group/medusa http://www.egroups.com - Simplifying group communications From jim@d... Wed May 26 15:12:41 1999 From: jim@d... (Jim Fulton) Date: Wed, 26 May 1999 10:12:41 -0400 Subject: [medusa] Re: Strange exception in ZServer/medusa/asynchat.py References: <199905260156.VAA29018@c...> Message-ID: <374C0159.E4A1291F@d...> Skip Montanaro wrote: > > I'm using Medusa by way of Zope's ZServer. I have ZServer sitting behind an > Apache proxy and am bombing it from a threaded client script that reads a > web server log file as input and throws requests at the Apache server. I > can vary the number of threads to adjust the load. I normally run it with > five simultaneous threads. > > By default, the system runs briefly then (as they say), "Whammo! Blammo!", I > get the following error: (snip) > Any ideas? This was a bug in ZServer that was fixed in the most recent Zope release (2.0 alpha 1). Basically, the push method in ZHTTPServer.zhttp_channel has a second argument that must be passed a false value when called from a thread other than the main (medusa) thread. The ZServer code that runs in separate threads now supplies this argument. > Is there a way to easily turn off threading? Not in ZServer. It would be a ZServer bug if a problem resulted from use of threads. This particular bug has been fixed. Jim -- Jim Fulton mailto:jim@d... Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. ------------------------------------------------------------------------ eGroups.com home: http://www.egroups.com/group/medusa http://www.egroups.com - Simplifying group communications From rushing@n... Thu May 27 00:20:02 1999 From: rushing@n... (rushing@n...) Date: Wed, 26 May 1999 16:20:02 -0700 (PDT) Subject: [medusa] Re: HTTPS available ? In-Reply-To: <7ifa4i$j9kc@e...> References: Message-ID: <14156.33124.873581.649238@s...> pwelliso@i... writes: > wrote: > Original Article: http://www.egroups.com/group/medusa/?start=36 > > Is there a https module available ? > A quick solution to this is STunnel: > http://mike.daewoo.com.pl/computer/stunnel/ > > It took about 2 hours and it was up and running. > > The software acts as a wrapper, you hit port 443 with a https connection, > the software handles the SSL part then passes off the plain text to the > normal web server running on port 80. The only problem is that it passes > the local host IP address to the web server so logging is not very useful. Great, thanks for the info! I'll put a note about this up on the web site. -Sam ------------------------------------------------------------------------ eGroups.com home: http://www.egroups.com/group/medusa http://www.egroups.com - Simplifying group communications From rushing@n... Sat Jun 12 02:25:52 1999 From: rushing@n... (Sam Rushing) Date: Fri, 11 Jun 1999 18:25:52 -0700 (PDT) Subject: [medusa] UDP gotcha Message-ID: <14177.46448.901418.666637@s...> This took quite a while to figure out, so I thought I'd share it with the list in case anyone else runs up against the same problem. If you are using asyncore for UDP, you need to make sure your handle_read() method calls recvfrom() in a loop, otherwise you will read only one packet per trip through the polling loop. If you are trying to pump many packets through UDP while at the same time servicing lots of other channels for other protocols (like HTTP or SMTP), then your OS will drop incoming packets if you don't read them fast enough. def handle_read (self): # read UDP in a loop try: while 1: reply, whence = self.recvfrom (2048) self.handle_reply (reply) except socket.error, why: if why[0] != asyncore.EWOULDBLOCK: raise socket.error, why It also helps to raise your send and recv buffer sizes: self.setsockopt (socket.SOL_SOCKET, socket.SO_SNDBUF, 200 * 1024) self.setsockopt (socket.SOL_SOCKET, socket.SO_RCVBUF, 200 * 1024) Thanks are due to Dru Nelson of eGroups for helping me track this down. -Sam ------------------------------------------------------------------------ eGroups.com home: http://www.egroups.com/group/medusa http://www.egroups.com - Simplifying group communications From Amos@d... Thu Jun 17 17:56:46 1999 From: Amos@d... (Amos Latteier) Date: Thu, 17 Jun 1999 12:56:46 -0400 Subject: [medusa] select error on win32 Message-ID: <613145F79272D211914B0020AFF640190AD23E@g...> Hi, A ZServer user forwarded this error to me: Traceback (innermost last): File "start.py", line 155, in ? asyncore.loop() File "medusa\asyncore.py", line 104, in loop poll_fun (timeout) File "medusa\asyncore.py", line 52, in poll (r,w,e) = select.select (r,w,e, timeout) ValueError: too many file descriptors in select() They are using ZServer on win32 on a fairly heavily trafficked site. This error halts the server. According to "File Descriptor Limitations in Various Operating Systems" http://www.nightmare.com/medusa/descriptor_limits.html NT does not have a hard limit to the number of fd's that can be selected, and poll is not available. Does anyone (I mean does Sam ;-) have any ideas about what may be causing the select loop to fail, and what ZServer may be able to do to mitigate these problems? Thanks. -Amos -- Amos Latteier mailto:amos@d... Digital Creations http://www.digicool.com ------------------------------------------------------------------------ eGroups.com home: http://www.egroups.com/group/medusa http://www.egroups.com - Simplifying group communications From jody@s... Thu Jun 17 19:58:04 1999 From: jody@s... (jody@s...) Date: Thu, 17 Jun 1999 18:58:04 -0000 Subject: [medusa] Need an example client that does not read from stdin and exits the main loop Message-ID: <7kbgfs$kb2g@e...> I've written a server in medusa that works like ftp. That is: command space args and the server responds with: code space results This code works using telnet and using a medusa client that reads from stdin. My questions are: 1. How do I send a fixed set of commands to the server from a client? What I have tried and failed with is in the client's handle_connect to use self.send(). 2. After all of the commands are sent and all responces processed, how do I exit from the main loop? ------------------------------------------------------------------------ eGroups.com home: http://www.egroups.com/group/medusa http://www.egroups.com - Simplifying group communications From rushing@n... Thu Jun 17 17:26:35 1999 From: rushing@n... (Sam Rushing) Date: Thu, 17 Jun 1999 09:26:35 -0700 (PDT) Subject: [medusa] select error on win32 In-Reply-To: <613145F79272D211914B0020AFF640190AD23E@g...> References: <613145F79272D211914B0020AFF640190AD23E@g...> Message-ID: <14185.8381.95775.851844@s...> Amos Latteier writes: > They are using ZServer on win32 on a fairly heavily trafficked site. > This error halts the server. > Does anyone (I mean does Sam ;-) have any ideas about what may be > causing the select loop to fail, and what ZServer may be able to do to > mitigate these problems? There are no hard limits on NT, but you have to recompile your application. In this case you need to have another version of socket.pyd compiled with FD_SETSIZE=1024 (or some other larger number). This is true of most platforms, because FD_SETSIZE is a macro. On FreeBSD I'm using FD_SETSIZE=8192 without any troubles. Pay no attention to the WSAStartup 'MaxSock' parameter, it's supposed to be ignored if you're using winsock2. -Sam ------------------------------------------------------------------------ eGroups.com home: http://www.egroups.com/group/medusa http://www.egroups.com - Simplifying group communications From rushing@n... Thu Jun 17 17:33:01 1999 From: rushing@n... (Sam Rushing) Date: Thu, 17 Jun 1999 09:33:01 -0700 (PDT) Subject: [medusa] Need an example client that does not read from stdin and exits the main loop In-Reply-To: <7kbgfs$kb2g@e...> References: <7kbgfs$kb2g@e...> Message-ID: <14185.8754.516841.546019@s...> jody@s... writes: > I've written a server in medusa that works like ftp. That is: > command space args > and the server responds with: > code space results > > This code works using telnet and using a medusa client that reads > from stdin. My questions are: > > 1. How do I send a fixed set of commands to the server from a > client? What I have tried and failed with is in the client's > handle_connect to use self.send(). In the __init__ method for you client class, do something like this: self.push ('200 Go Ahead, Make My Day\r\n') You should never use self.send(), because that will bypass async_chat's outgoing-buffer scheme. If you push() data even before the connection is made, that's ok, it will be sent as soon as it does connect. Look at the class inmonitor.py for an example. > 2. After all of the commands are sent and all responces > processed, how do I exit from the main loop? I usually exit by calling asyncore.close_all(), which clears out the socket map. This causes asyncore.loop() to exit. Let me know if you have any other trouble! -Sam ------------------------------------------------------------------------ eGroups.com home: http://www.egroups.com/group/medusa http://www.egroups.com - Simplifying group communications From jody@l... Fri Jun 18 13:59:01 1999 From: jody@l... (Jody Winston) Date: Fri, 18 Jun 1999 08:59:01 -0400 (EDT) Subject: [medusa] Re: Need an example client that does not read from stdin and exits the main loop In-Reply-To: <14185.8754.516841.546019@s...> (message from Message-ID: <199906181259.IAA23066@h...> >>>>> "Sam" == Sam Rushing writes: Sam> jody@s... writes: >> I've written a server in medusa that works like ftp. That is: >> command space args and the server responds with: code space >> results >> >> This code works using telnet and using a medusa client that >> reads from stdin. My questions are: >> >> 1. How do I send a fixed set of commands to the server from a >> client? What I have tried and failed with is in the client's >> handle_connect to use self.send(). Sam> In the __init__ method for you client class, do something Sam> like this: Sam> self.push ('200 Go Ahead, Make My Day\r\n') The problem with my code was that I had forgot to add the terminator. Sam> You should never use self.send(), because that will bypass Sam> async_chat's outgoing-buffer scheme. If you push() data even Sam> before the connection is made, that's ok, it will be sent as Sam> soon as it does connect. Look at the class Sam> inmonitor.py for an example. I'll change all my self.send to self.push >> 2. After all of the commands are sent and all responces >> processed, how do I exit from the main loop? Sam> I usually exit by calling asyncore.close_all(), which clears Sam> out the socket map. This causes asyncore.loop() to exit. When I was using self.send, calling asyncore.close_all happened before all the processing was finished on the client side. So I sent the quit messesage to the server which broke the connection to the client. Sam> Let me know if you have any other trouble! ------------------------------------------------------------------------ eGroups.com home: http://www.egroups.com/group/medusa http://www.egroups.com - Simplifying group communications From wolf@s... Thu Jun 24 00:49:27 1999 From: wolf@s... (Wolf Logan) Date: Wed, 23 Jun 1999 16:49:27 -0700 Subject: [medusa] Re: select error on win32 Message-ID: <219B76713101D31183B000902762824E104D@t...> microsoft's "official" position seems to be that you're not allowed to know. there's no defined way to know the "real" limit without stress-testing. however, the "real" limit on listen queues under NT is as follows: NT 3.5 : 100 NT 3.51 : 100 NT 4.0 (wks) : 5 NT 4.0 (srv) : 200 i don't know what the limits are for 95/98, but win CE (!) has a backlog of 5 over serial, and 2 over IR. asking for a queue larger than the size shown will set the queue to its maximum. -----Original Message----- From: Sam Rushing [mailto:rushing@n...] Sent: Tuesday, June 22, 1999 3:57 PM Amos Latteier writes: > Can NT not handle a listen queue greater than 1? I'm drawing a blank... Either NT ignores the listen queue size, or at one point it (or more likely win95) would throw an exception. ------------------------------------------------------------------------ eGroups.com home: http://www.egroups.com/group/medusa http://www.egroups.com - Simplifying group communications From rushing@n... Thu Jun 24 16:18:20 1999 From: rushing@n... (Sam Rushing) Date: Thu, 24 Jun 1999 08:18:20 -0700 (PDT) Subject: [medusa] asyncore.py descriptor-map hack Message-ID: <14194.19098.366275.507242@s...> I changed asyncore.dispatcher to cache fileno() information in a dictionary stored at the class level. This dictionary can be used to feed select() file descriptors directly, therefore removing its need to call the fileno() method of each socket object. [this was doubly bad because it would trigger a __getattr__ call]. It's possible that this will remove a significant amount of overhead from poll(), but I don't have time to thoroughly test it for speed right now. For those who might be interested, it's checked in as revision 2.41. [2.40 and 2.42 are exactly the same, I couldn't figure out how to correctly make a branch with CVS within 10 minutes so I gave up] The following command will do the trick: $ cvs up -r2.41 asyncore.py -Sam ------------------------------------------------------------------------ eGroups.com home: http://www.egroups.com/group/medusa http://www.egroups.com - Simplifying group communications From brian.brown@s... Tue Jul 27 20:23:58 1999 From: brian.brown@s... (brian.brown@s...) Date: Tue, 27 Jul 1999 12:23:58 -0700 Subject: [medusa] Virtual Hosting? Message-ID: <7nl10e$hp2n@e...> Does medusa support binding to a specific IP address? Also, does it support serving up different document roots based on DNS domain information contained in the http 1.1 header? thanks! Brian From rushing@n... Tue Jul 27 22:34:58 1999 From: rushing@n... (Sam Rushing) Date: Tue, 27 Jul 1999 14:34:58 -0700 (PDT) Subject: [medusa] Virtual Hosting? In-Reply-To: <7nl10e$hp2n@e...> References: <7nl10e$hp2n@e...> Message-ID: <14238.9475.715207.13060@s...> brian.brown@s... writes: > Does medusa support binding to a specific IP address? Also, does it > support serving up different document roots based on DNS domain > information contained in the http 1.1 header? Medusa supports both types of virtual hosting: check out the file 'virtual_handler.py'. Here's how you might use it: [in your copy of 'start_medusa.py'] fs = filesys.os_filesystem ('/var/www/scary') dh = default_handler.default_handler (fs) vh = virtual_handler.virtual_handler_with_host (dh, 'freddy.nightmare.com') hs = http_server.http_server ('', 80, rs, lg) hs.install_handler (vh) -Sam From brian@b... Wed Jul 28 07:24:11 1999 From: brian@b... (brian@b...) Date: Tue, 27 Jul 1999 23:24:11 -0700 Subject: [medusa] Re: Virtual Hosting? In-Reply-To: <14238.9475.715207.13060@s...> Message-ID: <7nm7mb$h5dp@e...> <14238.9475.715207.1306-@s...> wrote: original article:http://www.egroups.com/group/medusa/?start=66 > brian.brown@s... writes: > > Does medusa support binding to a specific IP address? Also, does it > > support serving up different document roots based on DNS domain > > information contained in the http 1.1 header? > > Medusa supports both types of virtual hosting: check out the file > 'virtual_handler.py'. > > Here's how you might use it: > > [in your copy of 'start_medusa.py'] > > fs = filesys.os_filesystem ('/var/www/scary') > dh = default_handler.default_handler (fs) > vh = virtual_handler.virtual_handler_with_host (dh, 'freddy.nightmare.com') > hs = http_server.http_server ('', 80, rs, lg) > hs.install_handler (vh) > > -Sam > Thanks, Sam! Now to figure out how to do it in the Zopeified version ;) From amos@a... Tue Aug 3 00:29:30 1999 From: amos@a... (Amos Latteier) Date: Mon, 02 Aug 1999 16:29:30 -0700 Subject: [medusa] IE5/medusa response delays Message-ID: <3.0.5.32.19990802162930.013973c0@m...> Hi, A Zope is experiencing a medusa problem that I am having trouble diagnosing. He is using Zope on NT4 SP3. When he connects via HTTP to the server using IE5 he sometimes gets delayed responses, especially after waiting a while between requests. In his words, Using netstat -an, I can see that ie5 keeps 2 connections open to the server and they're ESTABLISHED. As soon as the connections go to TIME_WAIT, IE5 will hang. I'm not sure which end is closing the connection. After the original connection goes to TIME_WAIT, I make a request from IE5 and see the long delay, netstat shows that a new connection has been opened, but there is still a long delay getting a reply from the server. If I open another connection from another browser, the "hanging" connection "wakes up" and services the request. Ideas? -Amos From francois_le_coguiec@h... Thu Aug 26 13:52:10 1999 From: francois_le_coguiec@h... (francois_le_coguiec@h...) Date: Thu, 26 Aug 1999 05:52:10 -0700 Subject: [medusa] Porting Medusa to Ruby Message-ID: <7q3d9q$sg3e@e...> An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail-21/medusa-dev/attachments/205c3119/attachment.html From cmenzel@s... Fri Aug 27 07:52:03 1999 From: cmenzel@s... (cmenzel@s...) Date: Thu, 26 Aug 1999 23:52:03 -0700 Subject: [medusa] drop in medusa? Message-ID: <7q5cij$72v6@e...> Hi, I administer a site running apache with basic .htaccess authorization. I would like to 'drop in' medusa but need to use the existing htaccess/ htpasswd files already in place. Is there a module to use or is this built into medusa? Thanks From jam@n... Fri Aug 27 17:49:47 1999 From: jam@n... (jam) Date: Fri, 27 Aug 1999 12:49:47 -0400 Subject: [medusa] Re: drop in medusa? In-Reply-To: <7q5cij$72v6@e...>; from cmenzel@s... on Thu, Aug 26, 1999 at 11:52:03PM -0700 References: <7q5cij$72v6@e...> Message-ID: <19990827124947.A27011@t...> On Thu, Aug 26, 1999 at 11:52:03PM -0700, cmenzel@s... wrote: > > Hi, > I administer a site running apache with basic .htaccess authorization. > I would like to 'drop in' medusa but need to use the existing htaccess/ > htpasswd files already in place. Is there a module to use or is this > built into medusa? > > Thanks > medusa is a framework in and of itself.. it doesn't have support for apache-like htaccess or htpasswd files that I am aware of, though it wouldn't be all that hard to write a module that could parse them or at least convert it to something useable. could you describe a little more about the application, and maybe we can work something out? regards, J -- || visit gfd || psa member #293 || New Image Systems & Services, Inc. From rushing@n... Mon Aug 30 11:34:29 1999 From: rushing@n... (Sam Rushing) Date: Mon, 30 Aug 1999 03:34:29 -0700 (PDT) Subject: [medusa] drop in medusa? In-Reply-To: <7q5cij$72v6@e...> References: <7q5cij$72v6@e...> Message-ID: <14282.24117.67050.810574@s...> cmenzel@s... writes: > I administer a site running apache with basic .htaccess authorization. > I would like to 'drop in' medusa but need to use the existing htaccess/ > htpasswd files already in place. Is there a module to use or is this > built into medusa? The best approach depends on how you are using .htaccess files. If you have a site with thousands of users maintaining their own .htaccess files, then you'll probably want a specialized handler that looks for these and parses them on the fly. If on the other hand you have a simple global password file I would suggest ignoring .htaccess itself (and writing a simple dictionary-interface wrapper for the actual password file - see how unix authentication is done in ftp_server.py). If someone else declares interest in this I will throw together a quick htpasswd-based auth handler. -Sam From cmenzel@s... Mon Aug 30 20:28:07 1999 From: cmenzel@s... (cmenzel@s...) Date: Mon, 30 Aug 1999 12:28:07 -0700 Subject: [medusa] Re: drop in medusa? In-Reply-To: <19990827124947.A27011@t...> Message-ID: <7qem07$c2rr@e...> cmenze-@s... wrote: original article:http://www.egroups.com/group/medusa/?start=71 > On Thu, Aug 26, 1999 at 11:52:03PM -0700, cmenzel@s... wrote: > > > > Hi, > > I administer a site running apache with basic .htaccess authorization. > > I would like to 'drop in' medusa but need to use the existing htaccess/ > > htpasswd files already in place. Is there a module to use or is this > > built into medusa? > > > > Thanks > > > > medusa is a framework in and of itself.. it doesn't have support for > apache-like htaccess or htpasswd files that I am aware of, though it > wouldn't be all that hard to write a module that could parse them or at > least convert it to something useable. could you describe a little more > about the application, and maybe we can work something out? > > regards, > J > -- > || visit gfd > || psa member #293 > || New Image Systems & Services, Inc. Hi, Thanks for the reply. The site is commercial in that customers pay for access to certain directories. Upon payment a cgi program adds their name and password to the .htpasswd file. Having discovered python and medusa I would very much like to be able to replace apache. I am just begining my learning curve with python so I need to come up to speed. The site delivers 800 meg a month and the files are between 30 and 200k. I feel sure Medusa woould be of benefit to my clients. Thanks, Chris From cmenzel@s... Mon Aug 30 20:36:18 1999 From: cmenzel@s... (cmenzel@s...) Date: Mon, 30 Aug 1999 12:36:18 -0700 Subject: [medusa] Re: drop in medusa? In-Reply-To: <14282.24117.67050.810574@s...> Message-ID: <7qemfi$c30p@e...> cmenze-@s... wrote: original article:http://www.egroups.com/group/medusa/?start=72 > cmenzel@s... writes: > > I administer a site running apache with basic .htaccess authorization. > > I would like to 'drop in' medusa but need to use the existing htaccess/ > > htpasswd files already in place. Is there a module to use or is this > > built into medusa? > > The best approach depends on how you are using .htaccess files. If > you have a site with thousands of users maintaining their own .htaccess > files, then you'll probably want a specialized handler that looks for > these and parses them on the fly. > > If on the other hand you have a simple global password file I would > suggest ignoring .htaccess itself (and writing a simple > dictionary-interface wrapper for the actual password file - see how > unix authentication is done in ftp_server.py). > > If someone else declares interest in this I will throw together a > quick htpasswd-based auth handler. > > -Sam > Hi Sam, I think I get it! Maintain the .htpasswd file with a Python module. It is the same idea as maintaining it with the shell script that I use now. I am new to Python but will try to write the wrapper. Thanks, Chris From jam@n... Mon Aug 30 23:18:20 1999 From: jam@n... (jam) Date: Mon, 30 Aug 1999 18:18:20 -0400 Subject: [medusa] Re: drop in medusa? In-Reply-To: <7qem07$c2rr@e...>; from cmenzel@s... on Mon, Aug 30, 1999 at 12:28:07PM -0700 References: <19990827124947.A27011@t...> <7qem07$c2rr@e...> Message-ID: <19990830181820.A863@t...> On Mon, Aug 30, 1999 at 12:28:07PM -0700, cmenzel@s... wrote: > > Hi, > Thanks for the reply. The site is commercial in that customers pay for > access to certain directories. Upon payment a cgi program adds their > name and password to the .htpasswd file. Having discovered python and > medusa I would very much like to be able to replace apache. I am just > begining > my learning curve with python so I need to come up to speed. The site > delivers 800 meg a month and the files are between 30 and 200k. I feel > sure > Medusa woould be of benefit to my clients. > > Thanks, > Chris > it is *certainly* of benefit to your clients, but just keep in mind as you move forward that it is *not* meant to be a drop-in replacement for apache. you ought to be able to get your system up and working-- are you able to change the payment processing code to do something slightly different? if you have control of that code, you can just make it write to some 'standard' file someplace, and write python code to read it in and process it.. in just thinking about it, it ought to be a really quick project. if I can answer any quesions or anything, feel free to fire away either directly or to the list.. this sounds like a cool project! ;) regards, J -- || visit gfd || psa member #293 || New Image Systems & Services, Inc. From ita.mis@e... Tue Aug 31 09:25:10 1999 From: ita.mis@e... (ita.mis@e...) Date: Tue, 31 Aug 1999 01:25:10 -0700 Subject: [medusa] problems with Internet Explorer Message-ID: <7qg3h6$sr7r@e...> Hi, I've experienced a problem similar to the one reported by Amos Latteier concerning IE. I've a medusa installation on a linux box and I've written some scripts to serve dynamic pages. I use the script_handler and persistent scripts. The test application is very simple. I just call a script for building on the fly a home page and than, clicking on the submit button, I post another request for building another form on the fly. When I use Nescape Navigator, everything works fine, but with IE (I user version 4.5 on a Macintosh) only the first call works. When I press the submit button of the home form the browser reports a "sending request" message on its status bar but nothing happens. I have to stop the execution. Has anyone experienced similar behaviour with persistent scripts using IE? Tasselli Marco From rushing@n... Tue Aug 31 09:44:09 1999 From: rushing@n... (Sam Rushing) Date: Tue, 31 Aug 1999 01:44:09 -0700 (PDT) Subject: [medusa] problems with Internet Explorer In-Reply-To: <7qg3h6$sr7r@e...> References: <7qg3h6$sr7r@e...> Message-ID: <14283.38361.900871.692692@s...> ita.mis@e... writes: > When I use Nescape Navigator, everything works fine, but with IE (I > user version 4.5 on a Macintosh) only the first call works. When I > press the submit button of the home form the browser reports a "sending > request" message on its status bar but nothing happens. > I have to stop the execution. This sounds like something that reported to me a while back (by Eric Parker eparker@z...). Apparently everyone but IE has been affixing an extra CRLF to the end of their POST/PUT data, and this is wrong. http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.1 I couldn't come up with a very clean fix for it at the time, and it just got buried. Are you using POST or GET for your form submission? -Sam From rushing@n... Tue Aug 31 10:50:17 1999 From: rushing@n... (Sam Rushing) Date: Tue, 31 Aug 1999 02:50:17 -0700 (PDT) Subject: [medusa] problems with Internet Explorer In-Reply-To: <14283.38361.900871.692692@s...> References: <7qg3h6$sr7r@e...> <14283.38361.900871.692692@s...> Message-ID: <14283.42329.298077.710128@s...> Sam Rushing writes: > I couldn't come up with a very clean fix for it at the time, and it > just got buried. w.r.t. 'extra CRLF' bug, please 'cvs up' these two files. I'm pretty sure it's fixed now (POST script tested with both NS & IE) http_server.py 3.35 script_handler.py 1.9 -Sam From skip@m... Tue Aug 31 23:31:11 1999 From: skip@m... (Skip Montanaro) Date: Tue, 31 Aug 1999 17:31:11 -0500 Subject: [medusa] Hideously long selects? Message-ID: <199908312231.RAA17133@d...> I'm using the ZServer component of Zope (but nothing else) behind an Apache serving as a proxy. I've noticed situations where it seems like I my ZServer process is hung, but when I connect to it with strace (Linux RH 5.2) I see it stalled in a select waiting on a lot of input file descriptors with an absurdly long timeout (typically 30 seconds): select(33, [6 9 10 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 32], [], [], {30, 0} I'm not much of a select expert, but it seems to me that this is what's causing my problem. Let's assume all those file descriptors are wacko web browsers (not a big stretch), so they aren't going to send input any time soon, so it will be a *long* time before Medusa falls out of the select call and gets around to thinking about new connections, right? Shouldn't the timeout be substantially shorter, 0.5-2.0 seconds perhaps? I haven't checked the medusa code yet (that's where I'm headed next), but I thought I'd check to see if I'm way off-base before I start messing around in detail. Thx, Skip Montanaro | http://www.mojam.com/ skip@m... | http://www.musi-cal.com/~skip/ 847-971-7098 | Python: Programming the way Guido indented... From skip@m... Tue Aug 31 23:33:57 1999 From: skip@m... (Skip Montanaro) Date: Tue, 31 Aug 1999 17:33:57 -0500 Subject: [medusa] Hideously long selects? Message-ID: <199908312233.RAA17359@d...> I'm using the ZServer component of Zope (but nothing else) behind an Apache serving as a proxy. I've noticed situations where it seems like I my ZServer process is hung, but when I connect to it with strace (Linux RH 5.2) I see it stalled in a select waiting on a lot of input file descriptors with an absurdly long timeout (typically 30 seconds): select(33, [6 9 10 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 32], [], [], {30, 0} I'm not much of a select expert, but it seems to me that this is what's causing my problem. Let's assume all those file descriptors are wacko web browsers (not a big stretch), so they aren't going to send input any time soon, so it will be a *long* time before Medusa falls out of the select call and gets around to thinking about new connections, right? Shouldn't the timeout be substantially shorter, 0.5-2.0 seconds perhaps? I haven't checked the medusa code yet (that's where I'm headed next), but I thought I'd ask and see if I'm way off-base before I start messing around in detail. Thx, Skip Montanaro | http://www.mojam.com/ skip@m... | http://www.musi-cal.com/~skip/ 847-971-7098 | Python: Programming the way Guido indented... From billtut@m... Tue Aug 31 23:38:57 1999 From: billtut@m... (Bill Tutt) Date: Tue, 31 Aug 1999 15:38:57 -0700 Subject: [medusa] Re: Hideously long selects? Message-ID: <4D0A23B3F74DD111ACCD00805F31D8100DB91161@R...> > From: Skip Montanaro [mailto:skip@m...] > > > I'm not much of a select expert, but it seems to me that this > is what's > causing my problem. Let's assume all those file descriptors > are wacko web > browsers (not a big stretch), so they aren't going to send > input any time > soon, so it will be a *long* time before Medusa falls out of > the select call > and gets around to thinking about new connections, right? No, the listen() socket should be in the file descriptor list passed into select(). select() should return when a new connection attempt is made, then handle_accept() usually gets called. (per handle_read_event() in asyncore.py) > Shouldn't the > timeout be substantially shorter, 0.5-2.0 seconds perhaps? > Not particularlily you don't really want to be eating up useless CPU if you don't have to. Bill From skip@m... Wed Sep 1 01:37:17 1999 From: skip@m... (skip@m...) Date: Tue, 31 Aug 1999 17:37:17 -0700 Subject: [medusa] Re: Hideously long selects? In-Reply-To: <199908312233.RAA17359@d...> Message-ID: <7qhsft$v69i@e...> [ long timeout ramblings deleted ] I tried shortening the select timeout to 2.0 seconds. Same problem. When I made the change I noticed that I picked the wrong version of asyncore.py to modify. When I compared them I got the following version info: The version from ZServer had: $Id: asyncore.py,v 1.2 1999/04/09 00:37:33 amos Exp $ The Python (1.5.2+) version had: $Id: asyncore.py,v 1.2 1999/06/08 13:20:05 guido Exp $ Any chance I'm seeing some version skew problems? Thanks, Skip From rushing@n... Wed Sep 1 03:43:42 1999 From: rushing@n... (Sam Rushing) Date: Tue, 31 Aug 1999 19:43:42 -0700 (PDT) Subject: [medusa] Re: Hideously long selects? In-Reply-To: <7qhsft$v69i@e...> References: <199908312233.RAA17359@d...> <7qhsft$v69i@e...> Message-ID: <14284.37598.265974.116161@s...> skip@m... writes: > [ long timeout ramblings deleted ] > > I tried shortening the select timeout to 2.0 seconds. Same problem. My guess is that your accept()ing socket is getting closed. In this kind of situation I usually add a 'print r' line just before the call to select(). This will tell you exactly which objects are waiting for read. If your server is not in there, then you have your culprit. > When I made the change I noticed that I picked the wrong version of > asyncore.py to modify. When I compared them I got the following > version info: > > The version from ZServer had: > $Id: asyncore.py,v 1.2 1999/04/09 00:37:33 amos Exp $ > > The Python (1.5.2+) version had: > $Id: asyncore.py,v 1.2 1999/06/08 13:20:05 guido Exp $ > > Any chance I'm seeing some version skew problems? Hmmm.. I get: $Id: asyncore.py,v 2.44 1999/07/27 00:50:06 rushing Exp $ 8^) We probably have three completely different CVS repositories at work here. To get the latest version from medusa: bash$ export CVSROOT=:pserver:medusa@s...:/usr/local/cvsroot bash$ cvs login ['medusa' is the pa**w**d] bash$ cvs checkout medusa/asyncore.py -Sam From jim@d... Wed Sep 1 13:06:54 1999 From: jim@d... (Jim Fulton) Date: Wed, 01 Sep 1999 08:06:54 -0400 Subject: [medusa] Re: Hideously long selects? References: <199908312233.RAA17359@d...> Message-ID: <37CD16DE.711ECAD6@d...> Skip Montanaro wrote: > > I'm using the ZServer component of Zope (but nothing else) behind an Apache > serving as a proxy. I've noticed situations where it seems like I my > ZServer process is hung, but when I connect to it with strace (Linux RH 5.2) > I see it stalled in a select waiting on a lot of input file descriptors with > an absurdly long timeout (typically 30 seconds): > > select(33, [6 9 10 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 32], [], [], {30, 0} > > I'm not much of a select expert, but it seems to me that this is what's > causing my problem. Let's assume all those file descriptors are wacko web > browsers (not a big stretch), so they aren't going to send input any time > soon, so it will be a *long* time before Medusa falls out of the select call > and gets around to thinking about new connections, right? Shouldn't the > timeout be substantially shorter, 0.5-2.0 seconds perhaps? > > I haven't checked the medusa code yet (that's where I'm headed next), but I > thought I'd ask and see if I'm way off-base before I start messing around in > detail. Skip, What version of Zope are you running? I beleave that this is due to a bug that was fixed in the Zope 2.0 beta 5. Jim -- Jim Fulton mailto:jim@d... Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From jim@d... Wed Sep 1 13:11:47 1999 From: jim@d... (Jim Fulton) Date: Wed, 01 Sep 1999 08:11:47 -0400 Subject: [medusa] Re: Hideously long selects? References: <7qhsft$v69i@e...> Message-ID: <37CD1803.44A9B587@d...> skip@m... wrote: > > [ long timeout ramblings deleted ] > > I tried shortening the select timeout to 2.0 seconds. Same problem. > When I made the change I noticed that I picked the wrong version of > asyncore.py to modify. When I compared them I got the following > version info: > > The version from ZServer had: > $Id: asyncore.py,v 1.2 1999/04/09 00:37:33 amos Exp $ This is the number it has in the Zope CVS repository. > The Python (1.5.2+) version had: > $Id: asyncore.py,v 1.2 1999/06/08 13:20:05 guido Exp $ > > Any chance I'm seeing some version skew problems? This is the number it has in the medusa CVS repository. Obviously, this should be handled better somehow...... but I'm not sure how. Jim -- Jim Fulton mailto:jim@d... Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From ray@u... Thu Sep 2 23:35:49 1999 From: ray@u... (ray@u...) Date: Thu, 02 Sep 1999 15:35:49 -0700 Subject: [medusa] Where is the lastest version? Message-ID: <7qmu45$e97m@e...> Is the latest Medusa version accessable via HTTP or FTP? The most current version I can find on nightmare.com is medusa-src-990517.tar.gz. Ray From rushing@n... Fri Sep 3 00:40:53 1999 From: rushing@n... (Sam Rushing) Date: Thu, 2 Sep 1999 16:40:53 -0700 (PDT) Subject: [medusa] Where is the lastest version? In-Reply-To: <7qmu45$e97m@e...> References: <7qmu45$e97m@e...> Message-ID: <14287.2821.802399.505716@s...> ray@u... writes: > Is the latest Medusa version accessable via HTTP or FTP? The most > current version I can find on nightmare.com is medusa-src-990517.tar.gz. I've just made a tarball of the latest, available via http or ftp. http://www.nightmare.com/medusa/medusa-src-990902.tar.gz ftp://www.nightmare.com/medusa/medusa-src-990902.tar.gz [note: the ftp link has moved] -Sam From ray@u... Fri Sep 3 00:55:14 1999 From: ray@u... (ray@u...) Date: Thu, 2 Sep 1999 16:55:14 -0700 Subject: [medusa] Re: Where is the lastest version? Message-ID: <882567E0.00832ED4.00@b...> Thanks Sam. Hey, the first Python user group meeting is here next week. Is there any chance I can get you down some time to demo/show/talkabout/ Medusa or eGroups? Anytime you and your family are in the area, you are welcome to stay at my house with me and my family. Ray Sam Rushing on 09/02/99 04:40:53 PM Please respond to medusa@egroups.com To: medusa@egroups.com cc: (bcc: Ray Price/UAI) Subject: [medusa] Where is the lastest version? ray@u... writes: > Is the latest Medusa version accessable via HTTP or FTP? The most > current version I can find on nightmare.com is medusa-src-990517.tar.gz. I've just made a tarball of the latest, available via http or ftp. http://www.nightmare.com/medusa/medusa-src-990902.tar.gz ftp://www.nightmare.com/medusa/medusa-src-990902.tar.gz [note: the ftp link has moved] -Sam ------------------------------------------------------------------------ MyPoints-Free Rewards When You're Online. Start with up to 150 Points for joining! http://clickhere.egroups.com/click/805 eGroups.com home: http://www.egroups.com/group/medusa http://www.egroups.com - Simplifying group communications From jam@n... Fri Sep 3 04:45:08 1999 From: jam@n... (jam) Date: Thu, 2 Sep 1999 23:45:08 -0400 Subject: [medusa] trouble with 19990902 snapshot: asyncore.loop() exists immediately Message-ID: <19990902234508.A9613@t...> greetings, I've been beating my head against the wall for an hour or two trying to figure out what I might be doing wrong, but to no avail.. hopefully someone on the list can shed some light on this.. I have a very simple 'httpd.py' script: -- cut here -- import sys import asyncore if __name__ == "__main__": from medusa import default_handler, filesys, http_server, resolver, logger rs = resolver.caching_resolver("127.0.0.1") lg = logger.file_logger(sys.stdout) fs = filesys.os_filesystem(".") dh = default_handler.default_handler(fs) hs = http_server.http_server('toast', 8081, rs, lg) hs.install_handler(dh) asyncore.loop() -- cut here -- when I try to run this script from the command line, it exists immediately, and doesn't wait for connections.. I get my prompt back. if I run 'http_server.py from the 'medusa' directory, everything seems to work fine, and it sits and waits for connections. my first thought is that I somehow set up something incorrectly, but the above is much simpler than what I had with the older version, and I've tried removing the various pieces to see if they might be causing a problem, but have not had any luck so far. the only thing that might be remotely unusual about my setup is that I am using python 1.5.2. to install this version of medusa, I renamed the 'asyncore' and 'asynchat' modules, and copied the ones from the distribution into /usr/lib/python1.5, to make sure I was using the right versions. could someone shed a clue on this problem? I'm almost certain it's something silly. any help appreciated. thanks! regards, J -- || visit gfd || psa member #293 || New Image Systems & Services, Inc. From jam@n... Fri Sep 3 05:16:52 1999 From: jam@n... (jam) Date: Fri, 3 Sep 1999 00:16:52 -0400 Subject: [medusa] Re: trouble with 19990902 snapshot: asyncore.loop() exists immediately In-Reply-To: <19990902234508.A9613@t...>; from jam on Thu, Sep 02, 1999 at 11:45:08PM -0400 References: <19990902234508.A9613@t...> Message-ID: <19990903001652.B9613@t...> On Thu, Sep 02, 1999 at 11:45:08PM -0400, jam wrote: > > greetings, > > I've been beating my head against the wall for an hour or two trying to > figure out what I might be doing wrong, but to no avail.. hopefully someone > on the list can shed some light on this.. > > -- cut here -- > import sys > import asyncore > > if __name__ == "__main__": > from medusa import default_handler, filesys, http_server, resolver, logger > rs = resolver.caching_resolver("127.0.0.1") > lg = logger.file_logger(sys.stdout) > fs = filesys.os_filesystem(".") > dh = default_handler.default_handler(fs) > hs = http_server.http_server('toast', 8081, rs, lg) > hs.install_handler(dh) > asyncore.loop() > -- cut here -- > [..snipped..] I have been plugging away a little more at this problem, and when I change the 'from medusa import ...' line to include 'asyncore', and remove it from the top of the script, everything works *fine*. this is odd because I copied the 'asyncore.py' file from the distribution into /usr/lib/python1.5, so there really shouldn't be a problem, should there? can this be handled by adding something to the '__init__.py' file in the medusa directory? it appears I've tickled some wierd feature of the interpeter, or perhaps less of a feature or *something*. further comments, questions, flames, etc, welcomed. regards, J -- || visit gfd || psa member #293 || New Image Systems & Services, Inc. From rushing@n... Fri Sep 3 05:51:43 1999 From: rushing@n... (Sam Rushing) Date: Thu, 2 Sep 1999 21:51:43 -0700 (PDT) Subject: [medusa] Re: trouble with 19990902 snapshot: asyncore.loop() exists immediately In-Reply-To: <19990903001652.B9613@t...> References: <19990902234508.A9613@t...> <19990903001652.B9613@t...> Message-ID: <14287.21471.552306.353000@s...> jam writes: > I have been plugging away a little more at this problem, and when I change > the 'from medusa import ...' line to include 'asyncore', and remove it from > the top of the script, everything works *fine*. this is odd because I copied > the 'asyncore.py' file from the distribution into /usr/lib/python1.5, so > there really shouldn't be a problem, should there? Ok, I figured it out... the package system is letting you introduce two different versions of the 'asyncore' module. The one used for 'asyncore.loop()' is not the one that knows about listening sockets. Change the code thus: > hs.install_handler(dh) > print asyncore, id(asyncore) > print http_server.asyncore, id(http_server.asyncore) > asyncore.loop() And you get this: | rushing@s...:/usr/src/nm/apps$ python strange.py | info: adding channel | info: adding channel | info: Medusa (V3.35) started at Thu Sep 2 21:48:29 1999 | Hostname: seattle | Port:8081 | | 135185384 | 135223912 sneaky, but not really the package system's fault: the whole idea of a package system is that you shouldn't have to worry about name collisions between unrelated packages. -Sam From jam@n... Fri Sep 3 06:26:58 1999 From: jam@n... (jam) Date: Fri, 3 Sep 1999 01:26:58 -0400 Subject: [medusa] Re: trouble with 19990902 snapshot: asyncore.loop() exists immediately In-Reply-To: <14287.21471.552306.353000@s...>; from Sam Rushing on Thu, Sep 02, 1999 at 09:51:43PM -0700 References: <19990902234508.A9613@t...> <19990903001652.B9613@t...> <14287.21471.552306.353000@s...> Message-ID: <19990903012658.C9613@t...> On Thu, Sep 02, 1999 at 09:51:43PM -0700, Sam Rushing wrote: > > Ok, I figured it out... the package system is letting you introduce > two different versions of the 'asyncore' module. The one used for > 'asyncore.loop()' is not the one that knows about listening sockets. > you are correct. when I renamed the copies of 'asyncore' and 'asynchat' in the medusa directory the problem goes away. thanks for the help ;) regards, J -- || visit gfd || psa member #293 || New Image Systems & Services, Inc. From skip@m... Fri Sep 3 18:50:10 1999 From: skip@m... (Skip Montanaro) Date: Fri, 3 Sep 1999 12:50:10 -0500 Subject: [medusa] Re: hideously long selects? Message-ID: <199909031750.MAA10640@d...> I'm baaack... Jim Fulton asked: > What version of Zope are you running? I beleave that this is due to a > bug that was fixed in the Zope 2.0 beta 5. I was running 1.something. I downloaded the 2.0.0 source today and installed it, then modified my old start.py to work with the new ZServer. I tried it out a bit on my development machine. Things were *real slow*, but nothing was happening, so I applied strace (fast becoming one of my favorite debugging tools) to both my database server and the ZServer process. The results were instructive. The database server trace looks like: 12:12:36 select(6, [5], [], [], {2, 500000}) = 0 (Timeout) 12:12:39 select(6, [5], [], [], {2, 500000}) = 1 (in [5], left {0, 280000}) 12:12:41 accept(5, {sin_family=AF_INET, sin_port=htons(4321), sin_addr=inet_addr("127.0.0.1")}, [16]) = 12 ... chunka chunka ... 12:12:41 read(13, "POST /RPC2 HTTP/1.0\r\nHost: loc"..., 1024) = 43 12:12:41 read(13, "User-Agent: xmlrpclib.py/0.9.8 ("..., 1024) = 56 12:12:41 read(13, "Content-Type: text/xml\r\nConten"..., 1024) = 170 ... chunka chunka ... 12:12:41 write(14, "HTTP/1.0 200 OK\r\nServer: BaseH"..., 1024) = 1024 ... bunch more writes ... 12:12:41 close(15) = 0 ... a few munmaps ... 12:12:41 select(6, [5], [], [], {2, 500000}) = 0 (Timeout) ... So, within the space of somewhat less than a second it accepts a connect, handles the request (in this case an XML-RPC call requesting the top 50 cities and performers the server has gotten requests for), then returns to its waiting state. Now, lets have a look at the trace for the ZServer: 12:12:39 accept(5, {sin_family=AF_INET, sin_port=htons(4319), sin_addr=inet_addr("127.0.0.2")}, [16]) = 6 12:12:39 write(1, "info: adding channel ) = 72 12:12:39 fcntl(6, F_GETFL) = 0x2 (flags O_RDWR) 12:12:39 fcntl(6, F_SETFL, O_RDWR|O_NONBLOCK) = 0 12:12:39 gettimeofday({936378759, 269643}, NULL) = 0 12:12:39 gettimeofday({936378759, 270168}, NULL) = 0 12:12:39 select(7, [5 6], [], [], {30, 0}) = 1 (in [6], left {30, 0}) 12:12:39 recv(6, "GET /filter?file=index.shtml HTT"..., 4096, 0) = 277 12:12:39 gettimeofday({936378759, 273572}, NULL) = 0 12:12:39 brk(0x827b000) = 0x827b000 12:12:39 pipe([11, 12]) = 0 12:12:39 clone() = 8159 12:12:39 write(12, "4\24%@\4\0\0\0\201\0\0\0 \205$@`"..., 148) = 148 12:12:39 sigprocmask(SIG_SETMASK, NULL, [USR1]) = 0 12:12:39 write(12, "4\24%@\0\0\0\0\0\0\0\0T\370\7\10"..., 148) = 148 12:12:39 sigprocmask(SIG_SETMASK, NULL, [USR1]) = 0 12:12:39 sigsuspend([] 12:12:39 --- SIGUSR1 (User defined signal 1) --- 12:12:39 <... sigsuspend resumed> ) = -1 EINTR (Interrupted system call) 12:12:39 sigreturn() = ? (mask now [USR1]) 12:12:39 kill(8160, SIGUSR1) = 0 12:12:39 kill(8160, SIGUSR1) = 0 12:12:39 sigprocmask(SIG_SETMASK, NULL, [USR1]) = 0 12:12:39 sigsuspend([] 12:12:39 --- SIGUSR1 (User defined signal 1) --- 12:12:39 <... sigsuspend resumed> ) = -1 EINTR (Interrupted system call) 12:12:39 sigreturn() = ? (mask now [USR1]) 12:12:39 select(7, [5 6], [], [], {30, 0}) = 0 (Timeout) 12:13:09 select(7, [5 6], [6], [], {30, 0}) = 1 (out [6], left {30, 0}) 12:13:09 send(6, "HTTP/1.0 200 OK\r\nServer: Zope/"..., 13297, 0) = 13297 12:13:09 select(7, [5 6], [6], [], {30, 0}) = 1 (out [6], left {30, 0}) 12:13:09 gettimeofday({936378789, 284258}, NULL) = 0 12:13:09 write(4, "127.0.0.2:4319 - - [03/Sep/1999:"..., 98) = 98 12:13:09 write(1, "info: closing channel 6:) = 99 12:13:09 close(6) = 0 12:13:09 close(-1) = -1 EBADF (Bad file descriptor) 12:13:09 select(6, [5], [], [], {30, 0}) = 0 (Timeout) I assume the chit-chat with the backend database server is happening in a separate thread, because I see no reads or writes around 12:12:41 in the trace. At any rate, strace seems not to be able to attach to those threads, so their chit-chat is unavailable. Note the back-to-back selects at 12:12:39 and 12:13:09. First it has fd 6 only in the read set and the select timed out after 30 seconds. Then it added 6 to the write set for the second select which returned immediately, and it sent the response back to Apache (which is serving as a proxy in this case) and from there to the user. Why'd it sit on that first select for 30 seconds? Any reason why fd 6 wasn't in the write set to begin with? I presume it is the socket which it uses to talk to the backend server. (This would be in httplib or stuff it calls.) To make sure I was getting appropriate versions of important modules, I had start.py print out sys.path and the locations of those modules I thought most important when ZServer was started: sys.path is (ordered first to last): 0: /usr/local/lib/python1.5/site-packages 1: /home/dolphin/skip/src/Zope-2.0.0-src/ZServer/../lib/python 2: /home/dolphin/skip/src/Zope-2.0.0-src/ZServer 3: /home/dolphin/skip/src/Zope-2.0.0-src/ZServer/.. 4: /home/dolphin/skip/src/Zope-2.0.0-src/ZServer 5: /home/dolphin/skip/src/vtk-linux/lib 6: /usr/local/lib/python1.5/ 7: /usr/local/lib/python1.5/plat-linux2 8: /usr/local/lib/python1.5/lib-tk 9: /usr/local/lib/python1.5/lib-dynload 10: /usr/local/lib/automatrix/python Here are the locations of some key modules, as discovered by imp.find_module: asyncore is at /home/dolphin/skip/src/Zope-2.0.0-src/ZServer/medusa/asyncore.py asynchat is at /home/dolphin/skip/src/Zope-2.0.0-src/ZServer/medusa/asynchat.py xmlrpclib is at /usr/local/lib/python1.5/site-packages/xmlrpclib.py sgmlop is builtin Now, if I shorten up the timeout for asyncore.loop, the pregnant pause goes away, but there's still this fd 6 in the read set followed by fd 6 in both the read and write sets: 12:39:13 sigreturn() = ? (mask now [USR1]) 12:39:13 select(7, [5 6], [], [], {0, 800000}) = 0 (Timeout) 12:39:14 select(7, [5 6], [6], [], {0, 800000}) = 1 (out [6], left {0, 800000}) 12:39:14 send(6, "HTTP/1.0 200 OK\r\nServer: Zope/"..., 37112, 0) = 37112 I would agree that long timeouts are okay if we could just eliminate that first select. Am I hosed because I'm mixing blocking (httplib) and non-blocking (asyncore) I/O? Should httplib be telling asyncore about its sockets? Skip Montanaro | http://www.mojam.com/ skip@m... | http://www.musi-cal.com/~skip/ 847-971-7098 | Python: Programming the way Guido indented... From jam@n... Fri Sep 3 21:18:17 1999 From: jam@n... (jam) Date: Fri, 3 Sep 1999 16:18:17 -0400 Subject: [medusa] script_handler: possible optimization? Message-ID: <19990903161817.E9613@t...> greetings, I'm working on (the forth or fifth generation of) my website on quark, and I'm using the latest-and-greatest medusa framework to do so, because I noticed this great handler called persistent_script_handler that I *really* wanted to use ;) I have managed to get things up and running at least to the point that I don't have any odd problems (like the server exiting immediately because I had imported two different instances of the asyncore module). the next phase of this is that I need to somehow scale the size of the site up by a factor of several. I have two modules so far, and my primary design concern at the moment is 'how do I hang other modules from different places in the directory tree?'.. for example: /gfd/index.html - handled by 'main.py' - also might handle things like '/gfd/about.html' or '/gfd/resume.html' /gfd/links/index.html - the 'match' routine in the 'links' module will need to know where it hangs on the 'tree' in order to determine if it should handle the request or not. this is where things get complicated, especially if the site uses a lot of these modules. one thought I had was to simplify things by *not* wrapping lots of different functions into one module by splitting each 'page' into it's own module ('about.py' handles '/gfd/about.html', 'resume.py' would handle '/gfd/resume.*', etc.) but that *also* seems like it would get complicated fast. I do have a small contribution to add to the cause, which anyone may use as they see fit: I made a new version of the persistent_script_handler class (called 'gfd_script_handler', below) that calls a method called 'match' in each module to let *it* tell the framework if it can deal with the request or not. I also changed the add_module routine to make it simpler to make the 'autoreload' functionality a possibility in the future (it's pretty easy once you start using __import__ and the ihooks module). comments, suggestions, strange looks, etc, accepted. -- cut here -- #!/usr/bin/env python # # this code uses medusa, available from # import sys import asyncore from cStringIO import StringIO from medusa import status_handler, default_handler, filesys, http_server, counter from medusa import script_handler, resolver, logger import config class gfd_script_handler: def __init__(self): self.exceptions = counter.counter() self.modules = {} def add_module(self, key): moduleref = __import__(key) self.modules[key] = moduleref def del_module(self, key): del self.modules[key] def match(self, request): for m in self.modules.values(): if hasattr(m, "match"): if m.match(request) == 1: print "found match", id(m) request.module = m return 1 return 0 def handle_request(self, request): if request.command in ( "put", "post"): cl = request.get_header("content-length") length = int(cl) if not cl: request.error(411) else: collector(self, length, request) else: self.continue_request(request, StringIO()) def continue_request(self, request, input_data): temp_files = input_data, StringIO(), StringIO() old_files = sys.stdin, sys.stdout, sys.stderr try: sys.stdin, sys.stdout, sys.stderr = temp_files request["content-type"] = "text/html" try: request.module.main(request) request.reply_code = 200 except: request.reply_code = 500 self.exceptions.increment() finally: sys.stdin, sys.stdout, sys.stderr = old_files i, o, e = temp_files if request.reply_code != 200: s = e.getvalue() else: s = o.getvalue() request["content-length"] = len(s) request.push(s) request.done() if __name__ == "__main__": sys.path.append("/home/jam/projects/thpwact/modules/") rs = resolver.resolver("neutron") lg = logger.file_logger(sys.stdout) hs = http_server.http_server("toast", 8080, rs, lg) # # the 'default_handler' tries to cover requests that aren't # handled by the script_handler (like /images) # fs = filesys.os_filesystem(config.SOFTROOT) dh = default_handler.default_handler(fs) hs.install_handler(dh) gsh = gfd_script_handler() gsh.add_module("about") gsh.add_module("main") hs.install_handler(gsh) sh = status_handler.status_extension( (hs, lg, gsh) ) hs.install_handler(sh) asyncore.loop() -- cut here -- then, as an example, here's my 'about.py' module, which uses DocumentTemplate and a few things from the 'config' module (which I'm not including lest I get shot for such a huge post ;)): -- cut here -- _RCSID = "$Id: $" import os import common import config def match(request): path, params, query, fragment = request.split_uri() parts = os.path.split(path) ## print "about: ", parts if len(parts) > 1 and parts[1] == "about.html": return 1 else: return 0 def main(request): print common.standard_header(title="about this site") print common.HTMLFile(config.DTMLROOT + "about.dtml")() print common.standard_footer() if __name__ == "__main__": # for testing main(None) -- cut here -- regards, J -- || visit gfd || psa member #293 || New Image Systems & Services, Inc. From jam@n... Tue Sep 7 15:27:42 1999 From: jam@n... (jam) Date: Tue, 7 Sep 1999 10:27:42 -0400 Subject: [medusa] running medusa in the background without exit Message-ID: <19990907102742.Q9613@t...> greetings! I've been using the 1999-09-02 rev of medusa for the last few days, and it's working *great*. the question I have, though, since medusa exits when it gets a SIGHUP, is how do I keep the server running in the background even when I log out? I have set up apache to redirect all requests on port 80 to the medusa server on port 8080.. this is fine when I start medusa from my shell prompt and let it go, and I could use 'nohup' to prevent it from exiting, but then I wouldn't be able to restart it without a fuss (something I need to be able to do until the 'reload' of modules on-the-fly is working. how do other folks handle this situation? any suggestions? regards, J -- || visit gfd || psa member #293 || New Image Systems & Services, Inc. From jim@d... Tue Sep 7 15:40:19 1999 From: jim@d... (Jim Fulton) Date: Tue, 07 Sep 1999 10:40:19 -0400 Subject: [medusa] Re: running medusa in the background without exit References: <19990907102742.Q9613@t...> Message-ID: <37D523D3.AA68C437@d...> jam wrote: > > greetings! > > I've been using the 1999-09-02 rev of medusa for the last few days, and it's > working *great*. > > the question I have, though, since medusa exits when it gets a SIGHUP, is > how do I keep the server running in the background even when I log out? I > have set up apache to redirect all requests on port 80 to the medusa server > on port 8080.. this is fine when I start medusa from my shell prompt and let > it go, and I could use 'nohup' to prevent it from exiting, but then I > wouldn't be able to restart it without a fuss (something I need to be able > to do until the 'reload' of modules on-the-fly is working. > > how do other folks handle this situation? any suggestions? The Zope distribution (www.zope.org) includes a module, zdaemon, for *easily* creating Unix daemons, which is what you want to do. We plan, eventually, to release this separately for use by other Python programs. In the mean time, feel free to grab it and use it. For an example of it's use, see the Zope ZServer startup script, z2.py. Jim -- Jim Fulton mailto:jim@d... Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From jam@n... Tue Sep 7 15:57:58 1999 From: jam@n... (jam) Date: Tue, 7 Sep 1999 10:57:58 -0400 Subject: [medusa] Re: running medusa in the background without exit In-Reply-To: <37D523D3.AA68C437@d...>; from Jim Fulton on Tue, Sep 07, 1999 at 10:40:19AM -0400 References: <19990907102742.Q9613@t...> <37D523D3.AA68C437@d...> Message-ID: <19990907105758.R9613@t...> On Tue, Sep 07, 1999 at 10:40:19AM -0400, Jim Fulton wrote: > > The Zope distribution (www.zope.org) includes a module, zdaemon, > for *easily* creating Unix daemons, which is what you want to do. > We plan, eventually, to release this separately for use by other > Python programs. In the mean time, feel free to grab it and use it. > For an example of it's use, see the Zope ZServer startup script, > z2.py. > > Jim > wow! thanks for the quick response! ;).. I'll give zope a shot. regards, J -- || visit gfd || psa member #293 || New Image Systems & Services, Inc. From Paul@d... Wed Sep 8 19:30:33 1999 From: Paul@d... (Paul Everitt) Date: Wed, 8 Sep 1999 14:30:33 -0400 Subject: [medusa] Q: Replacement URL for case study? Message-ID: <613145F79272D211914B0020AFF64019262A20@g...> Skip-- We have a link to concerts.calendar.com on the zope.org site. What should it be linked to these days? --Paul From Paul@d... Wed Sep 8 19:31:29 1999 From: Paul@d... (Paul Everitt) Date: Wed, 8 Sep 1999 14:31:29 -0400 Subject: [medusa] OOPS: My apologies Message-ID: <613145F79272D211914B0020AFF64019262A21@g...> Sorry for the email! I'm sure there are, errm, more direct ways to reach Skip. :^) --Paul From skip@m... Wed Sep 8 19:34:26 1999 From: skip@m... (Skip Montanaro) Date: Wed, 8 Sep 1999 13:34:26 -0500 (CDT) Subject: [medusa] Re: Q: Replacement URL for case study? In-Reply-To: <613145F79272D211914B0020AFF64019262A20@g...> References: <613145F79272D211914B0020AFF64019262A20@g...> Message-ID: <14294.43990.30179.213234@d...> Paul> We have a link to concerts.calendar.com on the zope.org site. Paul> What should it be linked to these days? Paul, Hopefully, concerts.calendar.com will be back on the air shortly, but the best place to link to is http://www.musi-cal.com/. We sold the calendar.com domain awhile back (tough to refuse cash you know) and are now using musi-cal.com. Skip From jkraai@m... Fri Sep 10 15:31:49 1999 From: jkraai@m... (jkraai@m...) Date: Fri, 10 Sep 1999 07:31:49 -0700 Subject: [medusa] Job: Async Message-ID: <7rb4ol$c4v8@e...> [Reposted from C.L.P.] Need someone interested in six month contract using Python. Two types of projects. 1. Implementing an HTTP'ish protocol via Sam Rushing's fine Async/Async Chat libraries. 2. Integration of existing C/C++ routines into Python core. We're located in Iowa. We are not asking for relocation. Demonstrable track record with async libs is required. Please reply with resume, rates, availability, and examples of successful projects. Thanks for the use of the list & for the great language & for the fantastic modules & for the beneavolent community & for ... Sincerely, --jim kraai President, MURL.COM From wkp@c... Wed Nov 10 23:49:53 1999 From: wkp@c... (wkp@c...) Date: Wed, 10 Nov 1999 15:49:53 -0800 Subject: [medusa] cookies Message-ID: <80d0b1$8bir@e...> I've been doing some CGI scripting with Python and Medusa. It's great! However, I'm not sure how to set and get cookies within my server-side scripts. Does anyone have any examples? Wesley Phoa. From robin@j... Sun Nov 14 12:53:41 1999 From: robin@j... (Robin Becker) Date: Sun, 14 Nov 1999 12:53:41 +0000 Subject: [medusa] strangeness in zope's version of ftpserver.py Message-ID: I'm using the medusa version distributed with the latest Zope and notice that ftpserver.py seems to have two versions of the close method for ftp_channel. Reason I'm interested is that I'm using the ftp server and seem to observe that abruptly killing an ftp client seems not to increment the closed_sessions_total counter thus leaving the open sessions count at 1 even when I don't have an active client. Has the session actually been closed? -- Robin Becker From rushing@n... Mon Nov 15 02:23:13 1999 From: rushing@n... (Sam Rushing) Date: Sun, 14 Nov 1999 18:23:13 -0800 (PST) Subject: [medusa] strangeness in zope's version of ftpserver.py In-Reply-To: References: Message-ID: <14383.28305.291751.634550@s...> Robin Becker writes: > I'm using the medusa version distributed with the latest Zope and notice > that ftpserver.py seems to have two versions of the close method for > ftp_channel. > > Reason I'm interested is that I'm using the ftp server and seem to > observe that abruptly killing an ftp client seems not to increment the > closed_sessions_total counter thus leaving the open sessions count at 1 > even when I don't have an active client. > > Has the session actually been closed? You can tell by hitting the 'channel list' link from the status page (does zope support the status page? see status_handler.py). It looks like there are two versions of the close method, each doing a little bit of the job. 8^) They most likely need to be made into a single version! -Sam From robin@j... Mon Nov 15 11:31:55 1999 From: robin@j... (Robin Becker) Date: Mon, 15 Nov 1999 11:31:55 +0000 Subject: [medusa] Re: strangeness in zope's version of ftpserver.py In-Reply-To: <14383.28305.291751.634550@s...> References: <14383.28305.291751.634550@s...> Message-ID: <972yTAAr8+L4EwlA@j...> In article <14383.28305.291751.634550@s...>, Sam Rushing writes >Robin Becker writes: > > I'm using the medusa version distributed with the latest Zope and notice > > that ftpserver.py seems to have two versions of the close method for > > ftp_channel. > > > > Reason I'm interested is that I'm using the ftp server and seem to > > observe that abruptly killing an ftp client seems not to increment the > > closed_sessions_total counter thus leaving the open sessions count at 1 > > even when I don't have an active client. > > > > Has the session actually been closed? > >You can tell by hitting the 'channel list' link from the status page >(does zope support the status page? see status_handler.py). > >It looks like there are two versions of the close method, each doing a >little bit of the job. 8^) They most likely need to be made into a >single version! > >-Sam > > ... I have hacked in a page which uses the status stuff. I don't know about the channel list though. I'll have to check that out. -- Robin Becker From robin@j... Mon Nov 15 12:30:05 1999 From: robin@j... (Robin Becker) Date: Mon, 15 Nov 1999 12:30:05 +0000 Subject: [medusa] Re: strangeness in zope's version of ftpserver.py In-Reply-To: <972yTAAr8+L4EwlA@j...> References: <14383.28305.291751.634550@s...> <972yTAAr8+L4EwlA@j...> Message-ID: In article <972yTAAr8+L4EwlA@j...>, Robin Becker writes ... >... >I have hacked in a page which uses the status stuff. I don't know about >the channel list though. I'll have to check that out. OK I checked my working remote zope with a modified status page and it seems as though the original zope ftp_server doesn't react to aborted ftp clients ie it leaves ftp_channels hanging. I'm not certain whether the recently posted ftp_server.py will fix this; I will install on my system and check. I'll need to restart the remote though. -- Robin Becker From rushing@n... Tue Nov 16 07:58:32 1999 From: rushing@n... (Sam Rushing) Date: Mon, 15 Nov 1999 23:58:32 -0800 (PST) Subject: [medusa] cookies In-Reply-To: <80d0b1$8bir@e...> References: <80d0b1$8bir@e...> Message-ID: <14385.3752.482888.238729@s...> wkp@c... writes: > I've been doing some CGI scripting with Python and Medusa. It's great! > However, I'm not sure how to set and get cookies within my server-side > scripts. Does anyone have any examples? You just need to monkey with the 'Cookie:' and 'Set-Cookie:' headers. If you are using the 'persistent_script_handler', then the request object is passed back to your module's main() function. You can access the request headers using 'request.get_header()' and you can set reply headers like this: request['Set-Cookie'] = "xxx" Here's a starting reference: http://www.netscape.com/newsref/std/cookie_spec.html -Sam From rushing@e... Fri Nov 19 11:38:55 1999 From: rushing@e... (Sam Rushing) Date: Fri, 19 Nov 1999 03:38:55 -0800 (PST) Subject: [medusa] tweaks to asyn{core,chat}.py Message-ID: <14389.14031.176921.629212@s...> I've been trying to tighten up the async modules a bit to fight an effect that shows up with lots of concurrent connections (>500). I've made some notes: http://www.nightmare.com/medusa/async_tweaks.html I think the changes are a win, so I'll be checking in this code over the next day or so. If anyone out there is violating the asyncore.socket_map abstraction (relying on the way the dictionary is used) or sees a problem I've missed, please let me know. I think the only code in medusa that cares is the stuff in http_server that kills zombie connections. Zope may have done something similar with the ftp server? -Sam From robin@j... Fri Nov 19 13:37:33 1999 From: robin@j... (Robin Becker) Date: Fri, 19 Nov 1999 13:37:33 +0000 Subject: [medusa] Re: tweaks to asyn{core,chat}.py In-Reply-To: <14389.14031.176921.629212@s...> References: <14389.14031.176921.629212@s...> Message-ID: In article <14389.14031.176921.629212@s...>, Sam Rushing writes >I've been trying to tighten up the async modules a bit to fight an >effect that shows up with lots of concurrent connections (>500). > >I've made some notes: > > http://www.nightmare.com/medusa/async_tweaks.html > >I think the changes are a win, so I'll be checking in this code over >the next day or so. If anyone out there is violating the >asyncore.socket_map abstraction (relying on the way the dictionary is >used) or sees a problem I've missed, please let me know. > >I think the only code in medusa that cares is the stuff in http_server >that kills zombie connections. Zope may have done something similar >with the ftp server? > >-Sam > ... I've been looking at how zombies are handled in Zope. seems that zope has an zhttp_channel analogous to http_channel; it has the zombie stuff, but without the maintainance thing so the kill_zombies method doesn't seem to get called. I noticed that ftp_server doesn't have a zombie timeout is there an obvious reason why not? -- Robin Becker From amos@a... Fri Nov 19 20:09:17 1999 From: amos@a... (Amos Latteier) Date: Fri, 19 Nov 1999 12:09:17 -0800 Subject: [medusa] Re: tweaks to asyn{core,chat}.py In-Reply-To: References: <14389.14031.176921.629212@s...> <14389.14031.176921.629212@s...> Message-ID: <3.0.5.32.19991119120917.015b83a0@m...> Robin Becker wrote: >Sam Rushing wrote: >>I've been trying to tighten up the async modules a bit to fight an >>effect that shows up with lots of concurrent connections (>500). >> >>I've made some notes: >> >> http://www.nightmare.com/medusa/async_tweaks.html >> >>I think the changes are a win, so I'll be checking in this code over >>the next day or so. If anyone out there is violating the >>asyncore.socket_map abstraction (relying on the way the dictionary is >>used) or sees a problem I've missed, please let me know. >> >>I think the only code in medusa that cares is the stuff in http_server >>that kills zombie connections. Zope may have done something similar >>with the ftp server? A few small changes will need to be made to Zope to accommodate this change to asycore.socket_map, but not many. I talked with Jim Fulton about ideas to improve asyncore/asychat select loop performance a while back. Let's see if I can remember some of his ideas. (Note, I may have gotten it slightly wrong, sorry Jim ;-) One problem Zope has with asyncore results from using worker threads in addition to a medusa thread. There are problems when the worker thread is ready to go, but the medusa thread is sitting in a select call, which potentially can take up to 30 secs to return. We get around this by using a select trigger to wake up select, but we would like to get rid of the need for a select trigger. One way to achieve this is to reduce the timeout on the select call to a small enough value that the worker threads can wait for a timeout without being overly inconvenienced. This would require the select loop being adequately optimized. One optimization Jim suggested is to pass integers to the select loop, not descriptor objects. I think that this is what you are suggesting as 'fdcache' optimization. Another optimization would be to have a system of re-entering the select call immediately after returning if nothing is ready--rather than checking the results and rebuilding the arguments to the select call. >... >I've been looking at how zombies are handled in Zope. seems that zope >has an zhttp_channel analogous to http_channel; it has the zombie stuff, >but without the maintainance thing so the kill_zombies method doesn't >seem to get called. I'm pretty sure that it *does* get called. zhttp_channel calls check_maintenance upon initialization. > I noticed that ftp_server doesn't have a zombie >timeout is there an obvious reason why not? I've wondered about this too. -Amos From rushing@n... Sat Nov 20 03:31:24 1999 From: rushing@n... (Sam Rushing) Date: Fri, 19 Nov 1999 19:31:24 -0800 (PST) Subject: [medusa] Re: tweaks to asyn{core,chat}.py In-Reply-To: <3.0.5.32.19991119120917.015b83a0@m...> References: <14389.14031.176921.629212@s...> <3.0.5.32.19991119120917.015b83a0@m...> Message-ID: <14390.5644.955061.401898@s...> Amos Latteier writes: > One problem Zope has with asyncore results from using worker > threads in addition to a medusa thread. There are problems when the > worker thread is ready to go, but the medusa thread is sitting in a > select call, which potentially can take up to 30 secs to return. We > get around this by using a select trigger to wake up select, but we > would like to get rid of the need for a select trigger. Lowering the timeout value just aggravates the CPU-abuse problem... imagine if you had a timeout of zero, then the CPU would stay pegged even if the server were completely idle. Why don't you like the trigger? For the uninitiated, the 'trigger' is a socket/descriptor that is always in the readable set for select. When select() is otherwise 'hung' waiting for a timeout, another thread can 'pull the trigger' and wake up the select call. On Unix, signals can also force select() to return, but it's not portable. > One way to achieve this is to reduce the timeout on the select call > to a small enough value that the worker threads can wait for a > timeout without being overly inconvenienced. This would require the > select loop being adequately optimized. Without a redesign that eliminated the calls to readable() and writable() I think we're stuck. Curiously, the cleanest 'redesign' that achieves this effect is to go to coroutines/fibers. But that's a whole 'nother paradigm... [coroutines combined with completion ports would be the real killer] > One optimization Jim suggested is to pass integers to the select > loop, not descriptor objects. I think that this is what you are > suggesting as 'fdcache' optimization. Waaay back (maybe last year) I did something similar, but I used a separate descriptor-map dictionary. It worked well, but would sometimes run into a problem where the one map wasn't in sync with the other. Now they're the same map... much cleaner. But there are other areas that need tweaking, the poll() overhead is no long on top of the list. I have modified a python here to do a type of statistical profiling (every VM insns a dictionary of {:} is updated). Here's what made me look at refill_buffer(): [...] 709 713 748 751 870 948 1019 1421 1582 1739 3258 4205 4211 4970 > Another optimization would be to have a system of re-entering the > select call immediately after returning if nothing is ready--rather > than checking the results and rebuilding the arguments to the > select call. This is a really good idea! it's getting into an interesting area, of trying to be smarter about scheduling the r/w sets. I think there are some things like that to be found here: http://www.kegel.com/c10k.html A problem I see is for users of the event_loop class, which supports timers. It would be possible (nay, likely) for a timer to modify the readable/writable condition for one or more objects, which would break the above fix. > > I noticed that ftp_server doesn't have a zombie > >timeout is there an obvious reason why not? > > I've wondered about this too. Lazy Sam. Every time I go to add it, I think - "there should be a general facility for this" and of course that allows me to put it off for a while. Lather. Rinse. Repeat. -Sam From robin@j... Sat Nov 20 11:06:56 1999 From: robin@j... (Robin Becker) Date: Sat, 20 Nov 1999 11:06:56 +0000 Subject: [medusa] Re: tweaks to asyn{core,chat}.py In-Reply-To: <3.0.5.32.19991119120917.015b83a0@m...> References: <14389.14031.176921.629212@s...> <14389.14031.176921.629212@s...> <3.0.5.32.19991119120917.015b83a0@m...> Message-ID: In article <3.0.5.32.19991119120917.015b83a0@m...>, Amos Latteier writes ... > >>... >>I've been looking at how zombies are handled in Zope. seems that zope >>has an zhttp_channel analogous to http_channel; it has the zombie stuff, >>but without the maintainance thing so the kill_zombies method doesn't >>seem to get called. > >I'm pretty sure that it *does* get called. zhttp_channel calls >check_maintenance upon initialization. OK I checked and you are right, but since the kill_zombies only happens every 500 creations things can persist a long time. My mistake. I guess in the worst case a large number of sockets (<500) could be zombies which would be bad for systems with small fd limits. > >> I noticed that ftp_server doesn't have a zombie >>timeout is there an obvious reason why not? > >I've wondered about this too. > -- Robin Becker From skip@m... Sat Nov 20 13:25:55 1999 From: skip@m... (Skip Montanaro) Date: Sat, 20 Nov 1999 07:25:55 -0600 (CST) Subject: [medusa] Re: tweaks to asyn{core,chat}.py In-Reply-To: <14390.5644.955061.401898@s...> References: <14389.14031.176921.629212@s...> <3.0.5.32.19991119120917.015b83a0@m...> <14390.5644.955061.401898@s...> Message-ID: <14390.41315.807375.200241@d...> Sam> But there are other areas that need tweaking, the poll() overhead Sam> is no long on top of the list. I have modified a python here to do Sam> a type of statistical profiling (every VM insns a Sam> dictionary of {:} is updated). Here's what Sam> made me look at refill_buffer(): Sam> [...] Sam> 4211 Sam> 4970 Have you sent this to Guido? A statistical profiler would be a nice complement to the regular (machine-eating) profiler. If you have a patch, I'd like to take a look at it. (I'd probably change the simple counter to a time accumulator the way gprof works.) >> Another optimization would be to have a system of re-entering the >> select call immediately after returning if nothing is ready--rather >> than checking the results and rebuilding the arguments to the select >> call. Sam> This is a really good idea! it's getting into an interesting area, Sam> of trying to be smarter about scheduling the r/w sets. I think Sam> there are some things like that to be found here: I've experienced cases where is sits for the timeout period (typically 30 seconds), then adds a fd that was in the read set to the write set as well (or was it the other way 'round?), after which the next select returns immediately. Skip Montanaro | http://www.mojam.com/ skip@m... | http://www.musi-cal.com/ 847-971-7098 | Python: Programming the way Guido indented... From rushing@n... Sat Nov 20 13:52:51 1999 From: rushing@n... (Sam Rushing) Date: Sat, 20 Nov 1999 05:52:51 -0800 (PST) Subject: [medusa] Re: tweaks to asyn{core,chat}.py In-Reply-To: <14390.41315.807375.200241@d...> References: <14389.14031.176921.629212@s...> <3.0.5.32.19991119120917.015b83a0@m...> <14390.5644.955061.401898@s...> <14390.41315.807375.200241@d...> Message-ID: <14390.42931.203540.780616@s...> Skip Montanaro writes: > Have you sent this to Guido? A statistical profiler would be a > nice complement to the regular (machine-eating) profiler. If you > have a patch, I'd like to take a look at it. (I'd probably change > the simple counter to a time accumulator the way gprof works.) Warning: This code ignores reference-counting. (not that there's anything wrong with that)... I've only used it once. I think it could be very useful for long-running servers, where 'real' profiling is tricky and expensive. This seems to me to have very low overhead; especially if I "sys.setcheckinterval(1000)" rather than 10. Python-1.5.2/Python/ceval.c: 379a380,384 > #ifdef STATISTICAL_PROFILING > static PyObject * profiling = NULL; > #endif > > 620a626,644 > #ifdef STATISTICAL_PROFILING > if (!profiling) { > profiling = PyDict_New(); > { > PyObject * sys_module = PyDict_GetItemString (PyImport_GetModuleDict(), "sys"); > PyObject * sys_dict = PyModule_GetDict (sys_module); > PyDict_SetItemString (sys_dict, "statistical_profiling_data", profiling); > } > } > { > PyObject * val = PyDict_GetItem (profiling, (PyObject *) co); > if (!val) { > PyDict_SetItem (profiling, (PyObject *) co, PyInt_FromLong (1)); > } else { > PyDict_SetItem (profiling, (PyObject *) co, PyInt_FromLong (PyInt_AsLong (val) + 1)); > } > } > #endif > > I've experienced cases where is sits for the timeout period > (typically 30 seconds), then adds a fd that was in the read set to > the write set as well (or was it the other way 'round?), after > which the next select returns immediately. You're saying there are two trips through poll() when there should be one? Or did a socket not 'fire' when it should have? -Sam From skip@m... Sat Nov 20 14:12:44 1999 From: skip@m... (Skip Montanaro) Date: Sat, 20 Nov 1999 08:12:44 -0600 (CST) Subject: [medusa] Re: tweaks to asyn{core,chat}.py In-Reply-To: <14390.42931.203540.780616@s...> References: <14389.14031.176921.629212@s...> <3.0.5.32.19991119120917.015b83a0@m...> <14390.5644.955061.401898@s...> <14390.41315.807375.200241@d...> <14390.42931.203540.780616@s...> Message-ID: <14390.44124.353420.835913@d...> Thanks for the profiling code. >> I've experienced cases where is sits for the timeout period >> (typically 30 seconds), then adds a fd that was in the read set to >> the write set as well (or was it the other way 'round?), after >> which the next select returns immediately. Sam> You're saying there are two trips through poll() when there should Sam> be one? Or did a socket not 'fire' when it should have? I dunno. I've only experienced it in test situations, where I feed a server log file back to the server as fast as possible, so I've not paid much attention to it. (Also, note that I'm using a somewhat oldish version of ZServer. YMMV.) I imagine you have something similar, but just in case, here's my beat-server.py script. It takes a medusa/ZServer log file on stdin and throws requests at the server given by the -h argument (default, localhost). by default, it runs 5 simultaneous threads but this can be adjusted up or down with the -t flag. Note the wierd time.sleep call at the bottom of the main loop. That was my feeble attempt to keep the server from getting into this catatonic state where it would only wake up and process one request every 30 seconds. Skip --[[application/octet-stream Content-Disposition: attachment; filename="beat-server.py"][quoted-printable]] #!/usr/bin/env python import sys, thread, getopt, string, httplib, urlparse, time, re, random from threading import * def get_page(url, sema, verbose): now =3D time.asctime(time.localtime(time.time())) if verbose =3D=3D 2: sys.stderr.write("%s: %s -> " % (now, url)); sys.stderr.flush() try: t =3D time.time() scheme, loc, path, params, query, frag =3D urlparse.urlparse(url)= pfx =3D scheme + "://" + loc h =3D httplib.HTTP(loc) server =3D string.split(loc, ":")[0] h.putrequest('GET', url[len(pfx):]) h.putheader('Accept', '*/*') h.putheader('Host', server) h.endheaders() reply, msg, hdrs =3D h.getreply() bytes =3D len(h.getfile().read()) t =3D time.time() - t except: if not verbose: sys.stderr.write("%s -> " % (url)); sys.stderr.flush() sys.stderr.write("failed\n"); sys.stderr.flush() time.sleep(2) else: if verbose: sys.stderr.write("%d (%d bytes, %.2f seconds)" % (reply, byte= s, t)) if verbose < 2 and (reply >=3D 400 or t > 10): sys.stderr.write(" %s" % (url)); sys.stderr.flush() sys.stderr.write("\n") sys.stderr.flush() sema.release() def build_url_from_log(loc, pat, f): line =3D f.readline() while line: m =3D pat.search(line) if m: url =3D "http://%s%s" % (loc, m.group(1)) return url line =3D f.readline() return line def main(): optlist, args =3D getopt.getopt(sys.argv[1:], "t:h:qv") maxthreads =3D 5 proxy =3D "" loc =3D "localhost" verbose =3D 1 for opt, arg in optlist: if opt =3D=3D "-t": maxthreads =3D string.atoi(arg) elif opt =3D=3D "-h": loc =3D arg elif opt =3D=3D "-q": verbose =3D 0 elif opt =3D=3D "-v": verbose =3D 2 = sema =3D Semaphore(maxthreads) pathpat =3D re.compile('GET\s([^\s]*)\s') try: while 1: url =3D build_url_from_log(loc, pathpat, sys.stdin) if not url: break sema.acquire() Thread(target=3Dget_page, args=3D(url, sema, verbose)).start(= ) time.sleep(random.uniform(0, 1.3)) except KeyboardInterrupt: return if __name__ =3D=3D "__main__": main() From rushing@n... Sun Nov 21 02:30:43 1999 From: rushing@n... (Sam Rushing) Date: Sat, 20 Nov 1999 18:30:43 -0800 (PST) Subject: [medusa] Re: tweaks to asyn{core,chat}.py In-Reply-To: <14390.44124.353420.835913@d...> References: <14389.14031.176921.629212@s...> <3.0.5.32.19991119120917.015b83a0@m...> <14390.5644.955061.401898@s...> <14390.41315.807375.200241@d...> <14390.42931.203540.780616@s...> <14390.44124.353420.835913@d...> Message-ID: <14391.22867.379464.147090@s...> Skip Montanaro writes: > Sam> You're saying there are two trips through poll() when there should > Sam> be one? Or did a socket not 'fire' when it should have? > > I dunno. I've only experienced it in test situations, where I feed a server > log file back to the server as fast as possible, so I've not paid much > attention to it. (Also, note that I'm using a somewhat oldish version of > ZServer. YMMV.) This could very well be a problem with listen() queue size. This always shows up in testing situations. Most code uses listen(5) as a default (I used to use 5 because I think some winsock stacks would complain if you used a higher value?)... but it should be as high as the greatest number of sockets you can handle. I've been defaulting to 1024 lately, which should be enough for most purposes. > I imagine you have something similar, but just in case, here's my > beat-server.py script. It takes a medusa/ZServer log file on stdin and > throws requests at the server given by the -h argument (default, localhost). > By default, it runs 5 simultaneous threads but this can be adjusted up or > down with the -t flag. Note the wierd time.sleep call at the bottom of the > main loop. That was my feeble attempt to keep the server from getting into > this catatonic state where it would only wake up and process one request > every 30 seconds. I've just grepped through Zope-2.0.1/ZServer, and it looks like the http server is using listen(1024), but I'd double-check your distribution. 309: self.listen(256) ./PCGIServer.py 537: self.listen (1024) ./medusa/http_server.py -Sam From sjz18@y... Wed Nov 24 04:34:25 1999 From: sjz18@y... (Stuart Zakon) Date: Tue, 23 Nov 1999 20:34:25 -0800 Subject: [medusa] any performance info? Message-ID: <81fpsh$7pka@e...> Hi, I am new to Medusa and have been intrigued by the difference in the asyncore model from the multi-thread model I've worked with for 10 years. From rushing@n... Wed Nov 24 05:11:07 1999 From: rushing@n... (Sam Rushing) Date: Tue, 23 Nov 1999 21:11:07 -0800 (PST) Subject: [medusa] any performance info? In-Reply-To: <81fpsh$7pka@e...> References: <81fpsh$7pka@e...> Message-ID: <14395.29547.740497.50771@s...> Stuart Zakon writes: > Hi, I am new to Medusa and have been intrigued by the difference in > the asyncore model from the multi-thread model I've worked with for > 10 years. From what I understand the model works well when the > units of work can be broken into relatively uniform durations. > Is the unit of work an HTTP exchange? The base 'unit' might be considered the http request, at least that's the way the http server in the distribution is designed. However, the actual I/O is done 'on demand'... a threaded perspective might say that the I/O is handled in its own thread[s]. Some systems based on Medusa take different approaches - for example Zope uses a thread pool to handle requests, but lets medusa handle I/O asynchronously. This frees up the precious resource of threads to do actual work, and lets the back end handle all those slow clients. > Are there performance comparisons of Medusa's asyncore model with web > servers using multi-threaded models? Under heavy load? The very best web servers (which in general provide performance waaaay over what the average user needs) combine the two approaches; for example using one thread per CPU, each doing event-driven i/o. Here are a couple of good references: http://www.acme.com/software/thttpd/benchmarks.html http://www.kegel.com/c10k.html Medusa can't handle the kind of loads that thttpd and Zeus can - for example, the file delivering code uses stdio, and... well, it *is* written in Python. 8^) But most people don't need 1000 hits/sec. [let's see.. that's 86 million hits/day?] One big problem with the unix select/poll model is that it *does* get inefficient once the # of connections gets large... it would be nice if unix were to add something like completion ports. I think there are folks adding an equivalent facility to Linux 2.3 [of course, it has to be completely different, how embarrassing to copy something from NT. 8^)] > Also, I heard that egroups uses the asyncore library to achieve > scalability. Is this so? Does egroups use Medusa? We have several scalable systems using Medusa for various things - usually we have a custom system built on top of asyn{core,chat} which uses the rest of medusa for a web interface. Probably the biggest 'show-off' piece is our mailing-list exploder, which juggles up to 10,000 SMTP client connections simultaneously. [actually, that's not the maximum it's capable of, we're just not fond of pushing the OS to the breaking point. 8^) The servers are configured with an FD_SETSIZE of 16K] These servers are capable of pushing 6-8 million msgs/day. We also have customized RPC servers, proxies, smtp servers, and other things based on the async lib. eGroups does not use Medusa for the front-end, though - all of the UI code is written in straightforward Python, and is not amenable to the async model [all our UI programmers would have to know how to do nasty callbacks, state machines, etc... to pull this off] Lately we have been migrating some of our back-end systems toward a coroutine-based solution. This combines the scalability/efficiency of async I/O with the ability to program straight-line code. It feels a lot like a multi-threaded system, but without the need to use locks. [some of our engineers call it 'co-operative multitasking'] We're in the process of releasing this code to the public. -Sam From sjz18@y... Fri Nov 26 17:08:47 1999 From: sjz18@y... (Stuart Zakon) Date: Fri, 26 Nov 1999 09:08:47 -0800 Subject: [medusa] Re: any performance info? In-Reply-To: <14395.29547.740497.50771@s...> Message-ID: <81meqv$5etq@e...> Sam, Thanks for the performance info. I've also pulled down some of the research papers (Proactor, Reactor patterns) which have a lot of the background. *So I don't have to dig down too deep myself, how do thttpd and Zeus achieve greater performance if they use an asynch scheme? Is it because they are closer to the kernel? *Has anybody done anything with asynch for reverse DNS lookups? *Is your overview "Programming in Python with Medusa and the Async Sockets Library", which I found very useful, directly accessible on a web site without having to open the distribution? (Would be helpful for linking to from other pages.) *Does your mailing list exploder use the poll() approach? Thanks, Stuart From rushing@n... Fri Nov 26 21:00:53 1999 From: rushing@n... (Sam Rushing) Date: Fri, 26 Nov 1999 13:00:53 -0800 (PST) Subject: [medusa] Re: any performance info? In-Reply-To: <81meqv$5etq@e...> References: <14395.29547.740497.50771@s...> <81meqv$5etq@e...> Message-ID: <14398.62725.578358.703796@s...> Stuart Zakon writes: > *So I don't have to dig down too deep myself, how do thttpd and > Zeus achieve greater performance if they use an asynch scheme? Is > it because they are closer to the kernel? The async approach allows them to mix I/O with computation. Serving static pages is one extreme... there's no computation at all, just network I/O. And it all takes place in 'parallel': making 30 non-blocking connect() calls is the equivalent of saying "connect these 30 sockets to these 30 hosts, and let me know when they get through". > *Has anybody done anything with asynch for reverse DNS lookups? I think Google uses async DNS lookups in their crawler. Newman (the eGroups mailing-list exploder) does as well. > *Is your overview "Programming in Python with Medusa and the Async > Sockets Library", which I found very useful, directly accessible on a > web site without having to open the distribution? (Would be helpful for > linking to from other pages.) This will work: http://www.nightmare.com:8080/nm/apps/medusa/docs/programming.html I think a few other people are linking to it in that location as well. > *Does your mailing list exploder use the poll() approach? Haven't bothered, yet. -Sam From jess@s... Mon Nov 29 22:55:15 1999 From: jess@s... (jess@s...) Date: Mon, 29 Nov 1999 14:55:15 -0800 Subject: [medusa] please help me Message-ID: <81v08j$tsfl@e...> I am doing a 15 minute presentation for my Operating Systems class on Medusa. I have been searching and searching and searching, but there is very little information available. My presentation is this Friday, hence why i am stressing out at the last minute! Can anyone help me? I have found some helpful websites, (such as www.nightmare.com/medusa) but the information i need for this presentation is: name company or organization(s) who created it major people involved when was it developed structure process control and scheduling memory management IO networking user interfaces (GUI, command line) why is this os important or different I have the name, company, and the major people involved. i have some other information on why this OS is better/different than others. What i really need is the structure, networking, and user interface. Anything anyone can help me with would be very useful. My email address is jess@s... Thank you in advance! Jessica From jam@n... Wed Dec 15 20:22:53 1999 From: jam@n... (Jonez) Date: Wed, 15 Dec 1999 12:22:53 -0800 Subject: [medusa] No Subject Message-ID: <838tat$b918@e...> greetings, as an FYI, the package that comes with redhat 6.1 has a 'quite' old version of asyncore, based on the CVS revision information. specifically, the 'log_info' method appears to be missing from the version that is installed by redhat. I solved the problem by copying the file from the 1999-09-02 distribution of medusa. how does one obtain access to the CVS version of medusa? I am interested in getting whatever 'tweaks' might be available. From rushing@n... Wed Dec 15 20:55:42 1999 From: rushing@n... (Sam Rushing) Date: Wed, 15 Dec 1999 12:55:42 -0800 (PST) Subject: [medusa] No Subject In-Reply-To: <838tat$b918@e...> References: <838tat$b918@e...> Message-ID: <14424.78.879406.394872@s...> From rushing@n... Wed Dec 15 21:00:40 1999 From: rushing@n... (Sam Rushing) Date: Wed, 15 Dec 1999 13:00:40 -0800 (PST) Subject: [medusa] No Subject In-Reply-To: <838tat$b918@e...> References: <838tat$b918@e...> Message-ID: <14424.376.392459.539029@s...> Jonez writes: > greetings, > > as an FYI, the package that comes with redhat 6.1 has a 'quite' old > version of asyncore, based on the CVS revision information. > specifically, the 'log_info' method appears to be missing from the > version that is installed by redhat. I solved the problem by > copying the file from the 1999-09-02 distribution of medusa. Yup, the core has been getting a few tweaks over the past year. The latest is a reworking of the way the socket_map dictionary works in order to improve performance with large numbers of sockets (>1000). I should probably get it checked in, a few minor changes still remaining. > how does one obtain access to the CVS version of medusa? I am > interested in getting whatever 'tweaks' might be available. This should do the trick on unix: bash$ export CVSROOT=:pserver:medusa@s...:/usr/local/cvsroot bash$ cvs login # password=='medusa' bash$ cvs checkout medusa [or] bash$ cvs checkout medusa/asyncore.py -Sam From jam@q... Wed Dec 15 21:23:25 1999 From: jam@q... (Jeff) Date: Wed, 15 Dec 1999 16:23:25 -0500 Subject: [medusa] Re: cvs access In-Reply-To: <14424.376.392459.539029@s...> References: <838tat$b918@e...> <14424.376.392459.539029@s...> Message-ID: <19991215162325.B1213@q...> On Wed, Dec 15, 1999 at 01:00:40PM -0800, Sam Rushing wrote: > This should do the trick on unix: > > bash$ export CVSROOT=:pserver:medusa@s...:/usr/local/cvsroot > bash$ cvs login > # password=='medusa' > bash$ cvs checkout medusa > [or] > bash$ cvs checkout medusa/asyncore.py > > -Sam > thanks.. you rock ;) regards, J -- || visit gfd || psa member -- || New Image Systems & Services, Inc. From zakons@e... Thu Dec 16 15:55:24 1999 From: zakons@e... (Stuart Zakon) Date: Thu, 16 Dec 1999 07:55:24 -0800 Subject: [medusa] IPC8? Message-ID: <83b21c$joda@e...> Will there be any Medusa related presentations at IPC8? BOF sessions? Stuart Zakon From rushing@n... Thu Jan 14 12:08:14 1999 From: rushing@n... (Samual M. Rushing) Date: 14 Jan 1999 12:08:14 -0000 Subject: [medusa] new code for CGI-ish thread pools Message-ID: <19990114120814.26127.qmail@g...> [ok, I'll make the first post] See the 'thread' subdirectory, you will find: thread_channel.py a good experiment, using pipes to bridge the sync/async gap. somewhat impractical because it wastes file descriptors select_trigger.py a special channel specifically designed for waking up the main select loop with output from other threads. a nice side-effect of this technique is that it uses a socket as a synchronization object; obviating the need for locks around medusa's i/o data structures. thread_handler.py a module for building cgi-like systems that use a pool of threads. uses select_trigger to create file-like objects that hide all the magic from you. -Sam p.s. a reminder, to access the read-only CVS repository: CVSROOT = :pserver:medusa@s...:/usr/local/cvsroot the password is 'medusa'. ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From rushing@n... Thu Jan 14 13:06:24 1999 From: rushing@n... (Samual M. Rushing) Date: 14 Jan 1999 13:06:24 -0000 Subject: [medusa] addition to the thread_handler demo Message-ID: <19990114130624.26323.qmail@g...> I've added a second test module, 'pi_module', that will compute a specified number of digits of pi. You can use it to test out the thread pool by taking up all the threads. If you have a request waiting for a thread, it will start up as soon as one of the others finishes. -Sam ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From rushing@n... Thu Jan 14 19:56:08 1999 From: rushing@n... (Samual M. Rushing) Date: 14 Jan 1999 19:56:08 -0000 Subject: [medusa] renaming of classes & methods Message-ID: <19990114195608.26512.qmail@g...> The names of the various classes, methods, variables, etc in the core library are mostly the result of accident; they were given quickly, and usually before their purpose was fully understood. For example, the 'dispatcher' class is hardly a dispatcher. If anything, it's a 'dispatchee'. Kinda screaming for a better name. I'm sure this doesn't help in trying to understand the code. I've thought for a while that the whole mess could use a renaming; if ever there were a time to do this it seems that *before* it went into the Python library would be it. Just wondering what others were thinking, along with any suggestions for more meaningful names, candidates for most-in-need-of-a-new-name, etc... -Sam ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From rushing@n... Sat Jan 16 16:19:21 1999 From: rushing@n... (Samual M. Rushing) Date: 16 Jan 1999 16:19:21 -0000 Subject: [medusa] rough draft of tutorial Message-ID: <19990116161921.16305.qmail@g...> I've filled in more of the tutorial. http://www.nightmare.com:8080/nm/apps/medusa/docs/programming.html Suggestions are welcome. The Reference section will come next. -Sam ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From rushing@n... Tue Jan 19 13:08:55 1999 From: rushing@n... (Samual M. Rushing) Date: 19 Jan 1999 13:08:55 -0000 Subject: [medusa] asyncore.py:compact_traceback(), handle_error() interfaces redesigned. Message-ID: <19990119130855.18284.qmail@g...> There is a warning about sys.exc_info() in the Python documentation. If only I'd thought to look there a long time ago, I would have saved myself much chasing of memory leaks. The basic problem is this: if you refer to a traceback from within an exception handler, you will automatically create a cycle (and thus leak every object in the entire stack frame) I've recently been peppering my code with try/finally/del sequences to avoid this, but a much easier solution is to move the call to sys.exc_info() out of the exception handler. It seems that in the old days (with sys.exc_type, sys.exc_value, sys.exc_traceback) you couldn't do this, which is why I've always passed the values around. This impacts not only asyncore.compact_traceback(), but also the handle_error() interface; which no longer takes any arguments. The necessary changes are fairly obvious, grep through the code for examples. -Sam ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From florent.heyworth@r... Fri Jan 22 01:47:27 1999 From: florent.heyworth@r... (florent.heyworth@r...) Date: Fri, 22 Jan 1999 01:47:27 -0000 Subject: [medusa] rough draft of tutorial Looks good. Looking forward to the reference section 8-) Cheers Florent Heyworth (keep up the good work) ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From carlosm@c... Wed Jan 27 18:28:47 1999 From: carlosm@c... (carlosm@c...) Date: Wed, 27 Jan 1999 18:28:47 -0000 Subject: [medusa] Medusa as web proxy server? Has anyone done any work on modifying Medusa so it can be used as a web proxy server? Thanks, Carlos ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From florent.heyworth@r... Thu Jan 21 08:02:15 1999 From: florent.heyworth@r... (florent.heyworth@r...) Date: Thu, 21 Jan 1999 08:02:15 -0000 Subject: [medusa] Re: rough draft of tutorial In-Reply-To: <19990116161921.16305.qmail@g...> <19990116161921.16305.qmai-@g...> wrote: Original Article: http://www.egroups.com/list/medusa/?start=4 > I've filled in more of the tutorial. > Looks good. Looking forward to seeing the reference section 8-) Cheers Florent Heyworth ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From florent.heyworth@r... Thu Jan 28 00:38:13 1999 From: florent.heyworth@r... (florent.heyworth@r...) Date: Thu, 28 Jan 1999 00:38:13 -0000 Subject: [medusa] Re: Medusa as web proxy server? Original Article: http://www.egroups.com/list/medusa/?start=7 > Has anyone done any work on modifying Medusa so it can be used as > a web proxy server? > > Thanks, > Carlos > Hi Carlos Sam Rushing has an example proxy server implementation in his Medusa tutorial. Check it out under: http://www.nightmare.com:8080/nm/apps/medusa/docs/programming.html Cheers Florent ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From robin@j... Thu Jan 28 05:45:23 1999 From: robin@j... (Robin Becker) Date: Thu, 28 Jan 1999 05:45:23 +0000 Subject: [medusa] How do I stop medusa gracefully Message-ID: <076c5GAzl$r2Ews4@j...> I'm just starting to play with medusa at home (Win95 OSR2) and find it does strange things to my dialup connection sometimes. I've just been killing the process; is there a better way? -- Robin Becker ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From florent.heyworth@r... Thu Jan 28 10:03:13 1999 From: florent.heyworth@r... (florent.heyworth@r...) Date: Thu, 28 Jan 1999 10:03:13 -0000 Subject: [medusa] Re: How do I stop medusa gracefully In-Reply-To: <076c5GAzl$r2Ews4@j...> <076c5gazl$r2ews-@j...> wrote: Original Article: http://www.egroups.com/list/medusa/?start=10 > I'm just starting to play with medusa at home (Win95 OSR2) and find it > does strange things to my dialup connection sometimes. I've just been > killing the process; is there a better way? > -- > Robin Becker > Hi I used to develop with Medusa on a Win95 machine. As the machine was a standalone it also had a dial-up connection to the Internet and never noticed any side effects. I'm now on NT and I've never noticed any side effects either. Could you be more specific as to what problems you've been encountering? Cheers Florent Heyworth ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From rushing@n... Thu Jan 28 18:21:54 1999 From: rushing@n... (rushing@n...) Date: Thu, 28 Jan 1999 10:21:54 -0800 (PST) Subject: [medusa] Re: How do I stop medusa gracefully In-Reply-To: References: <076c5GAzl$r2Ews4@j...> Message-ID: <14000.43652.959726.676173@g...> Robin Becker writes: > Occasionally my dialup works, but no connections can be established. I > suspected the pseudo DNS server, but can't think of any way it could do > this. When this has happened I later found that medusa was also a bit > strange; it would allow FTP, but not HTTP; a bit weird. by default start_medusa.py creates a DNS resolver pointing at the localhost. On a dialup connection you should change the '127.0.0.1' to point to your ISP's resolver. You can also just disable the resolving feature (you'll get IP numbers in your log). It should be possible to change the line rs = resolver.caching_resolver ('127.0.0.1') to read rs = None This affects the http and ftp servers, and is only used for logging, not for authentication. -Sam ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From florent.heyworth@r... Thu Jan 28 22:12:40 1999 From: florent.heyworth@r... (florent.heyworth@r...) Date: Thu, 28 Jan 1999 22:12:40 -0000 Subject: [medusa] Re: How do I stop medusa gracefully In-Reply-To: wrote: Original Article: http://www.egroups.com/list/medusa/?start=14 > thanks I tried this and it seemed to work fine; but what is the > preferred method of shutting medusa down? > -- > Robin Becker > If you're working with medusa by running it from Python.exe a Ctrl-C (Keyboard Interrupt) will attempt to shut down "gracefully" after a time out on the select loop. A more graceful, and fun, way of doing is by logging in to Medusa's monitor client. On Win32 you can call it as python monitor_client_win32.py [hostname] [port] on my machine I do: python monitor_client_win32.py 127.0.0.1 9999 at the password prompt enter the password (default is 'fnord') you can change it in start_medusa.py. on the interactive prompt enter: >>> import monitor >>> monitor.asyncore.close_all() That's it Cheers Florent ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From rushing@n... Fri Jan 29 02:57:51 1999 From: rushing@n... (rushing@n...) Date: Thu, 28 Jan 1999 18:57:51 -0800 (PST) Subject: [medusa] Re: How do I stop medusa gracefully In-Reply-To: References: Message-ID: <14001.8921.411033.535751@g...> Robin Becker writes: > I guessed close_all must be it; it doesn't seem to worry about timing > order of close at all though A more graceful close might shut down the servers only, then when the final client has exited (and thus there are no more sockets in the socket map) the loop should exit cleanly. This should also allow you to immediately restart a new set of servers, since the server ports will be available. -Sam ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From rushing@n... Sat Jan 30 01:43:22 1999 From: rushing@n... (rushing@n...) Date: Fri, 29 Jan 1999 17:43:22 -0800 (PST) Subject: [medusa] more uniform error handling in asyncore.dispatcher() ? Message-ID: <14002.25179.442430.872334@g...> Craig Silverstein has noticed that there's a discrepancy in the handling of errors in async_chat: calls to self.recv() are protected by a try/except clause, but calls to self.send() are not. While looking into this, I noticed that a possibly cleaner solution is to have the send() and recv() methods in asyncore.dispatcher call self.handle_error() directly, instead of re-raising unexpected errors that they see. I think this would relieve the responsibility to try/except from any derived class. Does anyone see a problem with this, or has anyone tried it already? -Sam -- rushing@n... rushing@e... ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From robin@j... Sat Jan 30 08:06:47 1999 From: robin@j... (Robin Becker) Date: Sat, 30 Jan 1999 08:06:47 +0000 Subject: [medusa] FTP permissions Message-ID: whilst playing with medusa's ftp server I noticed that even though I wasn't allowed to use the STOR command I was allowed to create remove/directories. Is this normal for FTP servers? -- Robin Becker ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From rushing@n... Sat Jan 30 20:28:37 1999 From: rushing@n... (rushing@n...) Date: Sat, 30 Jan 1999 12:28:37 -0800 (PST) Subject: [medusa] FTP permissions In-Reply-To: References: Message-ID: <14003.27219.235542.330350@g...> Robin Becker writes: > whilst playing with medusa's ftp server I noticed that even though I > wasn't allowed to use the STOR command I was allowed to create > remove/directories. Is this normal for FTP servers? No, this is major oops. There is a list of 'write' commands, and a comment to the affect that the list is checked against, but unfortunately the code that should reference it doesn't. It's a one-line fix, and I just checked it in: def check_command_authorization (self, command): if command in self.write_commands and self.read_only: return 0 else: return 1 Many thanks for pointing this out, Robin! -Sam ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From rushing@n... Sat Jan 30 21:32:13 1999 From: rushing@n... (rushing@n...) Date: Sat, 30 Jan 1999 13:32:13 -0800 (PST) Subject: [medusa] more uniform error handling in asyncore.dispatcher() ? In-Reply-To: <14002.25179.442430.872334@g...> References: <14002.25179.442430.872334@g...> Message-ID: <14003.31056.620896.457638@g...> rushing@n... writes: > While looking into this, I noticed that a possibly cleaner solution is > to have the send() and recv() methods in asyncore.dispatcher call > self.handle_error() directly, instead of re-raising unexpected errors > that they see. I think this would relieve the responsibility to > try/except from any derived class. On second thought, I don't think this is a good idea: it throws off the ability to exit correctly from a sequence of code: def method (self): n = self.send (data) [...other code...] If handle_error() is called automatically from send(), then there's no way for [...other code...] to know about the error condition so that it might exit the method (or choose a different branch of an if, etc...) -Sam ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From robin@j... Mon Feb 1 14:00:07 1999 From: robin@j... (Robin Becker) Date: Mon, 1 Feb 1999 14:00:07 +0000 Subject: [medusa] http error? Message-ID: I get the following error when requesting the same file repeatedly using a torture test. I don't get this error when re-loading as rapidly as possible with netscape. uncaptured python exception, closing channel (exceptions.AttributeError:'int' object has no attribute 'send' [c:\python\medusa\asyncore.py|poll|69] [c:\python\medusa\asyncore.py|handle_write_event|297] [c:\python\medusa\asynchat.py|handle_write|133] [c:\python\medusa\asynchat.py|initiate_send|185] [c:\python\medusa\http_server.py|send|351] [c:\python\medusa\asyncore.py|send|235] [|send|1]) -- Robin Becker ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From amos@a... Mon Feb 1 23:04:21 1999 From: amos@a... (Amos Latteier) Date: Mon, 01 Feb 1999 15:04:21 -0800 Subject: [medusa] FTP Limits Message-ID: <3.0.5.32.19990201150421.00b19c70@m...> One thing that's been discussed lately on the Zope list is the need for FTP limits to prevent denial of service attacks. The idea is that the server could be flooded with connections until it could no longer accept new legitimate connections. Right now it seems that Medusa's FTP server supports an authorizer object, but there do not seem to be any pre-built limit checking authorizers. I am looking for ideas on what sort of limits should be set, and how they could be enforced. It was suggested that limits could be set for different classes of users, for example, 'anonymous' users. This is slightly difficult in Zope's case, since authentication information is distributed, so the validity of a login cannot be verified without a directory in which to verify it. Thanks in advance for any pointers or experiences designing and implementing FTP limits. -Amos P.S. For those of you not familiar with Zope, it is an Open Source web application server which is using Medusa for HTTP and FTP services. http://www.zope.org/ ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From rushing@n... Mon Feb 8 22:24:02 1999 From: rushing@n... (rushing@n...) Date: Mon, 8 Feb 1999 14:24:02 -0800 (PST) Subject: [medusa] warning: server accept() threw an exception In-Reply-To: <88256712.00605040.00@b...> References: <88256712.00605040.00@b...> Message-ID: <14015.25153.666732.707340@g...> ray@u... writes: > Any idea what causes this this error to occur dozens of times in a row? > When it happens, the server will stop serving process and only print this > message.. sometimes hundreds of times and stopping http access for 2-30 > seconds. I have seen this sort of thing usually when bumping up against a file-descriptor limit. You may be able to verify this by looking at the number of open descriptors at the time the error is being thrown (sometimes you can get this from a /proc filesystem, otherwise you could add a bit of code to asyncore.loop() to see how many descriptors are being fed to select()). > I have noticed that the problem may be load related; it often begins when > the server is processing over 3k hits per hour, but does not incease in > frequency above that. But, it also occurs when hits are below 2k/hour. You might still be hitting descriptor limits under a lighter load if too many connections or files are staying open... > Using Medusa server (v3.19) serving only Bobo applications via the > bobo_handler (v1.03) on Solaris 2.6 Note also that if bobo is opening many files this will also take away from your budget. I've been told that Solaris is one of the better OS' for dynamically configuring around such limits; but I don't currently have a copy running to look into it myself. -Sam ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From ray@u... Wed Feb 24 00:05:12 1999 From: ray@u... (ray@u...) Date: Tue, 23 Feb 1999 16:05:12 -0800 Subject: [medusa] how to turn off dns resolving Message-ID: <88256722.000036D8.00@b...> We only want IP address in the logs. Thanks for you help. Ray ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From jam@n... Wed Feb 24 02:33:54 1999 From: jam@n... (Jonez) Date: Wed, 24 Feb 1999 02:33:54 +0000 Subject: [medusa] Re: how to turn off dns resolving References: <88256722.000036D8.00@b...> Message-ID: <36D36512.FB69E179@n...> ray@u... wrote: > > We only want IP address in the logs. > > Thanks for you help. > > Ray > greetings, it may take some small modifications to your application, but there is an 'unresolving_logger' object in logger.py.. you would probably need to derive from it to get the functionality you want, but it should be straight forward. perhaps you could post your startup script, and we can work on it from there? that will show how the logger object is being instantiated, and the modifications necessary will be clearer. hope that helps. regards, J ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From rushing@n... Wed Feb 24 03:13:12 1999 From: rushing@n... (rushing@n...) Date: Tue, 23 Feb 1999 19:13:12 -0800 (PST) Subject: [medusa] how to turn off dns resolving In-Reply-To: <88256722.000036D8.00@b...> References: <88256722.000036D8.00@b...> Message-ID: <14035.28151.461814.609649@g...> ray@u... writes: > > We only want IP address in the logs. The easiest way should be to change rs = resolver.caching_resolver ('127.0.0.1') to rs = None And then be sure to comment out the 'rs' in the list of status objects: status_objects = [ hs, ftp, ms, cs, # rs, lg ] -Sam ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From petrilli@a... Wed Feb 24 18:45:46 1999 From: petrilli@a... (Christopher G. Petrilli) Date: Wed, 24 Feb 1999 13:45:46 -0500 Subject: [medusa] asyncore.dispatcher methods Message-ID: <19990224134546.63759@a...> So can someone explain what the difference is between: handle_expt and handle_oob It seems only handle_expt is actually used, but maybe I missed something? THis is in asyncore.dispather Thanks! Chris -- | Christopher Petrilli ``Television is bubble-gum for | petrilli@a... the mind.''-Frank Lloyd Wright ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From rushing@n... Mon Mar 1 22:09:49 1999 From: rushing@n... (rushing@n...) Date: Mon, 1 Mar 1999 14:09:49 -0800 (PST) Subject: [medusa] asyncore.dispatcher methods In-Reply-To: <19990224134546.63759@a...> References: <19990224134546.63759@a...> Message-ID: <14043.4022.980510.779002@g...> Christopher G. Petrilli writes: > So can someone explain what the difference is between: > > handle_expt > > and > > handle_oob > > It seems only handle_expt is actually used, but maybe I missed > something? THis is in asyncore.dispather You're right. I think this is old code, probably there from the early days when I supported WSAAsyncSelect() on Win32 (which I think had an FD_OOB message). I'll take it out right now, thanks for the info! -Sam ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From amos@a... Tue Mar 16 19:12:41 1999 From: amos@a... (Amos Latteier) Date: Tue, 16 Mar 1999 11:12:41 -0800 Subject: [medusa] select_trigger.trigger requires posix Message-ID: <3.0.5.32.19990316111241.011b94f0@m...> I just noticed that the latest version of select_trigger.trigger inherits from asyncore.file_dispatcher, and thus is only available under posix. However, I would like to use select_trigger.trigger under NT, as well. Is there any way to allow select_trigger.trigger use under NT? Thanks. -Amos ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From rushing@n... Tue Mar 16 23:25:43 1999 From: rushing@n... (rushing@n...) Date: Tue, 16 Mar 1999 15:25:43 -0800 (PST) Subject: [medusa] select_trigger.trigger requires posix In-Reply-To: <3.0.5.32.19990316111241.011b94f0@m...> References: <3.0.5.32.19990316111241.011b94f0@m...> Message-ID: <14062.59402.933828.877673@s...> Amos Latteier writes: > I just noticed that the latest version of select_trigger.trigger inherits > from asyncore.file_dispatcher, and thus is only available under posix. > However, I would like to use select_trigger.trigger under NT, as well. > > Is there any way to allow select_trigger.trigger use under NT? This was an oversight... select_trigger should be able to use a loopback socket on win32. I will get this to you today. -Sam ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From ray@u... Fri Apr 2 18:34:28 1999 From: ray@u... (ray@u...) Date: Fri, 02 Apr 1999 18:34:28 -0000 Subject: [medusa] Changing logger.file_logger output How would I trim down the ipaddress in the log to eliminate the numbers after the ':'? Currently the first column in each row has the ipaddress formatted like this: 206.58.165.2:1206 and we would like it to be entered like this: 206.58.165.2 The curent format of the log files is causing our log analysis software to choke on the ipaddress calculations. Ray Price ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From rushing@n... Fri Apr 2 18:42:09 1999 From: rushing@n... (rushing@n...) Date: Fri, 2 Apr 1999 10:42:09 -0800 (PST) Subject: [medusa] Changing logger.file_logger output In-Reply-To: <199904021834.NAA03733@s...> References: <199904021834.NAA03733@s...> Message-ID: <14085.3886.43829.133624@s...> ray@u... writes: > How would I trim down the ipaddress in the log to eliminate the numbers after the ':'? > Currently the first column in each row has the ipaddress formatted like this: > 206.58.165.2:1206 > and we would like it to be entered like this: > 206.58.165.2 > > The curent format of the log files is causing our log analysis software to choke on the ipaddress calculations. This should do the trick: def log (self, bytes): self.channel.server.logger.log ( self.channel.addr[0], '%d - - [%s] "%s" %d %d\n' % ( self.channel.addr[1], self.log_date_string (time.time()), self.request, self.reply_code, bytes ) ) ==> def log (self, bytes): self.channel.server.logger.log ( self.channel.addr[0], ' - - [%s] "%s" %d %d\n' % ( self.log_date_string (time.time()), self.request, self.reply_code, bytes ) ) -Sam ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/list/medusa Free Web-based e-mail groups by eGroups.com From skip@c... Thu Apr 22 00:57:11 1999 From: skip@c... (skip@c...) Date: Wed, 21 Apr 1999 23:57:11 -0000 Subject: [medusa] Re: How do I stop medusa gracefully In-Reply-To: <14001.8921.411033.535751@g...> Message-ID: I'm just starting to use Medusa via the ZServer stuff with Zope. I'd like to be able to identify and kill my servers by process id. Accordingly, my ZServer/start.py looks like open(PID_FILE, "w").write("%d\n" % os.getpid()) ... yada yada yada ... asyncore.loop() try: os.unlink(PID_FILE) except os.error: pass The only problem is when I kill the process, asyncore.loop appears not to return in the usual fashion (calls sys.exit somewhere I suppose). I wind up with a lot of pid files laying about (for various reasons they currently have the pid in the file name). Is there some way I can get asyncore.loop to return in the usual manner? Thanks, Skip Montanaro ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/group/medusa Free Web-based e-mail groups by eGroups.com From rushing@n... Fri Apr 23 03:20:33 1999 From: rushing@n... (rushing@n...) Date: Thu, 22 Apr 1999 19:20:33 -0700 (PDT) Subject: [medusa] Re: How do I stop medusa gracefully In-Reply-To: References: <14001.8921.411033.535751@g...> Message-ID: <14111.55211.647805.600070@s...> skip@c... writes: > The only problem is when I kill the process, asyncore.loop appears > not to return in the usual fashion (calls sys.exit somewhere I > suppose). I wind up with a lot of pid files laying about (for > various reasons they currently have the pid in the file name). > Is there some way I can get asyncore.loop to return in the usual manner? It seems to me the only 'normal' way to avoid getting to your unlink would be if asyncore.loop() was exited via an exception... it may be that Zope is using an exception to signal exit in some way. If that's the case, you *should* be able to work around it by using a try/finally clause... -Sam ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/group/medusa Free Web-based e-mail groups by eGroups.com From pwelliso@i... Tue Apr 27 19:10:30 1999 From: pwelliso@i... (pwelliso@i...) Date: Tue, 27 Apr 1999 18:10:30 -0000 Subject: [medusa] HTTPS available ? Message-ID: Is there a https module available ? One that makes calls to SSLeay would be great ? I would be willing to help debug/test/write one if there is not one available. Peter ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/group/medusa Free Web-based e-mail groups by eGroups.com From rushing@n... Wed Apr 28 00:12:25 1999 From: rushing@n... (rushing@n...) Date: Tue, 27 Apr 1999 16:12:25 -0700 (PDT) Subject: [medusa] HTTPS available ? In-Reply-To: References: Message-ID: <14118.17195.189469.46089@s...> pwelliso@i... writes: > Is there a https module available ? > One that makes calls to SSLeay would be great ? I haven't had time to look into this seriously. If SSLeay doesn't support non-blocking i/o, then it would be difficult. One way it might work would be to have medusa's select() loop handle the descriptors for sockets that have SSL code wrapped around them. I.e., medusa sees that a socket is ready for reading, and calls the SSL version of recv() to get the data. We may have to add another layer in Python that handles the case of this recv() failing (say, because the SSL layer needs the rest of a packet of some kind). > I would be willing to help debug/test/write one if there is not one > available. That would be great! I'll help in any way I can. -Sam ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/group/medusa Free Web-based e-mail groups by eGroups.com From amos@a... Thu May 6 17:48:58 1999 From: amos@a... (Amos Latteier) Date: Thu, 06 May 1999 09:48:58 -0700 Subject: [medusa] ftp server on win95 Message-ID: <3.0.5.32.19990506094858.0121a050@m...> Hello, I've gotten several reports that Medusa's ftp server fails to run under win95 when not attached to a network. Here's a sample traceback: log: adding channel Traceback (innermost last): File "D:\WebSite\ZServer\start.py", line 96, in ? from FTPServer import FTPServer File "D:\WebSite\ZServer\FTPServer.py", line 140, in ? from medusa.ftp server import ftp_channel, ftp_server, recv_channel File "D:\WebSite\ZServer\medusa\ftp_server.py", line 58, in ? HOSTNAME = socket.gethostbyaddr (IP_ADDRESS)[0] socket.error: host not found (Note this user is using ZServer.) Apparently IP_ADDRESS is set to "127.0.0.1" and they have an entry for this IP in their hosts file. Any suggestions on how to keep socket.gethostbyaddr from failing? Alternately, any suggestions on avoiding this call to begin with--perhaps we could avoid lines 57, and 58 of ftp_server.py, since they seem to really be useful to calculate the HOSTNAME, which can be passed to the ftp_server upon initialization... Thanks. -Amos ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/group/medusa http://www.eGroups.com - Simplifying group communications From rushing@n... Fri May 7 01:54:13 1999 From: rushing@n... (rushing@n...) Date: Thu, 6 May 1999 17:54:13 -0700 (PDT) Subject: [medusa] ftp server on win95 In-Reply-To: <3.0.5.32.19990506094858.0121a050@m...> References: <3.0.5.32.19990506094858.0121a050@m...> Message-ID: <14130.14503.627295.836062@s...> Amos Latteier writes: > Any suggestions on how to keep socket.gethostbyaddr from failing? > Alternately, any suggestions on avoiding this call to begin > with--perhaps we could avoid lines 57, and 58 of ftp_server.py, > since they seem to really be useful to calculate the HOSTNAME, > which can be passed to the ftp_server upon initialization... The IP_ADDRESS setting used to be in there for figuring out what address to bind the PASV port to. That code was changed a while ago to use getsockname() instead, so it's not needed any more. I removed the IP_ADDRESS and HOSTNAME globals, and now defaults its hostname to socket.gethostname(), which shouldn't fail in any case - but even if it does it can be avoided by passing in the hostname parameter. There's actually no reason to even have the hostname (other than reporting status), but I'm loath to remove it completely... -Sam ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/group/medusa http://www.eGroups.com - Simplifying group communications From rushing@n... Mon May 10 18:28:12 1999 From: rushing@n... (rushing@n...) Date: Mon, 10 May 1999 10:28:12 -0700 (PDT) Subject: [medusa] cvs pserver back up Message-ID: <14135.5683.133110.390488@s...> seattle.nightmare.com suffered a Nasty Disk Failure on Friday. Nothing was lost, but I've had to build a whole new system over the past couple of days. I just brought the cvs read-only pserver back online, if anyone has any trouble with it please let me know. As a reminder: CVSROOT=:pserver:medusa@s...:/usr/local/cvsroot The password is 'medusa'. Thanks, -Sam ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/group/medusa http://www.eGroups.com - Simplifying group communications From ray@u... Wed May 12 18:54:00 1999 From: ray@u... (ray@u...) Date: Wed, 12 May 1999 17:54:00 -0000 Subject: [medusa] using auth_handler and bobo_handler Message-ID: Has anybody successfully used auth_handler.py and bobo_handler together? If so, any tips on what's involved. Ray ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/group/medusa http://www.eGroups.com - Simplifying group communications From rushing@n... Thu May 13 01:43:20 1999 From: rushing@n... (rushing@n...) Date: Wed, 12 May 1999 17:43:20 -0700 (PDT) Subject: [medusa] using auth_handler and bobo_handler In-Reply-To: References: Message-ID: <14138.7928.949989.373497@s...> ray@u... writes: > Has anybody successfully used auth_handler.py and bobo_handler > together? If so, any tips on what's involved. It shouldn't be any more difficult than wrapping the one with the other. There's an example in demo/publish.py, where an auth handler is wrapped around a put_handler: | # Supports the HTTP PUT method... | ph = put_handler.put_handler (fs, '/.*') | | # ... but be sure to wrap it with an auth handler: | ah = auth_handler.auth_handler (users, ph) | | [...] | | hs.install_handler (ah) # for PUT If this doesn't work, please let me know... -Sam ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/group/medusa http://www.eGroups.com - Simplifying group communications From ita.mis@e... Thu May 13 13:47:17 1999 From: ita.mis@e... (ita.mis@e...) Date: Thu, 13 May 1999 12:47:17 -0000 Subject: [medusa] non-threaded CGI apps Message-ID: Hi, is there an example of a script which does CGI parameter processing on a non-threaded python installation? is this at all possible right now? best regard Marco Tasselli ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/group/medusa http://www.eGroups.com - Simplifying group communications From rushing@n... Fri May 14 02:05:03 1999 From: rushing@n... (rushing@n...) Date: Thu, 13 May 1999 18:05:03 -0700 (PDT) Subject: [medusa] non-threaded CGI apps In-Reply-To: References: Message-ID: <14139.30319.442867.851030@s...> ita.mis@e... writes: > is there an example of a script which does CGI parameter processing > on a non-threaded python installation? by this do you mean parsing form data? If so, I believe I have used one of the Python library modules to handle it... -Sam ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/group/medusa http://www.eGroups.com - Simplifying group communications From rushing@n... Sat May 15 03:13:37 1999 From: rushing@n... (rushing@n...) Date: Fri, 14 May 1999 19:13:37 -0700 (PDT) Subject: [medusa] non-threaded CGI apps In-Reply-To: References: Message-ID: <14140.55038.246703.135936@s...> ita.mis@e... writes: > is there an example of a script which does CGI parameter processing > on a non-threaded python installation? > > is this at all possible right now? Well, *now* it is. 8^) I spruced up the script_handler a bit. New features: 1) restricted execution is a flag (off by default, so scripts will use execfile) 2) support for PUT/POST requests, data is handed directly to the script via stdin as a StringIO object. 3) persistent script handler. this is a bit like the bobo/zope system. modules can be added and deleted from the handler. the modules are persistent. the 'main' procedure of the module is handed the http_request object as its only parameter (rather than a bunch of CGI environment variable madness). A 'script_handler_demo' directory is also included, with lots of comments. It should run out-of-the-box. Note: restricted execution is off by default because apparently the cgi module cannot be imported? But many folks will want to use it for forms. -Sam ------------------------------------------------------------------------ eGroup home: http://www.eGroups.com/group/medusa http://www.eGroups.com - Simplifying group communications From amos@a... Tue May 25 19:50:48 1999 From: amos@a... (Amos Latteier) Date: Tue, 25 May 1999 11:50:48 -0700 Subject: [medusa] NT socket error, asyncore.py Message-ID: <3.0.5.32.19990525115048.00c7d250@m...> A non-text attachment was scrubbed... Name: not available Type: text/enriched Size: 1070 bytes Desc: not available Url : http://mail.python.org/pipermail-21/medusa-dev/attachments/3ae3ce41/attachment.bin From pwelliso@i... Wed May 26 00:01:38 1999 From: pwelliso@i... (pwelliso@i...) Date: Tue, 25 May 1999 23:01:38 -0000 Subject: [medusa] Re: HTTPS available ? In-Reply-To: Message-ID: <7ifa4i$j9kc@e...> wrote: Original Article: http://www.egroups.com/group/medusa/?start=36 > Is there a https module available ? > > One that makes calls to SSLeay would be great ? > > I would be willing to help debug/test/write one if there is not one > available. A quick solution to this is STunnel: http://mike.daewoo.com.pl/computer/stunnel/ It took about 2 hours and it was up and running. The software acts as a wrapper, you hit port 443 with a https connection, the software handles the SSL part then passes off the plain text to the normal web server running on port 80. The only problem is that it passes the local host IP address to the web server so logging is not very useful. ------------------------------------------------------------------------ eGroups.com home: http://www.egroups.com/group/medusa http://www.egroups.com - Simplifying group communications From skip@m... Wed May 26 02:56:52 1999 From: skip@m... (Skip Montanaro) Date: Tue, 25 May 1999 21:56:52 -0400 Subject: [medusa] Strange exception in ZServer/medusa/asynchat.py Message-ID: <199905260156.VAA29018@c...> I'm using Medusa by way of Zope's ZServer. I have ZServer sitting behind an Apache proxy and am bombing it from a threaded client script that reads a web server log file as input and throws requests at the Apache server. I can vary the number of threads to adjust the load. I normally run it with five simultaneous threads. by default, the system runs briefly then (as they say), "Whammo! Blammo!", I get the following error: Unhandled exception in thread: Traceback (innermost last): File "/home/dolphin/skip/src/Zope/ZServer/PubCore/ZServerPublisher.py", line 97, in __init__ response._finish() File "/home/dolphin/skip/src/Zope/ZServer/HTTPResponse.py", line 209, in _finish self.stdout.close() File "/home/dolphin/skip/src/Zope/ZServer/HTTPResponse.py", line 235, in close self._channel.push(CallbackProducer(self._channel.done)) File "/home/dolphin/skip/src/Zope/ZServer/HTTPServer.py", line 307, in push if send: self.initiate_send() File "/home/dolphin/skip/src/Zope/ZServer/medusa/asynchat.py", line 200, in initiate_send self.refill_buffer() File "/home/dolphin/skip/src/Zope/ZServer/medusa/asynchat.py", line 184, in refill_buffer self.producer_fifo.pop() File "/home/dolphin/skip/src/Zope/ZServer/medusa/asynchat.py", line 255, in pop del self.list[0] IndexError: list assignment index out of range If you look at the code in asynchat.fifo.pop, the error can only be explained by the fact that two threads are modifying the fifo simultaneously, since access to self.list[0] succeeded in the statement before the statement that throws the IndexError: def pop (self): if self.list: result = (1, self.list[0]) # non-null list here del self.list[0] # empty list here else: result = (0, None) return result I modified the fifo class to lock access to its list: class fifo: def __init__ (self, list=None): self.lock = thread.allocate_lock() self.lock.acquire() if not list: self.list = [] else: self.list = list self.lock.release() def __len__ (self): self.lock.acquire() l = len(self.list) self.lock.release() return l def first (self): self.lock.acquire() item = self.list[0] self.lock.release() return item def push (self, data): self.lock.acquire() self.list.append (data) self.lock.release() def pop (self): self.lock.acquire() if self.list: result = (1, self.list[0]) del self.list[0] else: result = (0, None) self.lock.release() return result but this didn't help: Unhandled exception in thread: Traceback (innermost last): File "/home/dolphin/skip/src/Zope/ZServer/PubCore/ZServerPublisher.py", line 97, in __init__ response._finish() File "/home/dolphin/skip/src/Zope/ZServer/HTTPResponse.py", line 209, in _finish self.stdout.close() File "/home/dolphin/skip/src/Zope/ZServer/HTTPResponse.py", line 235, in close self._channel.push(CallbackProducer(self._channel.done)) File "/home/dolphin/skip/src/Zope/ZServer/HTTPServer.py", line 307, in push if send: self.initiate_send() File "/home/dolphin/skip/src/Zope/ZServer/medusa/asynchat.py", line 200, in initiate_send self.refill_buffer() File "/home/dolphin/skip/src/Zope/ZServer/medusa/asynchat.py", line 175, in refill_buffer p = self.producer_fifo.first() File "/home/dolphin/skip/src/Zope/ZServer/medusa/asynchat.py", line 254, in first item = self.list[0] IndexError: list index out of range This error also looks like a threading problem. When refill_buffer tests self.producer.fifo, it sees something, but by the time it calls fifo.first, the list has evaporated: while 1: if len(self.producer_fifo): # non-null list here p = self.producer_fifo.first() # null list here Any ideas? Is there a way to easily turn off threading? It seems that the data structures are not properly protected. Skip Montanaro | Mojam: "Uniting the World of Music" http://www.mojam.com/ skip@m... | Musi-Cal: http://www.musi-cal.com/ 518-372-5583 ------------------------------------------------------------------------ eGroups.com home: http://www.egroups.com/group/medusa http://www.egroups.com - Simplifying group communications From jim@d... Wed May 26 15:12:41 1999 From: jim@d... (Jim Fulton) Date: Wed, 26 May 1999 10:12:41 -0400 Subject: [medusa] Re: Strange exception in ZServer/medusa/asynchat.py References: <199905260156.VAA29018@c...> Message-ID: <374C0159.E4A1291F@d...> Skip Montanaro wrote: > > I'm using Medusa by way of Zope's ZServer. I have ZServer sitting behind an > Apache proxy and am bombing it from a threaded client script that reads a > web server log file as input and throws requests at the Apache server. I > can vary the number of threads to adjust the load. I normally run it with > five simultaneous threads. > > By default, the system runs briefly then (as they say), "Whammo! Blammo!", I > get the following error: (snip) > Any ideas? This was a bug in ZServer that was fixed in the most recent Zope release (2.0 alpha 1). Basically, the push method in ZHTTPServer.zhttp_channel has a second argument that must be passed a false value when called from a thread other than the main (medusa) thread. The ZServer code that runs in separate threads now supplies this argument. > Is there a way to easily turn off threading? Not in ZServer. It would be a ZServer bug if a problem resulted from use of threads. This particular bug has been fixed. Jim -- Jim Fulton mailto:jim@d... Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. ------------------------------------------------------------------------ eGroups.com home: http://www.egroups.com/group/medusa http://www.egroups.com - Simplifying group communications From rushing@n... Thu May 27 00:20:02 1999 From: rushing@n... (rushing@n...) Date: Wed, 26 May 1999 16:20:02 -0700 (PDT) Subject: [medusa] Re: HTTPS available ? In-Reply-To: <7ifa4i$j9kc@e...> References: Message-ID: <14156.33124.873581.649238@s...> pwelliso@i... writes: > wrote: > Original Article: http://www.egroups.com/group/medusa/?start=36 > > Is there a https module available ? > A quick solution to this is STunnel: > http://mike.daewoo.com.pl/computer/stunnel/ > > It took about 2 hours and it was up and running. > > The software acts as a wrapper, you hit port 443 with a https connection, > the software handles the SSL part then passes off the plain text to the > normal web server running on port 80. The only problem is that it passes > the local host IP address to the web server so logging is not very useful. Great, thanks for the info! I'll put a note about this up on the web site. -Sam ------------------------------------------------------------------------ eGroups.com home: http://www.egroups.com/group/medusa http://www.egroups.com - Simplifying group communications From rushing@n... Sat Jun 12 02:25:52 1999 From: rushing@n... (Sam Rushing) Date: Fri, 11 Jun 1999 18:25:52 -0700 (PDT) Subject: [medusa] UDP gotcha Message-ID: <14177.46448.901418.666637@s...> This took quite a while to figure out, so I thought I'd share it with the list in case anyone else runs up against the same problem. If you are using asyncore for UDP, you need to make sure your handle_read() method calls recvfrom() in a loop, otherwise you will read only one packet per trip through the polling loop. If you are trying to pump many packets through UDP while at the same time servicing lots of other channels for other protocols (like HTTP or SMTP), then your OS will drop incoming packets if you don't read them fast enough. def handle_read (self): # read UDP in a loop try: while 1: reply, whence = self.recvfrom (2048) self.handle_reply (reply) except socket.error, why: if why[0] != asyncore.EWOULDBLOCK: raise socket.error, why It also helps to raise your send and recv buffer sizes: self.setsockopt (socket.SOL_SOCKET, socket.SO_SNDBUF, 200 * 1024) self.setsockopt (socket.SOL_SOCKET, socket.SO_RCVBUF, 200 * 1024) Thanks are due to Dru Nelson of eGroups for helping me track this down. -Sam ------------------------------------------------------------------------ eGroups.com home: http://www.egroups.com/group/medusa http://www.egroups.com - Simplifying group communications From Amos@d... Thu Jun 17 17:56:46 1999 From: Amos@d... (Amos Latteier) Date: Thu, 17 Jun 1999 12:56:46 -0400 Subject: [medusa] select error on win32 Message-ID: <613145F79272D211914B0020AFF640190AD23E@g...> Hi, A ZServer user forwarded this error to me: Traceback (innermost last): File "start.py", line 155, in ? asyncore.loop() File "medusa\asyncore.py", line 104, in loop poll_fun (timeout) File "medusa\asyncore.py", line 52, in poll (r,w,e) = select.select (r,w,e, timeout) ValueError: too many file descriptors in select() They are using ZServer on win32 on a fairly heavily trafficked site. This error halts the server. According to "File Descriptor Limitations in Various Operating Systems" http://www.nightmare.com/medusa/descriptor_limits.html NT does not have a hard limit to the number of fd's that can be selected, and poll is not available. Does anyone (I mean does Sam ;-) have any ideas about what may be causing the select loop to fail, and what ZServer may be able to do to mitigate these problems? Thanks. -Amos -- Amos Latteier mailto:amos@d... Digital Creations http://www.digicool.com ------------------------------------------------------------------------ eGroups.com home: http://www.egroups.com/group/medusa http://www.egroups.com - Simplifying group communications From jody@s... Thu Jun 17 19:58:04 1999 From: jody@s... (jody@s...) Date: Thu, 17 Jun 1999 18:58:04 -0000 Subject: [medusa] Need an example client that does not read from stdin and exits the main loop Message-ID: <7kbgfs$kb2g@e...> I've written a server in medusa that works like ftp. That is: command space args and the server responds with: code space results This code works using telnet and using a medusa client that reads from stdin. My questions are: 1. How do I send a fixed set of commands to the server from a client? What I have tried and failed with is in the client's handle_connect to use self.send(). 2. After all of the commands are sent and all responces processed, how do I exit from the main loop? ------------------------------------------------------------------------ eGroups.com home: http://www.egroups.com/group/medusa http://www.egroups.com - Simplifying group communications From rushing@n... Thu Jun 17 17:26:35 1999 From: rushing@n... (Sam Rushing) Date: Thu, 17 Jun 1999 09:26:35 -0700 (PDT) Subject: [medusa] select error on win32 In-Reply-To: <613145F79272D211914B0020AFF640190AD23E@g...> References: <613145F79272D211914B0020AFF640190AD23E@g...> Message-ID: <14185.8381.95775.851844@s...> Amos Latteier writes: > They are using ZServer on win32 on a fairly heavily trafficked site. > This error halts the server. > Does anyone (I mean does Sam ;-) have any ideas about what may be > causing the select loop to fail, and what ZServer may be able to do to > mitigate these problems? There are no hard limits on NT, but you have to recompile your application. In this case you need to have another version of socket.pyd compiled with FD_SETSIZE=1024 (or some other larger number). This is true of most platforms, because FD_SETSIZE is a macro. On FreeBSD I'm using FD_SETSIZE=8192 without any troubles. Pay no attention to the WSAStartup 'MaxSock' parameter, it's supposed to be ignored if you're using winsock2. -Sam ------------------------------------------------------------------------ eGroups.com home: http://www.egroups.com/group/medusa http://www.egroups.com - Simplifying group communications From rushing@n... Thu Jun 17 17:33:01 1999 From: rushing@n... (Sam Rushing) Date: Thu, 17 Jun 1999 09:33:01 -0700 (PDT) Subject: [medusa] Need an example client that does not read from stdin and exits the main loop In-Reply-To: <7kbgfs$kb2g@e...> References: <7kbgfs$kb2g@e...> Message-ID: <14185.8754.516841.546019@s...> jody@s... writes: > I've written a server in medusa that works like ftp. That is: > command space args > and the server responds with: > code space results > > This code works using telnet and using a medusa client that reads > from stdin. My questions are: > > 1. How do I send a fixed set of commands to the server from a > client? What I have tried and failed with is in the client's > handle_connect to use self.send(). In the __init__ method for you client class, do something like this: self.push ('200 Go Ahead, Make My Day\r\n') You should never use self.send(), because that will bypass async_chat's outgoing-buffer scheme. If you push() data even before the connection is made, that's ok, it will be sent as soon as it does connect. Look at the class inmonitor.py for an example. > 2. After all of the commands are sent and all responces > processed, how do I exit from the main loop? I usually exit by calling asyncore.close_all(), which clears out the socket map. This causes asyncore.loop() to exit. Let me know if you have any other trouble! -Sam ------------------------------------------------------------------------ eGroups.com home: http://www.egroups.com/group/medusa http://www.egroups.com - Simplifying group communications From jody@l... Fri Jun 18 13:59:01 1999 From: jody@l... (Jody Winston) Date: Fri, 18 Jun 1999 08:59:01 -0400 (EDT) Subject: [medusa] Re: Need an example client that does not read from stdin and exits the main loop In-Reply-To: <14185.8754.516841.546019@s...> (message from Message-ID: <199906181259.IAA23066@h...> >>>>> "Sam" == Sam Rushing writes: Sam> jody@s... writes: >> I've written a server in medusa that works like ftp. That is: >> command space args and the server responds with: code space >> results >> >> This code works using telnet and using a medusa client that >> reads from stdin. My questions are: >> >> 1. How do I send a fixed set of commands to the server from a >> client? What I have tried and failed with is in the client's >> handle_connect to use self.send(). Sam> In the __init__ method for you client class, do something Sam> like this: Sam> self.push ('200 Go Ahead, Make My Day\r\n') The problem with my code was that I had forgot to add the terminator. Sam> You should never use self.send(), because that will bypass Sam> async_chat's outgoing-buffer scheme. If you push() data even Sam> before the connection is made, that's ok, it will be sent as Sam> soon as it does connect. Look at the class Sam> inmonitor.py for an example. I'll change all my self.send to self.push >> 2. After all of the commands are sent and all responces >> processed, how do I exit from the main loop? Sam> I usually exit by calling asyncore.close_all(), which clears Sam> out the socket map. This causes asyncore.loop() to exit. When I was using self.send, calling asyncore.close_all happened before all the processing was finished on the client side. So I sent the quit messesage to the server which broke the connection to the client. Sam> Let me know if you have any other trouble! ------------------------------------------------------------------------ eGroups.com home: http://www.egroups.com/group/medusa http://www.egroups.com - Simplifying group communications From wolf@s... Thu Jun 24 00:49:27 1999 From: wolf@s... (Wolf Logan) Date: Wed, 23 Jun 1999 16:49:27 -0700 Subject: [medusa] Re: select error on win32 Message-ID: <219B76713101D31183B000902762824E104D@t...> microsoft's "official" position seems to be that you're not allowed to know. there's no defined way to know the "real" limit without stress-testing. however, the "real" limit on listen queues under NT is as follows: NT 3.5 : 100 NT 3.51 : 100 NT 4.0 (wks) : 5 NT 4.0 (srv) : 200 i don't know what the limits are for 95/98, but win CE (!) has a backlog of 5 over serial, and 2 over IR. asking for a queue larger than the size shown will set the queue to its maximum. -----Original Message----- From: Sam Rushing [mailto:rushing@n...] Sent: Tuesday, June 22, 1999 3:57 PM Amos Latteier writes: > Can NT not handle a listen queue greater than 1? I'm drawing a blank... Either NT ignores the listen queue size, or at one point it (or more likely win95) would throw an exception. ------------------------------------------------------------------------ eGroups.com home: http://www.egroups.com/group/medusa http://www.egroups.com - Simplifying group communications From rushing@n... Thu Jun 24 16:18:20 1999 From: rushing@n... (Sam Rushing) Date: Thu, 24 Jun 1999 08:18:20 -0700 (PDT) Subject: [medusa] asyncore.py descriptor-map hack Message-ID: <14194.19098.366275.507242@s...> I changed asyncore.dispatcher to cache fileno() information in a dictionary stored at the class level. This dictionary can be used to feed select() file descriptors directly, therefore removing its need to call the fileno() method of each socket object. [this was doubly bad because it would trigger a __getattr__ call]. It's possible that this will remove a significant amount of overhead from poll(), but I don't have time to thoroughly test it for speed right now. For those who might be interested, it's checked in as revision 2.41. [2.40 and 2.42 are exactly the same, I couldn't figure out how to correctly make a branch with CVS within 10 minutes so I gave up] The following command will do the trick: $ cvs up -r2.41 asyncore.py -Sam ------------------------------------------------------------------------ eGroups.com home: http://www.egroups.com/group/medusa http://www.egroups.com - Simplifying group communications From brian.brown@s... Tue Jul 27 20:23:58 1999 From: brian.brown@s... (brian.brown@s...) Date: Tue, 27 Jul 1999 12:23:58 -0700 Subject: [medusa] Virtual Hosting? Message-ID: <7nl10e$hp2n@e...> Does medusa support binding to a specific IP address? Also, does it support serving up different document roots based on DNS domain information contained in the http 1.1 header? thanks! Brian From rushing@n... Tue Jul 27 22:34:58 1999 From: rushing@n... (Sam Rushing) Date: Tue, 27 Jul 1999 14:34:58 -0700 (PDT) Subject: [medusa] Virtual Hosting? In-Reply-To: <7nl10e$hp2n@e...> References: <7nl10e$hp2n@e...> Message-ID: <14238.9475.715207.13060@s...> brian.brown@s... writes: > Does medusa support binding to a specific IP address? Also, does it > support serving up different document roots based on DNS domain > information contained in the http 1.1 header? Medusa supports both types of virtual hosting: check out the file 'virtual_handler.py'. Here's how you might use it: [in your copy of 'start_medusa.py'] fs = filesys.os_filesystem ('/var/www/scary') dh = default_handler.default_handler (fs) vh = virtual_handler.virtual_handler_with_host (dh, 'freddy.nightmare.com') hs = http_server.http_server ('', 80, rs, lg) hs.install_handler (vh) -Sam From brian@b... Wed Jul 28 07:24:11 1999 From: brian@b... (brian@b...) Date: Tue, 27 Jul 1999 23:24:11 -0700 Subject: [medusa] Re: Virtual Hosting? In-Reply-To: <14238.9475.715207.13060@s...> Message-ID: <7nm7mb$h5dp@e...> <14238.9475.715207.1306-@s...> wrote: original article:http://www.egroups.com/group/medusa/?start=66 > brian.brown@s... writes: > > Does medusa support binding to a specific IP address? Also, does it > > support serving up different document roots based on DNS domain > > information contained in the http 1.1 header? > > Medusa supports both types of virtual hosting: check out the file > 'virtual_handler.py'. > > Here's how you might use it: > > [in your copy of 'start_medusa.py'] > > fs = filesys.os_filesystem ('/var/www/scary') > dh = default_handler.default_handler (fs) > vh = virtual_handler.virtual_handler_with_host (dh, 'freddy.nightmare.com') > hs = http_server.http_server ('', 80, rs, lg) > hs.install_handler (vh) > > -Sam > Thanks, Sam! Now to figure out how to do it in the Zopeified version ;) From amos@a... Tue Aug 3 00:29:30 1999 From: amos@a... (Amos Latteier) Date: Mon, 02 Aug 1999 16:29:30 -0700 Subject: [medusa] IE5/medusa response delays Message-ID: <3.0.5.32.19990802162930.013973c0@m...> Hi, A Zope is experiencing a medusa problem that I am having trouble diagnosing. He is using Zope on NT4 SP3. When he connects via HTTP to the server using IE5 he sometimes gets delayed responses, especially after waiting a while between requests. In his words, Using netstat -an, I can see that ie5 keeps 2 connections open to the server and they're ESTABLISHED. As soon as the connections go to TIME_WAIT, IE5 will hang. I'm not sure which end is closing the connection. After the original connection goes to TIME_WAIT, I make a request from IE5 and see the long delay, netstat shows that a new connection has been opened, but there is still a long delay getting a reply from the server. If I open another connection from another browser, the "hanging" connection "wakes up" and services the request. Ideas? -Amos From francois_le_coguiec@h... Thu Aug 26 13:52:10 1999 From: francois_le_coguiec@h... (francois_le_coguiec@h...) Date: Thu, 26 Aug 1999 05:52:10 -0700 Subject: [medusa] Porting Medusa to Ruby Message-ID: <7q3d9q$sg3e@e...> An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail-21/medusa-dev/attachments/205c3119/attachment.html From cmenzel@s... Fri Aug 27 07:52:03 1999 From: cmenzel@s... (cmenzel@s...) Date: Thu, 26 Aug 1999 23:52:03 -0700 Subject: [medusa] drop in medusa? Message-ID: <7q5cij$72v6@e...> Hi, I administer a site running apache with basic .htaccess authorization. I would like to 'drop in' medusa but need to use the existing htaccess/ htpasswd files already in place. Is there a module to use or is this built into medusa? Thanks From jam@n... Fri Aug 27 17:49:47 1999 From: jam@n... (jam) Date: Fri, 27 Aug 1999 12:49:47 -0400 Subject: [medusa] Re: drop in medusa? In-Reply-To: <7q5cij$72v6@e...>; from cmenzel@s... on Thu, Aug 26, 1999 at 11:52:03PM -0700 References: <7q5cij$72v6@e...> Message-ID: <19990827124947.A27011@t...> On Thu, Aug 26, 1999 at 11:52:03PM -0700, cmenzel@s... wrote: > > Hi, > I administer a site running apache with basic .htaccess authorization. > I would like to 'drop in' medusa but need to use the existing htaccess/ > htpasswd files already in place. Is there a module to use or is this > built into medusa? > > Thanks > medusa is a framework in and of itself.. it doesn't have support for apache-like htaccess or htpasswd files that I am aware of, though it wouldn't be all that hard to write a module that could parse them or at least convert it to something useable. could you describe a little more about the application, and maybe we can work something out? regards, J -- || visit gfd || psa member #293 || New Image Systems & Services, Inc. From rushing@n... Mon Aug 30 11:34:29 1999 From: rushing@n... (Sam Rushing) Date: Mon, 30 Aug 1999 03:34:29 -0700 (PDT) Subject: [medusa] drop in medusa? In-Reply-To: <7q5cij$72v6@e...> References: <7q5cij$72v6@e...> Message-ID: <14282.24117.67050.810574@s...> cmenzel@s... writes: > I administer a site running apache with basic .htaccess authorization. > I would like to 'drop in' medusa but need to use the existing htaccess/ > htpasswd files already in place. Is there a module to use or is this > built into medusa? The best approach depends on how you are using .htaccess files. If you have a site with thousands of users maintaining their own .htaccess files, then you'll probably want a specialized handler that looks for these and parses them on the fly. If on the other hand you have a simple global password file I would suggest ignoring .htaccess itself (and writing a simple dictionary-interface wrapper for the actual password file - see how unix authentication is done in ftp_server.py). If someone else declares interest in this I will throw together a quick htpasswd-based auth handler. -Sam From cmenzel@s... Mon Aug 30 20:28:07 1999 From: cmenzel@s... (cmenzel@s...) Date: Mon, 30 Aug 1999 12:28:07 -0700 Subject: [medusa] Re: drop in medusa? In-Reply-To: <19990827124947.A27011@t...> Message-ID: <7qem07$c2rr@e...> cmenze-@s... wrote: original article:http://www.egroups.com/group/medusa/?start=71 > On Thu, Aug 26, 1999 at 11:52:03PM -0700, cmenzel@s... wrote: > > > > Hi, > > I administer a site running apache with basic .htaccess authorization. > > I would like to 'drop in' medusa but need to use the existing htaccess/ > > htpasswd files already in place. Is there a module to use or is this > > built into medusa? > > > > Thanks > > > > medusa is a framework in and of itself.. it doesn't have support for > apache-like htaccess or htpasswd files that I am aware of, though it > wouldn't be all that hard to write a module that could parse them or at > least convert it to something useable. could you describe a little more > about the application, and maybe we can work something out? > > regards, > J > -- > || visit gfd > || psa member #293 > || New Image Systems & Services, Inc. Hi, Thanks for the reply. The site is commercial in that customers pay for access to certain directories. Upon payment a cgi program adds their name and password to the .htpasswd file. Having discovered python and medusa I would very much like to be able to replace apache. I am just begining my learning curve with python so I need to come up to speed. The site delivers 800 meg a month and the files are between 30 and 200k. I feel sure Medusa woould be of benefit to my clients. Thanks, Chris From cmenzel@s... Mon Aug 30 20:36:18 1999 From: cmenzel@s... (cmenzel@s...) Date: Mon, 30 Aug 1999 12:36:18 -0700 Subject: [medusa] Re: drop in medusa? In-Reply-To: <14282.24117.67050.810574@s...> Message-ID: <7qemfi$c30p@e...> cmenze-@s... wrote: original article:http://www.egroups.com/group/medusa/?start=72 > cmenzel@s... writes: > > I administer a site running apache with basic .htaccess authorization. > > I would like to 'drop in' medusa but need to use the existing htaccess/ > > htpasswd files already in place. Is there a module to use or is this > > built into medusa? > > The best approach depends on how you are using .htaccess files. If > you have a site with thousands of users maintaining their own .htaccess > files, then you'll probably want a specialized handler that looks for > these and parses them on the fly. > > If on the other hand you have a simple global password file I would > suggest ignoring .htaccess itself (and writing a simple > dictionary-interface wrapper for the actual password file - see how > unix authentication is done in ftp_server.py). > > If someone else declares interest in this I will throw together a > quick htpasswd-based auth handler. > > -Sam > Hi Sam, I think I get it! Maintain the .htpasswd file with a Python module. It is the same idea as maintaining it with the shell script that I use now. I am new to Python but will try to write the wrapper. Thanks, Chris From jam@n... Mon Aug 30 23:18:20 1999 From: jam@n... (jam) Date: Mon, 30 Aug 1999 18:18:20 -0400 Subject: [medusa] Re: drop in medusa? In-Reply-To: <7qem07$c2rr@e...>; from cmenzel@s... on Mon, Aug 30, 1999 at 12:28:07PM -0700 References: <19990827124947.A27011@t...> <7qem07$c2rr@e...> Message-ID: <19990830181820.A863@t...> On Mon, Aug 30, 1999 at 12:28:07PM -0700, cmenzel@s... wrote: > > Hi, > Thanks for the reply. The site is commercial in that customers pay for > access to certain directories. Upon payment a cgi program adds their > name and password to the .htpasswd file. Having discovered python and > medusa I would very much like to be able to replace apache. I am just > begining > my learning curve with python so I need to come up to speed. The site > delivers 800 meg a month and the files are between 30 and 200k. I feel > sure > Medusa woould be of benefit to my clients. > > Thanks, > Chris > it is *certainly* of benefit to your clients, but just keep in mind as you move forward that it is *not* meant to be a drop-in replacement for apache. you ought to be able to get your system up and working-- are you able to change the payment processing code to do something slightly different? if you have control of that code, you can just make it write to some 'standard' file someplace, and write python code to read it in and process it.. in just thinking about it, it ought to be a really quick project. if I can answer any quesions or anything, feel free to fire away either directly or to the list.. this sounds like a cool project! ;) regards, J -- || visit gfd || psa member #293 || New Image Systems & Services, Inc. From ita.mis@e... Tue Aug 31 09:25:10 1999 From: ita.mis@e... (ita.mis@e...) Date: Tue, 31 Aug 1999 01:25:10 -0700 Subject: [medusa] problems with Internet Explorer Message-ID: <7qg3h6$sr7r@e...> Hi, I've experienced a problem similar to the one reported by Amos Latteier concerning IE. I've a medusa installation on a linux box and I've written some scripts to serve dynamic pages. I use the script_handler and persistent scripts. The test application is very simple. I just call a script for building on the fly a home page and than, clicking on the submit button, I post another request for building another form on the fly. When I use Nescape Navigator, everything works fine, but with IE (I user version 4.5 on a Macintosh) only the first call works. When I press the submit button of the home form the browser reports a "sending request" message on its status bar but nothing happens. I have to stop the execution. Has anyone experienced similar behaviour with persistent scripts using IE? Tasselli Marco From rushing@n... Tue Aug 31 09:44:09 1999 From: rushing@n... (Sam Rushing) Date: Tue, 31 Aug 1999 01:44:09 -0700 (PDT) Subject: [medusa] problems with Internet Explorer In-Reply-To: <7qg3h6$sr7r@e...> References: <7qg3h6$sr7r@e...> Message-ID: <14283.38361.900871.692692@s...> ita.mis@e... writes: > When I use Nescape Navigator, everything works fine, but with IE (I > user version 4.5 on a Macintosh) only the first call works. When I > press the submit button of the home form the browser reports a "sending > request" message on its status bar but nothing happens. > I have to stop the execution. This sounds like something that reported to me a while back (by Eric Parker eparker@z...). Apparently everyone but IE has been affixing an extra CRLF to the end of their POST/PUT data, and this is wrong. http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.1 I couldn't come up with a very clean fix for it at the time, and it just got buried. Are you using POST or GET for your form submission? -Sam From rushing@n... Tue Aug 31 10:50:17 1999 From: rushing@n... (Sam Rushing) Date: Tue, 31 Aug 1999 02:50:17 -0700 (PDT) Subject: [medusa] problems with Internet Explorer In-Reply-To: <14283.38361.900871.692692@s...> References: <7qg3h6$sr7r@e...> <14283.38361.900871.692692@s...> Message-ID: <14283.42329.298077.710128@s...> Sam Rushing writes: > I couldn't come up with a very clean fix for it at the time, and it > just got buried. w.r.t. 'extra CRLF' bug, please 'cvs up' these two files. I'm pretty sure it's fixed now (POST script tested with both NS & IE) http_server.py 3.35 script_handler.py 1.9 -Sam From skip@m... Tue Aug 31 23:31:11 1999 From: skip@m... (Skip Montanaro) Date: Tue, 31 Aug 1999 17:31:11 -0500 Subject: [medusa] Hideously long selects? Message-ID: <199908312231.RAA17133@d...> I'm using the ZServer component of Zope (but nothing else) behind an Apache serving as a proxy. I've noticed situations where it seems like I my ZServer process is hung, but when I connect to it with strace (Linux RH 5.2) I see it stalled in a select waiting on a lot of input file descriptors with an absurdly long timeout (typically 30 seconds): select(33, [6 9 10 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 32], [], [], {30, 0} I'm not much of a select expert, but it seems to me that this is what's causing my problem. Let's assume all those file descriptors are wacko web browsers (not a big stretch), so they aren't going to send input any time soon, so it will be a *long* time before Medusa falls out of the select call and gets around to thinking about new connections, right? Shouldn't the timeout be substantially shorter, 0.5-2.0 seconds perhaps? I haven't checked the medusa code yet (that's where I'm headed next), but I thought I'd check to see if I'm way off-base before I start messing around in detail. Thx, Skip Montanaro | http://www.mojam.com/ skip@m... | http://www.musi-cal.com/~skip/ 847-971-7098 | Python: Programming the way Guido indented... From skip@m... Tue Aug 31 23:33:57 1999 From: skip@m... (Skip Montanaro) Date: Tue, 31 Aug 1999 17:33:57 -0500 Subject: [medusa] Hideously long selects? Message-ID: <199908312233.RAA17359@d...> I'm using the ZServer component of Zope (but nothing else) behind an Apache serving as a proxy. I've noticed situations where it seems like I my ZServer process is hung, but when I connect to it with strace (Linux RH 5.2) I see it stalled in a select waiting on a lot of input file descriptors with an absurdly long timeout (typically 30 seconds): select(33, [6 9 10 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 32], [], [], {30, 0} I'm not much of a select expert, but it seems to me that this is what's causing my problem. Let's assume all those file descriptors are wacko web browsers (not a big stretch), so they aren't going to send input any time soon, so it will be a *long* time before Medusa falls out of the select call and gets around to thinking about new connections, right? Shouldn't the timeout be substantially shorter, 0.5-2.0 seconds perhaps? I haven't checked the medusa code yet (that's where I'm headed next), but I thought I'd ask and see if I'm way off-base before I start messing around in detail. Thx, Skip Montanaro | http://www.mojam.com/ skip@m... | http://www.musi-cal.com/~skip/ 847-971-7098 | Python: Programming the way Guido indented... From billtut@m... Tue Aug 31 23:38:57 1999 From: billtut@m... (Bill Tutt) Date: Tue, 31 Aug 1999 15:38:57 -0700 Subject: [medusa] Re: Hideously long selects? Message-ID: <4D0A23B3F74DD111ACCD00805F31D8100DB91161@R...> > From: Skip Montanaro [mailto:skip@m...] > > > I'm not much of a select expert, but it seems to me that this > is what's > causing my problem. Let's assume all those file descriptors > are wacko web > browsers (not a big stretch), so they aren't going to send > input any time > soon, so it will be a *long* time before Medusa falls out of > the select call > and gets around to thinking about new connections, right? No, the listen() socket should be in the file descriptor list passed into select(). select() should return when a new connection attempt is made, then handle_accept() usually gets called. (per handle_read_event() in asyncore.py) > Shouldn't the > timeout be substantially shorter, 0.5-2.0 seconds perhaps? > Not particularlily you don't really want to be eating up useless CPU if you don't have to. Bill From skip@m... Wed Sep 1 01:37:17 1999 From: skip@m... (skip@m...) Date: Tue, 31 Aug 1999 17:37:17 -0700 Subject: [medusa] Re: Hideously long selects? In-Reply-To: <199908312233.RAA17359@d...> Message-ID: <7qhsft$v69i@e...> [ long timeout ramblings deleted ] I tried shortening the select timeout to 2.0 seconds. Same problem. When I made the change I noticed that I picked the wrong version of asyncore.py to modify. When I compared them I got the following version info: The version from ZServer had: $Id: asyncore.py,v 1.2 1999/04/09 00:37:33 amos Exp $ The Python (1.5.2+) version had: $Id: asyncore.py,v 1.2 1999/06/08 13:20:05 guido Exp $ Any chance I'm seeing some version skew problems? Thanks, Skip From rushing@n... Wed Sep 1 03:43:42 1999 From: rushing@n... (Sam Rushing) Date: Tue, 31 Aug 1999 19:43:42 -0700 (PDT) Subject: [medusa] Re: Hideously long selects? In-Reply-To: <7qhsft$v69i@e...> References: <199908312233.RAA17359@d...> <7qhsft$v69i@e...> Message-ID: <14284.37598.265974.116161@s...> skip@m... writes: > [ long timeout ramblings deleted ] > > I tried shortening the select timeout to 2.0 seconds. Same problem. My guess is that your accept()ing socket is getting closed. In this kind of situation I usually add a 'print r' line just before the call to select(). This will tell you exactly which objects are waiting for read. If your server is not in there, then you have your culprit. > When I made the change I noticed that I picked the wrong version of > asyncore.py to modify. When I compared them I got the following > version info: > > The version from ZServer had: > $Id: asyncore.py,v 1.2 1999/04/09 00:37:33 amos Exp $ > > The Python (1.5.2+) version had: > $Id: asyncore.py,v 1.2 1999/06/08 13:20:05 guido Exp $ > > Any chance I'm seeing some version skew problems? Hmmm.. I get: $Id: asyncore.py,v 2.44 1999/07/27 00:50:06 rushing Exp $ 8^) We probably have three completely different CVS repositories at work here. To get the latest version from medusa: bash$ export CVSROOT=:pserver:medusa@s...:/usr/local/cvsroot bash$ cvs login ['medusa' is the pa**w**d] bash$ cvs checkout medusa/asyncore.py -Sam From jim@d... Wed Sep 1 13:06:54 1999 From: jim@d... (Jim Fulton) Date: Wed, 01 Sep 1999 08:06:54 -0400 Subject: [medusa] Re: Hideously long selects? References: <199908312233.RAA17359@d...> Message-ID: <37CD16DE.711ECAD6@d...> Skip Montanaro wrote: > > I'm using the ZServer component of Zope (but nothing else) behind an Apache > serving as a proxy. I've noticed situations where it seems like I my > ZServer process is hung, but when I connect to it with strace (Linux RH 5.2) > I see it stalled in a select waiting on a lot of input file descriptors with > an absurdly long timeout (typically 30 seconds): > > select(33, [6 9 10 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 32], [], [], {30, 0} > > I'm not much of a select expert, but it seems to me that this is what's > causing my problem. Let's assume all those file descriptors are wacko web > browsers (not a big stretch), so they aren't going to send input any time > soon, so it will be a *long* time before Medusa falls out of the select call > and gets around to thinking about new connections, right? Shouldn't the > timeout be substantially shorter, 0.5-2.0 seconds perhaps? > > I haven't checked the medusa code yet (that's where I'm headed next), but I > thought I'd ask and see if I'm way off-base before I start messing around in > detail. Skip, What version of Zope are you running? I beleave that this is due to a bug that was fixed in the Zope 2.0 beta 5. Jim -- Jim Fulton mailto:jim@d... Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From jim@d... Wed Sep 1 13:11:47 1999 From: jim@d... (Jim Fulton) Date: Wed, 01 Sep 1999 08:11:47 -0400 Subject: [medusa] Re: Hideously long selects? References: <7qhsft$v69i@e...> Message-ID: <37CD1803.44A9B587@d...> skip@m... wrote: > > [ long timeout ramblings deleted ] > > I tried shortening the select timeout to 2.0 seconds. Same problem. > When I made the change I noticed that I picked the wrong version of > asyncore.py to modify. When I compared them I got the following > version info: > > The version from ZServer had: > $Id: asyncore.py,v 1.2 1999/04/09 00:37:33 amos Exp $ This is the number it has in the Zope CVS repository. > The Python (1.5.2+) version had: > $Id: asyncore.py,v 1.2 1999/06/08 13:20:05 guido Exp $ > > Any chance I'm seeing some version skew problems? This is the number it has in the medusa CVS repository. Obviously, this should be handled better somehow...... but I'm not sure how. Jim -- Jim Fulton mailto:jim@d... Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From ray@u... Thu Sep 2 23:35:49 1999 From: ray@u... (ray@u...) Date: Thu, 02 Sep 1999 15:35:49 -0700 Subject: [medusa] Where is the lastest version? Message-ID: <7qmu45$e97m@e...> Is the latest Medusa version accessable via HTTP or FTP? The most current version I can find on nightmare.com is medusa-src-990517.tar.gz. Ray From rushing@n... Fri Sep 3 00:40:53 1999 From: rushing@n... (Sam Rushing) Date: Thu, 2 Sep 1999 16:40:53 -0700 (PDT) Subject: [medusa] Where is the lastest version? In-Reply-To: <7qmu45$e97m@e...> References: <7qmu45$e97m@e...> Message-ID: <14287.2821.802399.505716@s...> ray@u... writes: > Is the latest Medusa version accessable via HTTP or FTP? The most > current version I can find on nightmare.com is medusa-src-990517.tar.gz. I've just made a tarball of the latest, available via http or ftp. http://www.nightmare.com/medusa/medusa-src-990902.tar.gz ftp://www.nightmare.com/medusa/medusa-src-990902.tar.gz [note: the ftp link has moved] -Sam From ray@u... Fri Sep 3 00:55:14 1999 From: ray@u... (ray@u...) Date: Thu, 2 Sep 1999 16:55:14 -0700 Subject: [medusa] Re: Where is the lastest version? Message-ID: <882567E0.00832ED4.00@b...> Thanks Sam. Hey, the first Python user group meeting is here next week. Is there any chance I can get you down some time to demo/show/talkabout/ Medusa or eGroups? Anytime you and your family are in the area, you are welcome to stay at my house with me and my family. Ray Sam Rushing on 09/02/99 04:40:53 PM Please respond to medusa@egroups.com To: medusa@egroups.com cc: (bcc: Ray Price/UAI) Subject: [medusa] Where is the lastest version? ray@u... writes: > Is the latest Medusa version accessable via HTTP or FTP? The most > current version I can find on nightmare.com is medusa-src-990517.tar.gz. I've just made a tarball of the latest, available via http or ftp. http://www.nightmare.com/medusa/medusa-src-990902.tar.gz ftp://www.nightmare.com/medusa/medusa-src-990902.tar.gz [note: the ftp link has moved] -Sam ------------------------------------------------------------------------ MyPoints-Free Rewards When You're Online. Start with up to 150 Points for joining! http://clickhere.egroups.com/click/805 eGroups.com home: http://www.egroups.com/group/medusa http://www.egroups.com - Simplifying group communications From jam@n... Fri Sep 3 04:45:08 1999 From: jam@n... (jam) Date: Thu, 2 Sep 1999 23:45:08 -0400 Subject: [medusa] trouble with 19990902 snapshot: asyncore.loop() exists immediately Message-ID: <19990902234508.A9613@t...> greetings, I've been beating my head against the wall for an hour or two trying to figure out what I might be doing wrong, but to no avail.. hopefully someone on the list can shed some light on this.. I have a very simple 'httpd.py' script: -- cut here -- import sys import asyncore if __name__ == "__main__": from medusa import default_handler, filesys, http_server, resolver, logger rs = resolver.caching_resolver("127.0.0.1") lg = logger.file_logger(sys.stdout) fs = filesys.os_filesystem(".") dh = default_handler.default_handler(fs) hs = http_server.http_server('toast', 8081, rs, lg) hs.install_handler(dh) asyncore.loop() -- cut here -- when I try to run this script from the command line, it exists immediately, and doesn't wait for connections.. I get my prompt back. if I run 'http_server.py from the 'medusa' directory, everything seems to work fine, and it sits and waits for connections. my first thought is that I somehow set up something incorrectly, but the above is much simpler than what I had with the older version, and I've tried removing the various pieces to see if they might be causing a problem, but have not had any luck so far. the only thing that might be remotely unusual about my setup is that I am using python 1.5.2. to install this version of medusa, I renamed the 'asyncore' and 'asynchat' modules, and copied the ones from the distribution into /usr/lib/python1.5, to make sure I was using the right versions. could someone shed a clue on this problem? I'm almost certain it's something silly. any help appreciated. thanks! regards, J -- || visit gfd || psa member #293 || New Image Systems & Services, Inc. From jam@n... Fri Sep 3 05:16:52 1999 From: jam@n... (jam) Date: Fri, 3 Sep 1999 00:16:52 -0400 Subject: [medusa] Re: trouble with 19990902 snapshot: asyncore.loop() exists immediately In-Reply-To: <19990902234508.A9613@t...>; from jam on Thu, Sep 02, 1999 at 11:45:08PM -0400 References: <19990902234508.A9613@t...> Message-ID: <19990903001652.B9613@t...> On Thu, Sep 02, 1999 at 11:45:08PM -0400, jam wrote: > > greetings, > > I've been beating my head against the wall for an hour or two trying to > figure out what I might be doing wrong, but to no avail.. hopefully someone > on the list can shed some light on this.. > > -- cut here -- > import sys > import asyncore > > if __name__ == "__main__": > from medusa import default_handler, filesys, http_server, resolver, logger > rs = resolver.caching_resolver("127.0.0.1") > lg = logger.file_logger(sys.stdout) > fs = filesys.os_filesystem(".") > dh = default_handler.default_handler(fs) > hs = http_server.http_server('toast', 8081, rs, lg) > hs.install_handler(dh) > asyncore.loop() > -- cut here -- > [..snipped..] I have been plugging away a little more at this problem, and when I change the 'from medusa import ...' line to include 'asyncore', and remove it from the top of the script, everything works *fine*. this is odd because I copied the 'asyncore.py' file from the distribution into /usr/lib/python1.5, so there really shouldn't be a problem, should there? can this be handled by adding something to the '__init__.py' file in the medusa directory? it appears I've tickled some wierd feature of the interpeter, or perhaps less of a feature or *something*. further comments, questions, flames, etc, welcomed. regards, J -- || visit gfd || psa member #293 || New Image Systems & Services, Inc. From rushing@n... Fri Sep 3 05:51:43 1999 From: rushing@n... (Sam Rushing) Date: Thu, 2 Sep 1999 21:51:43 -0700 (PDT) Subject: [medusa] Re: trouble with 19990902 snapshot: asyncore.loop() exists immediately In-Reply-To: <19990903001652.B9613@t...> References: <19990902234508.A9613@t...> <19990903001652.B9613@t...> Message-ID: <14287.21471.552306.353000@s...> jam writes: > I have been plugging away a little more at this problem, and when I change > the 'from medusa import ...' line to include 'asyncore', and remove it from > the top of the script, everything works *fine*. this is odd because I copied > the 'asyncore.py' file from the distribution into /usr/lib/python1.5, so > there really shouldn't be a problem, should there? Ok, I figured it out... the package system is letting you introduce two different versions of the 'asyncore' module. The one used for 'asyncore.loop()' is not the one that knows about listening sockets. Change the code thus: > hs.install_handler(dh) > print asyncore, id(asyncore) > print http_server.asyncore, id(http_server.asyncore) > asyncore.loop() And you get this: | rushing@s...:/usr/src/nm/apps$ python strange.py | info: adding channel | info: adding channel | info: Medusa (V3.35) started at Thu Sep 2 21:48:29 1999 | Hostname: seattle | Port:8081 | | 135185384 | 135223912 sneaky, but not really the package system's fault: the whole idea of a package system is that you shouldn't have to worry about name collisions between unrelated packages. -Sam From jam@n... Fri Sep 3 06:26:58 1999 From: jam@n... (jam) Date: Fri, 3 Sep 1999 01:26:58 -0400 Subject: [medusa] Re: trouble with 19990902 snapshot: asyncore.loop() exists immediately In-Reply-To: <14287.21471.552306.353000@s...>; from Sam Rushing on Thu, Sep 02, 1999 at 09:51:43PM -0700 References: <19990902234508.A9613@t...> <19990903001652.B9613@t...> <14287.21471.552306.353000@s...> Message-ID: <19990903012658.C9613@t...> On Thu, Sep 02, 1999 at 09:51:43PM -0700, Sam Rushing wrote: > > Ok, I figured it out... the package system is letting you introduce > two different versions of the 'asyncore' module. The one used for > 'asyncore.loop()' is not the one that knows about listening sockets. > you are correct. when I renamed the copies of 'asyncore' and 'asynchat' in the medusa directory the problem goes away. thanks for the help ;) regards, J -- || visit gfd || psa member #293 || New Image Systems & Services, Inc. From skip@m... Fri Sep 3 18:50:10 1999 From: skip@m... (Skip Montanaro) Date: Fri, 3 Sep 1999 12:50:10 -0500 Subject: [medusa] Re: hideously long selects? Message-ID: <199909031750.MAA10640@d...> I'm baaack... Jim Fulton asked: > What version of Zope are you running? I beleave that this is due to a > bug that was fixed in the Zope 2.0 beta 5. I was running 1.something. I downloaded the 2.0.0 source today and installed it, then modified my old start.py to work with the new ZServer. I tried it out a bit on my development machine. Things were *real slow*, but nothing was happening, so I applied strace (fast becoming one of my favorite debugging tools) to both my database server and the ZServer process. The results were instructive. The database server trace looks like: 12:12:36 select(6, [5], [], [], {2, 500000}) = 0 (Timeout) 12:12:39 select(6, [5], [], [], {2, 500000}) = 1 (in [5], left {0, 280000}) 12:12:41 accept(5, {sin_family=AF_INET, sin_port=htons(4321), sin_addr=inet_addr("127.0.0.1")}, [16]) = 12 ... chunka chunka ... 12:12:41 read(13, "POST /RPC2 HTTP/1.0\r\nHost: loc"..., 1024) = 43 12:12:41 read(13, "User-Agent: xmlrpclib.py/0.9.8 ("..., 1024) = 56 12:12:41 read(13, "Content-Type: text/xml\r\nConten"..., 1024) = 170 ... chunka chunka ... 12:12:41 write(14, "HTTP/1.0 200 OK\r\nServer: BaseH"..., 1024) = 1024 ... bunch more writes ... 12:12:41 close(15) = 0 ... a few munmaps ... 12:12:41 select(6, [5], [], [], {2, 500000}) = 0 (Timeout) ... So, within the space of somewhat less than a second it accepts a connect, handles the request (in this case an XML-RPC call requesting the top 50 cities and performers the server has gotten requests for), then returns to its waiting state. Now, lets have a look at the trace for the ZServer: 12:12:39 accept(5, {sin_family=AF_INET, sin_port=htons(4319), sin_addr=inet_addr("127.0.0.2")}, [16]) = 6 12:12:39 write(1, "info: adding channel ) = 72 12:12:39 fcntl(6, F_GETFL) = 0x2 (flags O_RDWR) 12:12:39 fcntl(6, F_SETFL, O_RDWR|O_NONBLOCK) = 0 12:12:39 gettimeofday({936378759, 269643}, NULL) = 0 12:12:39 gettimeofday({936378759, 270168}, NULL) = 0 12:12:39 select(7, [5 6], [], [], {30, 0}) = 1 (in [6], left {30, 0}) 12:12:39 recv(6, "GET /filter?file=index.shtml HTT"..., 4096, 0) = 277 12:12:39 gettimeofday({936378759, 273572}, NULL) = 0 12:12:39 brk(0x827b000) = 0x827b000 12:12:39 pipe([11, 12]) = 0 12:12:39 clone() = 8159 12:12:39 write(12, "4\24%@\4\0\0\0\201\0\0\0 \205$@`"..., 148) = 148 12:12:39 sigprocmask(SIG_SETMASK, NULL, [USR1]) = 0 12:12:39 write(12, "4\24%@\0\0\0\0\0\0\0\0T\370\7\10"..., 148) = 148 12:12:39 sigprocmask(SIG_SETMASK, NULL, [USR1]) = 0 12:12:39 sigsuspend([] 12:12:39 --- SIGUSR1 (User defined signal 1) --- 12:12:39 <... sigsuspend resumed> ) = -1 EINTR (Interrupted system call) 12:12:39 sigreturn() = ? (mask now [USR1]) 12:12:39 kill(8160, SIGUSR1) = 0 12:12:39 kill(8160, SIGUSR1) = 0 12:12:39 sigprocmask(SIG_SETMASK, NULL, [USR1]) = 0 12:12:39 sigsuspend([] 12:12:39 --- SIGUSR1 (User defined signal 1) --- 12:12:39 <... sigsuspend resumed> ) = -1 EINTR (Interrupted system call) 12:12:39 sigreturn() = ? (mask now [USR1]) 12:12:39 select(7, [5 6], [], [], {30, 0}) = 0 (Timeout) 12:13:09 select(7, [5 6], [6], [], {30, 0}) = 1 (out [6], left {30, 0}) 12:13:09 send(6, "HTTP/1.0 200 OK\r\nServer: Zope/"..., 13297, 0) = 13297 12:13:09 select(7, [5 6], [6], [], {30, 0}) = 1 (out [6], left {30, 0}) 12:13:09 gettimeofday({936378789, 284258}, NULL) = 0 12:13:09 write(4, "127.0.0.2:4319 - - [03/Sep/1999:"..., 98) = 98 12:13:09 write(1, "info: closing channel 6:) = 99 12:13:09 close(6) = 0 12:13:09 close(-1) = -1 EBADF (Bad file descriptor) 12:13:09 select(6, [5], [], [], {30, 0}) = 0 (Timeout) I assume the chit-chat with the backend database server is happening in a separate thread, because I see no reads or writes around 12:12:41 in the trace. At any rate, strace seems not to be able to attach to those threads, so their chit-chat is unavailable. Note the back-to-back selects at 12:12:39 and 12:13:09. First it has fd 6 only in the read set and the select timed out after 30 seconds. Then it added 6 to the write set for the second select which returned immediately, and it sent the response back to Apache (which is serving as a proxy in this case) and from there to the user. Why'd it sit on that first select for 30 seconds? Any reason why fd 6 wasn't in the write set to begin with? I presume it is the socket which it uses to talk to the backend server. (This would be in httplib or stuff it calls.) To make sure I was getting appropriate versions of important modules, I had start.py print out sys.path and the locations of those modules I thought most important when ZServer was started: sys.path is (ordered first to last): 0: /usr/local/lib/python1.5/site-packages 1: /home/dolphin/skip/src/Zope-2.0.0-src/ZServer/../lib/python 2: /home/dolphin/skip/src/Zope-2.0.0-src/ZServer 3: /home/dolphin/skip/src/Zope-2.0.0-src/ZServer/.. 4: /home/dolphin/skip/src/Zope-2.0.0-src/ZServer 5: /home/dolphin/skip/src/vtk-linux/lib 6: /usr/local/lib/python1.5/ 7: /usr/local/lib/python1.5/plat-linux2 8: /usr/local/lib/python1.5/lib-tk 9: /usr/local/lib/python1.5/lib-dynload 10: /usr/local/lib/automatrix/python Here are the locations of some key modules, as discovered by imp.find_module: asyncore is at /home/dolphin/skip/src/Zope-2.0.0-src/ZServer/medusa/asyncore.py asynchat is at /home/dolphin/skip/src/Zope-2.0.0-src/ZServer/medusa/asynchat.py xmlrpclib is at /usr/local/lib/python1.5/site-packages/xmlrpclib.py sgmlop is builtin Now, if I shorten up the timeout for asyncore.loop, the pregnant pause goes away, but there's still this fd 6 in the read set followed by fd 6 in both the read and write sets: 12:39:13 sigreturn() = ? (mask now [USR1]) 12:39:13 select(7, [5 6], [], [], {0, 800000}) = 0 (Timeout) 12:39:14 select(7, [5 6], [6], [], {0, 800000}) = 1 (out [6], left {0, 800000}) 12:39:14 send(6, "HTTP/1.0 200 OK\r\nServer: Zope/"..., 37112, 0) = 37112 I would agree that long timeouts are okay if we could just eliminate that first select. Am I hosed because I'm mixing blocking (httplib) and non-blocking (asyncore) I/O? Should httplib be telling asyncore about its sockets? Skip Montanaro | http://www.mojam.com/ skip@m... | http://www.musi-cal.com/~skip/ 847-971-7098 | Python: Programming the way Guido indented... From jam@n... Fri Sep 3 21:18:17 1999 From: jam@n... (jam) Date: Fri, 3 Sep 1999 16:18:17 -0400 Subject: [medusa] script_handler: possible optimization? Message-ID: <19990903161817.E9613@t...> greetings, I'm working on (the forth or fifth generation of) my website on quark, and I'm using the latest-and-greatest medusa framework to do so, because I noticed this great handler called persistent_script_handler that I *really* wanted to use ;) I have managed to get things up and running at least to the point that I don't have any odd problems (like the server exiting immediately because I had imported two different instances of the asyncore module). the next phase of this is that I need to somehow scale the size of the site up by a factor of several. I have two modules so far, and my primary design concern at the moment is 'how do I hang other modules from different places in the directory tree?'.. for example: /gfd/index.html - handled by 'main.py' - also might handle things like '/gfd/about.html' or '/gfd/resume.html' /gfd/links/index.html - the 'match' routine in the 'links' module will need to know where it hangs on the 'tree' in order to determine if it should handle the request or not. this is where things get complicated, especially if the site uses a lot of these modules. one thought I had was to simplify things by *not* wrapping lots of different functions into one module by splitting each 'page' into it's own module ('about.py' handles '/gfd/about.html', 'resume.py' would handle '/gfd/resume.*', etc.) but that *also* seems like it would get complicated fast. I do have a small contribution to add to the cause, which anyone may use as they see fit: I made a new version of the persistent_script_handler class (called 'gfd_script_handler', below) that calls a method called 'match' in each module to let *it* tell the framework if it can deal with the request or not. I also changed the add_module routine to make it simpler to make the 'autoreload' functionality a possibility in the future (it's pretty easy once you start using __import__ and the ihooks module). comments, suggestions, strange looks, etc, accepted. -- cut here -- #!/usr/bin/env python # # this code uses medusa, available from # import sys import asyncore from cStringIO import StringIO from medusa import status_handler, default_handler, filesys, http_server, counter from medusa import script_handler, resolver, logger import config class gfd_script_handler: def __init__(self): self.exceptions = counter.counter() self.modules = {} def add_module(self, key): moduleref = __import__(key) self.modules[key] = moduleref def del_module(self, key): del self.modules[key] def match(self, request): for m in self.modules.values(): if hasattr(m, "match"): if m.match(request) == 1: print "found match", id(m) request.module = m return 1 return 0 def handle_request(self, request): if request.command in ( "put", "post"): cl = request.get_header("content-length") length = int(cl) if not cl: request.error(411) else: collector(self, length, request) else: self.continue_request(request, StringIO()) def continue_request(self, request, input_data): temp_files = input_data, StringIO(), StringIO() old_files = sys.stdin, sys.stdout, sys.stderr try: sys.stdin, sys.stdout, sys.stderr = temp_files request["content-type"] = "text/html" try: request.module.main(request) request.reply_code = 200 except: request.reply_code = 500 self.exceptions.increment() finally: sys.stdin, sys.stdout, sys.stderr = old_files i, o, e = temp_files if request.reply_code != 200: s = e.getvalue() else: s = o.getvalue() request["content-length"] = len(s) request.push(s) request.done() if __name__ == "__main__": sys.path.append("/home/jam/projects/thpwact/modules/") rs = resolver.resolver("neutron") lg = logger.file_logger(sys.stdout) hs = http_server.http_server("toast", 8080, rs, lg) # # the 'default_handler' tries to cover requests that aren't # handled by the script_handler (like /images) # fs = filesys.os_filesystem(config.SOFTROOT) dh = default_handler.default_handler(fs) hs.install_handler(dh) gsh = gfd_script_handler() gsh.add_module("about") gsh.add_module("main") hs.install_handler(gsh) sh = status_handler.status_extension( (hs, lg, gsh) ) hs.install_handler(sh) asyncore.loop() -- cut here -- then, as an example, here's my 'about.py' module, which uses DocumentTemplate and a few things from the 'config' module (which I'm not including lest I get shot for such a huge post ;)): -- cut here -- _RCSID = "$Id: $" import os import common import config def match(request): path, params, query, fragment = request.split_uri() parts = os.path.split(path) ## print "about: ", parts if len(parts) > 1 and parts[1] == "about.html": return 1 else: return 0 def main(request): print common.standard_header(title="about this site") print common.HTMLFile(config.DTMLROOT + "about.dtml")() print common.standard_footer() if __name__ == "__main__": # for testing main(None) -- cut here -- regards, J -- || visit gfd || psa member #293 || New Image Systems & Services, Inc. From jam@n... Tue Sep 7 15:27:42 1999 From: jam@n... (jam) Date: Tue, 7 Sep 1999 10:27:42 -0400 Subject: [medusa] running medusa in the background without exit Message-ID: <19990907102742.Q9613@t...> greetings! I've been using the 1999-09-02 rev of medusa for the last few days, and it's working *great*. the question I have, though, since medusa exits when it gets a SIGHUP, is how do I keep the server running in the background even when I log out? I have set up apache to redirect all requests on port 80 to the medusa server on port 8080.. this is fine when I start medusa from my shell prompt and let it go, and I could use 'nohup' to prevent it from exiting, but then I wouldn't be able to restart it without a fuss (something I need to be able to do until the 'reload' of modules on-the-fly is working. how do other folks handle this situation? any suggestions? regards, J -- || visit gfd || psa member #293 || New Image Systems & Services, Inc. From jim@d... Tue Sep 7 15:40:19 1999 From: jim@d... (Jim Fulton) Date: Tue, 07 Sep 1999 10:40:19 -0400 Subject: [medusa] Re: running medusa in the background without exit References: <19990907102742.Q9613@t...> Message-ID: <37D523D3.AA68C437@d...> jam wrote: > > greetings! > > I've been using the 1999-09-02 rev of medusa for the last few days, and it's > working *great*. > > the question I have, though, since medusa exits when it gets a SIGHUP, is > how do I keep the server running in the background even when I log out? I > have set up apache to redirect all requests on port 80 to the medusa server > on port 8080.. this is fine when I start medusa from my shell prompt and let > it go, and I could use 'nohup' to prevent it from exiting, but then I > wouldn't be able to restart it without a fuss (something I need to be able > to do until the 'reload' of modules on-the-fly is working. > > how do other folks handle this situation? any suggestions? The Zope distribution (www.zope.org) includes a module, zdaemon, for *easily* creating Unix daemons, which is what you want to do. We plan, eventually, to release this separately for use by other Python programs. In the mean time, feel free to grab it and use it. For an example of it's use, see the Zope ZServer startup script, z2.py. Jim -- Jim Fulton mailto:jim@d... Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From jam@n... Tue Sep 7 15:57:58 1999 From: jam@n... (jam) Date: Tue, 7 Sep 1999 10:57:58 -0400 Subject: [medusa] Re: running medusa in the background without exit In-Reply-To: <37D523D3.AA68C437@d...>; from Jim Fulton on Tue, Sep 07, 1999 at 10:40:19AM -0400 References: <19990907102742.Q9613@t...> <37D523D3.AA68C437@d...> Message-ID: <19990907105758.R9613@t...> On Tue, Sep 07, 1999 at 10:40:19AM -0400, Jim Fulton wrote: > > The Zope distribution (www.zope.org) includes a module, zdaemon, > for *easily* creating Unix daemons, which is what you want to do. > We plan, eventually, to release this separately for use by other > Python programs. In the mean time, feel free to grab it and use it. > For an example of it's use, see the Zope ZServer startup script, > z2.py. > > Jim > wow! thanks for the quick response! ;).. I'll give zope a shot. regards, J -- || visit gfd || psa member #293 || New Image Systems & Services, Inc. From Paul@d... Wed Sep 8 19:30:33 1999 From: Paul@d... (Paul Everitt) Date: Wed, 8 Sep 1999 14:30:33 -0400 Subject: [medusa] Q: Replacement URL for case study? Message-ID: <613145F79272D211914B0020AFF64019262A20@g...> Skip-- We have a link to concerts.calendar.com on the zope.org site. What should it be linked to these days? --Paul From Paul@d... Wed Sep 8 19:31:29 1999 From: Paul@d... (Paul Everitt) Date: Wed, 8 Sep 1999 14:31:29 -0400 Subject: [medusa] OOPS: My apologies Message-ID: <613145F79272D211914B0020AFF64019262A21@g...> Sorry for the email! I'm sure there are, errm, more direct ways to reach Skip. :^) --Paul From skip@m... Wed Sep 8 19:34:26 1999 From: skip@m... (Skip Montanaro) Date: Wed, 8 Sep 1999 13:34:26 -0500 (CDT) Subject: [medusa] Re: Q: Replacement URL for case study? In-Reply-To: <613145F79272D211914B0020AFF64019262A20@g...> References: <613145F79272D211914B0020AFF64019262A20@g...> Message-ID: <14294.43990.30179.213234@d...> Paul> We have a link to concerts.calendar.com on the zope.org site. Paul> What should it be linked to these days? Paul, Hopefully, concerts.calendar.com will be back on the air shortly, but the best place to link to is http://www.musi-cal.com/. We sold the calendar.com domain awhile back (tough to refuse cash you know) and are now using musi-cal.com. Skip From jkraai@m... Fri Sep 10 15:31:49 1999 From: jkraai@m... (jkraai@m...) Date: Fri, 10 Sep 1999 07:31:49 -0700 Subject: [medusa] Job: Async Message-ID: <7rb4ol$c4v8@e...> [Reposted from C.L.P.] Need someone interested in six month contract using Python. Two types of projects. 1. Implementing an HTTP'ish protocol via Sam Rushing's fine Async/Async Chat libraries. 2. Integration of existing C/C++ routines into Python core. We're located in Iowa. We are not asking for relocation. Demonstrable track record with async libs is required. Please reply with resume, rates, availability, and examples of successful projects. Thanks for the use of the list & for the great language & for the fantastic modules & for the beneavolent community & for ... Sincerely, --jim kraai President, MURL.COM From wkp@c... Wed Nov 10 23:49:53 1999 From: wkp@c... (wkp@c...) Date: Wed, 10 Nov 1999 15:49:53 -0800 Subject: [medusa] cookies Message-ID: <80d0b1$8bir@e...> I've been doing some CGI scripting with Python and Medusa. It's great! However, I'm not sure how to set and get cookies within my server-side scripts. Does anyone have any examples? Wesley Phoa. From robin@j... Sun Nov 14 12:53:41 1999 From: robin@j... (Robin Becker) Date: Sun, 14 Nov 1999 12:53:41 +0000 Subject: [medusa] strangeness in zope's version of ftpserver.py Message-ID: I'm using the medusa version distributed with the latest Zope and notice that ftpserver.py seems to have two versions of the close method for ftp_channel. Reason I'm interested is that I'm using the ftp server and seem to observe that abruptly killing an ftp client seems not to increment the closed_sessions_total counter thus leaving the open sessions count at 1 even when I don't have an active client. Has the session actually been closed? -- Robin Becker From rushing@n... Mon Nov 15 02:23:13 1999 From: rushing@n... (Sam Rushing) Date: Sun, 14 Nov 1999 18:23:13 -0800 (PST) Subject: [medusa] strangeness in zope's version of ftpserver.py In-Reply-To: References: Message-ID: <14383.28305.291751.634550@s...> Robin Becker writes: > I'm using the medusa version distributed with the latest Zope and notice > that ftpserver.py seems to have two versions of the close method for > ftp_channel. > > Reason I'm interested is that I'm using the ftp server and seem to > observe that abruptly killing an ftp client seems not to increment the > closed_sessions_total counter thus leaving the open sessions count at 1 > even when I don't have an active client. > > Has the session actually been closed? You can tell by hitting the 'channel list' link from the status page (does zope support the status page? see status_handler.py). It looks like there are two versions of the close method, each doing a little bit of the job. 8^) They most likely need to be made into a single version! -Sam From robin@j... Mon Nov 15 11:31:55 1999 From: robin@j... (Robin Becker) Date: Mon, 15 Nov 1999 11:31:55 +0000 Subject: [medusa] Re: strangeness in zope's version of ftpserver.py In-Reply-To: <14383.28305.291751.634550@s...> References: <14383.28305.291751.634550@s...> Message-ID: <972yTAAr8+L4EwlA@j...> In article <14383.28305.291751.634550@s...>, Sam Rushing writes >Robin Becker writes: > > I'm using the medusa version distributed with the latest Zope and notice > > that ftpserver.py seems to have two versions of the close method for > > ftp_channel. > > > > Reason I'm interested is that I'm using the ftp server and seem to > > observe that abruptly killing an ftp client seems not to increment the > > closed_sessions_total counter thus leaving the open sessions count at 1 > > even when I don't have an active client. > > > > Has the session actually been closed? > >You can tell by hitting the 'channel list' link from the status page >(does zope support the status page? see status_handler.py). > >It looks like there are two versions of the close method, each doing a >little bit of the job. 8^) They most likely need to be made into a >single version! > >-Sam > > ... I have hacked in a page which uses the status stuff. I don't know about the channel list though. I'll have to check that out. -- Robin Becker From robin@j... Mon Nov 15 12:30:05 1999 From: robin@j... (Robin Becker) Date: Mon, 15 Nov 1999 12:30:05 +0000 Subject: [medusa] Re: strangeness in zope's version of ftpserver.py In-Reply-To: <972yTAAr8+L4EwlA@j...> References: <14383.28305.291751.634550@s...> <972yTAAr8+L4EwlA@j...> Message-ID: In article <972yTAAr8+L4EwlA@j...>, Robin Becker writes ... >... >I have hacked in a page which uses the status stuff. I don't know about >the channel list though. I'll have to check that out. OK I checked my working remote zope with a modified status page and it seems as though the original zope ftp_server doesn't react to aborted ftp clients ie it leaves ftp_channels hanging. I'm not certain whether the recently posted ftp_server.py will fix this; I will install on my system and check. I'll need to restart the remote though. -- Robin Becker From rushing@n... Tue Nov 16 07:58:32 1999 From: rushing@n... (Sam Rushing) Date: Mon, 15 Nov 1999 23:58:32 -0800 (PST) Subject: [medusa] cookies In-Reply-To: <80d0b1$8bir@e...> References: <80d0b1$8bir@e...> Message-ID: <14385.3752.482888.238729@s...> wkp@c... writes: > I've been doing some CGI scripting with Python and Medusa. It's great! > However, I'm not sure how to set and get cookies within my server-side > scripts. Does anyone have any examples? You just need to monkey with the 'Cookie:' and 'Set-Cookie:' headers. If you are using the 'persistent_script_handler', then the request object is passed back to your module's main() function. You can access the request headers using 'request.get_header()' and you can set reply headers like this: request['Set-Cookie'] = "xxx" Here's a starting reference: http://www.netscape.com/newsref/std/cookie_spec.html -Sam From rushing@e... Fri Nov 19 11:38:55 1999 From: rushing@e... (Sam Rushing) Date: Fri, 19 Nov 1999 03:38:55 -0800 (PST) Subject: [medusa] tweaks to asyn{core,chat}.py Message-ID: <14389.14031.176921.629212@s...> I've been trying to tighten up the async modules a bit to fight an effect that shows up with lots of concurrent connections (>500). I've made some notes: http://www.nightmare.com/medusa/async_tweaks.html I think the changes are a win, so I'll be checking in this code over the next day or so. If anyone out there is violating the asyncore.socket_map abstraction (relying on the way the dictionary is used) or sees a problem I've missed, please let me know. I think the only code in medusa that cares is the stuff in http_server that kills zombie connections. Zope may have done something similar with the ftp server? -Sam From robin@j... Fri Nov 19 13:37:33 1999 From: robin@j... (Robin Becker) Date: Fri, 19 Nov 1999 13:37:33 +0000 Subject: [medusa] Re: tweaks to asyn{core,chat}.py In-Reply-To: <14389.14031.176921.629212@s...> References: <14389.14031.176921.629212@s...> Message-ID: In article <14389.14031.176921.629212@s...>, Sam Rushing writes >I've been trying to tighten up the async modules a bit to fight an >effect that shows up with lots of concurrent connections (>500). > >I've made some notes: > > http://www.nightmare.com/medusa/async_tweaks.html > >I think the changes are a win, so I'll be checking in this code over >the next day or so. If anyone out there is violating the >asyncore.socket_map abstraction (relying on the way the dictionary is >used) or sees a problem I've missed, please let me know. > >I think the only code in medusa that cares is the stuff in http_server >that kills zombie connections. Zope may have done something similar >with the ftp server? > >-Sam > ... I've been looking at how zombies are handled in Zope. seems that zope has an zhttp_channel analogous to http_channel; it has the zombie stuff, but without the maintainance thing so the kill_zombies method doesn't seem to get called. I noticed that ftp_server doesn't have a zombie timeout is there an obvious reason why not? -- Robin Becker From amos@a... Fri Nov 19 20:09:17 1999 From: amos@a... (Amos Latteier) Date: Fri, 19 Nov 1999 12:09:17 -0800 Subject: [medusa] Re: tweaks to asyn{core,chat}.py In-Reply-To: References: <14389.14031.176921.629212@s...> <14389.14031.176921.629212@s...> Message-ID: <3.0.5.32.19991119120917.015b83a0@m...> Robin Becker wrote: >Sam Rushing wrote: >>I've been trying to tighten up the async modules a bit to fight an >>effect that shows up with lots of concurrent connections (>500). >> >>I've made some notes: >> >> http://www.nightmare.com/medusa/async_tweaks.html >> >>I think the changes are a win, so I'll be checking in this code over >>the next day or so. If anyone out there is violating the >>asyncore.socket_map abstraction (relying on the way the dictionary is >>used) or sees a problem I've missed, please let me know. >> >>I think the only code in medusa that cares is the stuff in http_server >>that kills zombie connections. Zope may have done something similar >>with the ftp server? A few small changes will need to be made to Zope to accommodate this change to asycore.socket_map, but not many. I talked with Jim Fulton about ideas to improve asyncore/asychat select loop performance a while back. Let's see if I can remember some of his ideas. (Note, I may have gotten it slightly wrong, sorry Jim ;-) One problem Zope has with asyncore results from using worker threads in addition to a medusa thread. There are problems when the worker thread is ready to go, but the medusa thread is sitting in a select call, which potentially can take up to 30 secs to return. We get around this by using a select trigger to wake up select, but we would like to get rid of the need for a select trigger. One way to achieve this is to reduce the timeout on the select call to a small enough value that the worker threads can wait for a timeout without being overly inconvenienced. This would require the select loop being adequately optimized. One optimization Jim suggested is to pass integers to the select loop, not descriptor objects. I think that this is what you are suggesting as 'fdcache' optimization. Another optimization would be to have a system of re-entering the select call immediately after returning if nothing is ready--rather than checking the results and rebuilding the arguments to the select call. >... >I've been looking at how zombies are handled in Zope. seems that zope >has an zhttp_channel analogous to http_channel; it has the zombie stuff, >but without the maintainance thing so the kill_zombies method doesn't >seem to get called. I'm pretty sure that it *does* get called. zhttp_channel calls check_maintenance upon initialization. > I noticed that ftp_server doesn't have a zombie >timeout is there an obvious reason why not? I've wondered about this too. -Amos From rushing@n... Sat Nov 20 03:31:24 1999 From: rushing@n... (Sam Rushing) Date: Fri, 19 Nov 1999 19:31:24 -0800 (PST) Subject: [medusa] Re: tweaks to asyn{core,chat}.py In-Reply-To: <3.0.5.32.19991119120917.015b83a0@m...> References: <14389.14031.176921.629212@s...> <3.0.5.32.19991119120917.015b83a0@m...> Message-ID: <14390.5644.955061.401898@s...> Amos Latteier writes: > One problem Zope has with asyncore results from using worker > threads in addition to a medusa thread. There are problems when the > worker thread is ready to go, but the medusa thread is sitting in a > select call, which potentially can take up to 30 secs to return. We > get around this by using a select trigger to wake up select, but we > would like to get rid of the need for a select trigger. Lowering the timeout value just aggravates the CPU-abuse problem... imagine if you had a timeout of zero, then the CPU would stay pegged even if the server were completely idle. Why don't you like the trigger? For the uninitiated, the 'trigger' is a socket/descriptor that is always in the readable set for select. When select() is otherwise 'hung' waiting for a timeout, another thread can 'pull the trigger' and wake up the select call. On Unix, signals can also force select() to return, but it's not portable. > One way to achieve this is to reduce the timeout on the select call > to a small enough value that the worker threads can wait for a > timeout without being overly inconvenienced. This would require the > select loop being adequately optimized. Without a redesign that eliminated the calls to readable() and writable() I think we're stuck. Curiously, the cleanest 'redesign' that achieves this effect is to go to coroutines/fibers. But that's a whole 'nother paradigm... [coroutines combined with completion ports would be the real killer] > One optimization Jim suggested is to pass integers to the select > loop, not descriptor objects. I think that this is what you are > suggesting as 'fdcache' optimization. Waaay back (maybe last year) I did something similar, but I used a separate descriptor-map dictionary. It worked well, but would sometimes run into a problem where the one map wasn't in sync with the other. Now they're the same map... much cleaner. But there are other areas that need tweaking, the poll() overhead is no long on top of the list. I have modified a python here to do a type of statistical profiling (every VM insns a dictionary of {:} is updated). Here's what made me look at refill_buffer(): [...] 709 713 748 751 870 948 1019 1421 1582 1739 3258 4205 4211 4970 > Another optimization would be to have a system of re-entering the > select call immediately after returning if nothing is ready--rather > than checking the results and rebuilding the arguments to the > select call. This is a really good idea! it's getting into an interesting area, of trying to be smarter about scheduling the r/w sets. I think there are some things like that to be found here: http://www.kegel.com/c10k.html A problem I see is for users of the event_loop class, which supports timers. It would be possible (nay, likely) for a timer to modify the readable/writable condition for one or more objects, which would break the above fix. > > I noticed that ftp_server doesn't have a zombie > >timeout is there an obvious reason why not? > > I've wondered about this too. Lazy Sam. Every time I go to add it, I think - "there should be a general facility for this" and of course that allows me to put it off for a while. Lather. Rinse. Repeat. -Sam From robin@j... Sat Nov 20 11:06:56 1999 From: robin@j... (Robin Becker) Date: Sat, 20 Nov 1999 11:06:56 +0000 Subject: [medusa] Re: tweaks to asyn{core,chat}.py In-Reply-To: <3.0.5.32.19991119120917.015b83a0@m...> References: <14389.14031.176921.629212@s...> <14389.14031.176921.629212@s...> <3.0.5.32.19991119120917.015b83a0@m...> Message-ID: In article <3.0.5.32.19991119120917.015b83a0@m...>, Amos Latteier writes ... > >>... >>I've been looking at how zombies are handled in Zope. seems that zope >>has an zhttp_channel analogous to http_channel; it has the zombie stuff, >>but without the maintainance thing so the kill_zombies method doesn't >>seem to get called. > >I'm pretty sure that it *does* get called. zhttp_channel calls >check_maintenance upon initialization. OK I checked and you are right, but since the kill_zombies only happens every 500 creations things can persist a long time. My mistake. I guess in the worst case a large number of sockets (<500) could be zombies which would be bad for systems with small fd limits. > >> I noticed that ftp_server doesn't have a zombie >>timeout is there an obvious reason why not? > >I've wondered about this too. > -- Robin Becker From skip@m... Sat Nov 20 13:25:55 1999 From: skip@m... (Skip Montanaro) Date: Sat, 20 Nov 1999 07:25:55 -0600 (CST) Subject: [medusa] Re: tweaks to asyn{core,chat}.py In-Reply-To: <14390.5644.955061.401898@s...> References: <14389.14031.176921.629212@s...> <3.0.5.32.19991119120917.015b83a0@m...> <14390.5644.955061.401898@s...> Message-ID: <14390.41315.807375.200241@d...> Sam> But there are other areas that need tweaking, the poll() overhead Sam> is no long on top of the list. I have modified a python here to do Sam> a type of statistical profiling (every VM insns a Sam> dictionary of {:} is updated). Here's what Sam> made me look at refill_buffer(): Sam> [...] Sam> 4211 Sam> 4970 Have you sent this to Guido? A statistical profiler would be a nice complement to the regular (machine-eating) profiler. If you have a patch, I'd like to take a look at it. (I'd probably change the simple counter to a time accumulator the way gprof works.) >> Another optimization would be to have a system of re-entering the >> select call immediately after returning if nothing is ready--rather >> than checking the results and rebuilding the arguments to the select >> call. Sam> This is a really good idea! it's getting into an interesting area, Sam> of trying to be smarter about scheduling the r/w sets. I think Sam> there are some things like that to be found here: I've experienced cases where is sits for the timeout period (typically 30 seconds), then adds a fd that was in the read set to the write set as well (or was it the other way 'round?), after which the next select returns immediately. Skip Montanaro | http://www.mojam.com/ skip@m... | http://www.musi-cal.com/ 847-971-7098 | Python: Programming the way Guido indented... From rushing@n... Sat Nov 20 13:52:51 1999 From: rushing@n... (Sam Rushing) Date: Sat, 20 Nov 1999 05:52:51 -0800 (PST) Subject: [medusa] Re: tweaks to asyn{core,chat}.py In-Reply-To: <14390.41315.807375.200241@d...> References: <14389.14031.176921.629212@s...> <3.0.5.32.19991119120917.015b83a0@m...> <14390.5644.955061.401898@s...> <14390.41315.807375.200241@d...> Message-ID: <14390.42931.203540.780616@s...> Skip Montanaro writes: > Have you sent this to Guido? A statistical profiler would be a > nice complement to the regular (machine-eating) profiler. If you > have a patch, I'd like to take a look at it. (I'd probably change > the simple counter to a time accumulator the way gprof works.) Warning: This code ignores reference-counting. (not that there's anything wrong with that)... I've only used it once. I think it could be very useful for long-running servers, where 'real' profiling is tricky and expensive. This seems to me to have very low overhead; especially if I "sys.setcheckinterval(1000)" rather than 10. Python-1.5.2/Python/ceval.c: 379a380,384 > #ifdef STATISTICAL_PROFILING > static PyObject * profiling = NULL; > #endif > > 620a626,644 > #ifdef STATISTICAL_PROFILING > if (!profiling) { > profiling = PyDict_New(); > { > PyObject * sys_module = PyDict_GetItemString (PyImport_GetModuleDict(), "sys"); > PyObject * sys_dict = PyModule_GetDict (sys_module); > PyDict_SetItemString (sys_dict, "statistical_profiling_data", profiling); > } > } > { > PyObject * val = PyDict_GetItem (profiling, (PyObject *) co); > if (!val) { > PyDict_SetItem (profiling, (PyObject *) co, PyInt_FromLong (1)); > } else { > PyDict_SetItem (profiling, (PyObject *) co, PyInt_FromLong (PyInt_AsLong (val) + 1)); > } > } > #endif > > I've experienced cases where is sits for the timeout period > (typically 30 seconds), then adds a fd that was in the read set to > the write set as well (or was it the other way 'round?), after > which the next select returns immediately. You're saying there are two trips through poll() when there should be one? Or did a socket not 'fire' when it should have? -Sam From skip@m... Sat Nov 20 14:12:44 1999 From: skip@m... (Skip Montanaro) Date: Sat, 20 Nov 1999 08:12:44 -0600 (CST) Subject: [medusa] Re: tweaks to asyn{core,chat}.py In-Reply-To: <14390.42931.203540.780616@s...> References: <14389.14031.176921.629212@s...> <3.0.5.32.19991119120917.015b83a0@m...> <14390.5644.955061.401898@s...> <14390.41315.807375.200241@d...> <14390.42931.203540.780616@s...> Message-ID: <14390.44124.353420.835913@d...> Thanks for the profiling code. >> I've experienced cases where is sits for the timeout period >> (typically 30 seconds), then adds a fd that was in the read set to >> the write set as well (or was it the other way 'round?), after >> which the next select returns immediately. Sam> You're saying there are two trips through poll() when there should Sam> be one? Or did a socket not 'fire' when it should have? I dunno. I've only experienced it in test situations, where I feed a server log file back to the server as fast as possible, so I've not paid much attention to it. (Also, note that I'm using a somewhat oldish version of ZServer. YMMV.) I imagine you have something similar, but just in case, here's my beat-server.py script. It takes a medusa/ZServer log file on stdin and throws requests at the server given by the -h argument (default, localhost). by default, it runs 5 simultaneous threads but this can be adjusted up or down with the -t flag. Note the wierd time.sleep call at the bottom of the main loop. That was my feeble attempt to keep the server from getting into this catatonic state where it would only wake up and process one request every 30 seconds. Skip --[[application/octet-stream Content-Disposition: attachment; filename="beat-server.py"][quoted-printable]] #!/usr/bin/env python import sys, thread, getopt, string, httplib, urlparse, time, re, random from threading import * def get_page(url, sema, verbose): now =3D time.asctime(time.localtime(time.time())) if verbose =3D=3D 2: sys.stderr.write("%s: %s -> " % (now, url)); sys.stderr.flush() try: t =3D time.time() scheme, loc, path, params, query, frag =3D urlparse.urlparse(url)= pfx =3D scheme + "://" + loc h =3D httplib.HTTP(loc) server =3D string.split(loc, ":")[0] h.putrequest('GET', url[len(pfx):]) h.putheader('Accept', '*/*') h.putheader('Host', server) h.endheaders() reply, msg, hdrs =3D h.getreply() bytes =3D len(h.getfile().read()) t =3D time.time() - t except: if not verbose: sys.stderr.write("%s -> " % (url)); sys.stderr.flush() sys.stderr.write("failed\n"); sys.stderr.flush() time.sleep(2) else: if verbose: sys.stderr.write("%d (%d bytes, %.2f seconds)" % (reply, byte= s, t)) if verbose < 2 and (reply >=3D 400 or t > 10): sys.stderr.write(" %s" % (url)); sys.stderr.flush() sys.stderr.write("\n") sys.stderr.flush() sema.release() def build_url_from_log(loc, pat, f): line =3D f.readline() while line: m =3D pat.search(line) if m: url =3D "http://%s%s" % (loc, m.group(1)) return url line =3D f.readline() return line def main(): optlist, args =3D getopt.getopt(sys.argv[1:], "t:h:qv") maxthreads =3D 5 proxy =3D "" loc =3D "localhost" verbose =3D 1 for opt, arg in optlist: if opt =3D=3D "-t": maxthreads =3D string.atoi(arg) elif opt =3D=3D "-h": loc =3D arg elif opt =3D=3D "-q": verbose =3D 0 elif opt =3D=3D "-v": verbose =3D 2 = sema =3D Semaphore(maxthreads) pathpat =3D re.compile('GET\s([^\s]*)\s') try: while 1: url =3D build_url_from_log(loc, pathpat, sys.stdin) if not url: break sema.acquire() Thread(target=3Dget_page, args=3D(url, sema, verbose)).start(= ) time.sleep(random.uniform(0, 1.3)) except KeyboardInterrupt: return if __name__ =3D=3D "__main__": main() From rushing@n... Sun Nov 21 02:30:43 1999 From: rushing@n... (Sam Rushing) Date: Sat, 20 Nov 1999 18:30:43 -0800 (PST) Subject: [medusa] Re: tweaks to asyn{core,chat}.py In-Reply-To: <14390.44124.353420.835913@d...> References: <14389.14031.176921.629212@s...> <3.0.5.32.19991119120917.015b83a0@m...> <14390.5644.955061.401898@s...> <14390.41315.807375.200241@d...> <14390.42931.203540.780616@s...> <14390.44124.353420.835913@d...> Message-ID: <14391.22867.379464.147090@s...> Skip Montanaro writes: > Sam> You're saying there are two trips through poll() when there should > Sam> be one? Or did a socket not 'fire' when it should have? > > I dunno. I've only experienced it in test situations, where I feed a server > log file back to the server as fast as possible, so I've not paid much > attention to it. (Also, note that I'm using a somewhat oldish version of > ZServer. YMMV.) This could very well be a problem with listen() queue size. This always shows up in testing situations. Most code uses listen(5) as a default (I used to use 5 because I think some winsock stacks would complain if you used a higher value?)... but it should be as high as the greatest number of sockets you can handle. I've been defaulting to 1024 lately, which should be enough for most purposes. > I imagine you have something similar, but just in case, here's my > beat-server.py script. It takes a medusa/ZServer log file on stdin and > throws requests at the server given by the -h argument (default, localhost). > By default, it runs 5 simultaneous threads but this can be adjusted up or > down with the -t flag. Note the wierd time.sleep call at the bottom of the > main loop. That was my feeble attempt to keep the server from getting into > this catatonic state where it would only wake up and process one request > every 30 seconds. I've just grepped through Zope-2.0.1/ZServer, and it looks like the http server is using listen(1024), but I'd double-check your distribution. 309: self.listen(256) ./PCGIServer.py 537: self.listen (1024) ./medusa/http_server.py -Sam From sjz18@y... Wed Nov 24 04:34:25 1999 From: sjz18@y... (Stuart Zakon) Date: Tue, 23 Nov 1999 20:34:25 -0800 Subject: [medusa] any performance info? Message-ID: <81fpsh$7pka@e...> Hi, I am new to Medusa and have been intrigued by the difference in the asyncore model from the multi-thread model I've worked with for 10 years. From rushing@n... Wed Nov 24 05:11:07 1999 From: rushing@n... (Sam Rushing) Date: Tue, 23 Nov 1999 21:11:07 -0800 (PST) Subject: [medusa] any performance info? In-Reply-To: <81fpsh$7pka@e...> References: <81fpsh$7pka@e...> Message-ID: <14395.29547.740497.50771@s...> Stuart Zakon writes: > Hi, I am new to Medusa and have been intrigued by the difference in > the asyncore model from the multi-thread model I've worked with for > 10 years. From what I understand the model works well when the > units of work can be broken into relatively uniform durations. > Is the unit of work an HTTP exchange? The base 'unit' might be considered the http request, at least that's the way the http server in the distribution is designed. However, the actual I/O is done 'on demand'... a threaded perspective might say that the I/O is handled in its own thread[s]. Some systems based on Medusa take different approaches - for example Zope uses a thread pool to handle requests, but lets medusa handle I/O asynchronously. This frees up the precious resource of threads to do actual work, and lets the back end handle all those slow clients. > Are there performance comparisons of Medusa's asyncore model with web > servers using multi-threaded models? Under heavy load? The very best web servers (which in general provide performance waaaay over what the average user needs) combine the two approaches; for example using one thread per CPU, each doing event-driven i/o. Here are a couple of good references: http://www.acme.com/software/thttpd/benchmarks.html http://www.kegel.com/c10k.html Medusa can't handle the kind of loads that thttpd and Zeus can - for example, the file delivering code uses stdio, and... well, it *is* written in Python. 8^) But most people don't need 1000 hits/sec. [let's see.. that's 86 million hits/day?] One big problem with the unix select/poll model is that it *does* get inefficient once the # of connections gets large... it would be nice if unix were to add something like completion ports. I think there are folks adding an equivalent facility to Linux 2.3 [of course, it has to be completely different, how embarrassing to copy something from NT. 8^)] > Also, I heard that egroups uses the asyncore library to achieve > scalability. Is this so? Does egroups use Medusa? We have several scalable systems using Medusa for various things - usually we have a custom system built on top of asyn{core,chat} which uses the rest of medusa for a web interface. Probably the biggest 'show-off' piece is our mailing-list exploder, which juggles up to 10,000 SMTP client connections simultaneously. [actually, that's not the maximum it's capable of, we're just not fond of pushing the OS to the breaking point. 8^) The servers are configured with an FD_SETSIZE of 16K] These servers are capable of pushing 6-8 million msgs/day. We also have customized RPC servers, proxies, smtp servers, and other things based on the async lib. eGroups does not use Medusa for the front-end, though - all of the UI code is written in straightforward Python, and is not amenable to the async model [all our UI programmers would have to know how to do nasty callbacks, state machines, etc... to pull this off] Lately we have been migrating some of our back-end systems toward a coroutine-based solution. This combines the scalability/efficiency of async I/O with the ability to program straight-line code. It feels a lot like a multi-threaded system, but without the need to use locks. [some of our engineers call it 'co-operative multitasking'] We're in the process of releasing this code to the public. -Sam From sjz18@y... Fri Nov 26 17:08:47 1999 From: sjz18@y... (Stuart Zakon) Date: Fri, 26 Nov 1999 09:08:47 -0800 Subject: [medusa] Re: any performance info? In-Reply-To: <14395.29547.740497.50771@s...> Message-ID: <81meqv$5etq@e...> Sam, Thanks for the performance info. I've also pulled down some of the research papers (Proactor, Reactor patterns) which have a lot of the background. *So I don't have to dig down too deep myself, how do thttpd and Zeus achieve greater performance if they use an asynch scheme? Is it because they are closer to the kernel? *Has anybody done anything with asynch for reverse DNS lookups? *Is your overview "Programming in Python with Medusa and the Async Sockets Library", which I found very useful, directly accessible on a web site without having to open the distribution? (Would be helpful for linking to from other pages.) *Does your mailing list exploder use the poll() approach? Thanks, Stuart From rushing@n... Fri Nov 26 21:00:53 1999 From: rushing@n... (Sam Rushing) Date: Fri, 26 Nov 1999 13:00:53 -0800 (PST) Subject: [medusa] Re: any performance info? In-Reply-To: <81meqv$5etq@e...> References: <14395.29547.740497.50771@s...> <81meqv$5etq@e...> Message-ID: <14398.62725.578358.703796@s...> Stuart Zakon writes: > *So I don't have to dig down too deep myself, how do thttpd and > Zeus achieve greater performance if they use an asynch scheme? Is > it because they are closer to the kernel? The async approach allows them to mix I/O with computation. Serving static pages is one extreme... there's no computation at all, just network I/O. And it all takes place in 'parallel': making 30 non-blocking connect() calls is the equivalent of saying "connect these 30 sockets to these 30 hosts, and let me know when they get through". > *Has anybody done anything with asynch for reverse DNS lookups? I think Google uses async DNS lookups in their crawler. Newman (the eGroups mailing-list exploder) does as well. > *Is your overview "Programming in Python with Medusa and the Async > Sockets Library", which I found very useful, directly accessible on a > web site without having to open the distribution? (Would be helpful for > linking to from other pages.) This will work: http://www.nightmare.com:8080/nm/apps/medusa/docs/programming.html I think a few other people are linking to it in that location as well. > *Does your mailing list exploder use the poll() approach? Haven't bothered, yet. -Sam From jess@s... Mon Nov 29 22:55:15 1999 From: jess@s... (jess@s...) Date: Mon, 29 Nov 1999 14:55:15 -0800 Subject: [medusa] please help me Message-ID: <81v08j$tsfl@e...> I am doing a 15 minute presentation for my Operating Systems class on Medusa. I have been searching and searching and searching, but there is very little information available. My presentation is this Friday, hence why i am stressing out at the last minute! Can anyone help me? I have found some helpful websites, (such as www.nightmare.com/medusa) but the information i need for this presentation is: name company or organization(s) who created it major people involved when was it developed structure process control and scheduling memory management IO networking user interfaces (GUI, command line) why is this os important or different I have the name, company, and the major people involved. i have some other information on why this OS is better/different than others. What i really need is the structure, networking, and user interface. Anything anyone can help me with would be very useful. My email address is jess@s... Thank you in advance! Jessica From jam@n... Wed Dec 15 20:22:53 1999 From: jam@n... (Jonez) Date: Wed, 15 Dec 1999 12:22:53 -0800 Subject: [medusa] No Subject Message-ID: <838tat$b918@e...> greetings, as an FYI, the package that comes with redhat 6.1 has a 'quite' old version of asyncore, based on the CVS revision information. specifically, the 'log_info' method appears to be missing from the version that is installed by redhat. I solved the problem by copying the file from the 1999-09-02 distribution of medusa. how does one obtain access to the CVS version of medusa? I am interested in getting whatever 'tweaks' might be available. From rushing@n... Wed Dec 15 20:55:42 1999 From: rushing@n... (Sam Rushing) Date: Wed, 15 Dec 1999 12:55:42 -0800 (PST) Subject: [medusa] No Subject In-Reply-To: <838tat$b918@e...> References: <838tat$b918@e...> Message-ID: <14424.78.879406.394872@s...> From rushing@n... Wed Dec 15 21:00:40 1999 From: rushing@n... (Sam Rushing) Date: Wed, 15 Dec 1999 13:00:40 -0800 (PST) Subject: [medusa] No Subject In-Reply-To: <838tat$b918@e...> References: <838tat$b918@e...> Message-ID: <14424.376.392459.539029@s...> Jonez writes: > greetings, > > as an FYI, the package that comes with redhat 6.1 has a 'quite' old > version of asyncore, based on the CVS revision information. > specifically, the 'log_info' method appears to be missing from the > version that is installed by redhat. I solved the problem by > copying the file from the 1999-09-02 distribution of medusa. Yup, the core has been getting a few tweaks over the past year. The latest is a reworking of the way the socket_map dictionary works in order to improve performance with large numbers of sockets (>1000). I should probably get it checked in, a few minor changes still remaining. > how does one obtain access to the CVS version of medusa? I am > interested in getting whatever 'tweaks' might be available. This should do the trick on unix: bash$ export CVSROOT=:pserver:medusa@s...:/usr/local/cvsroot bash$ cvs login # password=='medusa' bash$ cvs checkout medusa [or] bash$ cvs checkout medusa/asyncore.py -Sam From jam@q... Wed Dec 15 21:23:25 1999 From: jam@q... (Jeff) Date: Wed, 15 Dec 1999 16:23:25 -0500 Subject: [medusa] Re: cvs access In-Reply-To: <14424.376.392459.539029@s...> References: <838tat$b918@e...> <14424.376.392459.539029@s...> Message-ID: <19991215162325.B1213@q...> On Wed, Dec 15, 1999 at 01:00:40PM -0800, Sam Rushing wrote: > This should do the trick on unix: > > bash$ export CVSROOT=:pserver:medusa@s...:/usr/local/cvsroot > bash$ cvs login > # password=='medusa' > bash$ cvs checkout medusa > [or] > bash$ cvs checkout medusa/asyncore.py > > -Sam > thanks.. you rock ;) regards, J -- || visit gfd || psa member -- || New Image Systems & Services, Inc. From zakons@e... Thu Dec 16 15:55:24 1999 From: zakons@e... (Stuart Zakon) Date: Thu, 16 Dec 1999 07:55:24 -0800 Subject: [medusa] IPC8? Message-ID: <83b21c$joda@e...> Will there be any Medusa related presentations at IPC8? BOF sessions? Stuart Zakon