From Daryl.Tester at iocane.com.au Thu Jun 1 01:34:58 2006 From: Daryl.Tester at iocane.com.au (Daryl Tester) Date: Thu, 01 Jun 2006 09:04:58 +0930 Subject: [sapug] Another meeting In-Reply-To: <1149053397.6213.2.camel@beast64.localnet> References: <447AF65E.6010305@uanywhere.com.au> <447C5CC9.8050002@afoyi.com> <1149053397.6213.2.camel@beast64.localnet> Message-ID: <447E2822.8060504@iocane.com.au> (Resuming from hibernate) George Patterson wrote: > Not this week but the week after next is good for me (rotating shift > work is annoying for scheduling stuff) How about Tuesday next week? We met are Marcellina's last time; anyone want something different this time? > Otherwise, start having sapug meetings at the same time each > month and I will be there at some. That's the problem for me; I'd rather turn up knowing there are going to be a bunch of others turning up. I personally don't feel we'd have the numbers for a sustained monthly effort (heck, I keep missing the other monthly meetings I want to go to). -- Regards, Daryl Tester, IOCANE Pty. Ltd. From ryan at uanywhere.com.au Thu Jun 1 08:42:16 2006 From: ryan at uanywhere.com.au (Ryan Verner) Date: Thu, 01 Jun 2006 16:12:16 +0930 Subject: [sapug] Another meeting In-Reply-To: <447E2822.8060504@iocane.com.au> References: <447AF65E.6010305@uanywhere.com.au> <447C5CC9.8050002@afoyi.com> <1149053397.6213.2.camel@beast64.localnet> <447E2822.8060504@iocane.com.au> Message-ID: <1149144136.21655.1.camel@legowelt> On Thu, 2006-06-01 at 09:04 +0930, Daryl Tester wrote: > (Resuming from hibernate) > > George Patterson wrote: > > > Not this week but the week after next is good for me (rotating shift > > work is annoying for scheduling stuff) > > How about Tuesday next week? We met are Marcellina's last time; > anyone want something different this time? Sounds good to me. From steve at adam.com.au Fri Jun 2 10:42:30 2006 From: steve at adam.com.au (stephen white) Date: Fri, 2 Jun 2006 18:12:30 +0930 Subject: [sapug] Another meeting In-Reply-To: <447E2822.8060504@iocane.com.au> References: <447AF65E.6010305@uanywhere.com.au> <447C5CC9.8050002@afoyi.com> <1149053397.6213.2.camel@beast64.localnet> <447E2822.8060504@iocane.com.au> Message-ID: On 01/06/2006, at 9:04 AM, Daryl Tester wrote: > (Resuming from hibernate) For those of you struggling without the benefits of Objective C, Smalltalk or Ruby: http://www.devx.com/opensource/Article/31482 http://www.devx.com/opensource/Article/31593 This may help you write some real code. :) -- steve at adam.com.au From trent.murray at gmail.com Fri Jun 2 13:27:30 2006 From: trent.murray at gmail.com (T Murray) Date: Fri, 2 Jun 2006 21:27:30 +1000 Subject: [sapug] Another Meeting Message-ID: Hey guys, My name is Trent and I am a member of SLUG (Sydney Linux Users Group) We are trying to start up a Snakes and Rubies Special Interest Group and are looking for participants. If you dont have the numbers why not come along to one of our monthly meetings held in the UTS building on the last friday of each month and help us get this SIG off the ground. We also hold regular CodeFests which i think would be of interest to many of you. SLUG Website is: www.slug.org.au or chat with us at irc.freenode.net on channel #slug. Hope to see you there! -- Regards, Trent Murray (diazepam) From Daryl.Tester at iocane.com.au Sat Jun 3 00:23:41 2006 From: Daryl.Tester at iocane.com.au (Daryl Tester) Date: Sat, 03 Jun 2006 07:53:41 +0930 Subject: [sapug] Another Meeting In-Reply-To: References: Message-ID: <4480BA6D.5090203@iocane.com.au> T Murray wrote: > My name is Trent and I am a member of SLUG (Sydney Linux Users Group) We are > trying to start up a Snakes and Rubies Special Interest Group and are > looking for participants. If you dont have the numbers why not come along > to one of our monthly meetings held in the UTS building on the last friday > of each month and help us get this SIG off the ground. It might be a bit of a long walk - the "SA" part in SAPUG stands for "South Australia". But thank you for the offer. :-) -- Regards, Daryl Tester, IOCANE Pty. Ltd. From Daryl.Tester at iocane.com.au Sun Jun 11 09:14:20 2006 From: Daryl.Tester at iocane.com.au (Daryl Tester) Date: Sun, 11 Jun 2006 16:44:20 +0930 Subject: [sapug] Stateless Iteration Message-ID: <448BC2CC.6010102@iocane.com.au> Sub-Subject: Sad man seeks life on long weekend. I'm puttering around with XML-RPC, which is all relatively straight forward and such, and have come across an issue that I've struck before in the past and never adequately resolved, and that is the burning issue of today - "Scalable APIs". I want to provide an access method for a large list of objects (say, Customers, or Accounts, or records of some type - they will be uniform types). If I provide a method that says "return list", then it may crunch away on a a very large dataset and occupy: 1) a large amount of bandwidth traversing the network (although if the data is required, the data is required), so this reason is a bit of a furfy. B) a large amount of time to transfer the result (variation on 1, but more important). III) a large amount of memory to store the result on the client. (also an inability to stick to a consistent numbering format, but that's the fault of the author's). B & III are the ones I'm most concerned about. If the results were driving (for example) a GUI listbox, I'd like the results to start populating that immediately rather than waiting for the entire result set to be returned. If I'm driving an export or conversion process then I don't need to store the entire result set on the client, I can process each record as it comes in. Cool, so some form of iterator would probably be ideal. Unfortunately, to use an iterator I would need to retain some form of state (insert dramatic chord: or would I?) on the server, which, like cliches, I try to avoid like the plague. I could probably "cheat" (although I'm not really sure that it's cheating) and provide an interface that not only returns the data in question, but some form of nonce that I could use to retrieve the next (or next 'x') records from. The nonce could probably even be the primary key of the next record, if it were against a database table that had primary keys; I tend to use views a lot (which may not have primary keys), and the result set may not even be the output of a database. So I was wondering if anyone has ever come across this problem, found any naff solutions, or sunk any thought into the problem? -- Regards, Daryl Tester, IOCANE Pty. Ltd. From chris at inetd.com.au Sun Jun 11 14:47:40 2006 From: chris at inetd.com.au (Chris Foote) Date: Sun, 11 Jun 2006 22:17:40 +0930 (CST) Subject: [sapug] Stateless Iteration In-Reply-To: <448BC2CC.6010102@iocane.com.au> References: <448BC2CC.6010102@iocane.com.au> Message-ID: On Sun, 11 Jun 2006, Daryl Tester wrote: > I'm puttering around with XML-RPC, which is all > relatively straight forward and such, and have come > across an issue that I've struck before in the past > and never adequately resolved, and that is the > burning issue of today - "Scalable APIs". I do like ithe simplicity of XMLRPC, but recently I've become a real fan of Pyro: http://pyro.sourceforge.net It has lots of advantages over XMLRPC, particularly speed and TCP connection reuse, but has just one disadvantage - Python specificity. > I want to provide an access method for a large list > of objects (say, Customers, or Accounts, or records > of some type - they will be uniform types). If I > provide a method that says "return list", then it > may crunch away on a a very large dataset and occupy: > > 1) a large amount of bandwidth traversing the network > (although if the data is required, the data is > required), so this reason is a bit of a furfy. > > B) a large amount of time to transfer the result > (variation on 1, but more important). > > III) a large amount of memory to store the result > on the client. > > (also an inability to stick to a consistent numbering > format, but that's the fault of the author's). > > B & III are the ones I'm most concerned about. I've found the Judy arrays are good for memory usage. There's a Python interface to Judy called PyJudy: http://www.dalkescientific.com/Python/PyJudy.html If you can't fit it into RAM, then maybe a fast disk based storage system such as Metakit might be useable: http://www.equi4.com/metakit/python.html > If the results were driving (for example) a GUI listbox, > I'd like the results to start populating that immediately > rather than waiting for the entire result set to be > returned. If I'm driving an export or conversion process > then I don't need to store the entire result set on the > client, I can process each record as it comes in. A good way might be to have the drop down populate itself from a blocking queue, where the data in the queue is populated from another thread getting the results back from the remote procedure call and adding each one to the queue. see: http://www.python.org/doc/current/lib/module-Queue.html & http://www.python.org/doc/current/lib/QueueObjects.html I'm using threads & Queues to implement an RPC logging server which allows the client to continue getting its work done whilst logging happens asynchronously. Your problem might be similar. > Cool, so some form of iterator would probably be ideal. You could get the yield statement to return the value of a Queue's get(block=True) method. Hope that helps, -- Chris Foote Inetd Pty Ltd T/A HostExpress Web: http://www.hostexpress.com.au Blog: http://www.hostexpress.com.au/drupal/chris Phone: (08) 8410 4566 From Daryl.Tester at iocane.com.au Mon Jun 12 04:20:23 2006 From: Daryl.Tester at iocane.com.au (Daryl Tester) Date: Mon, 12 Jun 2006 11:50:23 +0930 Subject: [sapug] Stateless Iteration In-Reply-To: References: <448BC2CC.6010102@iocane.com.au> Message-ID: <448CCF67.7000303@iocane.com.au> Chris Foote wrote: > http://pyro.sourceforge.net > It has lots of advantages over XMLRPC, particularly speed and TCP > connection reuse, but has just one disadvantage - Python specificity. D'oh. My fault, I should have been more specific in my generalisation (bear in mind I'm thinking more abstractly than concretely here - I'm after a general protocol to realise this on any number of technologies). XML-RPC highlights the issue, but isn't the cause as such. One of the reasons I'm looking at using XML-RPC is to allow me to get Python into some areas where it hasn't previously been allowed, so while Pyro is cool, it won't solve that aspect of the problem. The client most likely won't be written in Python. I guess (and I'm still mulling over the problem definition in my head) that what I'm aiming for is to retain as little state as possible in the transport (could be XML-RPC, but could also easily be traditional RPC, or REST) and the backend server so that the server could be restarted without loss of state, because it won't have any to lose (helps minimise the need for persistent session management). Dammit, this where my thinking is breaking down - I'm not able to articulate the problem in all its gory details. OK, a concrete example. I offer the following service interface: customer.fetch() customer.fetch(next) customer.fetch(next, limit) customer.fetch() returns the first customer (as an array of 1) and an ID of some sort. This returned ID can be fed into customer.fetch() as parameter "next" to return the next customer ID (in this example, it could be the primary key of the customer - it would map easily) and also return another ID, which if subsequently fed into customer.fetch() returns the next customer, and so forth. The "limit" parameter allows you to return a bunch of customers, as round trips can be expensive (and as before, the returned ID would point to the customer immediately after "limit" customers). In this example, no state is maintained by the server, so it could be rebooted in between requests and the client could be none the wiser. If the client was rebooted (or otherwise disappeared), the server doesn't have any session details to clean up. Sweet and simple. (There are other issues here that I'm glossing over, concurrent updates leaping to mind). The problem arises when I'm talking to something that's not a table with a primary key, like some sort of computed list (or a table without a primary key :-). "next" needs to be something that can be used by the server to reconstruct enough state to be able to resume the list evaluation from where it left off. "Next" is a nonce, and given a complicated enough nonce this could be done - i.e. the nonce is the remainder of the list - but then the client wouldn't need to go back to the server to fetch the remainder of the list when it already has it (it could be hidden by the client API, but then my goal in the first place was to lazy evaluate the list so it wasn't pulled across the link in its entirety). So I guess what I'm after is some method/pattern where, given an arbitrary function running on a server, I can extract enough state from it to be handed over to the client (ignoring for the moment issues like security) so that when the client says "give me more data", the server can reconstruct the previous state from the client supplied data, and take off from there. And it's taken me 2.5 hours to get to this point, but I think I understand my problem better now (you're all acting as my personal sounding board :-). -- Regards, Daryl Tester, IOCANE Pty. Ltd. From steve at adam.com.au Mon Jun 12 05:44:40 2006 From: steve at adam.com.au (stephen white) Date: Mon, 12 Jun 2006 13:14:40 +0930 Subject: [sapug] Stateless Iteration In-Reply-To: <448CCF67.7000303@iocane.com.au> References: <448BC2CC.6010102@iocane.com.au> <448CCF67.7000303@iocane.com.au> Message-ID: On 12/06/2006, at 11:50 AM, Daryl Tester wrote: > So I guess what I'm after is some method/pattern where, given > an arbitrary function running on a server, I can extract enough > state from it to be handed over to the client (ignoring for the While you're using XML, there doesn't seem to be a way around the problem as the answer is to stream the data, whereas XML is formatted in blocks. Given that constraint, the answer seems to be to use an intermediate server providing the API you want. That is to say, instead of a client/server arrangement, you want a client/api-server/server arrangement... where the database has no state, but the api-server does. That arrangement is scalable as you simply bung in more machines as you start running out of memory/CPU/ network, without the cost of a database per server. Needless to say, I think XML is a re-invention of the wheel, with spanners and swiss-army knives poking out of the treads, and I refuse to touch it. :) -- steve at adam.com.au From Daryl.Tester at iocane.com.au Mon Jun 12 06:09:57 2006 From: Daryl.Tester at iocane.com.au (Daryl Tester) Date: Mon, 12 Jun 2006 13:39:57 +0930 Subject: [sapug] Stateless Iteration In-Reply-To: References: <448BC2CC.6010102@iocane.com.au> <448CCF67.7000303@iocane.com.au> Message-ID: <448CE915.5050401@iocane.com.au> stephen white wrote: > While you're using XML, there doesn't seem to be a way around the > problem as the answer is to stream the data, whereas XML is formatted > in blocks. I'd strike the same issue if I used traditional RPC, UDP or REST. The absence of a streaming connection means that the session data that would be implicit to the connection (e.g. holding a cursor to a DB open) has to reside elsewhere. Streaming XML isn't out of the question; just view a document as a rather large packet. Then all you need is some session information ... and I'm kinda back to where I started. It's related to SYN cookies, except I may need to recover more state (I'm not sure; I don't have a formal method for achieving this yet). -- Regards, Daryl Tester, IOCANE Pty. Ltd. From chris at inetd.com.au Mon Jun 12 14:50:48 2006 From: chris at inetd.com.au (Chris Foote) Date: Mon, 12 Jun 2006 22:20:48 +0930 (CST) Subject: [sapug] Stateless Iteration In-Reply-To: <448CCF67.7000303@iocane.com.au> References: <448BC2CC.6010102@iocane.com.au> <448CCF67.7000303@iocane.com.au> Message-ID: On Mon, 12 Jun 2006, Daryl Tester wrote: > One of the reasons I'm looking at using XML-RPC is to allow me to > get Python into some areas where it hasn't previously been > allowed [...] That's a good idea. I had a recent need to use a piece of commercial software where the only interface was a Perl module, so I wrapped it in an XML-RPC server so I could use it from Python :-) > I guess (and I'm still mulling over the problem definition in > my head) that what I'm aiming for is to retain as little state > as possible in the transport (could be XML-RPC, but could also > easily be traditional RPC, or REST) and the backend server so > that the server could be restarted without loss of state, > because it won't have any to lose (helps minimise the need > for persistent session management). > > Dammit, this where my thinking is breaking down - I'm not > able to articulate the problem in all its gory details. > > OK, a concrete example. I offer the following service > interface: > > customer.fetch() > customer.fetch(next) > customer.fetch(next, limit) > > customer.fetch() returns the first customer (as an array of > 1) and an ID of some sort. This returned ID can be fed into > customer.fetch() as parameter "next" to return the next > customer ID (in this example, it could be the primary key > of the customer - it would map easily) and also return > another ID, which if subsequently fed into customer.fetch() > returns the next customer, and so forth. The "limit" > parameter allows you to return a bunch of customers, as > round trips can be expensive (and as before, the returned > ID would point to the customer immediately after "limit" > customers). In this example, no state is maintained by the > server, so it could be rebooted in between requests and the > client could be none the wiser. If the client was rebooted > (or otherwise disappeared), the server doesn't have any > session details to clean up. Sweet and simple. (There > are other issues here that I'm glossing over, concurrent > updates leaping to mind). > > The problem arises when I'm talking to something that's not > a table with a primary key, like some sort of computed list > (or a table without a primary key :-). "next" needs to be > something that can be used by the server to reconstruct > enough state to be able to resume the list evaluation from > where it left off. "Next" is a nonce, and given a complicated > enough nonce this could be done - i.e. the nonce is the remainder > of the list - but then the client wouldn't need to go back > to the server to fetch the remainder of the list when it > already has it (it could be hidden by the client API, but > then my goal in the first place was to lazy evaluate the list > so it wasn't pulled across the link in its entirety). > > So I guess what I'm after is some method/pattern where, given > an arbitrary function running on a server, I can extract enough > state from it to be handed over to the client (ignoring for the > moment issues like security) so that when the client says "give > me more data", the server can reconstruct the previous state > from the client supplied data, and take off from there. Is maintaining the list of objects on the server for each client a real resource problem ? If not, and if your desired outcome as you say is: "my goal in the first place was to lazy evaluate the list so it wasn't pulled across the link in its entirety" then what about putting the smarts into the client ? e.g. If you created a server XMLRPC method customers.getsome(limit=1) and get the client to manage loading of the data via a seperate thread where it puts items from the list into a queue[1], then you could watch the size of the queue and call customers.getsome(limit=50) in advance of the queue being empty. If the client doesn't need all of the data, it just needs to toggle a flag on the queue loader to not bother calling customers.getsome() anymore. [1] By queue, I mean something like Queue.Queue() which has locking support so that different threads can safely use it. Cheers, -- Chris Foote Inetd Pty Ltd T/A HostExpress Web: http://www.hostexpress.com.au Blog: http://www.hostexpress.com.au/drupal/chris Phone: (08) 8410 4566 From Daryl.Tester at iocane.com.au Tue Jun 13 13:51:55 2006 From: Daryl.Tester at iocane.com.au (Daryl Tester) Date: Tue, 13 Jun 2006 21:21:55 +0930 Subject: [sapug] Stateless Iteration In-Reply-To: References: <448BC2CC.6010102@iocane.com.au> <448CCF67.7000303@iocane.com.au> Message-ID: <448EA6DB.4040405@iocane.com.au> Chris Foote wrote: > Is maintaining the list of objects on the server for each client a real > resource problem ? Well, it could be, and I say could because at the moment I have no session state to maintain. If I keep it this way, or I recover the session completely from client supplied data, then I could achieve naff things like not having to time out session data on the server due to clients disappearing (and thus making arbitrary decisions about what constitutes client "liveliness"), and load balancing amongst a set of servers without having to replicate the session between them because the client provides it. A client could come back after a year, and provided the underlying result set hadn't changed still be able to pick up their iteration from where they left off. I've found that Microsoft are doing a similar thing under ASP; if you have a look at the Foxtel TV guide page you'll find a hidden form variable called "__VIEWSTATE" with a honking long base64 encoded string, which is apparently the session state serialised and passed to the client for safe keeping so that the server doesn't have to maintain it. > If not, and if your desired outcome as you say is: > "my goal in the first place was to lazy evaluate the list so it > wasn't pulled across the link in its entirety" I should have written "a goal"; I'm just envisioning that lazy evaluation will naturally fall out of this scheme, and the client having the placeholder so it can tell the server "btw, this is where you got up to in the list you were iterating for me". I've seen some indication that currying functions might provide me with a solution for what I'm looking for, so I'm currently trying to wrap my head around that. > e.g. If you created a server XMLRPC method customers.getsome(limit=1) > and get the client to manage loading of the data via a seperate thread > where it puts items from the list into a queue[1], then you could watch > the size of the queue and call customers.getsome(limit=50) in advance > of the queue being empty. If the client doesn't need all of the data, > it just needs to toggle a flag on the queue loader to not bother calling > customers.getsome() anymore. Yep, that should also just naturally fall out of this, once I've got a working solution of what "this" actually is. :-) Pre-emptive lazy loading ... I think I've bored everyone witless for now; I'm off to cut code ... Cheers. -- Regards, Daryl Tester, IOCANE Pty. Ltd. From spam at afoyi.com Tue Jun 20 09:47:43 2006 From: spam at afoyi.com (Darryl Ross) Date: Tue, 20 Jun 2006 17:17:43 +0930 Subject: [sapug] Simple Sequence Question Message-ID: <4497A81F.2050401@afoyi.com> Hey All, Got a rather simple question about formatting a string. Say I have a string: s = '1234567812345678' and I want to format it like: t = '1234:5678:1234:5678' Is there a simple slicing operation? Or do I need to do something like: t = '' for i in xrange(len(s)): if not i == 0 and i % 4 == 0: t += ':' t += s[i] Regards Darryl -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 187 bytes Desc: OpenPGP digital signature Url : http://mail.python.org/pipermail/sapug/attachments/20060620/8ff6186f/attachment.pgp From Daryl.Tester at iocane.com.au Tue Jun 20 10:33:04 2006 From: Daryl.Tester at iocane.com.au (Daryl Tester) Date: Tue, 20 Jun 2006 18:03:04 +0930 Subject: [sapug] Simple Sequence Question In-Reply-To: <4497A81F.2050401@afoyi.com> References: <4497A81F.2050401@afoyi.com> Message-ID: <4497B2C0.7080607@iocane.com.au> Darryl Ross wrote: > s = '1234567812345678' > > and I want to format it like: > > t = '1234:5678:1234:5678' > > Is there a simple slicing operation? Or do I need to do something like: > > t = '' > for i in xrange(len(s)): > if not i == 0 and i % 4 == 0: > t += ':' > t += s[i] Well, it's looping and slicing: s = '1234567812345678' t, s = s[:4], s[4:] while s: t, s = t + ':' + s[:4], s[4:] -- Regards, Daryl Tester, IOCANE Pty. Ltd. From chris.were at gmail.com Tue Jun 20 11:06:18 2006 From: chris.were at gmail.com (Chris Were) Date: Tue, 20 Jun 2006 18:36:18 +0930 Subject: [sapug] Simple Sequence Question In-Reply-To: <4497B2C0.7080607@iocane.com.au> References: <4497A81F.2050401@afoyi.com> <4497B2C0.7080607@iocane.com.au> Message-ID: <35bb42690606200206teccc24cwaf6ed6c151547dae@mail.gmail.com> Or you could do the following if you don't know how many characters the string will have: t = "" while len(s) > 0: t += s[:4]+"." s = s[4:] t = t[:-1] > > Is there a simple slicing operation? Or do I need to do something like: > > > > t = '' > > for i in xrange(len(s)): > > if not i == 0 and i % 4 == 0: > > t += ':' > > t += s[i] > > Well, it's looping and slicing: > > s = '1234567812345678' > t, s = s[:4], s[4:] > while s: > t, s = t + ':' + s[:4], s[4:] -- http://www.chriswere.com/ From Daryl.Tester at iocane.com.au Tue Jun 20 11:36:42 2006 From: Daryl.Tester at iocane.com.au (Daryl Tester) Date: Tue, 20 Jun 2006 19:06:42 +0930 Subject: [sapug] Simple Sequence Question In-Reply-To: <35bb42690606200206teccc24cwaf6ed6c151547dae@mail.gmail.com> References: <4497A81F.2050401@afoyi.com> <4497B2C0.7080607@iocane.com.au> <35bb42690606200206teccc24cwaf6ed6c151547dae@mail.gmail.com> Message-ID: <4497C1AA.4030803@iocane.com.au> Chris Were wrote: > Or you could do the following if you don't know how many characters > the string will have: Not sure what you mean by that; the "while s" construct would have run until s became empty (equivalent to "while len(s) > 0"). -- Regards, Daryl Tester, IOCANE Pty. Ltd.