From ed at pythoncharmers.com Sun Jul 1 07:44:17 2018 From: ed at pythoncharmers.com (Ed Schofield) Date: Sun, 1 Jul 2018 21:44:17 +1000 Subject: [melbourne-pug] Next Melbourne Python meeting - tomorrow - Monday 2 July Message-ID: Hi all! We're looking forward to the next meetup of the Melbourne Python user group tomorrow evening. Do come and join us! We have two main talks planned: *1. Rory Hart: Just Enough OpenID Connect* (~25 minutes) [Rescheduled from June] Many of us encounter authentication protocols as a side effect of just trying to get things done. The simple task of accessing data from a client API or integrating with a third parties balloons out into a yak shaving session filled with obtuse authentication failure messages. Rory will walk through a number of scenarios involving OpenID Connect and offer practical guidance on productively using Python to work with OpenID Connect. *2. Ed Schofield: Pandas -- under the fur * (~20 minutes) I will talk about some advanced topics in Pandas such as: - performance: which operations are slow and what to do about it - internals: views and copies, chained indexing - avoiding gotchas with data types and the design decisions behind Pandas 2. *3. Lightning talks?* *4. Announcements and pizza* *When:* 5.45pm for mingling; talks from 6pm to ~7.15pm, pizza afterwards *Where: *Outcome-Hub Co-Working Space, Suite 1, 121 Cardigan Street, Carlton *How to get there: *Walk 12 minutes north from Melbourne Central station. *Afterwards:* pizza! *Sponsorship:* many thanks to Outcome Hub for providing the venue and Python Charmers for ongoing sponsorship. *RSVP:* Please respond on Meetup.com so we can track numbers: https://www. meetup.com/Melbourne-Python-Meetup-Group/ We hope to see you there! :-D Best wishes, Ed -- Dr. Edward Schofield Python Charmers http://pythoncharmers.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From fps-pd at optusnet.com.au Sun Jul 1 13:51:21 2018 From: fps-pd at optusnet.com.au (Peter Dwyer) Date: Mon, 02 Jul 2018 03:51:21 +1000 Subject: [melbourne-pug] Next Melbourne Python meeting - tomorrow - Monday 2 July In-Reply-To: References: Message-ID: <1380F12F-8148-4776-B02B-01AB1B5CD0C6@optusnet.com.au> Hello Ed I haven?t attended any meetings. Is there a fee? Cheers Peter From: melbourne-pug on behalf of Ed Schofield Reply-To: Melbourne Python Users Group Date: Monday, 2 July 2018 at 3:42 am To: Melbourne Python Users Group Subject: [melbourne-pug] Next Melbourne Python meeting - tomorrow - Monday 2 July Hi all! We're looking forward to the next meetup of the Melbourne Python user group tomorrow evening. Do come and join us! We have two main talks planned: 1. Rory Hart: Just Enough OpenID Connect (~25 minutes) [Rescheduled from June] Many of us encounter authentication protocols as a side effect of just trying to get things done. The simple task of accessing data from a client API or integrating with a third parties balloons out into a yak shaving session filled with obtuse authentication failure messages. Rory will walk through a number of scenarios involving OpenID Connect and offer practical guidance on productively using Python to work with OpenID Connect. 2. Ed Schofield: Pandas -- under the fur (~20 minutes) I will talk about some advanced topics in Pandas such as: - performance: which operations are slow and what to do about it - internals: views and copies, chained indexing - avoiding gotchas with data types and the design decisions behind Pandas 2. 3. Lightning talks? 4. Announcements and pizza When: 5.45pm for mingling; talks from 6pm to ~7.15pm, pizza afterwards Where: Outcome-Hub Co-Working Space, Suite 1, 121 Cardigan Street, Carlton How to get there: Walk 12 minutes north from Melbourne Central station. Afterwards: pizza! Sponsorship: many thanks to Outcome Hub for providing the venue and Python Charmers for ongoing sponsorship. RSVP: Please respond on Meetup.com so we can track numbers: https://www.meetup.com/Melbourne-Python-Meetup-Group/ We hope to see you there! :-D Best wishes, Ed -- Dr. Edward Schofield Python Charmers http://pythoncharmers.com _______________________________________________ melbourne-pug mailing list melbourne-pug at python.org https://mail.python.org/mailman/listinfo/melbourne-pug -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian at linuxpenguins.xyz Sun Jul 1 17:59:36 2018 From: brian at linuxpenguins.xyz (Brian May) Date: Mon, 02 Jul 2018 07:59:36 +1000 Subject: [melbourne-pug] Next Melbourne Python meeting - tomorrow - Monday 2 July In-Reply-To: <1380F12F-8148-4776-B02B-01AB1B5CD0C6@optusnet.com.au> References: <1380F12F-8148-4776-B02B-01AB1B5CD0C6@optusnet.com.au> Message-ID: <87wouety5z.fsf@silverfish.pri> Peter Dwyer writes: > I haven?t attended any meetings. Is there a fee? No fees. Unless you want pizza, which case you might be asked to donate for the cause. -- Brian May https://linuxpenguins.xyz/brian/ From hartror at gmail.com Mon Jul 2 20:23:52 2018 From: hartror at gmail.com (Rory Hart) Date: Tue, 3 Jul 2018 10:23:52 +1000 Subject: [melbourne-pug] Next Melbourne Python meeting - tomorrow - Monday 2 July In-Reply-To: <87wouety5z.fsf@silverfish.pri> References: <1380F12F-8148-4776-B02B-01AB1B5CD0C6@optusnet.com.au> <87wouety5z.fsf@silverfish.pri> Message-ID: Slides from my OpenID talk last night. https://speakerdeck.com/hartror/just-enough-openid-connect-at-mpug On 2 July 2018 at 07:59, Brian May wrote: > Peter Dwyer writes: > > > I haven?t attended any meetings. Is there a fee? > > No fees. Unless you want pizza, which case you might be asked to donate > for the cause. > -- > Brian May > https://linuxpenguins.xyz/brian/ > _______________________________________________ > melbourne-pug mailing list > melbourne-pug at python.org > https://mail.python.org/mailman/listinfo/melbourne-pug > -------------- next part -------------- An HTML attachment was scrubbed... URL: From b.obrien at me.com Thu Jul 5 03:57:47 2018 From: b.obrien at me.com (Brendan O'Brien) Date: Thu, 05 Jul 2018 17:57:47 +1000 Subject: [melbourne-pug] job seeking Message-ID: <8F217CE4-D664-4C69-8A1C-DF69F81D1541@me.com> Hi All: The contract I?m on will expire soon so it?s time to start beating the bushes to find my next awesome gig. A short bio: My background is in academic science. After receiving my PhD, I?ve worked at several high powered research universities and institutes around the world (U Washington, Brown University, Max Planck Institute, Uni Auckland, ANU and U Melbourne). After leaving academia I concentrated my efforts to launch a career in software engineering. As an academic I used mainly the engineering powerhouse, Labview, for coding data collection and analysis routines. Since then, I?ve also developed skills in Python, Ruby and Java. Most recently, I?ve been working on developing a machine learning algorithm to help track child protection offenders nationwide through data matching across states. I am particularly keen to continue in the data science space but I?m open to other amazing opportunities. Best regards, Brendan CV attached _____________________________ Brendan J. O?Brien PhD Melbourne | Australia b.obrien at me.com +61 433 325 679 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: linkedin-icon-32x32.png Type: image/png Size: 1933 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: octocat32.png Type: image/png Size: 1858 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: amazon.png Type: image/png Size: 1528 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: TC icon32.png Type: image/png Size: 4515 bytes Desc: not available URL: From b.obrien at me.com Thu Jul 5 03:58:50 2018 From: b.obrien at me.com (Brendan O'Brien) Date: Thu, 05 Jul 2018 17:58:50 +1000 Subject: [melbourne-pug] job seeking Message-ID: <323FD792-EB0C-4F9E-B24E-A888CE6237E0@me.com> Hi All: The contract I?m on will expire soon so it?s time to start beating the bushes to find my next awesome gig. A short bio: My background is in academic science. After receiving my PhD, I?ve worked at several high powered research universities and institutes around the world (U Washington, Brown University, Max Planck Institute, Uni Auckland, ANU and U Melbourne). After leaving academia I concentrated my efforts to launch a career in software engineering. As an academic I used mainly the engineering powerhouse, Labview, for coding data collection and analysis routines. Since then, I?ve also developed skills in Python, Ruby and Java. Most recently, I?ve been working on developing a machine learning algorithm to help track child protection offenders nationwide through data matching across states. I am particularly keen to continue in the data science space but I?m open to other amazing opportunities. Best regards, Brendan CV attached _____________________________ Brendan J. O?Brien PhD Melbourne | Australia b.obrien at me.com +61 433 325 679 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: O'Brien_CV_JUL-2018-brief.pdf Type: application/pdf Size: 243497 bytes Desc: not available URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: linkedin-icon-32x32.png Type: image/png Size: 1933 bytes Desc: not available URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: octocat32.png Type: image/png Size: 1858 bytes Desc: not available URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: amazon.png Type: image/png Size: 1528 bytes Desc: not available URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: TC icon32.png Type: image/png Size: 4515 bytes Desc: not available URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at adversary.org Mon Jul 9 20:57:23 2018 From: ben at adversary.org (Ben McGinnes) Date: Tue, 10 Jul 2018 10:57:23 +1000 Subject: [melbourne-pug] Speakers wanted! In-Reply-To: References: Message-ID: <20180710005723.ttuaywi7adobdwz2@adversary.org> On Thu, Apr 26, 2018 at 07:56:52AM +1000, Ed Schofield wrote: > Hi everyone! > > We're looking for more speakers for the Melbourne Python user group > (in May and beyond). This is an opportunity to connect with the > Python community, get the word out about something important, or get > feedback. You may be interested if ... To quote _John Wick_, ?A talk, you say ? ? Actually things have been insanely hectic this year and are likely to remain that way for quite some time. Maybe they'll settle down a bit later in the year, but probably not until next year at the earliest (it depends on too many external factors to be sure). In fact, I'd not only forgotten I was on this list, I'd forgotten it even existed, and only rediscovered it when checking for something else entirely. Still, given the cause of some of my hectic schedule and your questions it seemed appropriate to say something ? > You're working on an interesting Python-related project that the > world should know about? Oh my word, yes indeed ? > You've discovered an amazing new Python-related tool recently? Does it count as a discovery if you're responsible for porting it from 2 to 3 and then getting that version reincorporated into the parent project as official bindings for that parent project? ;) > You'd like to share your experiences learning Python ... or using it > in your workplace? Well, I certainly learned a lot (though somewhat beyond learning the language) and indeed still am. As for the workplace part, the aforementioned parent project is now employing me so that counts too. > You'd like a trial run for a potential talk at PyCon AU in August? There is no chance of me even attending a PyCon for the foreseeable future; for reasons tangentially related to aspects of why I work on cryptographic projects in the first place. > If so .. please let me (or the list) know! Right, time to stop being cryptic. In 2015 I ported John Goerzen's old PyME project (version 0.9.0) from Python 2 to Python 3. PyME utilised SWIG to generate Python to C bindings for GPGME, the C API for the GNU Privacy Guard (aka the GnuPG Project, aka GPG). As PyME had always been licensed under the same dual licensing model as the rest of the GnuPG Project (GPLv2+ and LGPLv2.1+), it was able to be seamlessly committed back into the parent project a few months after waving it under Werner's nose. The following year it received a significant overhaul by Justus Winter, formerly of g10code, in order to make it work with both Python 2.7 and from Python 3.4 and above. It also gained a module rename (to "gpg") and the addition of a more pythonic layer between the lower level bindings and developers. When Justus moved on from g10code towards the end of last year, Werner needed a Python maintainer and I returned to take up stewardship of the bindings. Since the package utilises SWIG to dynamically generate these bindings, they are the most feature complete bindings to GPGME of any language aside, of course, from C itself. With Python being what it is, the options to provide easier access for other languages without needing to write specific bindings for them is already planned. The module works on any POSIX system and though there are frequent reports of failures on Windows system, recent investigations indicate that may be solved soon (assuming Andre and I are right about the cause). Now the juicy part, the brief demonstration of why you should care. With the newer and more pythonic layer, this module is actually easier to use than either the python-gnupg wrapper module or Isis Lovecruft's fork. There is no need to set any configuration parameters since it will find your default ~/.gnupg directory, though that can be overridden if you want; unlike both of those wrapper modules. There's also no risk of shell related exploits since it hooks directly into GPGME itself, likewise there is no need to configure a binary file for GPG since the functions aren't called that way. Oh, also no risk of passing your passphrase into Python's memory since GPGME invokes gpg-agent and pinentry. As for how easy it is, well, here's how to use the information in the headers of this email to import my key and then export it to a file. For this demonstration I'm also using an alternate temporary configuration directory made a few minutes prior to running these commands: Python 3.7.0 (default, Jun 29 2018, 13:09:59) >>> import gpg >>> import requests >>> c = gpg.Context(home_dir="~/.gnupg-temp") >>> url = "http://www.adversary.org/ben-key.asc" >>> id = "DB4724E6FA4286C92B4E55C4321E4E2373590E5D" >>> r = requests.get(url) >>> incoming = c.key_import(r.content) >>> incoming ImportResult(considered=1, imported=1, imported_rsa=0, imports=[ImportStatus(fpr='DB4724E6FA4286C92B4E55C4321E4E2373590E5D', result=0, status=1)], new_revocations=0, new_signatures=0, new_sub_keys=0, new_user_ids=0, no_user_id=0, not_imported=0, secret_imported=0, secret_read=0, secret_unchanged=0, skipped_new_keys=0, skipped_v3_keys=0, unchanged=0) >>> incoming.imported 1 >>> incoming.imports[0].fpr == id True >>> with open("/tmp/ben_key.gpg", "wb") as f: ... f.write(c.key_export(pattern=id)) ... 75439 >>> Then to check in a shell: bash-4.4$ ls -l /tmp/ben_key.gpg -rw-r--r-- 1 ben wheel 75439 10 Jul 09:57 /tmp/ben_key.gpg bash-4.4$ file /tmp/ben_key.gpg /tmp/ben_key.gpg: PGP/GPG key public ring (v4) created Sat Jul 28 22:57:31 2012 RSA (Encrypt or Sign) 4096 bits MPI=0xc3b0a5435cfc5b63... bash-4.4$ Which, of course, matches: bash-4.4$ gpg --homedir .gnupg-temp -k /home/ben/.gnupg-temp/pubring.kbx ---------------------------------- pub rsa4096 2012-07-28 [SC] DB4724E6FA4286C92B4E55C4321E4E2373590E5D uid [ unknown] Ben McGinnes uid [ unknown] Ben McGinnes sub rsa3072 2012-07-28 [S] sub elg4096 2012-07-28 [E] sub dsa3072 2017-12-27 [S] bash-4.4$ And back in my real homedir: bash-4.4$ gpg -K DB4724E6FA4286C92B4E55C4321E4E2373590E5D sec rsa4096/0x321E4E2373590E5D 2012-07-28 [SC] DB4724E6FA4286C92B4E55C4321E4E2373590E5D uid [ultimate] Ben McGinnes uid [ultimate] Ben McGinnes ssb rsa3072/0x7FF2D37135C7553C 2012-07-28 [S] ssb elg4096/0xC98BAA1862E4484D 2012-07-28 [E] ssb dsa3072/0x48723A7579041EC6 2017-12-27 [S] bash-4.4$ Note: three UIDs have been removed and obviously two have been modified to delay the spambot brigade by half a second or so ... For those of you wondering about secret key exports. Yes, you can and it's as easy as c.key_export_secret(). It still requires the passphrase, though (of course). Not only that, but it even comes with a shiny new-ish HOWTO. The official version of which is in the lang/python/docs/ directory in GPGME distributions, but the unofficial draft version (sometimes containing material slightly ahead of what goes into the official org-mode file) lives over here: http://files.au.adversary.org/crypto/gpgme-python-howto/webhelp/index.html Alternatively, if you don't mind accepting wildcard SSL certificates for AWS S3 buckets for other domains, you can also access it here: https://files.au.adversary.org/crypto/gpgme-python-howto/webhelp/index.html Those links are probably a little easier to follow since they also experiment with alternative documentation methods. The python bindings source is here: https://dev.gnupg.org/source/gpgme/browse/master/lang/python/ The official docs are here: https://dev.gnupg.org/source/gpgme/browse/master/lang/python/docs/ And the scripts referenced in the HOWTO are here: https://dev.gnupg.org/source/gpgme/browse/master/lang/python/examples/howto/ Anyway, as I said, things are very busy for me at the moment and I can't guarantee when or even if I'd be able to deliver a talk even locally anytime soon, but there should be enough here for my fellow Melbournian pythonistas to start taking advantage of this work. There's already very clear examples of encryption, decryption, signing and verifying and a little more besides (the HOWTO includes creating keys and subkeys). It is, of course, very good for securing any data at rest and requiring a user to authorise its use. No doubt there are plenty of scenarios for which that will be appropriate (including the one which led me to perform the port in the first place, though this work is *far* more important than that project was). If nothing else you can all at least share the joke every time Malcolm Turnbull or Peter Dutton start talking about how they're going to control cryptography in Australia. ;) If you do find it useful, though, or even if you've just found GPG useful over the years, consider donating to the project. https://gnupg.org/donate/index.html It very nearly didn't survive and would have ended a few years ago, but only Edward Snowden's revelations and highlighting its value saved it at the last moment. Given how much really depends on it (e.g. all your favourite package managers), it really should feature a little higher on the donations priorities. Regards, Ben P.S. Yes, I have completed the ITAR compliance questionaire through the ADF website and this is all fine. There's a GPG signed copy of the results in one of my branches on git.gnupg.org (but I can't remember which branch it's in). -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 228 bytes Desc: not available URL: From miked at dewhirst.com.au Mon Jul 16 01:38:03 2018 From: miked at dewhirst.com.au (Mike Dewhirst) Date: Mon, 16 Jul 2018 15:38:03 +1000 Subject: [melbourne-pug] Nobel prize for former BDFL Message-ID: I would like to see Guido nominated for a Nobel prize. Literature? Science? And perhaps canonicalisation* after he drops off the perch :) Yes Mike * Involves taxidermy and feathers -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at adversary.org Sat Jul 21 04:25:19 2018 From: ben at adversary.org (Ben McGinnes) Date: Sat, 21 Jul 2018 18:25:19 +1000 Subject: [melbourne-pug] Nobel prize for former BDFL In-Reply-To: References: Message-ID: <20180721082519.7wv34gmxakd372wk@adversary.org> On Mon, Jul 16, 2018 at 03:38:03PM +1000, Mike Dewhirst wrote: > I would like to see Guido nominated for a Nobel prize. > > Literature? > > Science? How about peace? I mean if Arafat can get it and Obama managed to just on the basis of being elected POTUS and the hope that carried with it, then surely choosing to voluntarily give up a dictatorial position, benevolent or otherwise, ought to do so too. Right? Regards, Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 228 bytes Desc: not available URL: From miked at dewhirst.com.au Sun Jul 22 20:22:19 2018 From: miked at dewhirst.com.au (Mike Dewhirst) Date: Mon, 23 Jul 2018 10:22:19 +1000 Subject: [melbourne-pug] Nobel prize for former BDFL In-Reply-To: <20180721082519.7wv34gmxakd372wk@adversary.org> References: <20180721082519.7wv34gmxakd372wk@adversary.org> Message-ID: On 21/07/2018 6:25 PM, Ben McGinnes wrote: > On Mon, Jul 16, 2018 at 03:38:03PM +1000, Mike Dewhirst wrote: >> I would like to see Guido nominated for a Nobel prize. >> >> Literature? >> >> Science? > How about peace? > > I mean if Arafat can get it and Obama managed to just on the basis of > being elected POTUS and the hope that carried with it, then surely > choosing to voluntarily give up a dictatorial position, benevolent or > otherwise, ought to do so too. Right? Excellent point! > > > Regards, > Ben > > > _______________________________________________ > melbourne-pug mailing list > melbourne-pug at python.org > https://mail.python.org/mailman/listinfo/melbourne-pug -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew.stuart at supercoders.com.au Mon Jul 23 17:19:04 2018 From: andrew.stuart at supercoders.com.au (Andrew Stuart) Date: Tue, 24 Jul 2018 07:19:04 +1000 Subject: [melbourne-pug] Spawn new process to handle inbound websocket connection Message-ID: <507B1958-DCB5-4EB9-9642-6E4A8384498C@supercoders.com.au> I have servers that send a sequence of PNG images which need to be processed via a sequence of commands. My idea is this: A Python websockets server is listening for connections from the servers. When a websocket connection is received, a new Python process is spawned - presumably using https://docs.python.org/3/library/multiprocessing.html and somehow the websocket is given to that process (how?). The spawned process listens to the websocket, receives the PNG images and then processes them. After completion, the spawned process dies and the websocket is closed. So the spawned Python processes might run for up to a few minutes each. Is this a workable idea in Python? One open question is how would I hand the websocket to the spawned process? Also, coud such an approach support tens, hundreds, thousands of connections on a for example 8 core machine with 16 gig RAM ( know this is how long is a piece of string, but just finger in the air guessing to make sure there?s no huge gotcha in there that would impose a huge constraint & invalidate the idea)? I was looking at this gist as inspiration for a start point https://gist.github.com/gdamjan/d7333a4d9069af96fa4d Any ideas/feedback valued. thanks Andrew From tim at growthpath.com.au Mon Jul 23 18:32:08 2018 From: tim at growthpath.com.au (Tim Richardson) Date: Tue, 24 Jul 2018 08:32:08 +1000 Subject: [melbourne-pug] Spawn new process to handle inbound websocket connection In-Reply-To: <507B1958-DCB5-4EB9-9642-6E4A8384498C@supercoders.com.au> References: <507B1958-DCB5-4EB9-9642-6E4A8384498C@supercoders.com.au> Message-ID: I'm sure lots of people will give this sort of answer a) you normally don't want a webserver keeping connections open for minutes, processing a request, although if you were sending progress messages through the websockets I guess it's ok b) to handle large connection numbers, you want to use asyncio and the aiohttp which builds on top of it. This gives you a pure python websocket server which can handle lots of connections. Process or thread-spawning servers don't scale. I made a toy implementation here to learn about it https://github.com/timrichardson/asyncio_bus_timetable In production, most people would run it behind nginx which moves your solution to serious volume. The solution to (a) is to use a background task processor. The most common python choice is probably celery (to do the work, this can spawn multiple processes) and redis (to communicate). Although arguably if you are going to be CPU bound anyway, maybe this conventional advice isn't going to help much. Because you are CPU bound and each request takes two to three minutes, I can't see how are you are going to reach thousands of connections on one machine. The approach I sketched can scale across multiple machines; nginx can run multiple websocket servers, and you can distribute celery/redis. You can go to upwork, find an expert, specify you want a python solution, and you'll learn a lot from them for $1000 or so. On Tue, 24 Jul 2018 at 07:27, Andrew Stuart < andrew.stuart at supercoders.com.au> wrote: > I have servers that send a sequence of PNG images which need to be > processed via a sequence of commands. > > My idea is this: > > A Python websockets server is listening for connections from the servers. > When a websocket connection is received, a new Python process is spawned - > presumably using https://docs.python.org/3/library/multiprocessing.html > and somehow the websocket is given to that process (how?). The spawned > process listens to the websocket, receives the PNG images and then > processes them. After completion, the spawned process dies and the > websocket is closed. So the spawned Python processes might run for up to a > few minutes each. > > Is this a workable idea in Python? > > One open question is how would I hand the websocket to the spawned process? > > Also, coud such an approach support tens, hundreds, thousands of > connections on a for example 8 core machine with 16 gig RAM ( know this is > how long is a piece of string, but just finger in the air guessing to make > sure there?s no huge gotcha in there that would impose a huge constraint & > invalidate the idea)? > > I was looking at this gist as inspiration for a start point > https://gist.github.com/gdamjan/d7333a4d9069af96fa4d > > Any ideas/feedback valued. > > thanks > > Andrew > _______________________________________________ > melbourne-pug mailing list > melbourne-pug at python.org > https://mail.python.org/mailman/listinfo/melbourne-pug > -- *Tim Richardson CPA, Director* GrowthPath. Finance transformation for SMEs via Cloud ERP, advanced reporting, CRM Mobile: +61 423 091 732 Office: +61 3 8678 1850. Chat via https://hangouts.google.com or Skype: te.richardson GrowthPath Pty Ltd ABN 76 133 733 963 -------------- next part -------------- An HTML attachment was scrubbed... URL: From hartror at gmail.com Mon Jul 23 18:41:02 2018 From: hartror at gmail.com (Rory Hart) Date: Tue, 24 Jul 2018 08:41:02 +1000 Subject: [melbourne-pug] Spawn new process to handle inbound websocket connection In-Reply-To: <507B1958-DCB5-4EB9-9642-6E4A8384498C@supercoders.com.au> References: <507B1958-DCB5-4EB9-9642-6E4A8384498C@supercoders.com.au> Message-ID: What you are talking about is known as a stateful service and they're great if your problem has the right dimensions and you invest in good controls and monitoring/alerting. The main issue is they are tricky design to scale and plan disaster recovery for. Beyond that as a general rule mixing your batch processing (PNG image processing) and your web services isn't a good idea. For example the load spikes from the batch processing can cause response timeouts and as web services don't generally have high CPU load you'll have trouble using the CPU cycles you are paying for. Do you have access to one of the cloud services? This is the sort of problem their FaaS (AWS Lambda etc) systems are well suited to. For example I would do something like this with Amazon's cloud: 1. Web service accepting PNGs via S3 (AWS's file storage service). 2. Web service triggering Lambda to do processing. 3. Client polling for processing completion. 4.a Client download image from S3. 4.b or Client tell web service new command on PNG triggering another lambda. You could also do some similar with without the cloud with Celery and a shared filesystem such as glusterfs. On Tue, 24 Jul 2018 at 07:27, Andrew Stuart < andrew.stuart at supercoders.com.au> wrote: > I have servers that send a sequence of PNG images which need to be > processed via a sequence of commands. > > My idea is this: > > A Python websockets server is listening for connections from the servers. > When a websocket connection is received, a new Python process is spawned - > presumably using https://docs.python.org/3/library/multiprocessing.html > and somehow the websocket is given to that process (how?). The spawned > process listens to the websocket, receives the PNG images and then > processes them. After completion, the spawned process dies and the > websocket is closed. So the spawned Python processes might run for up to a > few minutes each. > > Is this a workable idea in Python? > > One open question is how would I hand the websocket to the spawned process? > > Also, coud such an approach support tens, hundreds, thousands of > connections on a for example 8 core machine with 16 gig RAM ( know this is > how long is a piece of string, but just finger in the air guessing to make > sure there?s no huge gotcha in there that would impose a huge constraint & > invalidate the idea)? > > I was looking at this gist as inspiration for a start point > https://gist.github.com/gdamjan/d7333a4d9069af96fa4d > > Any ideas/feedback valued. > > thanks > > Andrew > _______________________________________________ > melbourne-pug mailing list > melbourne-pug at python.org > https://mail.python.org/mailman/listinfo/melbourne-pug > -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew.stuart at supercoders.com.au Thu Jul 26 02:11:05 2018 From: andrew.stuart at supercoders.com.au (Andrew Stuart) Date: Thu, 26 Jul 2018 16:11:05 +1000 Subject: [melbourne-pug] Spawn new process to handle inbound websocket connection In-Reply-To: <507B1958-DCB5-4EB9-9642-6E4A8384498C@supercoders.com.au> References: <507B1958-DCB5-4EB9-9642-6E4A8384498C@supercoders.com.au> Message-ID: <2733E6DC-892E-4EDE-AAFA-92F6437323B6@supercoders.com.au> Thanks to Tim and Rory and Ben Finney who all chipped in with ideas. Just to follow through, here?s the solution I put together in the end. It spawns a netcat process in response to a POST request (Python 3.6). I managed to avoid implementing a queueing system and lots of file juggling. It?s just plain TCP sockets, not websockets. Send this to get a new netcat process spawned on a port: curl -X POST -H "Content-Type: text/plain" --data ?example data' localhost:8000 from http.server import HTTPServer, BaseHTTPRequestHandler import socket import os import io host = '0.0.0.0' port = 8000 def find_free_port(): with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s: s.bind(('', 0)) return s.getsockname()[1] def spawn_netcat(port): command = "/bin/nc" params = f"nc -l {port}" spawnDaemon(command, params.split()) def spawnDaemon(path_to_executable, params): # Do first fork pid = os.fork() if (pid != 0): os.waitid(os.P_PID, pid, os.WEXITED) return # Decouple from parent environment os.chdir("/opt") os.setsid() os.umask(0) # Do second fork pid = os.fork() if (pid != 0): os._exit(0) # exec this process into netcat..... os.execv(path_to_executable, params) class Server(BaseHTTPRequestHandler): def do_POST(self): netcat_listen_port = find_free_port() spawn_netcat(netcat_listen_port) self.send_response(200) self.send_header("Content-type", "text/plain") self.end_headers() response = io.BytesIO() response.write(str(netcat_listen_port).encode()) self.wfile.write(response.getvalue()) if __name__ == "__main__": httpd = HTTPServer((host, port), Server) httpd.serve_forever() From hartror at gmail.com Thu Jul 26 02:50:21 2018 From: hartror at gmail.com (Rory Hart) Date: Thu, 26 Jul 2018 16:50:21 +1000 Subject: [melbourne-pug] Spawn new process to handle inbound websocket connection In-Reply-To: <2733E6DC-892E-4EDE-AAFA-92F6437323B6@supercoders.com.au> References: <507B1958-DCB5-4EB9-9642-6E4A8384498C@supercoders.com.au> <2733E6DC-892E-4EDE-AAFA-92F6437323B6@supercoders.com.au> Message-ID: <19EFFB89-E232-4C2A-A702-C5A29FC2F80F@gmail.com> Your implementation has a race condition in it. If two threads run at the same time and get the same port from ?find_free_port? one of your nc calls will fail. Also why are you not using the subprocess module? > On 26 Jul 2018, at 4:11 pm, Andrew Stuart wrote: > > Thanks to Tim and Rory and Ben Finney who all chipped in with ideas. > > Just to follow through, here?s the solution I put together in the end. It spawns a netcat process in response to a POST request (Python 3.6). I managed to avoid implementing a queueing system and lots of file juggling. It?s just plain TCP sockets, not websockets. > > Send this to get a new netcat process spawned on a port: > curl -X POST -H "Content-Type: text/plain" --data ?example data' localhost:8000 > > > from http.server import HTTPServer, BaseHTTPRequestHandler > import socket > import os > import io > > host = '0.0.0.0' > port = 8000 > > def find_free_port(): > with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s: > s.bind(('', 0)) > return s.getsockname()[1] > > def spawn_netcat(port): > command = "/bin/nc" > params = f"nc -l {port}" > spawnDaemon(command, params.split()) > > def spawnDaemon(path_to_executable, params): > > # Do first fork > pid = os.fork() > if (pid != 0): > os.waitid(os.P_PID, pid, os.WEXITED) > return > > # Decouple from parent environment > os.chdir("/opt") > os.setsid() > os.umask(0) > > # Do second fork > pid = os.fork() > if (pid != 0): > os._exit(0) > > # exec this process into netcat..... > os.execv(path_to_executable, params) > > class Server(BaseHTTPRequestHandler): > > def do_POST(self): > netcat_listen_port = find_free_port() > spawn_netcat(netcat_listen_port) > self.send_response(200) > self.send_header("Content-type", "text/plain") > self.end_headers() > response = io.BytesIO() > response.write(str(netcat_listen_port).encode()) > self.wfile.write(response.getvalue()) > > if __name__ == "__main__": > httpd = HTTPServer((host, port), Server) > httpd.serve_forever() > > > > > > > _______________________________________________ > melbourne-pug mailing list > melbourne-pug at python.org > https://mail.python.org/mailman/listinfo/melbourne-pug From andrew.stuart at supercoders.com.au Thu Jul 26 02:58:41 2018 From: andrew.stuart at supercoders.com.au (Andrew Stuart) Date: Thu, 26 Jul 2018 16:58:41 +1000 Subject: [melbourne-pug] Spawn new process to handle inbound websocket connection In-Reply-To: <19EFFB89-E232-4C2A-A702-C5A29FC2F80F@gmail.com> References: <507B1958-DCB5-4EB9-9642-6E4A8384498C@supercoders.com.au> <2733E6DC-892E-4EDE-AAFA-92F6437323B6@supercoders.com.au> <19EFFB89-E232-4C2A-A702-C5A29FC2F80F@gmail.com> Message-ID: >> Your implementation has a race condition in it. If two threads run at the same time and get the same port from ?find_free_port? one of your nc calls will fail. Hmmm. I?ll have to think on that. Kinda hard to avoid because I need to know the port number so I can feed it back to the client, and so I can tell the netcat process which port to start on. And it is possible that the port number might be grabbed by a subsequent request. Maybe I can do something hacky like keep track of recently allocated port numbers for a few seconds to give the netcat process time to start on that port. >> Also why are you not using the subprocess module? Because I want the netcat processes to be entirely independent of the Python server that spawns them, so I do a double fork. From sambvfx at gmail.com Thu Jul 26 03:36:40 2018 From: sambvfx at gmail.com (Sam Bourne) Date: Thu, 26 Jul 2018 17:36:40 +1000 Subject: [melbourne-pug] Greetings From Sam Message-ID: Hello Melbourne Python Aficionados, My name is Sam, I'm a dev over at a VFX company Luma Pictures primarily working on our pipeline. I have around 8 years of experience at this point - most of which has been spent writing python. I recently moved to Melbourne with my wife from Los Angeles - so I joined this mailing list hoping to connect with some other developers here. Happy to chat! -Sam -------------- next part -------------- An HTML attachment was scrubbed... URL: From hartror at gmail.com Thu Jul 26 04:18:25 2018 From: hartror at gmail.com (Rory Hart) Date: Thu, 26 Jul 2018 18:18:25 +1000 Subject: [melbourne-pug] Spawn new process to handle inbound websocket connection In-Reply-To: References: <507B1958-DCB5-4EB9-9642-6E4A8384498C@supercoders.com.au> <2733E6DC-892E-4EDE-AAFA-92F6437323B6@supercoders.com.au> <19EFFB89-E232-4C2A-A702-C5A29FC2F80F@gmail.com> Message-ID: <6FF7534D-EFA3-4FE7-A266-C009D57390C0@gmail.com> You are still at risk of another process grabbing the port, you need to open the socket collect the port number in the web host and hold on to it until the fork. Then the child process has the open socket after the fork. I feel the need to point out at this point the architecture you have chosen is non-standard and will be challenging for others to grok quickly and support. For example by opening random ports for incoming connections creating effective firewall rules is next to impossible, as is monitoring traffic. I?m not sure what the purpose of this project is so maybe these maintainability considerations do not apply? On 26 Jul 2018, at 4:58 pm, Andrew Stuart wrote: >>> Your implementation has a race condition in it. If two threads run at the same time and get the same port from ?find_free_port? one of your nc calls will fail. > > Hmmm. I?ll have to think on that. Kinda hard to avoid because I need to know the port number so I can feed it back to the client, and so I can tell the netcat process which port to start on. And it is possible that the port number might be grabbed by a subsequent request. Maybe I can do something hacky like keep track of recently allocated port numbers for a few seconds to give the netcat process time to start on that port. > >>> Also why are you not using the subprocess module? > > Because I want the netcat processes to be entirely independent of the Python server that spawns them, so I do a double fork. > > _______________________________________________ > melbourne-pug mailing list > melbourne-pug at python.org > https://mail.python.org/mailman/listinfo/melbourne-pug From william.leslie.ttg at gmail.com Thu Jul 26 04:20:17 2018 From: william.leslie.ttg at gmail.com (William ML Leslie) Date: Thu, 26 Jul 2018 18:20:17 +1000 Subject: [melbourne-pug] Spawn new process to handle inbound websocket connection In-Reply-To: <507B1958-DCB5-4EB9-9642-6E4A8384498C@supercoders.com.au> References: <507B1958-DCB5-4EB9-9642-6E4A8384498C@supercoders.com.au> Message-ID: On 24 July 2018 at 07:19, Andrew Stuart wrote: > I have servers that send a sequence of PNG images which need to be processed via a sequence of commands. > > My idea is this: > > A Python websockets server is listening for connections from the servers. When a websocket connection is received, a new Python process is spawned - presumably using https://docs.python.org/3/library/multiprocessing.html and somehow the websocket is given to that process (how?). The spawned process listens to the websocket, receives the PNG images and then processes them. After completion, the spawned process dies and the websocket is closed. So the spawned Python processes might run for up to a few minutes each. > > Is this a workable idea in Python? > > One open question is how would I hand the websocket to the spawned process? > > Also, coud such an approach support tens, hundreds, thousands of connections on a for example 8 core machine with 16 gig RAM ( know this is how long is a piece of string, but just finger in the air guessing to make sure there?s no huge gotcha in there that would impose a huge constraint & invalidate the idea)? > > I was looking at this gist as inspiration for a start point https://gist.github.com/gdamjan/d7333a4d9069af96fa4d > > Any ideas/feedback valued. > Is this a batch process - someone uploads all their images at once, and you run the same algorithm on them? I wonder if websockets are a good fit. Normally you'd use them where you have a constant stream of messages or an extremely interactive process. -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely MAY reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to DENY YOU THOSE RIGHTS would be illegal without prior contractual agreement. From hartror at gmail.com Thu Jul 26 04:20:57 2018 From: hartror at gmail.com (Rory Hart) Date: Thu, 26 Jul 2018 18:20:57 +1000 Subject: [melbourne-pug] Greetings From Sam In-Reply-To: References: Message-ID: Welcome to Melbourne Sam. Any chance you will be making it to MPUG next month. Or PyCon AU in Sydney later in the month? > On 26 Jul 2018, at 5:36 pm, Sam Bourne wrote: > > Hello Melbourne Python Aficionados, > > My name is Sam, I'm a dev over at a VFX company Luma Pictures primarily working on our pipeline. I have around 8 years of experience at this point - most of which has been spent writing python. > > I recently moved to Melbourne with my wife from Los Angeles - so I joined this mailing list hoping to connect with some other developers here. > > Happy to chat! > -Sam > _______________________________________________ > melbourne-pug mailing list > melbourne-pug at python.org > https://mail.python.org/mailman/listinfo/melbourne-pug -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew.stuart at supercoders.com.au Thu Jul 26 05:08:06 2018 From: andrew.stuart at supercoders.com.au (Andrew Stuart) Date: Thu, 26 Jul 2018 19:08:06 +1000 Subject: [melbourne-pug] Spawn new process to handle inbound websocket connection In-Reply-To: <6FF7534D-EFA3-4FE7-A266-C009D57390C0@gmail.com> References: <507B1958-DCB5-4EB9-9642-6E4A8384498C@supercoders.com.au> <2733E6DC-892E-4EDE-AAFA-92F6437323B6@supercoders.com.au> <19EFFB89-E232-4C2A-A702-C5A29FC2F80F@gmail.com> <6FF7534D-EFA3-4FE7-A266-C009D57390C0@gmail.com> Message-ID: >>> I?m not sure what the purpose of this project is so maybe these maintainability considerations do not apply? Yes its a purely internal server to server system that has the objective of being as close to real time as possible - it?s not a batch processing system so the idea is to do everything to avoid the disk, pass the data as directly and quickly as possible, and to be in a position to benefit from higher powered multi CPU servers to crunch through the data as fast as possible. So there?s also no need to worry about preserving/losing data if there is a fault because the data is real time live video data and not critical - focus is on being as fast as practical (for long running processes). >>> the architecture you have chosen is non-standard and will be challenging for others to grok quickly and support I?m the only person working on it so that?s ok and I take notes about what I built and why :-) nothing quite like coming backl to your own code six months later and wondering how it works. >>> You are still at risk of another process grabbing the port, you need to open the socket collect the port number in the web host and hold on to it until the fork. Then the child process has the open socket after the fork. Good pickup Rory I?ll do some work in this area. It also appears I might be able to instruct the OS to reserve a range of ports so I can directly control allocations via ip_local_reserved_ports ref: https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt this would also ensure the OS or some other process does not grab ports that I allocated to a spawned process. Thanks for your feedback it?s nice to collaborate when in a team of one! On 26 Jul 2018, at 6:18 pm, Rory Hart wrote: You are still at risk of another process grabbing the port, you need to open the socket collect the port number in the web host and hold on to it until the fork. Then the child process has the open socket after the fork. I feel the need to point out at this point the architecture you have chosen is non-standard and will be challenging for others to grok quickly and support. For example by opening random ports for incoming connections creating effective firewall rules is next to impossible, as is monitoring traffic. I?m not sure what the purpose of this project is so maybe these maintainability considerations do not apply? On 26 Jul 2018, at 4:58 pm, Andrew Stuart wrote: >>> Your implementation has a race condition in it. If two threads run at the same time and get the same port from ?find_free_port? one of your nc calls will fail. > > Hmmm. I?ll have to think on that. Kinda hard to avoid because I need to know the port number so I can feed it back to the client, and so I can tell the netcat process which port to start on. And it is possible that the port number might be grabbed by a subsequent request. Maybe I can do something hacky like keep track of recently allocated port numbers for a few seconds to give the netcat process time to start on that port. > >>> Also why are you not using the subprocess module? > > Because I want the netcat processes to be entirely independent of the Python server that spawns them, so I do a double fork. > > _______________________________________________ > melbourne-pug mailing list > melbourne-pug at python.org > https://mail.python.org/mailman/listinfo/melbourne-pug _______________________________________________ melbourne-pug mailing list melbourne-pug at python.org https://mail.python.org/mailman/listinfo/melbourne-pug From sambvfx at gmail.com Thu Jul 26 06:43:15 2018 From: sambvfx at gmail.com (Sam Bourne) Date: Thu, 26 Jul 2018 20:43:15 +1000 Subject: [melbourne-pug] Greetings From Sam In-Reply-To: References:

Message-ID: I do plan on attending MPUG and am excited to meet some of you there. I'm unsure about pycon at the moment... On Thu, Jul 26, 2018, 6:21 PM Rory Hart wrote: > Welcome to Melbourne Sam. Any chance you will be making it to MPUG next > month. Or PyCon AU in Sydney later in the month? > > On 26 Jul 2018, at 5:36 pm, Sam Bourne wrote: > > Hello Melbourne Python Aficionados, > > My name is Sam, I'm a dev over at a VFX company Luma Pictures > primarily working on our pipeline. I have > around 8 years of experience at this point - most of which has been spent > writing python. > > I recently moved to Melbourne with my wife from Los Angeles - so I joined > this mailing list hoping to connect with some other developers here. > > Happy to chat! > -Sam > > _______________________________________________ > melbourne-pug mailing list > melbourne-pug at python.org > https://mail.python.org/mailman/listinfo/melbourne-pug > > _______________________________________________ > melbourne-pug mailing list > melbourne-pug at python.org > https://mail.python.org/mailman/listinfo/melbourne-pug > -------------- next part -------------- An HTML attachment was scrubbed... URL: