From javier at candeira.com Fri Nov 1 00:02:57 2013 From: javier at candeira.com (Javier Candeira) Date: Fri, 1 Nov 2013 10:02:57 +1100 Subject: [melbourne-pug] R to Pandas Cookbook In-Reply-To: References: Message-ID: That's amazing, thanks, Lex! J On Fri, Nov 1, 2013 at 9:45 AM, Lex H wrote: > If you're not aware of the Pandas project it's Python's answer to R, and > it's awesome. > > http://pandas.pydata.org/ > http://blog.wesmckinney.com/ (Pandas author's blog) > > A while back I started making some notes on how to do the various recipes in > O'Reilly's R Cookbook (http://shop.oreilly.com/product/9780596809164.do) > with Numpy, Pandas, Scipy. > > I haven't had time to complete it so I'm sharing it in it's current state, > and trying to get some community help to fill in the gaps. > > I think this could be an extremely useful resource to encourage and help > transition lots of people from R to Pandas. > > So here's the notes: > > http://notes.lexual.com/tech/r_numpy_pandas_cookbook.html > > And here's the github repo, patches more than welcome! > > https://github.com/lexual/sphinx-notes/blob/master/source/tech/r_numpy_pandas_cookbook.rst > > Cheers, > > Lex. > > > > _______________________________________________ > melbourne-pug mailing list > melbourne-pug at python.org > https://mail.python.org/mailman/listinfo/melbourne-pug > From sam at nipl.net Fri Nov 1 02:26:12 2013 From: sam at nipl.net (Sam Watkins) Date: Fri, 1 Nov 2013 12:26:12 +1100 Subject: [melbourne-pug] R to Pandas Cookbook In-Reply-To: References: Message-ID: <20131101012612.GJ23669@opal.nipl.net> On Fri, Nov 01, 2013 at 09:45:08AM +1100, Lex H wrote: > Pandas project > http://notes.lexual.com/tech/r_numpy_pandas_cookbook.html Good stuff. I'll remember Pandas and this WIP cookbook next time I need to do some stats / modelling. Sam From javier at candeira.com Mon Nov 4 04:31:13 2013 From: javier at candeira.com (Javier Candeira) Date: Mon, 4 Nov 2013 14:31:13 +1100 Subject: [melbourne-pug] November MPUG meeting: Machine Vision and Django CMSs, Tonight 6PM - Inspire 9, 41 Stewart St, Richmond Message-ID: Dear Melbourne Pythonistas, As previously announced, the next meeting of the Melbourne Python Users Group will be tonight 4 November at 6PM. Venue: Inspire 9, 41 Stewart St. Richmond. 50m from Richmond Train Station. And this is the current talk lineup: # Lars Yencken -- Machine Vision with SimpleCV. # Nicole Harris -- Mezzanine, the best Django CMS. Note the change of programme, and thanks to Nicole for stepping up! We can still fit in a 5 minute short talk for this session, so please volunteer or dob in a friend! You can do it anonymously using our wiki: https://wiki.python.org/moin/MelbournePUG Javier & the MPUG organizers. From lex.lists at gmail.com Tue Nov 5 00:53:05 2013 From: lex.lists at gmail.com (Lex H) Date: Tue, 5 Nov 2013 10:53:05 +1100 Subject: [melbourne-pug] Melbourne Pythonista wins Django Prize Message-ID: This is very cool! https://www.djangoproject.com/weblog/2013/nov/04/announcing-inaugural-winner-malcolm-tredinnick-mem/ Lex. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bianca.rachel.gibson at gmail.com Tue Nov 5 00:58:42 2013 From: bianca.rachel.gibson at gmail.com (Bianca Gibson) Date: Tue, 5 Nov 2013 10:58:42 +1100 Subject: [melbourne-pug] Fwd: [Linux-aus] Linux Australia Travel Grants In-Reply-To: References: Message-ID: "Linux Australia and linux.conf.au 2014 are pleased to announce the 2014 travel grants programme. linux.conf.au strives to be a conference that is open and accessible to the whole open source community. We realise however that, no matter how we arrange the pricing for the conference, it will still be unaffordable for some members of the community. These grants aim to reduce the financial barriers to attending linux.conf.au2014, by subsidising the registration and travel costs of contributors to the community who would otherwise have difficulty affording the cost of attending the conference. Please go to http://travelgrants.linux.org.au/ to read more and/or put in an application. Applications are due by 15 November, and winners will be informed by 1 December. LCA Travel Grants Organisers Michael Still and Kylie Willison" -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew.stuart at supercoders.com.au Mon Nov 4 07:03:17 2013 From: andrew.stuart at supercoders.com.au (Andrew Stuart) Date: Mon, 4 Nov 2013 17:03:17 +1100 Subject: [melbourne-pug] Melbourne job: Python developer(s) Message-ID: <656B3663-E6C9-488A-85DB-03B0CC438736@supercoders.com.au> Hello folks, We have several Python jobs available at the moment. There are several positions available paying $85K to $100K plus super. The positions are using Python 2.7. For the most part these jobs entail working with very large databases, formulating fairly sophisticated data retrieval queries, munging and mashing data into a final form. Extracting, sifting, straining, crunching and squashing until crispy. Ideally you'll have a solid grasp of Postgres and an interest in NoSQL, map reduce, Hadoop and such stuff. Shell scripting is not required but would be nice to have. Experience with other languages such as C++ also nice to have but not required. Understanding of REST, JSON, XML is helpful. To apply, send your resume to apply at supercoders.com.au and ensure your subject line includes "Python direct" Inner Melbourne (south) location. Questions to andrew.stuart at supercoders.com.au thanks Andrew Stuart From tleeuwenburg at gmail.com Wed Nov 6 03:51:49 2013 From: tleeuwenburg at gmail.com (Tennessee Leeuwenburg) Date: Wed, 6 Nov 2013 13:51:49 +1100 Subject: [melbourne-pug] Melbourne Pythonista wins Django Prize In-Reply-To: References: Message-ID: Wonderful! Fantastic to see recognition going to people who deserve it. Well done to Curtis / FunkyBob. 2013/11/5 Lex H > This is very cool! > > > https://www.djangoproject.com/weblog/2013/nov/04/announcing-inaugural-winner-malcolm-tredinnick-mem/ > > Lex. > > _______________________________________________ > melbourne-pug mailing list > melbourne-pug at python.org > https://mail.python.org/mailman/listinfo/melbourne-pug > > -- -------------------------------------------------- Tennessee Leeuwenburg http://myownhat.blogspot.com/ "Don't believe everything you think" -------------- next part -------------- An HTML attachment was scrubbed... URL: From bianca.rachel.gibson at gmail.com Mon Nov 11 23:20:43 2013 From: bianca.rachel.gibson at gmail.com (Bianca Gibson) Date: Tue, 12 Nov 2013 09:20:43 +1100 Subject: [melbourne-pug] OWOOT Melbourne picnic Message-ID: Hi all, There is an Oceania Women Of Open Technology (OWOOT) picnic coming up in December. Partners and children welcome. ---------- Forwarded message ---------- Hi all, It's going to be at 1pm Saturday 14th of December at the western lawn in the botanic gardens. RSVP, dietary requirements and information in the following pad: http://piratepad.net/HsR3wzMLDH Cheers, Bianca -------------- next part -------------- An HTML attachment was scrubbed... URL: From bianca.rachel.gibson at gmail.com Tue Nov 12 08:36:36 2013 From: bianca.rachel.gibson at gmail.com (Bianca Gibson) Date: Tue, 12 Nov 2013 18:36:36 +1100 Subject: [melbourne-pug] Fwd: [Linux-aus] Three days left: Nominate a community contributor for the 2014 Rusty Wrench In-Reply-To: <20131111222000.GA18156@flay.puzzling.org> References: <20131111222000.GA18156@flay.puzzling.org> Message-ID: ---------- Forwarded message ---------- From: Mary Gardiner Nominations for the Linux Australia 2014 Rusty Wrench award are open for just THREE MORE DAYS! This year, thanks to Linux Australia Council, the Rusty Wrench winner will receive admission to linux.conf.au 2014 in Perth and travel costs from Australian or New Zealand locations. Who is giving to the Free and Open Source Software community? Creating it, making us stronger and representing our interests? This is your chance to acknowledge them at http://linux.org.au/vote/rusty-wrench-nomination-form. ** Timeline ** Nominations close THIS Friday November 15, 2013 Winner announced at linux.conf.conf.au 2014 (6th to 10th January) ** More information ** See http://linux.org.au/projects/awards for the history of the award and http://linux.org.au/rusty-wrench-selection-infomation for more information about what community contributions the Rusty Wrench award recognises! Make your nomination at http://linux.org.au/vote/rusty-wrench-nomination-form Existing 2014 nominations can be found at http://linux.org.au/vote/rusty-wrench-nomination-form/nominations/2014 We *encourage* duplicate nominations of the same people, because they can add more information. -Mary Gardiner (with Rusty Russell, Pia Waugh, Kimberlee Weatherall, Donna Benjamin) _______________________________________________ linux-aus mailing list linux-aus at lists.linux.org.au http://lists.linux.org.au/listinfo/linux-aus -------------- next part -------------- An HTML attachment was scrubbed... URL: From bianca.rachel.gibson at gmail.com Wed Nov 13 02:23:29 2013 From: bianca.rachel.gibson at gmail.com (Bianca Gibson) Date: Wed, 13 Nov 2013 12:23:29 +1100 Subject: [melbourne-pug] Fwd: [Linux-aus] Linux Australia Travel Grants In-Reply-To: References: Message-ID: 2 days left to apply for Linux.conf.au travel grants. On 5 Nov 2013 10:58, "Bianca Gibson" wrote: > > "Linux Australia and linux.conf.au 2014 are pleased to announce the 2014 > travel grants programme. > > linux.conf.au strives to be a conference that is open and accessible to > the whole open source community. We realise however that, no matter how we > arrange the pricing for the conference, it will still be unaffordable for > some members of the community. > > These grants aim to reduce the financial barriers to attending > linux.conf.au 2014, by subsidising the registration and travel costs of > contributors to the community who would otherwise have difficulty affording > the cost of attending the conference. > > Please go to http://travelgrants.linux.org.au/ to read more and/or put in > an application. Applications are due by 15 November, and winners will be > informed by 1 December. > > LCA Travel Grants Organisers > Michael Still and Kylie Willison" > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rfkelly at mozilla.com Wed Nov 13 06:44:19 2013 From: rfkelly at mozilla.com (Ryan Kelly) Date: Wed, 13 Nov 2013 16:44:19 +1100 Subject: [melbourne-pug] [JOB] Mozilla Cloud Services is hiring Message-ID: <528311B3.6040406@mozilla.com> Hi All, Mozilla's Cloud Services team is looking for engineers to help us bring the philosophy of the Open Web out of the Browser and into the Cloud: https://careers.mozilla.org/en-US/position/oFn3XfwM We're a distributed team of server-side engineers who write a lot of Python, although we also have some projects in JavaScript and Go. Everything we write is open-source and is built on open standards - in fact you can view most of it right here: https://github.com/mozilla-services/ And of course my favorite part: it's code that has the potential to impact millions of Firefox users around the world. Interested? Please reply off-list for more details or a chat. Oh, I'm told that applying via this special link will flag you as a referral in our hiring software, which can help grease the wheels a little with our HR department: http://hire.jobvite.com/Jobvite/jobvite.aspx?b=nkfsClwt Cheers, Ryan From bianca.rachel.gibson at gmail.com Tue Nov 19 02:27:44 2013 From: bianca.rachel.gibson at gmail.com (Bianca Gibson) Date: Tue, 19 Nov 2013 12:27:44 +1100 Subject: [melbourne-pug] Fwd: [Linux-aus] FLOSS 2013 Survey you may be interested in In-Reply-To: <5289DA8D.2010207@linux.org.au> References: <5289DA8D.2010207@linux.org.au> Message-ID: ---------- Forwarded message ---------- Hi everyone, You may be interested in the survey below The FLOSS 2013 survey In 2002, the first FLOSS survey was launched. With over 2500 participants, it was the first large survey of Free/Libre/Open Source developers around the world. This survey had major impact in the community, academia and politics. 10 years later we would like to replicate the survey in order to see how the community has changed. http://t.co/1df8p71n2A Cheers, Bianca -------------- next part -------------- An HTML attachment was scrubbed... URL: From alecclews at gmail.com Tue Nov 19 04:48:24 2013 From: alecclews at gmail.com (Alec Clews) Date: Tue, 19 Nov 2013 14:48:24 +1100 Subject: [melbourne-pug] Social Software Hacking Sun 8th December In-Reply-To: <1288871041.1384730600464.JavaMail.root@jobs1.meetup.com> References: <1288871041.1384730600464.JavaMail.root@jobs1.meetup.com> Message-ID: <528ADF88.1050505@gmail.com> A Hackday at the Melbourne Hackspace (NB Limited places) 1) Open to all, no charge 2) Software only -- no loud tools to stop people thinking. 4) Limited to 14 people. Please RSVP below 5) Suggested activities (to be organised by attendees) a) Project and social hacking. Singly or in groups. Bring your project b) Code Dojo http://codingdojo.org/cgi-bin/wiki.pl?WhatIsCodingDojo (NB Not the Coder dojo for young kids) c) Short workshops in the back room, organised barcamp style Sunday, December 8, 2013 at 10:00 AM Hackspace has Power, WiFi, Ethernet, cold drinks ($1) and public transport. There are a few cafes etc around Glenferrie open on a Sunday as well. Please RSVP at http://www.meetup.com/Connected-Community-HackerSpace-Melbourne/events/151411092/ If it's successful we might run them monthly in 2014 Regards -- Alec Clews http://about.me/alecthegeek From chrisjrn at gmail.com Tue Nov 19 22:34:23 2013 From: chrisjrn at gmail.com (Chris Neugebauer) Date: Wed, 20 Nov 2013 08:34:23 +1100 Subject: [melbourne-pug] =?iso-8859-1?q?Travel_Grants_for_Australians_and_?= =?iso-8859-1?q?New_Zealanders_to_PyCon_US_in_Montr=E9al=2C_April_2?= =?iso-8859-1?q?014?= Message-ID: PyCon Australia and Linux Australia are very pleased to announce that we're contributing to the travel assistance scheme of PyCon US, the biggest Python event in the world. PyCon US is next being held in Montr?al, Canada in April. This means that PyCon will be able to help fund the attendance of more members of the Python community from Australia and New Zealand. Financial assistance for PyCon is offered to help people who otherwise couldn't afford to attend the conference, and is offered in the form of travel subsidies, discounted or free registration, and accommodation. Basically, if you'll benefit from being at PyCon, or the Python community will benefit from your attendance, then we want to make sure you can make it. For further details about the programme, and to apply for funding, visit https://us.pycon.org/2014/assistance/ There's no extra process for qualifying for the extra funding that we're putting up -- just note that you're from Australia or New Zealand in your application. Applications close on 1 January, 2014. If you've got any further queries about the programme, or PyCon Australia's involvement in it, send me an e-mail at chrisjrn at gmail.com -- if I can't answer it, I'll make sure someone from PyCon US does! We're looking forward to seeing a record number of Aussies and Kiwis at PyCon in Montr?al! -- --Christopher Neugebauer Jabber: chrisjrn at gmail.com -- IRC: chrisjrn on irc.freenode.net -- AIM: chrisjrn157 -- MSN: chris at neugebauer.id.au -- WWW: http://chris.neugebauer.id.au -- Twitter/Identi.ca: @chrisjrn From brian at microcomaustralia.com.au Wed Nov 20 06:33:42 2013 From: brian at microcomaustralia.com.au (Brian May) Date: Wed, 20 Nov 2013 16:33:42 +1100 Subject: [melbourne-pug] django db race conditions In-Reply-To: <52609941.1050602@openminddev.net> References: <52609941.1050602@openminddev.net> Message-ID: On 18 October 2013 13:13, Rasjid Wilcox wrote: > The insert into get_data_guard should be completely atomic - it will only > ever succeed for one caller, and so it should eliminate the race condition. > Hmmm. Wonder how this will work with transactions... Just tried running the following test code: --- cut --- #!/usr/bin/env python import os os.environ['DJANGO_SETTINGS_MODULE'] = 'kgadmin.conf.settings' import datetime import time import django.db.transaction from karaage.machines.models import Machine from karaage.cache.models import MachineCache @django.db.transaction.commit_on_success def meow(): start=datetime.date(year=2003,month=12,day=25) end=datetime.date(year=2003,month=12,day=25) print "meowing" machine = Machine.objects.get(pk=1) print "barking" MachineCache.objects.create(machine=machine, start=start, end=end, cpu_hours=10, no_jobs=10) print "sleeping" time.sleep(10) print "eating" raise RuntimeError("oops") meow() --- cut --- If I run one copy, it works fine: meowing barking sleeping eating Traceback (most recent call last): File "./test", line 24, in meow() File "/usr/lib/python2.7/dist-packages/django/db/transaction.py", line 223, in inner return func(*args, **kwargs) File "./test", line 22, in meow raise RuntimeError("oops") RuntimeError: oops If I run another copy while the first one is sleeping, the 2nd copy gets locked out until the first one finishes: meowing barking [ hangs here ] sleeping eating Traceback (most recent call last): File "./test", line 24, in meow() File "/usr/lib/python2.7/dist-packages/django/db/transaction.py", line 223, in inner return func(*args, **kwargs) File "./test", line 22, in meow raise RuntimeError("oops") RuntimeError: oops I wasn't aware a transaction could lock out other processes like this. Something to be careful of when doing http requests that should return quickly. Would be interested to know just what is happening here, and what the scope is of this "lock". (also note this is Django 1.5, apparently transactions support is different in 1.6; I don't really understand what has changed however). Will continue looking at this tomorrow. -- Brian May -------------- next part -------------- An HTML attachment was scrubbed... URL: From miked at dewhirst.com.au Wed Nov 20 08:51:08 2013 From: miked at dewhirst.com.au (Mike Dewhirst) Date: Wed, 20 Nov 2013 18:51:08 +1100 Subject: [melbourne-pug] django db race conditions In-Reply-To: References: <52609941.1050602@openminddev.net> Message-ID: <528C69EC.8010303@dewhirst.com.au> On 20/11/2013 4:33pm, Brian May wrote: > If I run another copy while the first one is sleeping, the 2nd copy gets > locked out until the first one finishes: > If you are running on the dev server, I believe it is only capable of dealing with one request at a time. Not saying this is the problem - just a thought. Mike From rasjidw at openminddev.net Wed Nov 20 14:25:26 2013 From: rasjidw at openminddev.net (Rasjid Wilcox) Date: Thu, 21 Nov 2013 00:25:26 +1100 Subject: [melbourne-pug] django db race conditions In-Reply-To: References: <52609941.1050602@openminddev.net> Message-ID: <528CB846.1010302@openminddev.net> On 20/11/2013 4:33 PM, Brian May wrote: > On 18 October 2013 13:13, Rasjid Wilcox > wrote: > > The insert into get_data_guard should be completely atomic - it > will only ever succeed for one caller, and so it should eliminate > the race condition. > > > Hmmm. Wonder how this will work with transactions... > > Just tried running the following test code: I can't really comment a lot, since I've avoided django after looking at it a number of years ago and deciding that its ORM was too limited for my needs. You may need to drop down to raw sql, rather than relying on the django ORM. At the very least you will want to look at exactly what sql is being sent to the server. See https://docs.djangoproject.com/en/dev/faq/models/#how-can-i-see-the-raw-sql-queries-django-is-running for info on that, although I don't know if that will tell you when each transaction is starting and stopping. However, looking at your code below, I'm guessing that the issue is you are wrapping the entire process up in a single transaction. I think you would be better off with two steps or at least two transactions: My original post suggested having a new and completely separate table just to do the guarding, and that may be the simpler way. If you don't want to do that, I would suggest using two transactions. In transaction number one you attempt to insert your key (start + stop + machine?) into your table, with a flag saying that the data is being generated. This should either succeed (if you are the first) or fail (for subsequent attempts) almost instantly. If the key insert is successful, you generate the data (outside a database transaction). You then (in transaction two) store the data into the table and clear the 'being generated' flag. If the key insert fails, then someone else has already or is currently generating the data. You read the table - if the 'being generated' flag is clear, your data should already be there. If the flag is true, you wait a bit and try again. (With Postgresql you could use NOTIFY and LISTEN rather than polling.) With either approach, you want a transaction just around the insert, not the whole data generation process. If this is not easy to code using Django's ORM, then I would suggest going the separate 'guard table' route and using raw sql (just for that table). Cheers, Rasjid. > > --- cut --- > #!/usr/bin/env python > > import os > os.environ['DJANGO_SETTINGS_MODULE'] = 'kgadmin.conf.settings' > > import datetime > import time > import django.db.transaction > from karaage.machines.models import Machine > from karaage.cache.models import MachineCache > > @django.db.transaction.commit_on_success > def meow(): > start=datetime.date(year=2003,month=12,day=25) > end=datetime.date(year=2003,month=12,day=25) > print "meowing" > machine = Machine.objects.get(pk=1) > print "barking" > MachineCache.objects.create(machine=machine, start=start, end=end, > cpu_hours=10, no_jobs=10) > print "sleeping" > time.sleep(10) > print "eating" > raise RuntimeError("oops") > > meow() > --- cut --- > > If I run one copy, it works fine: > > meowing > barking > sleeping > eating > Traceback (most recent call last): > File "./test", line 24, in > meow() > File "/usr/lib/python2.7/dist-packages/django/db/transaction.py", > line 223, in inner > return func(*args, **kwargs) > File "./test", line 22, in meow > raise RuntimeError("oops") > RuntimeError: oops > > If I run another copy while the first one is sleeping, the 2nd copy > gets locked out until the first one finishes: > > meowing > barking > [ hangs here ] > sleeping > eating > Traceback (most recent call last): > File "./test", line 24, in > meow() > File "/usr/lib/python2.7/dist-packages/django/db/transaction.py", > line 223, in inner > return func(*args, **kwargs) > File "./test", line 22, in meow > raise RuntimeError("oops") > RuntimeError: oops > > I wasn't aware a transaction could lock out other processes like this. > Something to be careful of when doing http requests that should return > quickly. > > Would be interested to know just what is happening here, and what the > scope is of this "lock". > > (also note this is Django 1.5, apparently transactions support is > different in 1.6; I don't really understand what has changed however). > > Will continue looking at this tomorrow. > -- > Brian May > > > > _______________________________________________ > melbourne-pug mailing list > melbourne-pug at python.org > https://mail.python.org/mailman/listinfo/melbourne-pug -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian at microcomaustralia.com.au Fri Nov 22 00:12:24 2013 From: brian at microcomaustralia.com.au (Brian May) Date: Fri, 22 Nov 2013 10:12:24 +1100 Subject: [melbourne-pug] django db race conditions In-Reply-To: <528C69EC.8010303@dewhirst.com.au> References: <52609941.1050602@openminddev.net> <528C69EC.8010303@dewhirst.com.au> Message-ID: On 20 November 2013 18:51, Mike Dewhirst wrote: > If you are running on the dev server, I believe it is only capable of > dealing with one request at a time. Not saying this is the problem - just a > thought. > Suspect what you are referring to is if you run the server like this: ./manage.py runserver You will get a single process single threaded server that can only process one HTTP request at the time. However in my case, I was running two separate processes at the same time, and neither process knows about the other process in any way. -- Brian May -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian at microcomaustralia.com.au Fri Nov 22 00:56:57 2013 From: brian at microcomaustralia.com.au (Brian May) Date: Fri, 22 Nov 2013 10:56:57 +1100 Subject: [melbourne-pug] django db race conditions In-Reply-To: <528CB846.1010302@openminddev.net> References: <52609941.1050602@openminddev.net> <528CB846.1010302@openminddev.net> Message-ID: On 21 November 2013 00:25, Rasjid Wilcox wrote: > I can't really comment a lot, since I've avoided django after looking at > it a number of years ago and deciding that its ORM was too limited for my > needs. You may need to drop down to raw sql, rather than relying on the > django ORM. At the very least you will want to look at exactly what sql is > being sent to the server. See > https://docs.djangoproject.com/en/dev/faq/models/#how-can-i-see-the-raw-sql-queries-django-is-runningfor info on that, although I don't know if that will tell you when each > transaction is starting and stopping. > You might find any limitations you once had with Django ORM no longer exist now. One limitation I have noticed though, is that above technique (and the other logging technique) for logging SQL don't log session start up / transaction stuff :-( This is Django 1.5 however, can't really report a bug until I have tested it with Django 1.6, a lot has changed in the db layer. So, I am restricted to using wireshark. Lets see: Process #1: SET NAMES utf8 set autocommit=0 SET SQL_AUTO_IS_NULL = 0 SELECT `machine`.`id`, `machine`.`password`, `machine`.`last_login`, `machine`.`name`, `machine`.`no_cpus`, `machine`.`no_nodes`, `machine`.`type`, `machine`.`category_id`, `machine`.`start_date`, `machine`.`end_date`, `machine`.`pbs_server_host`, `machine`.`mem_per_core`, `machine`.`scaling_factor` FROM `machine` WHERE `machine`.`id` = 1 Ping INSERT INTO `cache_machinecache` (`date`, `start`, `end`, `cpu_hours`, `no_jobs`, `machine_id`) VALUES ('2013-11-22', '2003-12-25', '2003-12-25', '10.00', 10, 1) [ 10 second sleep ] Process #2: SET NAMES utf8 set autocommit=0 SET SQL_AUTO_IS_NULL = 0 SELECT `machine`.`id`, `machine`.`password`, `machine`.`last_login`, `machine`.`name`, `machine`.`no_cpus`, `machine`.`no_nodes`, `machine`.`type`, `machine`.`category_id`, `machine`.`start_date`, `machine`.`end_date`, `machine`.`pbs_server_host`, `machine`.`mem_per_core`, `machine`.`scaling_factor` FROM `machine` WHERE `machine`.`id` = 1 Ping INSERT INTO `cache_machinecache` (`date`, `start`, `end`, `cpu_hours`, `no_jobs`, `machine_id`) VALUES ('2013-11-22', '2003-12-25', '2003-12-25', '10.00', 10, 1) [ doesn't return result yet ] 10 second delay Process #1: rollback Process #2 gets result rollback So I guess the magic line must be "set autocommit=0". Still not sure I understand how this locks other sessions however. However, looking at your code below, I'm guessing that the issue is you are > wrapping the entire process up in a single transaction. > That is the recommended practise - enable the middleware that wraps every web request in a transaction. Can be changed or a per method basis however. I think you would be better off with two steps or at least two transactions: > > My original post suggested having a new and completely separate table just > to do the guarding, and that may be the simpler way. > This is just a test to make sure I understand correctly what would happen. I don't want a web request to hang around for ages waiting for the cache to be filled. Which in turn requires Celery. So my latest attempt at the real code is: def gen_machine_cache(request, start, end): try: tc = TaskCache.objects.get(machine_category__isnull=True, date=datetime.date.today(), start=start, end=end) if tc.ready: return None result = Task.AsyncResult(tc.celery_task_id) if result.ready(): result.forget() tc.ready = True tc.save() return None except TaskCache.DoesNotExist: result = tasks.gen_machine_cache.delay(start, end) TaskCache.objects.create(machine_category=None, date=datetime.date.today(), start=start, end=end, celery_task_id=result.task_id) return render_to_response( 'usage/progress.html', { 'task_id': result.task_id }, context_instance=RequestContext(request)) As per your suggestion, am thinking of changing it to something like: def gen_machine_cache(request, start, end): try: tc = TaskCache.objects.create(machine_category=None, date=datetime.date.today(), start=start, end=end, celery_task_id="") result = tasks.gen_machine_cache.delay(start, end) tc.celery_task_id = result.task_id tc.save() except TaskCache.IntegrityError: tc = TaskCache.objects.get(machine_category=None, date=datetime.date.today(), start=start, end=end) if tc.ready: return None result = Task.AsyncResult(tc.celery_task_id) if result.ready(): result.forget() tc.ready = True tc.save() return None return render_to_response( 'usage/progress.html', { 'task_id': result.task_id }, context_instance=RequestContext(request)) The IntegrityError case doesn't cope if celery_task_id hasn't been initialized, however if my early tests are anything to go by, the 2nd call to create will not return until the first mysql session is complete, so celery_task_id should always be set at this point. With either approach, you want a transaction just around the insert, not > the whole data generation process. If this is not easy to code using > Django's ORM, then I would suggest going the separate 'guard table' route > and using raw sql (just for that table). > Not sure this is required. Actually, like I said, this stuff has changed in Django 1.6, so it might be better off testing Django 1.6 now. -- Brian May -------------- next part -------------- An HTML attachment was scrubbed... URL: From rasjidw at openminddev.net Fri Nov 22 02:35:07 2013 From: rasjidw at openminddev.net (Rasjid Wilcox) Date: Fri, 22 Nov 2013 12:35:07 +1100 Subject: [melbourne-pug] django db race conditions In-Reply-To: References: <52609941.1050602@openminddev.net> <528CB846.1010302@openminddev.net> Message-ID: <528EB4CB.2040506@openminddev.net> On 22/11/2013 10:56 AM, Brian May wrote: > On 21 November 2013 00:25, Rasjid Wilcox > wrote: > > I can't really comment a lot, since I've avoided django after > looking at it a number of years ago and deciding that its ORM was > too limited for my needs. You may need to drop down to raw sql, > rather than relying on the django ORM. At the very least you will > want to look at exactly what sql is being sent to the server. See > https://docs.djangoproject.com/en/dev/faq/models/#how-can-i-see-the-raw-sql-queries-django-is-runningfor > info on that, although I don't know if that will tell you when > each transaction is starting and stopping. > > > You might find any limitations you once had with Django ORM no longer > exist now. > https://code.djangoproject.com/ticket/373 - Multi-part keys are still not supported. (The ticket was opened 8 years ago.) That is a deal-breaker for me in the work that I do. Cheers, Rasjid. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian at microcomaustralia.com.au Fri Nov 22 05:18:29 2013 From: brian at microcomaustralia.com.au (Brian May) Date: Fri, 22 Nov 2013 15:18:29 +1100 Subject: [melbourne-pug] django db race conditions In-Reply-To: <528EB4CB.2040506@openminddev.net> References: <52609941.1050602@openminddev.net> <528CB846.1010302@openminddev.net> <528EB4CB.2040506@openminddev.net> Message-ID: On 22 November 2013 12:35, Rasjid Wilcox wrote: > https://code.djangoproject.com/ticket/373 - Multi-part keys are still > not supported. (The ticket was opened 8 years ago.) That is a > deal-breaker for me in the work that I do. > Very true. For some applications you can work around this by creating a unique index on the multiple values and keep the primary key as a single integer. Won't help if you need to need to support existing or legacy databases however. Seems this is one feature request that is deemed too hard, breaks too much existing code, and/or not enough interest to fix. There are a number of these irritations with Django that are considered "very hard" to fix, as mentioned at Django miniconf at PyCon AU 2013. Developer's don't want to rush in with quick fixes that are later determined to be just as broken as the original code. Or break compatibility with existing applications unnecessarily. -- Brian May -------------- next part -------------- An HTML attachment was scrubbed... URL: From javier at candeira.com Fri Nov 22 07:14:22 2013 From: javier at candeira.com (Javier Candeira) Date: Fri, 22 Nov 2013 17:14:22 +1100 Subject: [melbourne-pug] December MPUG meeting: LaTeX, Adventure Gaming, Supercomputers Message-ID: Dear Melbourne Pythonistas, This is the talks lineup for the next meeting of the Melbourne Python Users Group to be held on 2 December at 6PM. Venue: Inspire 9, 41 Stewart St. Richmond. 50m from Richmond Train Station. # Luke Miller -- My big gay adventure. Making, releasing and selling an indie game made in Python # Bianca Gibson -- Python and LaTeX # Nathan Faggian -- Python and Supercomputers: Celery, Redis, Mongo See you in 10 days, Javier & the MPUG organisers From bianca.rachel.gibson at gmail.com Wed Nov 27 06:07:45 2013 From: bianca.rachel.gibson at gmail.com (Bianca Gibson) Date: Wed, 27 Nov 2013 16:07:45 +1100 Subject: [melbourne-pug] December MPUG meeting: LaTeX, Adventure Gaming, Supercomputers In-Reply-To: References: Message-ID: Hi, I may need to cancel, I've done something to my foot. I'll let you know if I can't make it. Cheers, Bianca -------------- next part -------------- An HTML attachment was scrubbed... URL: From javier at candeira.com Wed Nov 27 06:16:02 2013 From: javier at candeira.com (Javier Candeira) Date: Wed, 27 Nov 2013 16:16:02 +1100 Subject: [melbourne-pug] December MPUG meeting: LaTeX, Adventure Gaming, Supercomputers In-Reply-To: References: Message-ID: No worries. We have two more speakers, so you can just tell us on the day. Take care of yourself! Hugs, J On Wed, Nov 27, 2013 at 4:07 PM, Bianca Gibson wrote: > Hi, > I may need to cancel, I've done something to my foot. > I'll let you know if I can't make it. > > Cheers, > Bianca > > _______________________________________________ > melbourne-pug mailing list > melbourne-pug at python.org > https://mail.python.org/mailman/listinfo/melbourne-pug > From matt at plot.ly Tue Nov 26 10:39:47 2013 From: matt at plot.ly (Matt Sundquist) Date: Tue, 26 Nov 2013 01:39:47 -0800 Subject: [melbourne-pug] Plotly: Python graphs and online command line Message-ID: Hi Python friends in Melbourne, Happy Tuesday! My name is Matt, and I'm a co-founder at Plot.ly, a graphing and analytics project. We just shipped a beta, and thought folks on this list might be interested. Plotly has a Python Shell (NumPy supported) that you can run in the browser, and a Python graphing library available here that lets you make interactive, browser-based graphs. You can also make interactive graphs with IPython with Plotly, like this (general documentation) and this (for multiple axes and subplots). All your graphs are stored with their data, and you can share them with a URL, stream data to them, download them, and collaboratively edit them online. Our gallery is below, and the code to make the graphs is available on our site. We're a brand new project, so any and all help, feedback, and support is massively appreciated. Thanks so much for any and all thoughts. Thanks so much, Matt [image: Inline image 1] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Gallery.png Type: image/png Size: 286629 bytes Desc: not available URL: From bianca.rachel.gibson at gmail.com Thu Nov 28 10:10:04 2013 From: bianca.rachel.gibson at gmail.com (Bianca Gibson) Date: Thu, 28 Nov 2013 20:10:04 +1100 Subject: [melbourne-pug] Fwd: [Chat] LCA2014 Official Announcement In-Reply-To: References: Message-ID: Lots of linux.conf.au 2014 details below. I'm really looking forward to it, I hope to see lots of you there. ---------- Forwarded message ---------- From: Paul Del Fante LCA2014 Official Announcement Announcing linux.conf.au 2014 Overview In January 2014, Perth, Western Australia will be playing host to the continents premier open source conference ? linux.conf.au. linux.conf.au draws attendees and speakers worldwide from all spheres of open source, geeks, hackers, professionals, hobbyists and business partners. Conference Format The linux.conf.au conference runs over 5 days from Monday 6th January to Friday 10th January. Conference tickets are available at Professional, Hobbyist and Student levels with associated pricing and perks. Over the first two days we host a series of miniconfs, small single day conferences covering particular topics. Intended as an incubator for related technologies and interests that could eventually hold their own conferences. The miniconfs are managed and run as independent conferences with their own call for papers and management committees. The remaining three days feature a packed conference schedule, in 2014 we will be running presentations over 5 simultaneous tracks. Each track will feature four 45 minute presentations per day. The full programme is available on our website at http://linux.conf.au/programme/schedule. On most days before the main sessions, the conference will open with a keynote presentation. The keynotes tend to follow a less technical approach and provide commentary and insight on broad global trends and open source challenges. Birds of a Feather Sessions Run outside the scope of the formal presentation agenda, linux.conf.au encourages delegates to arrange ?Birds of a Feather? - BOF - sessions. These draw together people with a specific area of interest and are run after hours. Typical BOF sessions range from Astronomy, Video capture, Running, Cycling, Hacking and many others. The BOF sessions are a great opportunity to meet and socialise with like minded people during the conference. Video and Streaming All presentations made at linux.conf.au will be recorded and the completed videos made available for download and online viewing. This year we will be attempting to provide a live stream of the proceedings too. Social Programme The linux.conf.au tries to be as accessible to participants as possible, we generally run a partners programme to provide some interest and entertainment to partners of attendees who are not interested in participating. We also arrange a creche facility to care for children while delegates are attending presentations. We also run several semi-formal networking sessions including a speakers dinner on Tuesday, a professional delegates networking function on the Thursday and a penguin dinner open to all delegates on the Wednesday. In addition to the above, various organisations will be hosting social gatherings during the conference and on free evenings. Keynotes At present, we've announced two keynote presenters, with a third still being determined. The confirmed keynote presenters so far are: Jonathon Oxer Jon will be presenting an overview of the latest developments to the ArduSat project as part of his keynote presentation during linux.conf.au 2014 in Perth, Western Australia. Jon has been hacking on both hardware and software since he was a little tacker. Most recently he's been focusing more on the Open Hardware side, co-founding Freetronics as a direct result of organising the first Arduino Miniconf at LCA2010. His books include "Ubuntu Hacks" and "Practical Arduino". Jon has been variously referred to as Australia's geekiest man and as a cyborg-in-progress. As part of his ?SuperHouse? home automation series, Jon has taken keyless entry to an all new level by embedding an RFID tag into his arm using a vet's chipping tool. Recently he has been working on ArduSat, a satellite that aims to give hobbyists, students and space enthusiasts an opportunity to design and run their own experiments in space. By choosing a standardised platform based on the hugely popular Arduino hardware design, ArduSat allows anyone to develop and prototype experiments at home using readily accessible parts and all based on a simple open source software environment. Kate Chapman Kate Chapman is the Executive Director of the Humanitarian OpenStreetMap Team. The Humanitarian OpenStreetMap Team (HOT) is an initiative of OpenStreetMap to apply the principles of open source and open data sharing towards humanitarian response and economic development. The Humanitarian OpenStreetMap Team believes that freely available, up-to-date geographic data can be vital in responding to a disaster. They help communities and responders to utilize the OpenStreetMap data, and encourage them to contribute and update the data. Over the past two years Kate has been primarily based in Indonesia where she has lead the AusAID funded HOT program. The program Scenario Development for Contingency Planning (SD4CP) is a joint initiative of the Indonesian and Australian governments through BNPB (Indonesian Disaster Management Agency) and AusAID that aims to teach disaster mangers how to develop realistic impact scenario for contingency planning. HOT teaches disaster mangers to spatially analysis the combination of scientific hazard model outputs with community knowledge through the open-source software InaSAFE (Indonesia Scenario Assessment For Emergencies). OpenStreetMap is one of the keys elements of this procedure, as it allows the disaster mangers to draw on community knowledge to identify important buildings such as schools and hospitals that potentially be impacted during a disaster. We're really pleased to have Kate coming to talk with us as her personal efforts and her teams work represents an organization that we can really hold up as a shining example of how the open source model truly does work. Miniconfs Each year we try to select a range of topics for our focused mini conferences. This year's miniconf sessions are listed below. Monday: Systems Administration Developer, Testing, Release and Continuous integration Automation Open Government Open Radio Tuesday: OpenStack Linux Kernel Haecksen Astronomy Multimedia and Music Arduino Conference The main conference runs over the remaining three days, and is a full programme of five streams of deep technical presentations and tutorials covering a wide range of topics. To view the full schedule please visit our website at http://linux.conf.au/programme/schedule. Conference Location linux.conf.au will be held in Perth, Western Australia this year. Perth is Australia's most remote capital city, but is also where most of the action is happening. Perth is experiencing an huge amount of growth, driven by the mining boom, but encompassing all aspects of city life. Perth now has a vibrant cultural scene, and our outdoor lifestyle and coastal location is the envy of the world. The last linux.conf.au held in Perth was ten years ago in 2003, so we are really looking forward to hosting the conference again. The linux.conf.au conference is traditionally held on a university campus, utilising the lecture rooms and accommodation facilities they have to offer ? 2014 is no exception, and we will be hosting the conference at the University of Western Australia (UWA). The UWA Crawley Campus is situated a short distance from the Perth CBD and adjacent to the magnificent Swan river and Kings Park public space. Accomodation Accommodation will be at the university colleges of St George's http://www.stgeorgescollege.com.au/ and Trinity http://www.trinity.uwa.edu.au/ for full details of the accomodation please refer to our wiki article https://lca2014.linux.org.au/wiki/Campus_accommodation. The accomodation is ideally situated just across the road from the conference and registration areas. Volunteering The conference is run as a voluntary effort. We rely on people to volunteer their time to organise and run the conference. During the conference we are always keen to welcome volunteers to come on board and help with the running of the conference. Interested people can sign up to volunteer at https://lca2014.linux.org.au/volunteer/new. We offer free registration for people who are willing to volunteer a significant portion of their time to the conference. Proposed Focus To provide a degree of consistency linux.conf.au normally selects a general focus to help guide the presenters in topic selection. For 2014 our focus is on ?linux on the frontier and deep technical content?. With the scope of Linux broadening at such a rapid pace this provides a broad scope for speakers topics and allows topics to be drawn from just about any sphere of computing, be it cybernetics, mobile operating systems, astronomy, big data, gaming etc. Contacts Please send all correspondence to the LCA2014 Organising Team team at lists.lca2014.linux.org.au Our website is accessible from lca2014.linux.org.au #lca2014 _______________________________________________ Chat mailing list Chat at lists.lca2014.linux.org.au http://lists.lca2014.linux.org.au/mailman/listinfo/chat_lists.lca2014.linux.org.au -------------- next part -------------- An HTML attachment was scrubbed... URL: From sam at nipl.net Fri Nov 29 00:09:02 2013 From: sam at nipl.net (Sam Watkins) Date: Fri, 29 Nov 2013 10:09:02 +1100 Subject: [melbourne-pug] Plotly: Python graphs and online command line In-Reply-To: References: Message-ID: <20131128230902.GQ19318@opal.nipl.net> hi Matt, plot.ly looks to be a valuable tool, thanks for sharing > Plotly has a Python Shell (NumPy supported) Wow. Did you guys code this up from scratch, or is it based on other work? How is it done? I guess you wrote a python interpreter or translator in javascript? Is your numpy performant? I'm thinking you'd have to use WebGL for that! Sam From william.leslie.ttg at gmail.com Fri Nov 29 00:30:46 2013 From: william.leslie.ttg at gmail.com (William ML Leslie) Date: Fri, 29 Nov 2013 10:30:46 +1100 Subject: [melbourne-pug] Plotly: Python graphs and online command line In-Reply-To: <20131128230902.GQ19318@opal.nipl.net> References: <20131128230902.GQ19318@opal.nipl.net> Message-ID: On 29 November 2013 10:09, Sam Watkins wrote: >> Plotly has a Python Shell (NumPy supported) > > Wow. Did you guys code this up from scratch, or is it based on other > work? How is it done? I guess you wrote a python interpreter or > translator in javascript? Is your numpy performant? I'm thinking you'd > have to use WebGL for that! Probably the same way geordi or http://codepad.org/about is done (no spoilers). -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely may reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to deny you those rights would be illegal without prior contractual agreement.