From r1chardj0n3s at gmail.com  Tue Feb  1 04:53:10 2011
From: r1chardj0n3s at gmail.com (Richard Jones)
Date: Tue, 1 Feb 2011 14:53:10 +1100
Subject: [melbourne-pug] Next meeting: Monday 7th March @ RMIT
Message-ID: <AANLkTi=-D-6StMJDOBNeu+kD_SP_pgDwpAGsA+XXVM=d@mail.gmail.com>

Hi all,

Thanks for the great meeting last night. It's always nice to see a
bunch of new faces!

I've confirmed the RMIT lecture room booking for the rest of the year
(well until November since I can't book December until September ...
er :-) The calendar has been updated with these meeting dates (get it
via http://bit.ly/mpug) Please start thinking about things you might
tell us all about. Stuff you discover, create or use over the next
month. Also there's several of us that'll take requests (I've
previously done some quick talks on various bits of Python
syntax/semantics and am happy to do so again).


     Richard

From gcross at fastmail.fm  Tue Feb  1 05:28:44 2011
From: gcross at fastmail.fm (Graeme Cross)
Date: Tue, 01 Feb 2011 15:28:44 +1100
Subject: [melbourne-pug] Next meeting: Monday 7th March @ RMIT
In-Reply-To: <AANLkTi=-D-6StMJDOBNeu+kD_SP_pgDwpAGsA+XXVM=d@mail.gmail.com>
References: <AANLkTi=-D-6StMJDOBNeu+kD_SP_pgDwpAGsA+XXVM=d@mail.gmail.com>
Message-ID: <1296534524.3169.1418286887@webmail.messagingengine.com>

On Tue, 01 Feb 2011 14:53 +1100, "Richard Jones"
<r1chardj0n3s at gmail.com> wrote:
> Hi all,
> 
> Thanks for the great meeting last night. It's always nice to see a
> bunch of new faces!
> 
> I've confirmed the RMIT lecture room booking for the rest of the year
> (well until November since I can't book December until September ...
> er :-) The calendar has been updated with these meeting dates (get it
> via http://bit.ly/mpug) Please start thinking about things you might
> tell us all about. Stuff you discover, create or use over the next
> month. Also there's several of us that'll take requests (I've
> previously done some quick talks on various bits of Python
> syntax/semantics and am happy to do so again).
> 
> 
>      Richard

Hi Richard.

It was a great meeting (and sorry for bailing before dinner, but I had
another commitment to get to).

Given the number of people who were (relatively or completely) new to
Python who came along, I think it would be a good idea to more
experienced people volunteer to give talks about parts of Python that
are (a) useful and (b) probably unfamiliar to programmers coming from
other languages.

Lots of ideas come to mind:

a) what's the difference between tuples, lists and dictionaries? when
should I use one or the other?
b) using decorators
c) generators, list comprehensions etc
d) what's the difference between Python 2.x and Python 3?
e) people's favourite modules that aren't in the standard library (eg. I
didn't know about the "e" module that you spoke about last night)

I'll put my money where my mouth is and do an overview of decorators at
next month's meeting.

Anyone else want to volunteer a talk to help Python newcomers?

Cheers
Graeme



From tleeuwenburg at gmail.com  Tue Feb  1 06:46:33 2011
From: tleeuwenburg at gmail.com (Tennessee Leeuwenburg)
Date: Tue, 1 Feb 2011 16:46:33 +1100
Subject: [melbourne-pug] Next meeting: Monday 7th March @ RMIT
In-Reply-To: <AANLkTi=-D-6StMJDOBNeu+kD_SP_pgDwpAGsA+XXVM=d@mail.gmail.com>
References: <AANLkTi=-D-6StMJDOBNeu+kD_SP_pgDwpAGsA+XXVM=d@mail.gmail.com>
Message-ID: <AANLkTimk76fr-deO7dVC52F-Py8sEtNcCxuAG+4McE1X@mail.gmail.com>

I'll volunteer a talk on "Pointless but fun 3D stuff with Python"

On Tue, Feb 1, 2011 at 2:53 PM, Richard Jones <r1chardj0n3s at gmail.com>wrote:

> Hi all,
>
> Thanks for the great meeting last night. It's always nice to see a
> bunch of new faces!
>
> I've confirmed the RMIT lecture room booking for the rest of the year
> (well until November since I can't book December until September ...
> er :-) The calendar has been updated with these meeting dates (get it
> via http://bit.ly/mpug) Please start thinking about things you might
> tell us all about. Stuff you discover, create or use over the next
> month. Also there's several of us that'll take requests (I've
> previously done some quick talks on various bits of Python
> syntax/semantics and am happy to do so again).
>
>
>     Richard
> _______________________________________________
> melbourne-pug mailing list
> melbourne-pug at python.org
> http://mail.python.org/mailman/listinfo/melbourne-pug
>



-- 
--------------------------------------------------
Tennessee Leeuwenburg
http://myownhat.blogspot.com/
"Don't believe everything you think"
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/melbourne-pug/attachments/20110201/56fa8911/attachment.html>

From jervisau at gmail.com  Tue Feb  1 06:54:23 2011
From: jervisau at gmail.com (Jervis Whitley)
Date: Tue, 1 Feb 2011 16:54:23 +1100
Subject: [melbourne-pug] Next meeting: Monday 7th March @ RMIT
In-Reply-To: <AANLkTimk76fr-deO7dVC52F-Py8sEtNcCxuAG+4McE1X@mail.gmail.com>
References: <AANLkTi=-D-6StMJDOBNeu+kD_SP_pgDwpAGsA+XXVM=d@mail.gmail.com>
	<AANLkTimk76fr-deO7dVC52F-Py8sEtNcCxuAG+4McE1X@mail.gmail.com>
Message-ID: <AANLkTin7qAL2T1uR=FMex2t1BJ6Or0dxnXoqbzRsw2K-@mail.gmail.com>

I'd like to hear a talk about pyweek!

On Tue, Feb 1, 2011 at 4:46 PM, Tennessee Leeuwenburg <
tleeuwenburg at gmail.com> wrote:

> I'll volunteer a talk on "Pointless but fun 3D stuff with Python"
>
>
> On Tue, Feb 1, 2011 at 2:53 PM, Richard Jones <r1chardj0n3s at gmail.com>wrote:
>
>> Hi all,
>>
>> Thanks for the great meeting last night. It's always nice to see a
>> bunch of new faces!
>>
>> I've confirmed the RMIT lecture room booking for the rest of the year
>> (well until November since I can't book December until September ...
>> er :-) The calendar has been updated with these meeting dates (get it
>> via http://bit.ly/mpug) Please start thinking about things you might
>> tell us all about. Stuff you discover, create or use over the next
>> month. Also there's several of us that'll take requests (I've
>> previously done some quick talks on various bits of Python
>> syntax/semantics and am happy to do so again).
>>
>>
>>     Richard
>> _______________________________________________
>> melbourne-pug mailing list
>> melbourne-pug at python.org
>> http://mail.python.org/mailman/listinfo/melbourne-pug
>>
>
>
>
> --
> --------------------------------------------------
> Tennessee Leeuwenburg
> http://myownhat.blogspot.com/
> "Don't believe everything you think"
>
> _______________________________________________
> melbourne-pug mailing list
> melbourne-pug at python.org
> http://mail.python.org/mailman/listinfo/melbourne-pug
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/melbourne-pug/attachments/20110201/d4331b27/attachment.html>

From dcrisp at netspace.net.au  Tue Feb  1 06:49:29 2011
From: dcrisp at netspace.net.au (David Crisp)
Date: Tue, 1 Feb 2011 16:49:29 +1100 (EST)
Subject: [melbourne-pug] Next meeting: Monday 7th March @ RMIT
In-Reply-To: <AANLkTimk76fr-deO7dVC52F-Py8sEtNcCxuAG+4McE1X@mail.gmail.com>
References: <AANLkTi=-D-6StMJDOBNeu+kD_SP_pgDwpAGsA+XXVM=d@mail.gmail.com>
	<AANLkTimk76fr-deO7dVC52F-Py8sEtNcCxuAG+4McE1X@mail.gmail.com>
Message-ID: <alpine.DEB.2.00.1102011648160.3480@thor.pearwood.mine.nu>

.... What sort of Pointless 3D stuff are you talking about?

It might not be so pointless!

Two things I wouldnt mind to hear are the different times and places for 
using tuples and directoies etc as previously mentioned,  And Pointless 3D 
stuff :)



On Tue, 1 Feb 2011, Tennessee Leeuwenburg wrote:

> I'll volunteer a talk on "Pointless but fun 3D stuff with Python"
>
> On Tue, Feb 1, 2011 at 2:53 PM, Richard Jones <r1chardj0n3s at gmail.com>wrote:
>
>> Hi all,
>>
>> Thanks for the great meeting last night. It's always nice to see a
>> bunch of new faces!
>>
>> I've confirmed the RMIT lecture room booking for the rest of the year
>> (well until November since I can't book December until September ...
>> er :-) The calendar has been updated with these meeting dates (get it
>> via http://bit.ly/mpug) Please start thinking about things you might
>> tell us all about. Stuff you discover, create or use over the next
>> month. Also there's several of us that'll take requests (I've
>> previously done some quick talks on various bits of Python
>> syntax/semantics and am happy to do so again).
>>
>>
>>     Richard
>> _______________________________________________
>> melbourne-pug mailing list
>> melbourne-pug at python.org
>> http://mail.python.org/mailman/listinfo/melbourne-pug
>>
>
>
>
>

From tleeuwenburg at gmail.com  Wed Feb  2 03:35:18 2011
From: tleeuwenburg at gmail.com (Tennessee Leeuwenburg)
Date: Wed, 2 Feb 2011 13:35:18 +1100
Subject: [melbourne-pug] Next meeting: Monday 7th March @ RMIT
In-Reply-To: <alpine.DEB.2.00.1102011648160.3480@thor.pearwood.mine.nu>
References: <AANLkTi=-D-6StMJDOBNeu+kD_SP_pgDwpAGsA+XXVM=d@mail.gmail.com>
	<AANLkTimk76fr-deO7dVC52F-Py8sEtNcCxuAG+4McE1X@mail.gmail.com>
	<alpine.DEB.2.00.1102011648160.3480@thor.pearwood.mine.nu>
Message-ID: <AANLkTinUNPGaAxb7+nbSimsG_MS9D2UV-CA0bXRHQXBo@mail.gmail.com>

Well, I grabbed the Enthought Python distribution (laziness, will re-do with
free stuff later) and used Mayavi to render a random graph plotted on a
sphere, then made a movie of it rotating. It was not that hard (trig aside)
and visually rewarding. It's cool to know that stuff is actually quite
accessible to anyone.

On Tue, Feb 1, 2011 at 4:49 PM, David Crisp <dcrisp at netspace.net.au> wrote:

> .... What sort of Pointless 3D stuff are you talking about?
>
> It might not be so pointless!
>
> Two things I wouldnt mind to hear are the different times and places for
> using tuples and directoies etc as previously mentioned,  And Pointless 3D
> stuff :)
>
>
>
>
> On Tue, 1 Feb 2011, Tennessee Leeuwenburg wrote:
>
>  I'll volunteer a talk on "Pointless but fun 3D stuff with Python"
>>
>> On Tue, Feb 1, 2011 at 2:53 PM, Richard Jones <r1chardj0n3s at gmail.com
>> >wrote:
>>
>>  Hi all,
>>>
>>> Thanks for the great meeting last night. It's always nice to see a
>>> bunch of new faces!
>>>
>>> I've confirmed the RMIT lecture room booking for the rest of the year
>>> (well until November since I can't book December until September ...
>>> er :-) The calendar has been updated with these meeting dates (get it
>>> via http://bit.ly/mpug) Please start thinking about things you might
>>> tell us all about. Stuff you discover, create or use over the next
>>> month. Also there's several of us that'll take requests (I've
>>> previously done some quick talks on various bits of Python
>>> syntax/semantics and am happy to do so again).
>>>
>>>
>>>    Richard
>>> _______________________________________________
>>> melbourne-pug mailing list
>>> melbourne-pug at python.org
>>> http://mail.python.org/mailman/listinfo/melbourne-pug
>>>
>>>
>>
>>
>>
>>  _______________________________________________
> melbourne-pug mailing list
> melbourne-pug at python.org
> http://mail.python.org/mailman/listinfo/melbourne-pug
>



-- 
--------------------------------------------------
Tennessee Leeuwenburg
http://myownhat.blogspot.com/
"Don't believe everything you think"
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/melbourne-pug/attachments/20110202/1c44c576/attachment.html>

From pwilliams at nswrdn.com.au  Thu Feb  3 03:24:53 2011
From: pwilliams at nswrdn.com.au (Peter Williams)
Date: Thu, 03 Feb 2011 13:24:53 +1100
Subject: [melbourne-pug] Job in Newcastle NSW
Message-ID: <4D4A11F5.6030003@nswrdn.com.au>

Hi all

The RDN seeks a PROGRAMMER/SYSTEMS ENGINEER for 12 months to be based in 
our Head Office in Newcastle?s CBD. This position will provide essential 
programming services to the Information Management team utilising 
high-level development languages and libraries such as SQL, HTML, 
Django, Python JavaScript, Linux, Ajax and CSS. RECENT GRADUATES are 
encouraged to apply. Employment is open to Australian citizens, 
permanent residents and other applicants with the appropriate visa 
authorisation to work in Australia. The person will be required to work 
at the RDN office.

http://www.nswrdn.com.au/site/index.cfm?display=57772

If you know anyone who might be interested, please direct them to the 
URL above. Applications close 17 Feb 2011.

Cheers

Peter

-- 
Peter J Williams
Information Manager
NSW Rural Doctors Network
Head Office
Suite 19, Level 3
133 King Street
Newcastle NSW 2300
Telephone: (02) 4924-8000
Facsimile: (02) 4924-8010
Mailto:pwilliams at nswrdn.com.au
Web: http://www.nswrdn.com.au




-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: disclaimer.txt
URL: <http://mail.python.org/pipermail/melbourne-pug/attachments/20110203/8bf2e0fe/attachment.txt>

From swifteye2 at msn.com  Thu Feb  3 09:51:17 2011
From: swifteye2 at msn.com (John Thornton)
Date: Thu, 3 Feb 2011 18:51:17 +1000
Subject: [melbourne-pug] python and operations research
Message-ID: <SNT125-W359DB2A8A06C98C176161CF1E70@phx.gbl>



                 I am new to python. I am studying some computer programming certificates including some python. I haven't got far into the python yet though. My broad intent is to use python for operations reserach problems. I have a particular interest in inventory models and the EOQ [Economic Order Quantity] formulae. When I did OR at university [mid 1990's] I had one software package in DOS: Management Scientist which came with my Anderson/Sweeney/Williams "Intro to Management Science" textbook. 

                 So, today I set up a Vmware Ubuntu 10.10 box and installed PuLP and the glpk solver. It has all passed the tests. [excluding of course gurobi tests etc because those solvers aren't there.] 

                  I am very reluctant to use any closed-source software for OR; it is a discipline that demands too much flexibility for commercial options. 

                  FYI I left school in the early 1990's with absolute computer illiteracy. I could not put a disk in the disk drive. I was in a nasty, nasty era where you never knew where you stood with computers. In 1980 things were clearcut. There was no home PC market in 1980 so you could study computers without needing one at home. But in the early 1990's things got nasty. You were expected to know how to use computers to pass VCE subjects. Yet nobody taught you a thing about computers. I had to drop a subject that I was getting straight A's for [geography] because year 12 would have had me creating heaps of pie graphs with a computer. Yet who on earth was going to teach me how to do those? In short people like myself were "technologised" out of subjects. I never really got a handle on computers until 2007[!]. It is a long way back from leaving school computer illiterate; there was the vicious circle of "I need to be more confident with computers to learn about them but I need to learn them to get more confident". 

                 Anyone else on this list into Python OR? Is PuLP with glpk a good way to go? What libraries does python have in OR? 

                Regards
               Andrew Thornton 

                  

                

                  

                   

                    
 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/melbourne-pug/attachments/20110203/b6aa71a6/attachment.html>

From r1chardj0n3s at gmail.com  Fri Feb  4 01:14:34 2011
From: r1chardj0n3s at gmail.com (Richard Jones)
Date: Fri, 4 Feb 2011 11:14:34 +1100
Subject: [melbourne-pug] python and operations research
In-Reply-To: <SNT125-W359DB2A8A06C98C176161CF1E70@phx.gbl>
References: <SNT125-W359DB2A8A06C98C176161CF1E70@phx.gbl>
Message-ID: <AANLkTi=aEbs=VoSOQKVB=O1pezd2FXo5=TFpjXAt3zTn@mail.gmail.com>

On Thu, Feb 3, 2011 at 7:51 PM, John Thornton <swifteye2 at msn.com> wrote:
> ???????????????? Anyone else on this list into Python OR? Is PuLP with glpk
> a good way to go? What libraries does python have in OR?

PuLP seems to be popular and the project is definitely active. You
might try asking this question in a larger forum - the main python
mailing list which you can get to through various means listed at
http://python.org/community/lists/


     Richard

From acalcium at yahoo.com.au  Fri Feb  4 22:49:20 2011
From: acalcium at yahoo.com.au (Chai Ang)
Date: Fri, 4 Feb 2011 13:49:20 -0800 (PST)
Subject: [melbourne-pug] Does Anyone In Australia Want a Free Python3 PyCon
	Tutorial?
In-Reply-To: <mailman.45.1296644403.12831.melbourne-pug@python.org>
Message-ID: <224640.74099.qm@web110106.mail.gq1.yahoo.com>

Pretty short notice though.

http://dabeaz.blogspot.com/2011/02/does-anyone-in-australia-want-free.html




      

From miked at dewhirst.com.au  Sat Feb  5 06:21:59 2011
From: miked at dewhirst.com.au (Mike Dewhirst)
Date: Sat, 05 Feb 2011 16:21:59 +1100
Subject: [melbourne-pug] Does Anyone In Australia Want a Free Python3
 PyCon	Tutorial?
In-Reply-To: <224640.74099.qm@web110106.mail.gq1.yahoo.com>
References: <224640.74099.qm@web110106.mail.gq1.yahoo.com>
Message-ID: <4D4CDE77.1000405@dewhirst.com.au>

On 5/02/2011 8:49am, Chai Ang wrote:
> Pretty short notice though.
>
> http://dabeaz.blogspot.com/2011/02/does-anyone-in-australia-want-free.html

If anyone from Melbourne is setting this up I'd like to see it but can 
only attend if it is Sunday 13 Feb.

Mike


>
>
>
>
>
> _______________________________________________
> melbourne-pug mailing list
> melbourne-pug at python.org
> http://mail.python.org/mailman/listinfo/melbourne-pug
>
>


From dcrisp at netspace.net.au  Mon Feb  7 03:15:42 2011
From: dcrisp at netspace.net.au (David Crisp)
Date: Mon, 7 Feb 2011 13:15:42 +1100
Subject: [melbourne-pug] Lists or Arrays - When do I use what and why - Or :
	Confused much? I am, Now
Message-ID: <000d01cbc66c$ec2115f0$c46341d0$@netspace.net.au>

I need a little bit of assistance understanding when to use python arrays
from scipy and when to use the lists.   In part,  my confusion comes about
from my use of arrays in C, Visual Basic, and perl. 

First let me explain what I am attempting to do.  I will use the language I
understand which may very well be incorrect python wording.

I have a large file of data which consists of  the following:  X,Y,Z,I,R,G,B
As sampled below.   There are a large number of files with variable lengths.
A few contain 100 million data points but mostly there are between 3 million
and 10 million points. 

(X,Y and Z are coordinates and can be positive or negative numbers
I which is intensity is also either positive or negative
R,G,B are Red, Green and Blue values between 0 and 255)

-54.677597 	1.153580 	8.334061 	-815 	255 253 255
-54.632614 	1.131516 	8.359695 	-1220 	255 255 255
-54.749954 	1.199265 	8.365860 	-357 	238 237 225
-54.509628 	1.210983 	8.305679 	-1306 	255 255 255
-54.558060 	1.146866 	8.275925 	-1006 	255 255 255
-54.512802 	1.125717 	8.295731 	-1176 	255 254 254
-54.524399 	1.146805 	8.338608 	-1054 	255 255 255
-54.528763 	1.154678 	8.362442 	-919 	255 255 255
-54.637314 	1.076492 	8.433945 	-910 	255 255 254

I wish to read this file in to memory and assign it into an array.   

There is a catch :  At variable locations through the file there appears
"point count entries"   lines which contain simply 342345d  which says that
there are that many points in the following section.   These can be ignored
and skipped BUT it does mean I cant simply say "file
---(go_and_populate)----> array"

I then wish to iterate through the array and 'bin'  the X values into a
range.  For instance every X value between -54 and -53 goes into bin 1,  -53
to -52 goes into bin 2

Then I wish to iterate through this bin array and sort the Y values into the
same sort of ranges. 

Then I wish to Sort the Z values into the same sort of ranges.   This should
give me an array of Array[X][Y][Z] Where Array[0][0][0] should give me the
very first cell  and I can reference any of the values simply by :

>>>print Array[100][100][10]  

and get the resulting Z values for that location. 

Now, I suspect that once I know how to do that or have had my "C concepts,
Visual Basic Concepts" of Arrays converted to Python views of arrays and
lists I should be able to cope with this. 

A couple of things ?I have already worked on: 

I already have a piece of code which reads the file into python lists.  It
runs out of memory quite badly if I try importing too many data points (more
than 20 million , or something... irrelevant.. it runs out of memory! )

The (cludgy) way I solved  ONE of my problems was to work out how many lines
the file was and then populate a list with < point_data=[None]*file_length
>    This was okay until I started filtering out the point count lines.
Then the array had more entries than actual data points and so when I did
something like

For line in array :
	Dosomething

I would run over the end of my data and get errors with the NONE entries.  

Now, I could probably write some checks in there to make sure I handle the
NONEs properly.... But that?s not the right way of doing this. 

I figured if  somebody could explain it clearly right at the start I would
remember how these things work for ever after. 

As I mentioned earlier, its actually quite amazing that for the last 25
years arrays have worked the same way in all the programing languages  I
have used and so the terminology and written form they take has always been
the same (almost similar)   and because python is slightly different,  my
brain is having great trouble interpreting what I should be doing! :)
Brain break! 

Hopefully somebody can explain it all clearly.  I'm sure there are other
people out there who need help as well! 

Regards, 
David  Crisp





From r1chardj0n3s at gmail.com  Mon Feb  7 03:44:51 2011
From: r1chardj0n3s at gmail.com (Richard Jones)
Date: Mon, 7 Feb 2011 13:44:51 +1100
Subject: [melbourne-pug] Lists or Arrays - When do I use what and why -
 Or : Confused much? I am, Now
In-Reply-To: <000d01cbc66c$ec2115f0$c46341d0$@netspace.net.au>
References: <000d01cbc66c$ec2115f0$c46341d0$@netspace.net.au>
Message-ID: <AANLkTimRXMdxo69DUsj4PR4bOomm7XS7XQh-aZP6stw3@mail.gmail.com>

On Mon, Feb 7, 2011 at 1:15 PM, David Crisp <dcrisp at netspace.net.au> wrote:
> I need a little bit of assistance understanding when to use python arrays
> from scipy and when to use the lists. ? In part, ?my confusion comes about
> from my use of arrays in C, Visual Basic, and perl.

Python arrays (http://docs.python.org/library/array.html and the basic
ones found in numpy) are almost identical to C arrays - you declare
them to contain a fixed amount of data of a certain type. Very
memory-efficient.

Python lists are resizable arrays that contain data of any type (it's
an array of pointers to PyObject). IIRC they are allocated using a 2x
expansion algorithm and really are not memory-efficient if you just
want to store a bunch of floats or ints (or ubytes!)

You could probably do something with arrays for the i, r, g and b
values in a nested list of lists. And sort into your quantised bins
when you read the data in, rather than quantise between in-memory
structures.

The scipy/numpy community have been dealing with these problems for a
while (using sparse arrays, submatrices, ...) and your best bet is to
ask in that community for further help
(http://www.scipy.org/Mailing_Lists).


     Richard

From william.leslie.ttg at gmail.com  Mon Feb  7 04:28:55 2011
From: william.leslie.ttg at gmail.com (William ML Leslie)
Date: Mon, 7 Feb 2011 14:28:55 +1100
Subject: [melbourne-pug] Lists or Arrays - When do I use what and why -
 Or : Confused much? I am, Now
In-Reply-To: <000d01cbc66c$ec2115f0$c46341d0$@netspace.net.au>
References: <000d01cbc66c$ec2115f0$c46341d0$@netspace.net.au>
Message-ID: <AANLkTikXtVhMbo=bvVQWWWNtTc1_=D-St2bB_bmV2J05@mail.gmail.com>

On 7 February 2011 13:15, David Crisp <dcrisp at netspace.net.au> wrote:
> I then wish to iterate through the array and 'bin' ?the X values into a
> range. ?For instance every X value between -54 and -53 goes into bin 1, ?-53
> to -52 goes into bin 2
>
> Then I wish to iterate through this bin array and sort the Y values into the
> same sort of ranges.
>
> Then I wish to Sort the Z values into the same sort of ranges. ? This should
> give me an array of Array[X][Y][Z] Where Array[0][0][0] should give me the
> very first cell ?and I can reference any of the values simply by :
>
>>>>print Array[100][100][10]
>
> and get the resulting Z values for that location.

A better data structure for an index like this might actually be a
dictionary. A dictionary can be keyed on (X, Y, Z) tuples.

You could do something just a little neater than than:

points = {}
for line in file:
    points = line.split(None, 6)
    if not points[2:]:
        # specifically handle blank lines and count lines
        continue
    x, y, z, i, r, g, b = map(float, points)
    region = points.setdefault((int(x), int(y), int(z)), [])
    region.append((x, y, z, i, r, g, b))

On tuples:

Tuples are (generally) for representing heterogeneous data; much like
structs or classes in other languages.  Each "column" of your table
represents a different concept; whether an intensity, a location, or
whatever, and it looks like they are even supposed to be different
types.  Whenever each column has a distinct meaning/type, a
heterogeneous collection type like a tuple, namedtuple or class is
probably the way to go.

On the other hand, you probably want to talk about "the list of
points"; dealing with them as a heterogeneous collection.  Each point
in the list, or row of your table, represents the same type of data.

Someone had suggested doing a talk next month on appropriate use of
data structures, so it is probably a good idea to get some questions
going on in here.  Using numpy for the detail rows is reasonable too,
but that means you either give up unboxing the floats (which unboxing
will halve the memory usage of this table on cpython) or give up
having different types for different columns; so it might not be
worthwhile.

-- 
William Leslie

From dcrisp at netspace.net.au  Mon Feb  7 08:08:55 2011
From: dcrisp at netspace.net.au (David Crisp)
Date: Mon, 7 Feb 2011 18:08:55 +1100 (EST)
Subject: [melbourne-pug] Lists or Arrays - When do I use what and why -
 Or : Confused much? I am, Now
In-Reply-To: <AANLkTikXtVhMbo=bvVQWWWNtTc1_=D-St2bB_bmV2J05@mail.gmail.com>
References: <000d01cbc66c$ec2115f0$c46341d0$@netspace.net.au>
	<AANLkTikXtVhMbo=bvVQWWWNtTc1_=D-St2bB_bmV2J05@mail.gmail.com>
Message-ID: <alpine.DEB.2.00.1102071805240.9770@thor.pearwood.mine.nu>

Thanks William,

I read up on the Dictionaries at one of the main points I spotted was a 
reference on one of the sites :

(a) More than one entry per key not allowed. Which means no duplicate key 
is allowed. When duplicate keys encountered during assignment, the last 
assignment wins.

By my reading that is saying that there can only be one X value of 10  if 
a new X value of ten comes along then it will supercede the previous.

Because this is 3D space these points represent,  there can actually be 
more than 1 X of Value 10 and indeed there can be more than 1 Y value of 
10 as well.

Or have i misunderstood the restrictions on Dictionaries.

Regards,
David



  On Mon, 7 Feb 2011, William ML Leslie wrote:

> On 7 February 2011 13:15, David Crisp <dcrisp at netspace.net.au> wrote:
>> I then wish to iterate through the array and 'bin' ?the X values into a
>> range. ?For instance every X value between -54 and -53 goes into bin 1, ?-53
>> to -52 goes into bin 2
>>
>> Then I wish to iterate through this bin array and sort the Y values into the
>> same sort of ranges.
>>
>> Then I wish to Sort the Z values into the same sort of ranges. ? This should
>> give me an array of Array[X][Y][Z] Where Array[0][0][0] should give me the
>> very first cell ?and I can reference any of the values simply by :
>>
>>>>> print Array[100][100][10]
>>
>> and get the resulting Z values for that location.
>
> A better data structure for an index like this might actually be a
> dictionary. A dictionary can be keyed on (X, Y, Z) tuples.
>
> You could do something just a little neater than than:
>
> points = {}
> for line in file:
>    points = line.split(None, 6)
>    if not points[2:]:
>        # specifically handle blank lines and count lines
>        continue
>    x, y, z, i, r, g, b = map(float, points)
>    region = points.setdefault((int(x), int(y), int(z)), [])
>    region.append((x, y, z, i, r, g, b))
>
> On tuples:
>
> Tuples are (generally) for representing heterogeneous data; much like
> structs or classes in other languages.  Each "column" of your table
A> represents a different concept; whether an intensity, a location, or
> whatever, and it looks like they are even supposed to be different
> types.  Whenever each column has a distinct meaning/type, a
> heterogeneous collection type like a tuple, namedtuple or class is
> probably the way to go.
>
> On the other hand, you probably want to talk about "the list of
> points"; dealing with them as a heterogeneous collection.  Each point
> in the list, or row of your table, represents the same type of data.
>
> Someone had suggested doing a talk next month on appropriate use of
> data structures, so it is probably a good idea to get some questions
> going on in here.  Using numpy for the detail rows is reasonable too,
> but that means you either give up unboxing the floats (which unboxing
> will halve the memory usage of this table on cpython) or give up
> having different types for different columns; so it might not be
> worthwhile.
>
>

From schweitzer.ubiquitous at gmail.com  Mon Feb  7 08:20:45 2011
From: schweitzer.ubiquitous at gmail.com (martin schweitzer)
Date: Mon, 7 Feb 2011 18:20:45 +1100
Subject: [melbourne-pug] Lists or Arrays - When do I use what and why -
 Or : Confused much? I am, Now
In-Reply-To: <alpine.DEB.2.00.1102071805240.9770@thor.pearwood.mine.nu>
References: <000d01cbc66c$ec2115f0$c46341d0$@netspace.net.au>
	<AANLkTikXtVhMbo=bvVQWWWNtTc1_=D-St2bB_bmV2J05@mail.gmail.com>
	<alpine.DEB.2.00.1102071805240.9770@thor.pearwood.mine.nu>
Message-ID: <AANLkTik59dygfehgka9Ks3F8O-T-SJLNSvwXCjiETF-v@mail.gmail.com>

Hi David

Dictionaries map a key onto a value - and each key must be unique.  However
the keys and values are 'things' and these things are not limited to
integers or strings.  In particular, the value can be a tuple or list... (or
even another dictionary).

So, we can have something like this:

d = {} # d is a dictionary...

d['x'] = (1, 3, 5)  # Tuple
d['y'] = 'abc'       # String
d['z'] = [2, 5, 6]  # list

We can also do something like:

d['x' 'z'] = 42 # degenerates into d['xz'] = 42.

We can also have a tuple as the key... eg.

d[(1,2)] = 42

Hope this helps.

Regards,
Martin




On Mon, Feb 7, 2011 at 6:08 PM, David Crisp <dcrisp at netspace.net.au> wrote:

> Thanks William,
>
> I read up on the Dictionaries at one of the main points I spotted was a
> reference on one of the sites :
>
> (a) More than one entry per key not allowed. Which means no duplicate key
> is allowed. When duplicate keys encountered during assignment, the last
> assignment wins.
>
> By my reading that is saying that there can only be one X value of 10  if a
> new X value of ten comes along then it will supercede the previous.
>
> Because this is 3D space these points represent,  there can actually be
> more than 1 X of Value 10 and indeed there can be more than 1 Y value of 10
> as well.
>
> Or have i misunderstood the restrictions on Dictionaries.
>
> Regards,
> David
>
>
>
>
>  On Mon, 7 Feb 2011, William ML Leslie wrote:
>
>  On 7 February 2011 13:15, David Crisp <dcrisp at netspace.net.au> wrote:
>>
>>> I then wish to iterate through the array and 'bin'  the X values into a
>>> range.  For instance every X value between -54 and -53 goes into bin 1,
>>>  -53
>>> to -52 goes into bin 2
>>>
>>> Then I wish to iterate through this bin array and sort the Y values into
>>> the
>>> same sort of ranges.
>>>
>>> Then I wish to Sort the Z values into the same sort of ranges.   This
>>> should
>>> give me an array of Array[X][Y][Z] Where Array[0][0][0] should give me
>>> the
>>> very first cell  and I can reference any of the values simply by :
>>>
>>>  print Array[100][100][10]
>>>>>>
>>>>>
>>> and get the resulting Z values for that location.
>>>
>>
>> A better data structure for an index like this might actually be a
>> dictionary. A dictionary can be keyed on (X, Y, Z) tuples.
>>
>> You could do something just a little neater than than:
>>
>> points = {}
>> for line in file:
>>   points = line.split(None, 6)
>>   if not points[2:]:
>>       # specifically handle blank lines and count lines
>>       continue
>>   x, y, z, i, r, g, b = map(float, points)
>>   region = points.setdefault((int(x), int(y), int(z)), [])
>>   region.append((x, y, z, i, r, g, b))
>>
>> On tuples:
>>
>> Tuples are (generally) for representing heterogeneous data; much like
>> structs or classes in other languages.  Each "column" of your table
>>
> A> represents a different concept; whether an intensity, a location, or
>
>  whatever, and it looks like they are even supposed to be different
>> types.  Whenever each column has a distinct meaning/type, a
>> heterogeneous collection type like a tuple, namedtuple or class is
>> probably the way to go.
>>
>> On the other hand, you probably want to talk about "the list of
>> points"; dealing with them as a heterogeneous collection.  Each point
>> in the list, or row of your table, represents the same type of data.
>>
>> Someone had suggested doing a talk next month on appropriate use of
>> data structures, so it is probably a good idea to get some questions
>> going on in here.  Using numpy for the detail rows is reasonable too,
>> but that means you either give up unboxing the floats (which unboxing
>> will halve the memory usage of this table on cpython) or give up
>> having different types for different columns; so it might not be
>> worthwhile.
>>
>>
>>  _______________________________________________
> melbourne-pug mailing list
> melbourne-pug at python.org
> http://mail.python.org/mailman/listinfo/melbourne-pug
>



-- 
Martin Schweitzer
Mobile: 0412 345 938
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/melbourne-pug/attachments/20110207/b6c4704b/attachment-0001.html>

From j.lee.nielsen at gmail.com  Mon Feb  7 08:23:50 2011
From: j.lee.nielsen at gmail.com (Jason)
Date: Mon, 07 Feb 2011 18:23:50 +1100
Subject: [melbourne-pug] Lists or Arrays - When do I use what and why -
 Or : Confused much? I am, Now
In-Reply-To: <alpine.DEB.2.00.1102071805240.9770@thor.pearwood.mine.nu>
References: <000d01cbc66c$ec2115f0$c46341d0$@netspace.net.au>
	<AANLkTikXtVhMbo=bvVQWWWNtTc1_=D-St2bB_bmV2J05@mail.gmail.com>
	<alpine.DEB.2.00.1102071805240.9770@thor.pearwood.mine.nu>
Message-ID: <op.vqiut0v7rlpctr@polybius>

You are correct, you just missed the subtle bit where he made the key of the dictionary out of the whole tuple not just x so the combined (x,y,z) would have to be unique to get a duplicate.

So your lookup would then be something like:

In [1]: points = {}
In [2]: points[1,5,7] = 'data'
In [3]: points
Out[3]: {(1, 5, 7): 'data'}

so the key is the combined data structure or tuple of (1, 5, 7)

This method is nice in that it should be a very fast lookup, Im not sure how much it would limit in your application the ability to lookup a whole dimension at once, ie:

Array[X][Y]

would return the whole Z axis using the array methodology where for this dictionary methodology it would be a bit harder to pull that info out.

Jason



On Mon, 07 Feb 2011 18:08:55 +1100, David Crisp <dcrisp at netspace.net.au> wrote:

> Thanks William,
>
> I read up on the Dictionaries at one of the main points I spotted was a
> reference on one of the sites :
>
> (a) More than one entry per key not allowed. Which means no duplicate key
> is allowed. When duplicate keys encountered during assignment, the last
> assignment wins.
>
> By my reading that is saying that there can only be one X value of 10  if
> a new X value of ten comes along then it will supercede the previous.
>
> Because this is 3D space these points represent,  there can actually be
> more than 1 X of Value 10 and indeed there can be more than 1 Y value of
> 10 as well.
>
> Or have i misunderstood the restrictions on Dictionaries.
>
> Regards,
> David
>
>
>
>   On Mon, 7 Feb 2011, William ML Leslie wrote:
>
>> On 7 February 2011 13:15, David Crisp <dcrisp at netspace.net.au> wrote:
>>> I then wish to iterate through the array and 'bin'  the X values into a
>>> range.  For instance every X value between -54 and -53 goes into bin 1,  -53
>>> to -52 goes into bin 2
>>>
>>> Then I wish to iterate through this bin array and sort the Y values into the
>>> same sort of ranges.
>>>
>>> Then I wish to Sort the Z values into the same sort of ranges.   This should
>>> give me an array of Array[X][Y][Z] Where Array[0][0][0] should give me the
>>> very first cell  and I can reference any of the values simply by :
>>>
>>>>>> print Array[100][100][10]
>>>
>>> and get the resulting Z values for that location.
>>
>> A better data structure for an index like this might actually be a
>> dictionary. A dictionary can be keyed on (X, Y, Z) tuples.
>>
>> You could do something just a little neater than than:
>>
>> points = {}
>> for line in file:
>>    points = line.split(None, 6)
>>    if not points[2:]:
>>        # specifically handle blank lines and count lines
>>        continue
>>    x, y, z, i, r, g, b = map(float, points)
>>    region = points.setdefault((int(x), int(y), int(z)), [])
>>    region.append((x, y, z, i, r, g, b))
>>
>> On tuples:
>>
>> Tuples are (generally) for representing heterogeneous data; much like
>> structs or classes in other languages.  Each "column" of your table
> A> represents a different concept; whether an intensity, a location, or
>> whatever, and it looks like they are even supposed to be different
>> types.  Whenever each column has a distinct meaning/type, a
>> heterogeneous collection type like a tuple, namedtuple or class is
>> probably the way to go.
>>
>> On the other hand, you probably want to talk about "the list of
>> points"; dealing with them as a heterogeneous collection.  Each point
>> in the list, or row of your table, represents the same type of data.
>>
>> Someone had suggested doing a talk next month on appropriate use of
>> data structures, so it is probably a good idea to get some questions
>> going on in here.  Using numpy for the detail rows is reasonable too,
>> but that means you either give up unboxing the floats (which unboxing
>> will halve the memory usage of this table on cpython) or give up
>> having different types for different columns; so it might not be
>> worthwhile.
>>
>>
> _______________________________________________
> melbourne-pug mailing list
> melbourne-pug at python.org
> http://mail.python.org/mailman/listinfo/melbourne-pug

From ben.dyer at taguchimail.com  Mon Feb  7 10:12:48 2011
From: ben.dyer at taguchimail.com (Ben Dyer)
Date: Mon, 7 Feb 2011 20:12:48 +1100
Subject: [melbourne-pug] Lists or Arrays - When do I use what and why -
	Or : Confused much? I am, Now
In-Reply-To: <alpine.DEB.2.00.1102071805240.9770@thor.pearwood.mine.nu>
References: <000d01cbc66c$ec2115f0$c46341d0$@netspace.net.au>
	<AANLkTikXtVhMbo=bvVQWWWNtTc1_=D-St2bB_bmV2J05@mail.gmail.com>
	<alpine.DEB.2.00.1102071805240.9770@thor.pearwood.mine.nu>
Message-ID: <853632FA-944F-470A-8A47-C3A31E0E3136@taguchimail.com>

David,

Assuming you need to be able to refer back to the X, Y, Z co-ordinates at full precision, storing 100 million records as a flat array of C structs (3x double, 1x int32, 3x uint8) would take 2.9GB of RAM.

Having run some tests using various native Python data structures to read a file of 100 million records, I'd say the only sane way to do this on PC-class hardware is to use numpy/scipy:
? Sparse matrix classes: http://docs.scipy.org/doc/scipy/reference/sparse.html
? numpy array data types: http://docs.scipy.org/doc/numpy/reference/arrays.dtypes.html

The appropriate structure will depend on the expected sparseness of the data (e.g. if your XYZ values are in the range (-100, 100) then you'll have 100,000,000 points spread across 8,000,000 possible bins so using a sparse data structure probably won't be worth it). Also, you need to take into account the fact that you'll have variable numbers of data points in each bin, so for the non-sparse approach you'd need to use something like a 3-dimensional array of 1-dimensional arrays of records containing 3x double, 1x uint32, 3x uint8 (since numpy doesn't support variable-length arrays, and using the "array of native Python types" functionality with tuples etc would probably consume too much memory).

> I have a large file of data which consists of  the following:  X,Y,Z,I,R,G,B
> As sampled below.   There are a large number of files with variable lengths.
> A few contain 100 million data points but mostly there are between 3 million
> and 10 million points. 

All the above does of course assume that when you say "[you] wish to read this file in to memory and assign it into an array" you're referring to one of the "? large number of file[s] ?" (of which, "[a] few contain 100 million data points ?"). If you're actually referring to the "? large file of data ?" at the start of that paragraph, taken to mean the entire contents of each data file, then there's no way you're going to fit all of that in memory on a system with "desktop" amounts of RAM, so you'd be better off processing on disk or using an extra-large (high memory) Amazon EC2 instance.

Regards,
Ben

On 07/02/2011, at 18:08 , David Crisp wrote:

> Thanks William,
> 
> I read up on the Dictionaries at one of the main points I spotted was a reference on one of the sites :
> 
> (a) More than one entry per key not allowed. Which means no duplicate key is allowed. When duplicate keys encountered during assignment, the last assignment wins.
> 
> By my reading that is saying that there can only be one X value of 10  if a new X value of ten comes along then it will supercede the previous.
> 
> Because this is 3D space these points represent,  there can actually be more than 1 X of Value 10 and indeed there can be more than 1 Y value of 10 as well.
> 
> Or have i misunderstood the restrictions on Dictionaries.
> 
> Regards,
> David
> 
> 
> 
> On Mon, 7 Feb 2011, William ML Leslie wrote:
> 
>> On 7 February 2011 13:15, David Crisp <dcrisp at netspace.net.au> wrote:
>>> I then wish to iterate through the array and 'bin'  the X values into a
>>> range.  For instance every X value between -54 and -53 goes into bin 1,  -53
>>> to -52 goes into bin 2
>>> 
>>> Then I wish to iterate through this bin array and sort the Y values into the
>>> same sort of ranges.
>>> 
>>> Then I wish to Sort the Z values into the same sort of ranges.   This should
>>> give me an array of Array[X][Y][Z] Where Array[0][0][0] should give me the
>>> very first cell  and I can reference any of the values simply by :
>>> 
>>>>>> print Array[100][100][10]
>>> 
>>> and get the resulting Z values for that location.
>> 
>> A better data structure for an index like this might actually be a
>> dictionary. A dictionary can be keyed on (X, Y, Z) tuples.
>> 
>> You could do something just a little neater than than:
>> 
>> points = {}
>> for line in file:
>>   points = line.split(None, 6)
>>   if not points[2:]:
>>       # specifically handle blank lines and count lines
>>       continue
>>   x, y, z, i, r, g, b = map(float, points)
>>   region = points.setdefault((int(x), int(y), int(z)), [])
>>   region.append((x, y, z, i, r, g, b))
>> 
>> On tuples:
>> 
>> Tuples are (generally) for representing heterogeneous data; much like
>> structs or classes in other languages.  Each "column" of your table
> A> represents a different concept; whether an intensity, a location, or
>> whatever, and it looks like they are even supposed to be different
>> types.  Whenever each column has a distinct meaning/type, a
>> heterogeneous collection type like a tuple, namedtuple or class is
>> probably the way to go.
>> 
>> On the other hand, you probably want to talk about "the list of
>> points"; dealing with them as a heterogeneous collection.  Each point
>> in the list, or row of your table, represents the same type of data.
>> 
>> Someone had suggested doing a talk next month on appropriate use of
>> data structures, so it is probably a good idea to get some questions
>> going on in here.  Using numpy for the detail rows is reasonable too,
>> but that means you either give up unboxing the floats (which unboxing
>> will halve the memory usage of this table on cpython) or give up
>> having different types for different columns; so it might not be
>> worthwhile.
>> 
>> 
> _______________________________________________
> melbourne-pug mailing list
> melbourne-pug at python.org
> http://mail.python.org/mailman/listinfo/melbourne-pug


From ben.dyer at taguchimail.com  Mon Feb  7 15:06:28 2011
From: ben.dyer at taguchimail.com (Ben Dyer)
Date: Tue, 8 Feb 2011 01:06:28 +1100
Subject: [melbourne-pug] Lists or Arrays - When do I use what and why -
	Or : Confused much? I am, Now
In-Reply-To: <853632FA-944F-470A-8A47-C3A31E0E3136@taguchimail.com>
References: <000d01cbc66c$ec2115f0$c46341d0$@netspace.net.au>
	<AANLkTikXtVhMbo=bvVQWWWNtTc1_=D-St2bB_bmV2J05@mail.gmail.com>
	<alpine.DEB.2.00.1102071805240.9770@thor.pearwood.mine.nu>
	<853632FA-944F-470A-8A47-C3A31E0E3136@taguchimail.com>
Message-ID: <1B38E98E-615F-4662-AA31-A3E6F1F121D0@taguchimail.com>

To put some concrete numbers to this, I ran some test scripts on an Amazon m2.4xlarge instance (68GB RAM). Loading 100m records with uniformly distributed coordinate values in the range (-100.0, 100.0) into a 200x200x200 numpy array (with each element of the 3d array containing a 1d numpy array of records) takes 42m13s of CPU time and 3.2GB RAM. Using a similar structure based on Python dicts, lists and tuples (3d sparse matrix using dicts, with each element being a list, containing a tuple per record) has consumed 106m17s of CPU time and 10GB RAM for 42.5m records so far ? and it's scaling much, much worse.

numpy arrays support easy access to records within a bin, e.g.:

> >>> data[5,1,2]
> array([ (5.3820352554321289, 1.9680783748626709, 2.4225616455078125, 8863, 83, 123, 155),
>        (5.9297022819519043, 1.285321831703186, 2.2762794494628906, 77, 135, 127, 190),
>        (5.1742649078369141, 1.0804551839828491, 2.246323823928833, 700, 126, 192, 215),
>        (5.7373204231262207, 1.5889065265655518, 2.0195131301879883, 1027, 113, 186, 87),
>        (5.2434444427490234, 1.3134119510650635, 2.1608052253723145, 8901, 29, 203, 36),
>        (5.6749391555786133, 1.5029317140579224, 2.0740163326263428, -4095, 187, 212, 183),
>        (5.874603271484375, 1.645256519317627, 2.7637510299682617, 8471, 81, 88, 57),
>        (5.076021671295166, 1.2720983028411865, 2.7047195434570312, -5897, 239, 228, 105),
>        (5.4287075996398926, 1.7871297597885132, 2.3681249618530273, 5266, 130, 119, 252),
>        (5.1363730430603027, 1.4398738145828247, 2.2409327030181885, -106, 101, 41, 236)], 
>       dtype=[('x', '<f4'), ('y', '<f4'), ('z', '<f4'), ('i', '<i4'), ('r', '|u1'), ('g', '|u1'), ('b', '|u1')])

Slicing is also supported, so "data[5,1]" returns all records for X=5, Y=1; "data[5,:,2]" returns everything for X=5, Z=2.

My numpy code was as follows:

> import numpy
> record_type = numpy.dtype({
>     'names': ['x', 'y', 'z', 'i', 'r', 'g', 'b'], 
>     'formats': ['f4', 'f4', 'f4', 'i4', 'u1', 'u1', 'u1']
> })
> data = numpy.empty((200, 200, 200), dtype='object')
> with open('/data/xyzirgb-test', 'r') as f:
>     for line in f:
>         line = line.split()
>         if len(line) < 7:
>             continue
>         
>         line = (
>             float(line[0]), float(line[1]), float(line[2]), # X, Y, Z
>             int(line[3]), # I
>             int(line[4]), int(line[5]), int(line[6]) # R, G, B
>         )
>         coord = int(line[0]), int(line[1]), int(line[2])
>         rec = numpy.array([line], dtype=record_type)
>         data[coord] = rec if data[coord] is None else numpy.append(data[coord], rec, axis=0)




On 07/02/2011, at 20:12 , Ben Dyer wrote:

> David,
> 
> Assuming you need to be able to refer back to the X, Y, Z co-ordinates at full precision, storing 100 million records as a flat array of C structs (3x double, 1x int32, 3x uint8) would take 2.9GB of RAM.
> 
> Having run some tests using various native Python data structures to read a file of 100 million records, I'd say the only sane way to do this on PC-class hardware is to use numpy/scipy:
> ? Sparse matrix classes: http://docs.scipy.org/doc/scipy/reference/sparse.html
> ? numpy array data types: http://docs.scipy.org/doc/numpy/reference/arrays.dtypes.html
> 
> The appropriate structure will depend on the expected sparseness of the data (e.g. if your XYZ values are in the range (-100, 100) then you'll have 100,000,000 points spread across 8,000,000 possible bins so using a sparse data structure probably won't be worth it). Also, you need to take into account the fact that you'll have variable numbers of data points in each bin, so for the non-sparse approach you'd need to use something like a 3-dimensional array of 1-dimensional arrays of records containing 3x double, 1x uint32, 3x uint8 (since numpy doesn't support variable-length arrays, and using the "array of native Python types" functionality with tuples etc would probably consume too much memory).
> 
>> I have a large file of data which consists of  the following:  X,Y,Z,I,R,G,B
>> As sampled below.   There are a large number of files with variable lengths.
>> A few contain 100 million data points but mostly there are between 3 million
>> and 10 million points. 
> 
> All the above does of course assume that when you say "[you] wish to read this file in to memory and assign it into an array" you're referring to one of the "? large number of file[s] ?" (of which, "[a] few contain 100 million data points ?"). If you're actually referring to the "? large file of data ?" at the start of that paragraph, taken to mean the entire contents of each data file, then there's no way you're going to fit all of that in memory on a system with "desktop" amounts of RAM, so you'd be better off processing on disk or using an extra-large (high memory) Amazon EC2 instance.
> 
> Regards,
> Ben
> 
> On 07/02/2011, at 18:08 , David Crisp wrote:
> 
>> Thanks William,
>> 
>> I read up on the Dictionaries at one of the main points I spotted was a reference on one of the sites :
>> 
>> (a) More than one entry per key not allowed. Which means no duplicate key is allowed. When duplicate keys encountered during assignment, the last assignment wins.
>> 
>> By my reading that is saying that there can only be one X value of 10  if a new X value of ten comes along then it will supercede the previous.
>> 
>> Because this is 3D space these points represent,  there can actually be more than 1 X of Value 10 and indeed there can be more than 1 Y value of 10 as well.
>> 
>> Or have i misunderstood the restrictions on Dictionaries.
>> 
>> Regards,
>> David
>> 
>> 
>> 
>> On Mon, 7 Feb 2011, William ML Leslie wrote:
>> 
>>> On 7 February 2011 13:15, David Crisp <dcrisp at netspace.net.au> wrote:
>>>> I then wish to iterate through the array and 'bin'  the X values into a
>>>> range.  For instance every X value between -54 and -53 goes into bin 1,  -53
>>>> to -52 goes into bin 2
>>>> 
>>>> Then I wish to iterate through this bin array and sort the Y values into the
>>>> same sort of ranges.
>>>> 
>>>> Then I wish to Sort the Z values into the same sort of ranges.   This should
>>>> give me an array of Array[X][Y][Z] Where Array[0][0][0] should give me the
>>>> very first cell  and I can reference any of the values simply by :
>>>> 
>>>>>>> print Array[100][100][10]
>>>> 
>>>> and get the resulting Z values for that location.
>>> 
>>> A better data structure for an index like this might actually be a
>>> dictionary. A dictionary can be keyed on (X, Y, Z) tuples.
>>> 
>>> You could do something just a little neater than than:
>>> 
>>> points = {}
>>> for line in file:
>>>  points = line.split(None, 6)
>>>  if not points[2:]:
>>>      # specifically handle blank lines and count lines
>>>      continue
>>>  x, y, z, i, r, g, b = map(float, points)
>>>  region = points.setdefault((int(x), int(y), int(z)), [])
>>>  region.append((x, y, z, i, r, g, b))
>>> 
>>> On tuples:
>>> 
>>> Tuples are (generally) for representing heterogeneous data; much like
>>> structs or classes in other languages.  Each "column" of your table
>> A> represents a different concept; whether an intensity, a location, or
>>> whatever, and it looks like they are even supposed to be different
>>> types.  Whenever each column has a distinct meaning/type, a
>>> heterogeneous collection type like a tuple, namedtuple or class is
>>> probably the way to go.
>>> 
>>> On the other hand, you probably want to talk about "the list of
>>> points"; dealing with them as a heterogeneous collection.  Each point
>>> in the list, or row of your table, represents the same type of data.
>>> 
>>> Someone had suggested doing a talk next month on appropriate use of
>>> data structures, so it is probably a good idea to get some questions
>>> going on in here.  Using numpy for the detail rows is reasonable too,
>>> but that means you either give up unboxing the floats (which unboxing
>>> will halve the memory usage of this table on cpython) or give up
>>> having different types for different columns; so it might not be
>>> worthwhile.
>>> 
>>> 
>> _______________________________________________
>> melbourne-pug mailing list
>> melbourne-pug at python.org
>> http://mail.python.org/mailman/listinfo/melbourne-pug
> 
> _______________________________________________
> melbourne-pug mailing list
> melbourne-pug at python.org
> http://mail.python.org/mailman/listinfo/melbourne-pug


From richard at python.org  Tue Feb  8 09:32:47 2011
From: richard at python.org (Richard Jones)
Date: Tue, 8 Feb 2011 19:32:47 +1100
Subject: [melbourne-pug] PyCon Australia 2011: 20th & 21st August,
	Sydney Masonic Center
Message-ID: <AANLkTimzoupTiZXG3eO_ZNupLsY-i2qM7rh=MSr-jCtV@mail.gmail.com>

The second PyCon Australia will be held in Sydney on the weekend of the 20th
and 21st of August at the Sydney Masonic Center.

The first PyCon Australia was held in June 2010 and attracted over 200 Python
programming enthusiasts. The second event is expected to
host over 250 attendees.

The weekend will see dozens of presentations introducing;

- Python programming and techniques,
- web programming,
- business applications,
- game development,
- education, science and mathematics,
- social issues,
- testing, databases, documentation and more!

We are hoping to organise sprints on the days following the conference proper.

International guests should note that Kiwi PyCon is to run on the
following weekend, making it a great opportunity to attend a couple of
awesome Down Under conferences and hopefully do some sprinting with
the locals.


    Richard Jones
    http://pycon-au.org/
    PyCon AU Committee

From brian.curtin at gmail.com  Wed Feb  9 02:01:04 2011
From: brian.curtin at gmail.com (Brian Curtin)
Date: Tue, 8 Feb 2011 19:01:04 -0600
Subject: [melbourne-pug] PyCon 2011 news - going strong, sign up today!
Message-ID: <AANLkTincHL3Zw7usv=kVr1Dd769jgkwpRsr=M3xT+_6o@mail.gmail.com>

Hello group!

PyCon 2011 looks like it may very well break every single record in the past
-
making it one of the biggest and best PyCons of all time. We've gone all out
this year - including Extreme Talks, a Startup Row, amazing talks,
tutorials,
Poster sessions.

Extreme talks:
http://us.pycon.org/2011/speaker/extreme/
Startup Row:
http://us.pycon.org/2011/blog/2011/01/19/announcing-startup-row-pycon-2011/

Also just announced - "Startup Stories":
http://us.pycon.org/2011/blog/2011/02/07/pycon-2011-announcing-startup-stories/

If you haven't bought your PyCon tickets, it's time to do so. This year
we're
limited to 1500 attendees, and with a faster than average sales rate
compared
to previous years, you may not have the luxury of leaving registration until
just before the conference -- places may sell out in advance! Check out
http://us.pycon.org/2011/tickets/ for rates and details.

Not sold on the conference? Not sure what it's all about? We are confident
this
year's conference will surpass anything that has gone before, and we think
there's something for everyone.

Looking for education? This year's tutorial selection features a range of
topics and an all-star cast of presenters. From the cloud to the GUI, from
beginner to advanced, we have it. Check out courses taught by book authors
Wesley Chun and David Beazley, web framework master Jacob Kaplan-Moss, core
developer and guru Raymond Hettinger, hacker extraordinaire Zed Shaw and
many
more: http://us.pycon.org/2011/schedule/tutorials/ -- and at flat fees that
wouldn't buy you an hour of training by less-qualified instructors
elsewhere!

The conference talks are some of the best we've seen in years, coming from a
record number of submissions. We've got talks on using Python for airplane
tuning and massive telescopes. All of the major web frameworks will be
discussed. Numerous databases are covered. PyPy, IronPython and Jython are
in.
We'll be talking about IDEs, testing, security, scalability, and
documentation.
Have a look: http://us.pycon.org/2011/schedule/lists/talks/

We've also got an amazing array of poster presentations - everything from
Arduino hacking, Geospatial Python, Open Government and Microsoft Kinect
hacking. Check out: http://us.pycon.org/2011/schedule/lists/posters/

Meanwhile we've got the awesome sprints  (http://us.pycon.org/2011/sprints/
),
and open spaces (http://us.pycon.org/2011/openspaces/) everyone knows and
loves.

This is shaping up to be the best PyCon yet, and we think it's a great
value,
both cost-wise, and community-wise. We can guarantee that you'll come away
with
a head full of knowledge and amazing new friends and contacts.

Get your tickets before they are gone: http://us.pycon.org/2011/tickets/

If you have any questions, comments or concerns - feel free to email Van
Lindberg (van at python.org) or Jesse Noller (jnoller at python.org) or
pycon-organizers (pycon-organizers at python.org).

See you in Atlanta,

Van Lindberg
Jesse Noller
The entire PyCon 2011 team
http://us.pycon.org/2011
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/melbourne-pug/attachments/20110208/dd47c7b4/attachment.html>

From brian.curtin at gmail.com  Wed Feb  9 15:41:08 2011
From: brian.curtin at gmail.com (Brian Curtin)
Date: Wed, 9 Feb 2011 08:41:08 -0600
Subject: [melbourne-pug] PyCon 2011 news - going strong, sign up today!
Message-ID: <AANLkTi=FDVSV-xDt1wLLYYmwTr845pLJzRd-J6F=SV=_@mail.gmail.com>

Hello group!

PyCon 2011 looks like it may very well break every single record in the past
-
making it one of the biggest and best PyCons of all time. We've gone all out
this year - including Extreme Talks, a Startup Row, amazing talks,
tutorials,
Poster sessions.

Extreme talks:
http://us.pycon.org/2011/speaker/extreme/
Startup Row:
http://us.pycon.org/2011/blog/2011/01/19/announcing-startup-row-pycon-2011/

Also just announced - "Startup Stories":
http://us.pycon.org/2011/blog/2011/02/07/pycon-2011-announcing-startup-stories/

If you haven't bought your PyCon tickets, it's time to do so. This year
we're
limited to 1500 attendees, and with a faster than average sales rate
compared
to previous years, you may not have the luxury of leaving registration until
just before the conference -- places may sell out in advance! Check out
http://us.pycon.org/2011/tickets/ for rates and details.

Not sold on the conference? Not sure what it's all about? We are confident
this
year's conference will surpass anything that has gone before, and we think
there's something for everyone.

Looking for education? This year's tutorial selection features a range of
topics and an all-star cast of presenters. From the cloud to the GUI, from
beginner to advanced, we have it. Check out courses taught by book authors
Wesley Chun and David Beazley, web framework master Jacob Kaplan-Moss, core
developer and guru Raymond Hettinger, hacker extraordinaire Zed Shaw and
many
more: http://us.pycon.org/2011/schedule/tutorials/ -- and at flat fees that
wouldn't buy you an hour of training by less-qualified instructors
elsewhere!

The conference talks are some of the best we've seen in years, coming from a
record number of submissions. We've got talks on using Python for airplane
tuning and massive telescopes. All of the major web frameworks will be
discussed. Numerous databases are covered. PyPy, IronPython and Jython are
in.
We'll be talking about IDEs, testing, security, scalability, and
documentation.
Have a look: http://us.pycon.org/2011/schedule/lists/talks/

We've also got an amazing array of poster presentations - everything from
Arduino hacking, Geospatial Python, Open Government and Microsoft Kinect
hacking. Check out: http://us.pycon.org/2011/schedule/lists/posters/

Meanwhile we've got the awesome sprints  (http://us.pycon.org/2011/sprints/
),
and open spaces (http://us.pycon.org/2011/openspaces/) everyone knows and
loves.

This is shaping up to be the best PyCon yet, and we think it's a great
value,
both cost-wise, and community-wise. We can guarantee that you'll come away
with
a head full of knowledge and amazing new friends and contacts.

Get your tickets before they are gone: http://us.pycon.org/2011/tickets/

If you have any questions, comments or concerns - feel free to email Van
Lindberg (van at python.org) or Jesse Noller (jnoller at python.org) or
pycon-organizers (pycon-organizers at python.org).

See you in Atlanta,

Van Lindberg
Jesse Noller
The entire PyCon 2011 team
http://us.pycon.org/2011
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/melbourne-pug/attachments/20110209/5471f2a6/attachment.html>

From acalcium at yahoo.com.au  Wed Feb  9 22:50:17 2011
From: acalcium at yahoo.com.au (Chai Ang)
Date: Wed, 9 Feb 2011 13:50:17 -0800 (PST)
Subject: [melbourne-pug] Does Anyone In Australia Want a Free Python3
	PyCon Tutorial?
In-Reply-To: <mailman.49.1296903602.22338.melbourne-pug@python.org>
Message-ID: <364766.33939.qm@web110104.mail.gq1.yahoo.com>

> On 5/02/2011 8:49am, Chai Ang wrote:
> > Pretty short notice though.
> >
> > http://dabeaz.blogspot.com/2011/02/does-anyone-in-australia-want-free.html
> 
> If anyone from Melbourne is setting this up I'd like to see
> it but can 
> only attend if it is Sunday 13 Feb.


In case anyone wasnt aware, it is happening in Melb but is on Sat the 12th.

Check the blog again for details.
 
http://dabeaz.blogspot.com/2011/02/does-anyone-in-australia-want-free.html


      

From dariusp686 at gmail.com  Wed Feb  9 23:43:17 2011
From: dariusp686 at gmail.com (Darius Powell)
Date: Thu, 10 Feb 2011 09:43:17 +1100
Subject: [melbourne-pug] Does Anyone In Australia Want a Free Python3
 PyCon Tutorial?
In-Reply-To: <364766.33939.qm@web110104.mail.gq1.yahoo.com>
References: <mailman.49.1296903602.22338.melbourne-pug@python.org>
	<364766.33939.qm@web110104.mail.gq1.yahoo.com>
Message-ID: <AANLkTimDdFQT=PK5T+X92tULiDqsJmni9tqHhLRSgQg6@mail.gmail.com>

Hi,

For anyone interested this is happening on Sat 12th at around 1:30pm.

Email Steven.Cyphers at ghd.com to request a place.

Dave does some great talks so I'm looking forward to this, even if I'm
still on 2.6.6!

On 10 February 2011 08:50, Chai Ang <acalcium at yahoo.com.au> wrote:
>> On 5/02/2011 8:49am, Chai Ang wrote:
>> > Pretty short notice though.
>> >
>> > http://dabeaz.blogspot.com/2011/02/does-anyone-in-australia-want-free.html
>>
>> If anyone from Melbourne is setting this up I'd like to see
>> it but can
>> only attend if it is Sunday 13 Feb.
>
>
> In case anyone wasnt aware, it is happening in Melb but is on Sat the 12th.
>
> Check the blog again for details.
>
> http://dabeaz.blogspot.com/2011/02/does-anyone-in-australia-want-free.html
>
>
>
> _______________________________________________
> melbourne-pug mailing list
> melbourne-pug at python.org
> http://mail.python.org/mailman/listinfo/melbourne-pug
>

From william.leslie.ttg at gmail.com  Thu Feb 10 08:19:18 2011
From: william.leslie.ttg at gmail.com (William ML Leslie)
Date: Thu, 10 Feb 2011 18:19:18 +1100
Subject: [melbourne-pug] Lists or Arrays - When do I use what and why -
 Or : Confused much? I am, Now
In-Reply-To: <alpine.DEB.2.00.1102071805240.9770@thor.pearwood.mine.nu>
References: <000d01cbc66c$ec2115f0$c46341d0$@netspace.net.au>
	<AANLkTikXtVhMbo=bvVQWWWNtTc1_=D-St2bB_bmV2J05@mail.gmail.com>
	<alpine.DEB.2.00.1102071805240.9770@thor.pearwood.mine.nu>
Message-ID: <AANLkTikcoiPibdQbitMA4yFPqyDVW+pX03EQSCcZKdGL@mail.gmail.com>

On 7 February 2011 18:08, David Crisp <dcrisp at netspace.net.au> wrote:
> (a) More than one entry per key not allowed. Which means no duplicate key is
> allowed. When duplicate keys encountered during assignment, the last
> assignment wins.
>
> By my reading that is saying that there can only be one X value of 10 ?if a
> new X value of ten comes along then it will supercede the previous.
>
> Because this is 3D space these points represent, ?there can actually be more
> than 1 X of Value 10 and indeed there can be more than 1 Y value of 10 as
> well.

That is why I used a dictionary of lists.  Dictionaries with sets or
lists as the value can be a very useful data structure.  This means
that points[3, 4, 5] is the list of all points just north-west-up of
(3, 4, 5).

If you do have different access patterns, you can build appropriate
indexes; like so:

position_by_x = {}
for x, y, z in points:
    index_by_x.setdefault(x, []).append((x, y, z))


On 8 February 2011 01:06, Ben Dyer <ben.dyer at taguchimail.com> wrote:
> To put some concrete numbers to this, I ran some test scripts on an Amazon m2.4xlarge instance (68GB RAM). Loading 100m records with uniformly distributed coordinate values in the range (-100.0, 100.0) into a 200x200x200 numpy array (with each element of the 3d array containing a 1d numpy array of records) takes 42m13s of CPU time and 3.2GB RAM. Using a similar structure based on Python dicts, lists and tuples (3d sparse matrix using dicts, with each element being a list, containing a tuple per record) has consumed 106m17s of CPU time and 10GB RAM for 42.5m records so far ? and it's scaling much, much worse.

Or even a database; depending on what you expect to do with all that
data.  Besides allowing better parallel access, an appropriately
indexed relational database may reduce the amount of paging and TLB
churn you get attempting to use such a large array mapped directly
into memory.

-- 
William Leslie

From richard at python.org  Mon Feb 21 03:28:15 2011
From: richard at python.org (Richard Jones)
Date: Mon, 21 Feb 2011 13:28:15 +1100
Subject: [melbourne-pug] PyCon Australia 2011 - Call for Participation
Message-ID: <AANLkTinPAQ9xy1zuMQpU2ed3Fgh=f01-0FwHF7MQy1Ge@mail.gmail.com>

The second PyCon AU will be held in Sydney on the weekend of the 20th
and 21st of August at the Sydney Masonic Center.

  http://pycon-au.org/

We are looking for proposals for Talks on all aspects of Python programming
from novice to advanced levels; applications and frameworks, or how you
have been involved in introducing Python into your organisation. We're
especially interested in short presentations that will teach conference-goers
something new and useful. Can you show attendees how to use a module?
Explore a Python language feature? Package an application?

We welcome first-time speakers; we are a community conference and we are
eager to hear about your experience. If you have friends or colleagues
who have something valuable to contribute, twist their arms to tell us
about it! Please also forward this Call for Proposals to anyone that you
feel may be interested.

To find out more go to the official Call for Proposals page here:

   http://pycon-au.org/2011/conference/proposals/

The deadline for proposal submission is the 2nd of May.


See you in Sydney in August!

     Richard Jones
     PyCon AU Program Chair