[Tutor] Pythonic way

[Avi Gross]

This is not a question or reply. Nor is it short. If not interested, feel free to delete.

 

It is an observation based on recent experiences.

 

We have had quite a few messages that pointed out how some people approach  solving a problem using subconscious paradigms inherited from their past. This includes people who never programmed and are thinking of how they might do it manually as well as people who are proficient in one or more other computer languages and their first attempt to visualize the solution may lead along paths that are possibly doable in Python but not optimal or even suggested.

 

I recently had to throw together a routine that would extract info from multiple SAS data files and join them together on one key into a large DataFrame (or data.frame or other such names for a tabular object.). Then I needed to write them out to disk either as a CSV or XLSX file for future use.

 

Since I have studied and used (not to mention abused) many programming languages, my first thought was to do this in R. It has lots of the tools needed to do such things including packages (sort of like modules you can import but not exactly) and I have done many data/graphics programs in it. I then redid it in Python after some thought.

 

The pseudocode outline is:

 

*	Read in all the files into a set of data.frame objects.
*	Trim back the variables/columns of some of them as many are not needed.
*	Join them together on a common index using a full or outer join.
*	Save the result on disk as a Comma Separated Values file.
*	Save the result on disk as a named tab in a new style EXCEL file.

 

I determined some of what I might us such as the needed built-in commands, packages and functions I could use for the early parts but ran into an annoyance as some of the files contained duplicate entries. Luckily, the R function reduce (not the same as map/reduce) is like many things in R and takes a list of items and makes it work. Also, by default, it renames duplicates so if you have ALPHA in multiple places, it names them ALPHA.x and ALPHA.x.x and other variations.

 

df_joined <- reduce(df_list, full_join, by = "COMMON")

 

Mind you, when I stripped away many of the columns not needed in some of the files, there were fewer duplicates and a much smaller output file.

 

But I ran into a wall later. Saving into a CSV is trivial. There are multiple packages meant to be used for saving into a XLSX file but they all failed for me. One wanted something in JAVA and another may want PERL and some may want packages I do not have installed. So, rather than bash my head against the wall, Itwisted and used the best XSLX maker there is. I opened the CSV file in EXCEL manually and did a SAVE AS …

 

Then I went to plan C (no, not the language C or its many extensions like C++) as I am still learning Python and have not used it much. As an exercise, I decided to learn how to do this in Python using tools/modules like numpy and pandas that I have not had much need for as well as additional tools for reading and writing files in other formats.

 

My first attempts gradually worked, after lots of mistakes and looking at manual pages. It followed an eclectic set of paradigms but worked.  Not immediately, as I ran into a problem in that the pandas version of a join did not tolerate duplicate column names when used on a list. I could get it to rename the left or right list (adding a suffix a suffix) when used on exactly two DataFrames. So, I needed to take the first df and do a df.join(second, …) then take that and join the third and so on. I also needed to keep telling it to set the index to the common value for each and every df including the newly joined series. And, due to size, I chose to keep deleting df no longer in use but that would not be garbage collected.

 

I then looked again at how to tighten it up in a more pythonic way. In English (my sixth language since we are talking about languages 😉 ) I did some things linearly then shifted it to a list method. I used lists of file names and lists of the df made from each file after removing unwanted columns. (NOTE: I use “column” but depending on language and context I mean variable or field or axis or many other ways to say a group of related information in a tabular structure that crosses rows or instances.)

 

So I was able to do my multi-step join more like this:

 

join_with_list = dflist[1:]

current = df1

suffix = 1

 

for df in join_with_list:

    current = current.join(df, how='outer', rsuffix='_'+str(suffix))

    suffix += 1

    current.set_index('ID')

 

In this formulation, the intermediate DataFrame objects held in current will silently be garbage collected as nothing points to them, for example. Did I mention these were huge files?

 

The old code was much longer and error prone as I had a df1, df2, … df8 as well as other intermediates and was easy to copy and paste then not edit the changes properly.

 

On to the main point. Saving as a CSV was trivial. Saving as an XLSX took some work BECAUSE I had pandas but apparently was missing some components it needed. I had to figure out what was needed and get it and finally got it working nicely.

 

In this case, I am sure I might have been able to figure out how to make my R environment work in my Windows 10 machine or installed the other software needed or move my development into Cygwin or even used Linux. 

 

Now for the main point. As hinted above, sometimes the functionality in a particular programming language like R or Python already requires the use of external resources that can include parts made in other languages. Python makes extensive use of internal parts being made in C for speed and the interpreter itself is in C or C++. Similarly R uses C and C++. So the next step can be to integrate the languages. They have strengths and weaknesses and I know both are used heavily in Machine Learning which is part of my current focus. Some things are best done when your language matches the method/algorithm you use. R has half a dozen object-oriented variations and generally they are not as useful as the elegant and consistent OO model in Python. But it has strengths that let me make weird graphics in at least three different major philosophies (base, lattice and ggplot) and lots of nice things and ways based on the underlying philosophy being that everything is a vector and most operations are vectorized or can be. Handy. It can lead to modes of thinking about a problem quite different than the pythonic way.

 

So, I can now load an R package that lets me shift back and forth in the same session to a now embedded python Interpreter. Many “objects” can be created and processed in one language, then passed through the veil to functionality in the other and back and forth. You can load modules within the python component and load packages in the R component. Within reason, you use the best tools for the job. If part is best done with sets or dictionaries or generators, do it on the python side. Want to use ggplot2 for graphics, do it on the R side.

 

In reality, there are many more programming languages (especially dormant ones) than are really needed. But they evolve and some are designed more for some tasks than others. So, if you want to use Python, why not see if you can use it the way it is loosely intended. If all you are doing is rewriting it to fit the mold of another language, why not use that, if still available.

 

Do I have a favorite language? Not really. I note that in attempts to improve Python (and other languages too) over the years, they keep adding and often in ways that sort of change enough so there is no longer as clear a philosophy. I can see heavy borrowing from many horse breeds and making a camel. So there isn’t really ONE pythonic way for many things. We have two completely separate ways to format strings that end up with fairly similar functionality. Actually, there is an implicit third way 😊

 

So I think there are multiple overlapping sets of what it means to be pythonic. If you come from a procedural background, you can do much without creating objects or using functional programming skills. If you come from an OO background, you can have fun making endless classes and subclasses to do just about everything including procedural things the hard way by creating one master object that controls lots of lave objects. If you lie functional programming with factories that churn out functions that may retain their variables, again, have fun. There are other such paradigms supported including lots of miniature sub-languages you can create with regular expressions being an example, as well as the print formatting methods. To be fluent in python, though, you need to be able to use other people’s existing code and perhaps be able to modify or emulate it. That effectively means being open to multiple ways so in a sense, being pythonic includes being flexible, to a point.

 

Good place to stop and resume my previously scheduled life.