Handy utilities = Friday Filosofical Finking

[Cameron Simpson]

On 02Apr2019 13:14, DL Neil <PythonList at DancesWithMice.info> wrote:
>On 2/04/19 11:57 AM, Cameron Simpson wrote:
>>On 29Mar2019 09:32, DL Neil <PythonList at DancesWithMice.info> wrote:
>>>Do you 'keep' these, or perhaps next time you need something you've 
>>>'done before' do you remember when/where a technique was last 
>>>used/burrow into 'history'?
>>>(else, code it from scratch, all over again)

I didn't say it before, but I _loathe_ the "files with code snippets" 
approach. Hard to search, hard to reuse, promotes cut/paste coding which 
means that bugfixes _don't_ make it into all the codebases, just the one 
you're hacking right now.

>>I put them into modules for import. I've got a tree of Python 
>>modules named "cs.this", "cs.that" for various things. Then just 
>>import stuff like any other library module.
>
>Agreed - but Python doesn't (natively) like imports from a 
>different/parallel/peer-level dir-tree. (see also Apache httpd)

Well, you have several options:

- modify $PYTHONPATH

- make a virtualenv

- make a symlink; Python imports from the current directory - not my 
  favourite feature, but very handy in the dev environment

>>For personal projects (if they aren't just part of that tree) I just 
>>need to symlink the tree into the local Python library as "cs".
>
>I was doing this.

This works really well for me. In particular, in my development tree I 
symlink to my _dev_ "cs" tree, so that if I modify things (fix a bug, 
add a feature etc) to a cs.* module the change is right there in the 
repo where I can tidy it up and commit it and possibly publish it.

>Much of my work is a simple-delivery, ie copying code from my dev m/c 
>to the client's PC or server - so I don't 'package it up'* because of 
>the (perceived) effort required cf the one-to-one (or maybe a couple) 
>machine relationships.

I'm doing something similar right now, but where I've used a cs.* module 
in their code it will go through PyPI as part of the prepare-for-deploy 
process so that they can reproduce the build themselves.

In my case the deploy.sh script makes a directory tree with a copy of 
the released code (from a repo tag name - "hg archive" for me, there's 
an equivalent git command).  That tree contains a prep.sh script which 
runs on the target machine to build the virtualenv and likewise the 
javascript side which is used for the web front end.

So deploy for me is:

- get the code ready (committed, tagged) at some suitable good phase

- run "./deploy.sh release-tag" which makes a deployable tree

- rsync onto the target (into a shiny new tree - I'll twiddle a symlink 
  when it goes "live")

- run "./prep.sh" in the target, which fetches components etc

The advantage of the prep.sh step is that (a) the target matches the 
target host (where that matters, for example the local virtualenv will 
run off the target host Python and so forth), and _also_ (b) it serves 
as doco for me on how to build the app: if prep.sh doesn't work, my 
client doesn't have a working recipe for building their app.

I still need to add some prechecks to the deploy, like "is the venv 
requirements file commented to the repo" etc.

>However, during 'delivery' to another machine, have to remember to 
>rsync/copy including the symlink, as well as to transfer both 
>dir-trees.

By making a local virtualenv and getting my modules through PyPI (pip 
install) this issue is bypassed: there's the client code library 
(rsynced) and the third party modules fetched via PyPI. (Not just my 
cs.* modules if any, but also all the other modules required.)

>Recently, stopped to organise the code into (more) modules (as also 
>suggested here) and moved to adding the 'utilities' directories into 
>PYTHONPATH. Now I have to remember to modify that on the/each target 
>m/c!

Script it. Include the script in the rsync.

>>If I get something well enough defined and sufficiently cleaned up for 
>>use beyond my own code (or just good enough that others might want 
>>it), up it goes to PyPI so that it can just be "pip install"ed.
>>
>>So I've got this huge suite of modules with stuff in them, and a 
>>subset of those modules are published to PyPI for third party reuse.
>
>Am dubious that any of the 'little bits' that I have collected are 
>sufficiently worthy, but that's 'proper' F/LOSSy thinking!

If you're reusing your little bits then they need organisation into 
modules so that you can include them with the main code without treading 
on others' namespaces.

Publishing to PyPI is a very very optional step; the critical thing is 
breaking stuff up into modules; rsyncing them is then easy and you have 
a common codebase which _were formerly_ snippets for reuse.

Also, putting them in modules and using them that way forces you to make 
you snippets reusable instead of cut/patse/adaptable. Win win.

>*will be interested to hear if you think I should stop 'being lazy' and 
>invest some time-and-effort into learning 'packaging' options and do 
>things 'properly'?professionally...

There's real pain there. The first time I did this (2015?) I basicly 
spent a whole week right after new year figuring out how to make a 
package and push it up; the doco was fragmented and confusing, and in 
some cases out of date, and there is/was more that one way to do things.

These days things are somewhat cleaner: start here:

  https://packaging.python.org/

>One of the points which intrigue me is that my colleagues don't keep 
>snippets/a library, preferring to remember (hah!) when/where they used 
>particular techniques in the past, and copying/duplicating, to fit the 
>new system's requirements. Am wondering if this is true beyond our 
>little band?

Among random people I suspect it is quite common. In larger teams you 
accrue a utilities library which contain this stuff.

>Yet, here on the list, interest was shown in 'being organised' (even if 
>few have actually weighed-in)...

Get your reused code organsed - it is a huge win. At least make a little 
library of modules in its own namespace (so you can just import it 
without conflict). What automatic/published/distribution processes you 
follow after that is your choice. But having that library? Invaluable.

Cheers,
Cameron Simpson <cs at cskk.id.au>