Handy utilities = Friday Filosofical Finking

[DL Neil]

On 2/04/19 1:56 PM, Cameron Simpson wrote:
> On 02Apr2019 13:14, DL Neil <PythonList at DancesWithMice.info> wrote:
>> On 2/04/19 11:57 AM, Cameron Simpson wrote:
>>> On 29Mar2019 09:32, DL Neil <PythonList at DancesWithMice.info> wrote:
>>>> Do you 'keep' these, or perhaps next time you need something you've 
>>>> 'done before' do you remember when/where a technique was last 
>>>> used/burrow into 'history'?
>>>> (else, code it from scratch, all over again)
> 
> I didn't say it before, but I _loathe_ the "files with code snippets" 
> approach. Hard to search, hard to reuse, promotes cut/paste coding which 
> means that bugfixes _don't_ make it into all the codebases, just the one 
> you're hacking right now.

...and I didn't say it earlier, but *my* unit-of-code in this 
conversation is a function/module and maybe a whole class. However, 
that's me, and directly relates to my background with "subroutine 
libraries" etc.

I'm not sure how much code can still be defined as a "snippet"? This 
idea - how small is too small, plus how large is too... was intended to 
be a part of the conversation...

As far as having unlabelled units of code, describable only as 'a bunch 
of lines which go-together': I agree, the cost of searching probably 
exceeds most peoples' will to live. I guess it is like many other 
decisions: do I spend time now, in the hopes of saving time later...

Yes, the inevitable evolution of this units of code, creates a 
maintenance issue in-and-of itself (another conversational topic I was 
hoping ppl might explore). Personally I don't trust Git sufficiently, 
and have started adding version numbers to the fileNMs so that whereas 
system-X was written to import widgetv1.py, I can update it to 
widgetv2.py as part of system-Y's dev process, and add the possibly/not 
needed update to system-X's backlog.
(well, best laid plans of mice and men...)

However, in my last greenfield project, everything include-d was taken 
as-is and not 'improved'. That might indicate a degree of 'maturity' and 
RoI. Alternatively, that the project was not particularly demanding (it 
was not!)

The other thought I have, in my learning to embrace OOP later in life, 
is that if the 'snippet' is a class, and extensions are needed, perhaps 
a sub-class will side-step such issues, per your warning?
(more best-laid plans, hopes, dreams...)


>>> I put them into modules for import. I've got a tree of Python modules 
>>> named "cs.this", "cs.that" for various things. Then just import stuff 
>>> like any other library module.
>> Agreed - but Python doesn't (natively) like imports from a 
>> different/parallel/peer-level dir-tree. (see also Apache httpd)
> 
> Well, you have several options:
> - modify $PYTHONPATH
> - make a virtualenv
> - make a symlink; Python imports from the current directory - not my 
>   favourite feature, but very handy in the dev environment

Yes, thanks for this. Ideas tossed-around earlier.

As you will have experienced, until recently virtual, Python 
environments had their disadvantages and quirks. I was one of the 
grumblers.

What came along before the recent Py3.? improvements, was VirtualBox. 
This became my preferred virtual modus-operandi, because it very cleanly 
separates one client/department/project from another (and separates 
non-Python components, eg databases and entire file systems). Thus, also 
Python appv2 from v1! If I've managed to convince the client to also run 
their live-system under VBox, then most of the delivery/memory problems 
discussed 'go away'! Sadly nirvana lies just out-there, somewhere beyond 
tomorrow...
(many of these systems sit on a central 'server' somewhere, likely 
sharing a central DB and/or source-data - which is a bit different from 
desktop systems delivered to nn-hundred users; plus web-server projects 
which are similarly 'centralised')


>>> For personal projects (if they aren't just part of that tree) I just 
>>> need to symlink the tree into the local Python library as "cs".
>> I was doing this.
> This works really well for me. In particular, in my development tree I 
> symlink to my _dev_ "cs" tree, so that if I modify things (fix a bug, 
> add a feature etc) to a cs.* module the change is right there in the 
> repo where I can tidy it up and commit it and possibly publish it.

That ability to update the 'repo' as required by the application-dev 
task, is key.


>> Much of my work is a simple-delivery, ie copying code from my dev m/c 
>> to the client's PC or server - so I don't 'package it up'* because of 
>> the (perceived) effort required cf the one-to-one (or maybe a couple) 
>> machine relationships.
> I'm doing something similar right now, but where I've used a cs.* module 
> in their code it will go through PyPI as part of the prepare-for-deploy 
> process so that they can reproduce the build themselves.

The last step is one I can avoid. In fact, most clients are keen for me 
to do all the computer-technical-stuff, so they can concentrate on their 
research/numbers... ("one-off tasks", discussed earlier)

However, just because it doesn't affect *me* today, still offers a 
learning experience!


> In my case the deploy.sh script makes a directory tree with a copy of 
> the released code (from a repo tag name - "hg archive" for me, there's 
> an equivalent git command).  That tree contains a prep.sh script which 
> runs on the target machine to build the virtualenv and likewise the 
> javascript side which is used for the web front end.
> So deploy for me is:
> - get the code ready (committed, tagged) at some suitable good phase
> - run "./deploy.sh release-tag" which makes a deployable tree
> - rsync onto the target (into a shiny new tree - I'll twiddle a symlink 
>   when it goes "live")
> - run "./prep.sh" in the target, which fetches components etc
> The advantage of the prep.sh step is that (a) the target matches the 
> target host (where that matters, for example the local virtualenv will 
> run off the target host Python and so forth), and _also_ (b) it serves 
> as doco for me on how to build the app: if prep.sh doesn't work, my 
> client doesn't have a working recipe for building their app.
> I still need to add some prechecks to the deploy, like "is the venv 
> requirements file commented to the repo" etc.
>> However, during 'delivery' to another machine, have to remember to 
>> rsync/copy including the symlink, as well as to transfer both dir-trees.
> By making a local virtualenv and getting my modules through PyPI (pip 
> install) this issue is bypassed: there's the client code library 
> (rsynced) and the third party modules fetched via PyPI. (Not just my 
> cs.* modules if any, but also all the other modules required.)

This makes perfect sense. I'll have to sit down and map-out how it would 
work/what would be needed, to (re-)implement a recent project, by way of 
comparison. Thanks!


>> Recently, stopped to organise the code into (more) modules (as also 
>> suggested here) and moved to adding the 'utilities' directories into 
>> PYTHONPATH. Now I have to remember to modify that on the/each target m/c!
> Script it. Include the script in the rsync.

But (devil's advocating), surely the PYTHONPATH approach is more 'pythonic'?


>>> If I get something well enough defined and sufficiently cleaned up 
>>> for use beyond my own code (or just good enough that others might 
>>> want it), up it goes to PyPI so that it can just be "pip install"ed.
>>> So I've got this huge suite of modules with stuff in them, and a 
>>> subset of those modules are published to PyPI for third party reuse.
>> Am dubious that any of the 'little bits' that I have collected are 
>> sufficiently worthy, but that's 'proper' F/LOSSy thinking!
> 
> If you're reusing your little bits then they need organisation into 
> modules so that you can include them with the main code without treading 
> on others' namespaces.

+1


> Publishing to PyPI is a very very optional step; the critical thing is 
> breaking stuff up into modules; rsyncing them is then easy and you have 
> a common codebase which _were formerly_ snippets for reuse.

Plus client concerns!

Might a private GitHub substitute for PyPI?
(at the expense of the convenience of pip...)


> Also, putting them in modules and using them that way forces you to make 
> you snippets reusable instead of cut/patse/adaptable. Win win.

+1


>> *will be interested to hear if you think I should stop 'being lazy' 
>> and invest some time-and-effort into learning 'packaging' options and 
>> do things 'properly'?professionally...
> There's real pain there. The first time I did this (2015?) I basicly 
> spent a whole week right after new year figuring out how to make a 
> package and push it up; the doco was fragmented and confusing, and in 
> some cases out of date, and there is/was more that one way to do things.

...as above. However, I wanted to learn, so I asked...


> These days things are somewhat cleaner: start here:
>   https://packaging.python.org/

Thanks!
(added to my reading list)


>> One of the points which intrigue me is that my colleagues don't keep 
>> snippets/a library, preferring to remember (hah!) when/where they used 
>> particular techniques in the past, and copying/duplicating, to fit the 
>> new system's requirements. Am wondering if this is true beyond our 
>> little band?
> Among random people I suspect it is quite common. In larger teams you 
> accrue a utilities library which contain this stuff.

Thank you! Yes, that simple observation probably explains quite a few of 
the different views with which I have challenged my 
(younger/more-recently trained/OOP-native) colleagues. Group-working cf 
programming as a solitary activity!
(I had puzzled over the observation that this old 'stick-in-the-mud'* 
often comes-out with ideas 'from the past' which are components of OOP, 
eg "re-use"; rather than the OOP-natives already applying them as SOP)

* one of the many, vaguely-insulting, comments sometimes levelled in my 
direction* - especially when I pose such questions... They are non-PC 
but are not taken to be particularly hurtful - part of a humor-filled, 
rough-and-tumble team. Besides, one of 'them' taking the mickey, simply 
invites me to riposte with a comment about 'those of us who have time to 
have grown-up'. Much raucous laughter, rather than serious intent!


>> Yet, here on the list, interest was shown in 'being organised' (even 
>> if few have actually weighed-in)...
> Get your reused code organsed - it is a huge win. At least make a little 
> library of modules in its own namespace (so you can just import it 
> without conflict). What automatic/published/distribution processes you 
> follow after that is your choice. But having that library? Invaluable.

Agreed. Thank you for adding motivation!

-- 
Regards =dn