[Tutor] Module Architecture

[Alan Gauld]

On 15/08/2022 17:07, Gordon Stewart wrote:
> I'd like some recommendations on how to design and layout a module.
> There are any number of sources to help with discrete coding problems (on
> this site for instance), but I'm interested in a higher level view on how
> to assemble these discrete components into something easily maintained and
> understood.  

There are many books on software design which is what you
are really asking about. However there are 2 fundamental
concepts that you need to understand and apply for good
design:

1) Cohesion is the amount of integrity that a module or
component has. Can it be used alone or does it need
other modules or components to make it work. In a
Python context that really means components outside
the standard library.

2) Coupling is the amount of interaction with other
components. It's the yin to cohesion's yang. In
particular, you want to avoid direct access to
other modules' internal data. access should be via
an API.

Ideally research these terms on Wikipedia for a more
detailed and precise understanding...

You want to minimise access to the internals of other
modules as much as possible, so minimize coupling and
maximize cohesion. One reason OOP became popular is
that classes do exactly that - they keep all the related
data and functions together in a single component.


> At the moment it is a single module with data definitions grouped towards
> the top, followed by many discrete functions followed by procedural code
> which invokes the functions as necessary.
> It uses functions rather than classes although I'd like to convert lists
> and dictionaries to use @dataclass where appropriate.
> 
> What are your recommendations as to how to structure the refactored module?

I'd suggest that you put your principle data structures
into separate modules along with the primitive functions
used to manage them (ie the CRUD - Create,Read,Update,
Delete functions plus any comparison and output/display
functions)

Then create another module for your interaction with
the external server - treat it as another type of data
store with the same principles.

Finally, create a module (probably the main app module) that
has the higher-level functions that call the lower level
ones..

If you do it well you will have data-oriented modules
that can be reused in other apps in the future and all
the applicatiion specific knowledge is in the app module.

If you use multiple instances of any of the data
structures then those are the ones to turn into
classes. Classes with a single instance are a
bit of a waste of effort in most cases.

> My current thinking is that I'll have three files:
> 
>    - my_data.py which contains all lists, dicts, dataclasses.
>    - my_funcs.py which for all function definitions.  Where possible
>    functions will have calling arguments and will 'return' their output
>    without modifying any external data.

I would strongly recommend not doing this since the
high level of dependencies(ie coupling) will make the
modules completely unusable without also importing
the other module too. To reuse any of the data
structures will require importing all of the others
plus all the functions too. That gets messy fast.
It will also make test/debug more complex(see below).

The idea of returning the data without modifying the
external data is sound functional programming practice.
In this application it may even be possible but in more
general terms it's usually impossible to avoid altering
the data at some point. The trick is to do so via a
function that can strictly control/validate the changes made.

>    - my_procedures.py which contains all the logic to run the functions in
>    my_funcs and update data in my_data.

Yes this is a good idea, it may well be your "main" module.

> If this is the way to go, I have a question about the my_funcs file.  Some
> of the functions are complex and I'd like to test them on a 'stand alone'
> basis.  I could use an ' if __name__ == "__main__ :" '  approach but only
> if the function is in its own file.  Should I split some or all functions
> into their own files or is there a better way to do it?

It is easier to test a module if the data it uses is in
the same module. But testing code is a whole topic by
itself with support modules for doing so. Try reading
up on unit testing and the python unittest module docs
(although many prefer other test tools such as pytest)

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos