Organizing modules and their code

[transreductionist]

On Friday, February 3, 2023 at 5:31:56 PM UTC-5, Thomas Passin wrote:
> On 2/3/2023 4:18 PM, transreductionist wrote: 
> > Here is the situation. There is a top-level module (see designs below) containing code, that as the name suggests, manages an ETL pipeline. A directory is created called etl_helpers that organizes several modules responsible for making up the pipeline. The discussion concerns the Python language, which supports OOP as well as Structural/Functional approaches to programming. 
> > 
> > I am interested in opinions on which design adheres best to standard architectural practices and the SOLID principles. I understand that this is one of those topics where people may have strong opinions one way or the other. I am interested in those opinions.
> Well, you have pretty well stacked the deck to make DESIGN 1 the 
> obviously preferred choice. I don't think it has much to do with Python 
> per se, or even with OO vs imperative style. 
> 
> As a practical matter, once you got into working with 
> extract_transform_load.py (for the other designs), I would expect that 
> you would start wanting to refactor it and eventually end up more like 
> DESIGN 1. So you might as well start out that way. 
> 
> The reasons are 1) what you said about separation of concerns, 2) a 
> desire to keep each module or file relatively coherent and easy to read, 
> and 3, as you also suggested, making each of them easier to test. 
> Decoupling is important too (one of the SOLID prescriptions), but you 
> can violate that with any architecture if you don't think carefully 
> about what you are doing. 
> 
> On the subject of OO, I think it is a very good approach to think about 
> architecture and design in object terms - meaning conceptual objects 
> from the users' point of view. For example, here you have a pipeline (a 
> metaphorical or userland object). It will need functionality to load, 
> transform, and output data so logically it can be composed of a loader, 
> one or more transformers, and one or more output formatters (more 
> objects). You may also need a scheduler and a configuration manager 
> (more objects). 
> 
> (*Please* let's not have any quibbling about "class" vs "object". We 
> are at a conceptual level here!) 
> 
> When it comes to implementation, you can choose to implement those 
> userland objects with either imperative, OO, or functional techniques, 
> or a mixture.
> > Allow me to give my thoughts. First, I don't think there would be much difference if I was using OOP for the functionality, or using a structural paradigm. A structural paradigm in my opinion, along the lines of Rich Hickey's comments on simple versus complex, would be a simpler implementation. In this case there is no reason to create a construct with state. So let's assume the code is structural and not OOP. 
> > 
> > I would go with Design I. Succinctly stated, Design I supports readability and maintainability at least as well, if not better than the other designs. The goal of the SOLID principles are the creation of mid-level software structures that (Software Architecture: SA Martin). I think Design I best adheres to these principles of: 
> > ---- Tolerate change, 
> > ---- Are easy to understand, and 
> > ---- Are the basis of components that can be used in many software systems. 
> > 
> > I could point to the Single Responsibility Principle which is defined as (SA Martin): a module should be responsible to one, and only one, actor. It should satisfy the Liskov Substitution Principle as well. Further, each module in the etl_helpers directory is at the same level of abstraction. 
> > 
> > I could also mention that as Dijkstra stressed, at every level, from the smallest function to the largest component, software is like a science and, therefore, is driven by falsifiability. Software architects strive to define modules, components, and services that are easily falsifiable (testable). To do so, they employ restrictive disciplines similar to structured programming, 
> > albeit at a much higher level (SA Martin). 
> > 
> > One can point to multiple reasons why Design I might be preferred, but what are the compelling reasons, if there are any, that would suggest another design was superior. 
> > 
> > Finally, let me reference an interesting research paper I read recently that seems to support the other designs as anti-patterns: Architecture_Anti-patterns_Automatically.pdf 
> > 
> > ---- (https://www.cs.drexel.edu/~yfcai/papers/2019/tse2019.pdf) 
> > 
> > SEVERAL DESIGNS FOR COMPARISON 
> > 
> > DESIGN I: 
> > 
> > ---- manage_the_etl_pipeline.py 
> > ---- etl_helpers 
> > ---- extract.py 
> > ---- transform.py 
> > ---- load.py 
> > 
> > Of course one could also 
> > 
> > DESIGN II: 
> > 
> > ---- manage_the_etl_pipeline.py 
> > ---- etl_helpers 
> > ---- extract_transform_load.py 
> > 
> > or probably even: 
> > 
> > DESIGN III: 
> > 
> > ---- manage_the_etl_pipeline.py 
> > ---- extract_transform_load.py


On point that I think is worth making ,and I forgot to make it, is that namespaces are ubiquitous in Python: Built-in, Global,  Function, and Enclosing namespaces, as well as user namespaces, e.g. dictionaries, the SimpleNamespace, and DataClasses to list just a few. Modules ARE namespaces. Namespaces organize programming constructs like classes, functions, variables, etc. into coherent groups of "things". To have a namespace that complects extract constructs with transform constructs, and load constructs in one module seems un-pythonistic.