Organizing modules and their code

[Weatherby,Gerard]

Well, first of all, while there is no doubt as to Dijkstra’s contribution to computer science, I don’t think his description of scientific thought is correct. The acceptance of Einstein’s theory of relativity has nothing to do with internal consistency or how easy or difficult to explain but rather repeatedly experimental results validating it. Or, more precisely, not disproving it. See Feynmann: https://www.youtube.com/watch?v=0KmimDq4cSU


Engineering is simply maximizing the ratio: benefit / cost. Highly recommend To Engineer is Human  by Henry Petroski.

Regarding the initial question: none of the suggested designs would work because they lack __init__.py file.

Once the __init__.py is added, the construct of the import statements within it will determine how the API looks. All three of Design I / Design II and Design III can be implemented with the same API. (I’m pretty sure that’s true. If it’s not, I’d be interested in a counterexample).





From: Python-list <python-list-bounces+gweatherby=uchc.edu at python.org> on behalf of transreductionist <transreductionist at gmail.com>
Date: Saturday, February 4, 2023 at 7:42 PM
To: python-list at python.org <python-list at python.org>
Subject: Re: Organizing modules and their code
*** Attention: This is an external email. Use caution responding, opening attachments or clicking on links. ***

Thank you for all the helpful replies and consideration. I do hope for other opinions

I would rather say it is more like engineering than art. Whether it is a matter of overthinking, or under thinking, is another matter. I enjoyed Dijkstra's letter to his colleagues on the role of scientific thought in computer programming. It is located at:

---- https://urldefense.com/v3/__https://www.cs.utexas.edu/users/EWD/transcriptions/EWD04xx/EWD447.html__;!!Cn_UX_p3!nME8OhiOxAzmzM3jzg6uXZU851dhWWD9JGB8ZRZIzyUzGkmCN-C6SSXrL59eA2KVIh-y-W0VycJSNb8aYcNnc3jd5Pi2fw$<https://urldefense.com/v3/__https:/www.cs.utexas.edu/users/EWD/transcriptions/EWD04xx/EWD447.html__;!!Cn_UX_p3!nME8OhiOxAzmzM3jzg6uXZU851dhWWD9JGB8ZRZIzyUzGkmCN-C6SSXrL59eA2KVIh-y-W0VycJSNb8aYcNnc3jd5Pi2fw$>

It is my academic training in physics that makes me enjoy picking up an idea and examining  it from all sides, and sharing thoughts with friends. Just inquisitive curiosity, and not a homework problem,. Thanks for the great link to the ETL site. That was a good read. A few years back I built a prod ETL application in Golang using gRPC with a multiprocessing pipeline throughout. It handled GB of data and was fast.

This analogy came to me the other day. For me, I would rather walk into a grocery store where the bananas, apples, and oranges are separated in to their own bins, instead of one common crate.


On Friday, February 3, 2023 at 4:18:57 PM UTC-5, transreductionist wrote:
> Here is the situation. There is a top-level module (see designs below) containing code, that as the name suggests, manages an ETL pipeline. A directory is created called etl_helpers that organizes several modules responsible for making up the pipeline. The discussion concerns the Python language, which supports OOP as well as Structural/Functional approaches to programming.
>
> I am interested in opinions on which design adheres best to standard architectural practices and the SOLID principles. I understand that this is one of those topics where people may have strong opinions one way or the other. I am interested in those opinions.
>
> Allow me to give my thoughts. First, I don't think there would be much difference if I was using OOP for the functionality, or using a structural paradigm. A structural paradigm in my opinion, along the lines of Rich Hickey's comments on simple versus complex, would be a simpler implementation. In this case there is no reason to create a construct with state. So let's assume the code is structural and not OOP.
>
> I would go with Design I. Succinctly stated, Design I supports readability and maintainability at least as well, if not better than the other designs. The goal of the SOLID principles are the creation of mid-level software structures that (Software Architecture: SA Martin). I think Design I best adheres to these principles of:
> ---- Tolerate change,
> ---- Are easy to understand, and
> ---- Are the basis of components that can be used in many software systems.
>
> I could point to the Single Responsibility Principle which is defined as (SA Martin): a module should be responsible to one, and only one, actor. It should satisfy the Liskov Substitution Principle as well. Further, each module in the etl_helpers directory is at the same level of abstraction.
>
> I could also mention that as Dijkstra stressed, at every level, from the smallest function to the largest component, software is like a science and, therefore, is driven by falsifiability. Software architects strive to define modules, components, and services that are easily falsifiable (testable). To do so, they employ restrictive disciplines similar to structured programming,
> albeit at a much higher level (SA Martin).
>
> One can point to multiple reasons why Design I might be preferred, but what are the compelling reasons, if there are any, that would suggest another design was superior.
>
> Finally, let me reference an interesting research paper I read recently that seems to support the other designs as anti-patterns: Architecture_Anti-patterns_Automatically.pdf
>
> ---- (https://urldefense.com/v3/__https://www.cs.drexel.edu/*yfcai/papers/2019/tse2019.pdf__;fg!!Cn_UX_p3!nME8OhiOxAzmzM3jzg6uXZU851dhWWD9JGB8ZRZIzyUzGkmCN-C6SSXrL59eA2KVIh-y-W0VycJSNb8aYcNnc3jaresNFQ$ )
>
> SEVERAL DESIGNS FOR COMPARISON
>
> DESIGN I:
>
> ---- manage_the_etl_pipeline.py
> ---- etl_helpers
> ---- extract.py
> ---- transform.py
> ---- load.py
>
> Of course one could also
>
> DESIGN II:
>
> ---- manage_the_etl_pipeline.py
> ---- etl_helpers
> ---- extract_transform_load.py
>
> or probably even:
>
> DESIGN III:
>
> ---- manage_the_etl_pipeline.py
> ---- extract_transform_load.py
--
https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!nME8OhiOxAzmzM3jzg6uXZU851dhWWD9JGB8ZRZIzyUzGkmCN-C6SSXrL59eA2KVIh-y-W0VycJSNb8aYcNnc3hpaHTfyQ$<https://urldefense.com/v3/__https:/mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!nME8OhiOxAzmzM3jzg6uXZU851dhWWD9JGB8ZRZIzyUzGkmCN-C6SSXrL59eA2KVIh-y-W0VycJSNb8aYcNnc3hpaHTfyQ$>