[Tutor] file accessibility across directories?

[dn]

James,

On 1/03/24 06:16, James Hartley wrote:
> This has been a useful discussion (at least for me).  Thank you all for
> your candor.

This is a common concern/problem. However, you outlined it well. Plus, 
@Mat's expansion is a very good discussion of the interplay between 
Python (the interpreter) and one's directory structure. Using "python -m 
..." is often a good way around. Thus, it's worth others reading (and 
learning)...


>  From these comments, it sounds like playing with SYSPATH is a common task.
> I have some database tables defined for SQLAlchemy which I have imported
> into some Pytest tests.  This works fine, but to import the same table
> definitions into scripts found in other directories suffers from the
> problems prompting this thread.  It appears that I will have to modify
> SYSPATH to make one (preferrable) definition work in all locations.

Yes and No!

If you are both customer and supplier, then you probably won't have a 
problem - either at coding-time or later in life.

If however, you distribute the code to others, these aspects take-on a 
whole new (and 'exciting') dimension!

BTW the sub-dir layout is 'standard' for some of the 'installers' used 
for Python distribution.


A first question though: if something has been put into a 'utility' 
directory - which makes it sound as if it will be import-ed from 
different parts of the code (even different projects), then why is it 
being executed as a script? Once the utility routines have been 
exercised and tested (routines either located in the project/test 
sub-dir or in project/utils/tests) their (only) access should be by 
import (from project/src routines)?


For further thought (as if there isn't enough swirling your brain - and 
bearing in-mind that I haven't used SQLAlchemy for a while) is that DRY 
is a concern for schema as much as it is for Python-code. There's no 
point in testing one and running another. Oops! Accordingly, a separate 
file (and/or potentially project sub-dir) which holds the SQL details 
and access methods, (hah! and here it comes...) accessible from both 
code and test modules.



> Thanks again for your comments!
> 
> On Thu, Feb 29, 2024 at 10:57 AM Mats Wichmann <mats at wichmann.us> wrote:
> 
>> On 2/28/24 18:45, dn via Tutor wrote:
>>> On 29/02/24 13:06, Mats Wichmann wrote:
>>>> On 2/28/24 15:39, James Hartley wrote:
>>>>> While in the project directory:
>>>>> $ pwd
>>>>> /home/me/project
>>>>> $ cat src/foo.py
>>>>> class Foo:
>>>>>       pass
>>>>> $ cat utils/baz.py
>>>>> from src.foo import Foo
>>>>> $ python utils/baz.py
>>>>> Traceback (most recent call last):
>>>>>     File "./baz.py", line 1 in <module>
>>>>>       from src.foo import Foo
>>>>> ModuleNotFoundError: No module named 'src'
>>>>
>>>>
>>>> When you run a script, the directory of the script is added to
>>>> sys.path. Not the directory you were in when you ran it.  You can
>>>> print out sys.path in utils/baz.py to convince yourself of this.
>>>> There's no src in that directory (utils).
>>>
>>> Whereas pytest works the other way around: run from the
>>> project-directory because its (internal) 'process of discovery' will
>>> find test* files by searching the entire directory-tree...
>>> (hence the questions in earlier reply)
>>
>>
>> Right... pytest takes a common situation and helps you out, in this case
>> (a) by discovering tests and (b) fiddling paths so those tests can
>> import cleanly. If you don't get that help by using (for example) a test
>> framework, you have to take care of it yourself.
>>
>> This import fails because for an absolute import (src.foo - doesn't
>> start with dots), the starting directory has to be in the search path
>> (sys.path), but it isn't. You can fix that by adding to sys.path in the
>> scripts in utils, or by using PYTHONPATH.  For example, this should work:
>>
>> PYTHONPATH=. python utils/baz.py
>>
>> You would think you could use relative paths. However, if you attempt a
>> relative import like:
>>
>> from ..src.foo import Foo
>>
>> it doesn't work, because the way the script was invoked, Python doesn't
>> think it's part of a package (invoked as a script, the name is
>> "__main__", and contains no dots, thus you must not be in a package),
>> and relative paths are package-hierarchy-relative, not
>> filesystem-relative, so you'll get one of Python's famous error messages:
>>
>> ImportError: attempted relative import with no known parent package
>>
>> However, you could run baz as a module, like:
>>
>> python -m utils.baz
>>
>> because now the name of the running module is "utils.baz", so it looks
>> like a package rooted at a level above "utils", and that directory will
>> be in sys.path, so "from src/foo import Foo" should work.  It just seems
>> a bit ugly to have to invoke a "top level script" as a module to get
>> imports to work.
>>
>> Python imports are... challenging.
>>
>> (yes, I'm assuming answers to the question @dn asked, so this could be
>> wrong if I made the wrong guess)
>>
>> _______________________________________________
>> Tutor maillist  -  Tutor at python.org
>> To unsubscribe or change subscription options:
>> https://mail.python.org/mailman/listinfo/tutor
>>
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor

-- 
Regards,
=dn