How to organize Python files in a (relatively) big project

[Claudio Grondi]

If I understand you right you need a concept in which you can put the files
of your project where you want, i.e. restructure the nesting of directories
storing your scripts without the problem of breaking the import statements.
This will give you not a solution to any problem you maybe have with
managing a large package but is a problem in itself, so I see that you have
a huge number of options to choose from:

1. best is you get a better structure of the project in your mind, so you
don't need frequent changes by resorting your modules into a various
directories (flat is better than nested). Set a size limit of a single
Python script it must reach before you split it into two separate files, so
that the number of your files decreases to a number you can handle with ease
(this depends on how good you are at it).

2. create e.g. an HTML file from which you can view, edit, execute the
scripts you have. In this file you create your nested structure and have
this way the overview you need. The Python files remain in one directory
storing the entire project. This allows you to keep together what belongs
together, but to work with it as the single files were spread over many
categories (I mean directories :).

3. create a Python script which automatically creates an HTML file giving
you the overview of all your scripts from walking the directories your
project files are stored below and put in each of your scripts a function
which on call returns the actual full path of passed name of the file to
import and updates the Python PATH, so that it can be found (e.g
updatePATH('nameOfTheModuleToImport')

4. start to choose from many tools for project management which will keep
you busy for a while and away from the actual work on your project ...

None of these above can solve the supposed actual problem, that the project
grows beyond the capacity to manage it in mind. I know about this limitation
and I have seen programmers failing on projects with increasing size of
them. Usually the dependencies between the single modules in large projects
get more and more complicated. Each next added module making the project
size larger must work with the entire already existing ones and after a
given size of project is reached it becomes really hard (and for some just
impossible) to add a new module to it making the necessary modifications to
existing ones in order to gain the new functionality. Such modifications
result usually in breaks in other parts of the project and in my eyes no
management system can really help in anticipating it at that point. Every
programmer seems to have a limit of a project size he can manage and in my
opinion no tool for structuring it or getting a better overview is of real
help here. The core of the problem is lack of skills required for managing
large projects and it seems, that not everyone who managed to learn to
program is able to acquire this skills necessary to increase the size of a
project over the by his intellectual capacity given limits. That is why
programmer need to work in teams and have a very good manager able to
coordinate their efforts in order to succeed in creating large applications.
In really large projects it is sometimes necessary to work half a year or
more full time on the existing library of code before becoming capable of
writing a single line of additional one.

Publish your code, let others work with it, try to get responses on it and
you will probably get much more out of this than by asking for ways of
managing it.

I am curious if others on this newsgroup agree with what I said above, so I
would be glad to hear about it.

Claudio



"TokiDoki" <neuromancer at wanadoo.fr> schrieb im Newsbeitrag
news:dj492l$3hf$1 at apollon.grec.isp.9tel.net...
> Hello there,
>
> I have been programming python for a little while, now. But as I am
> beginning to do more complex stuff, I am running into small organization
> problems.
> It is possible that what I want to obtain is not possible, but I would
> like the advice of more experienced python programmers.
>
> I am writing a relatively complex program in python that has now around
> 40 files.
> At first, I had all of my files in one single directory, but now, with
> the increasing number of files, it is becoming hard to browse my
directory.
> So, I would want to be able to divide the files between 8 directory,
> according to their purpose. The problem is that it breaks the 'import's
>    between my files. And besides,AFAIK, there is no easy way to import a
> file that is not in a subdirectory of the current file (I suppose I
> could adjust the os.path in every file, but that seems not very elegant
> to me).
> I thought about turning the whole into a package. But I have to change
> every 'import module_name' into
> 'from package_name.sub_directory_name import module_name'
> which is a little bit time consuming and not very flexible (if I change
> my mind about the subdirectory a file belongs to, I will have to track
> again every import to correct them)
>
> So, basically, here is the point: is there an 'elegant' way to keep my
> files in separate directory and still be able to import between them
> with simple 'import filename'?
>
> And if not, what would be the standard way to organize numerous python
> files belonging to the same project?
>
> Thank you,
>
> TokiDoki