Project organization and import

[Martin Unsal]

On Mar 5, 9:15 am, "Chris Mellon" <arka... at gmail.com> wrote:
> That's actually the exact benefit of unit testing, but I don't feel
> that you've actually made a case that this workflow is error prone.
> You often have multiple developers working on the same parts of the
> same module?

Protecting your head is the exact benefit of bike helmets, that
doesn't mean you should bike more more recklessly just because you're
wearing a helmet. :)

Doing text merges is more error prone than not doing them. :)

There are myriad other benefits of breaking up large files into
functional units. Integration history, refactoring, reuse, as I
mentioned. Better clarity of design. Easier communication and
coordination within a team. What's the down side? What's the advantage
of big files with many functional units?

> If you don't do this, you aren't really testing your changes, you're
> testing your reload() machinery.

Only because reload() is hard in Python! ;)

> You seem to have a lot of views about
> what the "Python way" should be and those are at odds with the actual
> way people work with Python. I'm not (necessarily) saying you're
> wrong, but you seem to be coming at this from a confrontational
> standpoint.

When I refer to "Pythonic" all I'm talking about is what I've read
here and observed in other people's code. I'm here looking for more
information about how other people work, to see if there are good
solutions to the problems I see.

However when I talk about what I think is "wrong" with the Pythonic
way, obviously that's just my opinion formed by my own experience.

> Your claim, for example, that the language shouldn't place constraints
> on how you manage your modules is questionable. I think it's more
> likely that you've developed a workflow based around the constraints
> (and abilities) of other languages and you're now expecting Python to
> conform to that instead of its own.

I don't think so; I'm observing things that are common to several
projects in several languages.

> I wonder if you've ever asked yourself why this is the case. I know
> from my own experience why it's done in traditional C++/C environments
> - it's because compiling is slow and breaking things into as many
> files (with as few interdependencies) as possible speeds up the
> compilation process.

I don't think that's actually true. Fewer, bigger compilation units
actually compile faster in C, at least in my experience.

> Absent this need (which doesn't exist in Python),

Python still takes time to load & "precompile". That time is becoming
significant for me even in a modest sized project; I imagine it would
be pretty awful in a multimillion line project.

No matter how fast it is, I'd rather reload one module than exit my
interpreter and reload the entire world.

This is not a problem for Python as scripting language. This is a real
problem for Python as world class application development language.

> In a package __init__, which exists expressly
> for the purpose of exposing it's interior namespaces as a single flat
> one, it makes perfect sense.

OK! That's good info, thanks.

> Nobody I know uses reload() for anything more than trivial "as
> you work" testing in the interpreter. It's not reliable or recommended
> for anything other than that.

That too... although I think that's unfortunate. If reload() were
reliable, would you use it? Do you think it's inherently unreliable,
that is, it couldn't be fixed without fundamentally breaking the
Python language core?

> This is
> still a much faster way than compiling any but the most trivial of
> C/C++ modules.

I'm with you there! I love Python and I'd never go back to C/C++. That
doesn't change my opinion that Python's import mechanism is an
impediment to developing large projects in the language.

> If you don't like working with explicit namespaces, you've probably
> chosen the wrong language.

I never said that. I like foo.Bar(), I just don't like typing
foo.Foo() and bar.Bar(), which is a waste of space; syntax without
semantics.

> I propose that the technique most amenable to source code management
> is for a single file (or RCS level module, if you have a locking RCS)
> to have everything that it makes sense to edit or change for a
> specific feature.

Oh, I agree completely. I think we're using the exact same criterion.
A class is a self-contained feature with a well defined interface,
just what you'd want to put in it's own file. (Obviously there are
trivial classes which don't implement features, and they don't need
their own files.)

> You're also placing far too much emphasis on reload. Focus yourself on
> unit tests and environment scripts instead. These are more reliable
> and easier to validate than reload() in a shell.

I think this is the crux of my frustration. I think reload() is
unreliable and hard to validate because Python's package management is
broken. I appreciate your suggestion of alternatives and I think I
need to come to terms with the fact that reload() is just broken. That
doesn't mean it has to be that way or that Python is blameless in this
problem.

Martin