Upgrading Python Breaks Extensions; Fix proposal

[Steve Holden]

"John Roth" <johnroth at ameritech.net> wrote in message
news:toopvfbr30t45d at news.supernews.com...
>
> "Ignacio Vazquez-Abrams" <ignacio at openservices.net> wrote in message
> news:mailman.999031543.25940.python-list at python.org...
> > On 28 Aug 2001, Bryan Olson wrote:
> >
> > > I've done a Google search and found that others have had the same
> > > problem I'm having.  Any upgrade of Python, even from 2.0 to 2.1,
breaks
> > > extension modules.  I'm working on MS-Windows, and the extension
cannot
> > > find python20.dll, which is not surprising because it was replace with
> > > python21.dll.  I suppose I could put back the old DLL, but I don't
know
> > > what that would do, since the rest of the system is using the newer
DLL.
> > >
> > > I don't the workings of Windows DLL's are to blame; Unix shared
> > > libraries would have the same problem.
> >
> > Actually they don't. In the Unix version the .so's link back to the
> > executable; presumably they don't do that in Windows. Does anyone know
> why?
>
> Yes, as a matter of fact. The design goals for Unix and Windows were
totally
> different. Under Unix, each program is its own address space, and .so
> modules are private resources once they're loaded. Under Windows, they
> are shared resources - the same physical memory copy can be used
> simultaniously by multiple users, and appears in multiple address spaces.
> Consequently, it's impossible for them to link back. Also, much of the
> thinking that went into the mechanisms came from CORBA, where they
> wanted to run an application on one system, and link to a module on
> another system.
>
OK, I'll bite. Aren't the *.so files shared libraries, and the *.a files the
ones which are linked in statically. I may not know much about this, but I
*do* remember Sun introducing *.so files as a way of ensuring that large
libraries (SunTools was the worst offender) could be loaded in once by the
kernel and shared between many processes.

When a program is linked against a shared library, a record is kept of the
major and minor version numbers. This work predated CORBA by a good few
years, but as you say the problems are essentially the same.
> > > If the interface to Python extensions actually changed, then certainly
> > > there's no avoiding updating the extensions.  The current system is
much
> > > worse; any upgrade to Python breaks extensions, in many cases
> > > needlessly.
> > >
> > > I don't think that giving different versions of dynamic libraries the
> > > same name, such as "python.dll", is a good idea.  It would just cause
> > > other problems.
> >
> > Not nearly as many problems as 5 different versions of msvcrt40.dll ;)
> >
Of course the *.so files are versioned, which makes it somewhat easier to
avoid DLL hell, because a program will request a library version of "at
least" some value. You can, if I remember correctly, load any shared library
with the right major version number and a minor version  number greater than
or equal to the version the program was linked against. And you can have
several major versions present at the same time. Took Microsoft a while
longer to catch on to versioning (amateurs :-).

> > > So here's my proposal for a solution: don't make the extension load
the
> > > library.  When the interpreter loads the extension, it should give it
a
> > > pointer through which it can find all the library objects. The pointer
> > > could just be the system-dependent library pointer/handle, but more
> > > elegant would be a table of functions, defined the same on all
> > > platforms.
>
> I agree. This would solve the problem rather neatly.
>
> > What should happen is that the extensions under Windows should have the
> same
> > behaviour as extensions on Unix; they should reach back to the
executable
> for
> > run-time bindings. IIRC, the DllMain() function is passed the handle of
> the
> > executable when the DLL is loaded; at that point the bindings should be
> done.
>
> As I mentioned above, it can't be done, at least without implementing a
> completely
> independent loader. I'd much rather see development resources go to
> something
> more useful.
>
> > > I think the change could be made without forcing code changes to
> > > extensions (though of course they'd need one last re-compile).  We
just
> > > have to change the statically-linked stubs library.  We need one
> > > function that always gets linked in, and that the interpreter will
call
> > > with the function-table pointer.  The utility functions that the
library
> > > offers would be stubs that get the function pointers from the table,
and
> > > call through them.
> > >
> > > Of course we should think this through and try to find a way to
> > > negotiate versions, so that extensions last as long as possible.
> > >
> > >
> > > Is this idea PEP-worthy?
>
> Go for it. Anything that saves me from having to distribute a new version
> of extension DLLs when it's technically unnecessary is a great idea.
>
If you can pull this off it would be of value to many people unconnected
with the Python world, too.

Of course, Microsoft's solution is to migrate to .NET, where you can have
umpteen different versions of just about anything.

regards
 Steve
--
http://www.holdenweb.com/