[Numpy-discussion] 64bit infrastructure

[David Cournapeau]

On Wed, Aug 22, 2012 at 4:12 PM, Ondřej Čertík <ondrej.certik at gmail.com> wrote:
> On Wed, Aug 22, 2012 at 7:35 AM, Travis Oliphant <travis at continuum.io> wrote:
>> On Aug 22, 2012, at 9:28 AM, David Cournapeau wrote:
>>
>>> On Wed, Aug 22, 2012 at 3:25 PM, Travis Oliphant <travis at continuum.io> wrote:
>>>>
>>>> On Aug 22, 2012, at 3:59 AM, Ralf Gommers wrote:
>>>>
>>>>
>>>>
>>>> On Tue, Aug 21, 2012 at 12:51 AM, Travis Oliphant <travis at continuum.io>
>>>> wrote:
>>>>>
>>>>> I'm actually not sure, why.   I think the issue is making sure that the
>>>>> release manager can actually "build" NumPy without having to buy a
>>>>> particular compiler.
>>>>
>>>>
>>>> That would help, yes. MS Express doesn't work under Wine last time I checked
>>>> by the way.
>>>>
>>>> However, the issue is more than just one license. There's a large number of
>>>> packages that depend on numpy and provide binaries. If they can't make those
>>>> compatible with numpy ones, that's a problem. Users will first install numpy
>>>> 64-bit, and then later find out that part of the scientific Python stack
>>>> isn't available to them anymore.
>>>>
>>>>
>>>>
>>>> As far as I understand, you don't *have* to build all downstream
>>>> dependencies with the same compiler that NumPy was built with unless your
>>>> extension relies on the way C-functions pass structures on the stack (not
>>>> pointers to them, but structures as a whole) or if it relies on the
>>>> representation of FILE*.      At one time all structures were passed as
>>>> pointers specifically for this reason.   The FILE* situation is a problem,
>>>> but most extensions don't use NumPy C-API calls that have a FILE* argument.
>>>
>>> It is much more pervasive than that, unfortunately. And for fortran,
>>> it is much worse, because if we build scipy or numpy with Intel
>>> Fortran, I think we pretty much force everyone to use intel fortran
>>> for *any* binary on top of them.
>>
>> Can you be more specific?   Does the calling convention for C-routines created with Intel Fortran differ so much?
>
>
> I have the same question as Travis. If you are interested about ABI
> for Fortran, I have created this FAQ:
>
> http://fortran90.org/src/faq.html#are-fortran-compilers-abi-compatible
>
> Since NumPy only calls the Fortran routines, but does not expose them,
> then the only issue is how to build NumPy with (let's say) Intel
> Fortran. That's a separate issue.
> Once NumPy is built, then nobody cares, because they only need to
> interface the C routines, if anything at all.
>
> As far as Fortran runtime library goes (which of course is different
> for Intel and gfortran), I am currently not sure whether it is
> possible to mix them, but I think you probably can, if numpy .so is
> using Intel, and my own .so is using gfortran.
>
>
> Finally, if NumPy is build using MSVC, does this force everybody to
> use the same C compiler? I thought that C compilers are ABI
> compatible, at least Intel C and gfortran C are ABI compatible.  Is
> MSVC different?
>
> Btw, to correctly call Fortran from C, one should always be using the
> iso_c_binding module, as explained here:
>
> http://fortran90.org/src/best-practices.html#interfacing-with-c
>
> Then the Fortran code becomes just like any other C library.

It is unfortunately more complicated than that.

 1 regarding fortran runtimes: I have never been able to link a
gfortran object file with Visual Studio linker (link.exe).
 2 mixing runtimes is never a good idea, because it becomes difficult
to avoid passing a pointer from one runtime to the other. Intel
fortran compiler obviously knows how to deal with the C runtime of
Visual Studio, but gfortran doesn't.
 3 gcc and visual studio are ABI compatible in a (very) restricted
sense: they share the same calling convention at least in C, but
that's pretty much it. Because having multiple copies of a runtime is
so common on windows, you cannot easily pass objects between dlls.
Travis mentioned FILE*, but that's also true for pointers returned by
malloc, file descriptors, etc... See this for example:
http://stackoverflow.com/questions/1052491/c-runtime-objects-dll-boundaries

Because of 1, if we have a binary with intel + visual studio, we are
effectively forcing everyone on windows to use intel fortran
compilers. I would rather have the official binaries using open source
toolchains.

cheers,

David