Call julia from Python: which package?

[Oscar Benjamin]

On Fri, 17 Dec 2021 at 22:40, Chris Angelico <rosuav at gmail.com> wrote:
>
> On Sat, Dec 18, 2021 at 9:24 AM Oscar Benjamin
> <oscar.j.benjamin at gmail.com> wrote:
> > When I timed the result in Julia and in Python I found that the Julia
> > code was slower than the Python code. Of course I don't know how to
> > optimise Julia code so I asked one of my colleagues who does (and who
> > likes to proselytise about Julia). He pointed me to here where the
> > creator of Julia says "BigInts are currently pretty slow in Julia":
> > https://stackoverflow.com/questions/37193586/bigints-seem-slow-in-julia#:~:text=BigInts%20are%20currently%20pretty%20slow,that%20basic%20operations%20are%20fast.
> > I should make clear here that I used the gmpy2 library in Python for
> > the basic integer operations which is a wrapper around the same gmp
> > library that is used by Julia. That means that the basic integer
> > operations were being done by the same gmp library (a C library) in
> > each case. The timing differences between Python and Julia are purely
> > about overhead around usage of the same underlying C library.
>
> Point of note: "Python actually uses a hybrid approach where small
> integer values are represented inline and only when values get too
> large are they represented as BigInts" might be sorta-kinda true for
> Python 2, but it's not true for Python 3. In all versions of CPython
> to date, all integers are objects, they're not "represented inline"
> (there *are* some languages that do this intrinsically, and I think
> that PyPy can sometimes unbox integers, but CPython never will); and
> the performance advantage of Py2's machine-sized integers clearly
> wasn't worth recreating in Py3. (It would be possible to implement it
> in Py3 as an optimization, while still having the same 'int' type for
> both, but nobody's done it.)
>
> So if Python's integers are faster than Julia's, it's not because
> there's any sort of hybrid approach, it's simply because CPython is
> more heavily optimized for that sort of work.
>
> (That said: I have no idea whether a StackOverflow answer from 2016 is
> still applicable.)

To be clear: I wasn't using Python's int type. I used the gmpy2.mpz
type which is precisely the same mpz from the same gmp library that
Julia uses. I'm told by my colleague that Julia has a lot of overhead
when using "heap types" which is possibly the cause of the problem.

The other library I referenced (flint) does have a super-optimised
approach for handling small integers natively as machine types at the
same time as supporting large integers and even includes hand-written
assembly for the most basic operations. That library is significantly
faster than Python's int or gmpy2/gmp.

In any case what it shows is that Julia does not have reduced overhead
*in general* (i.e. for all situations) as compared to Python. I
expected that but I also thought that this particular example would be
within the range of things where Julia out-performed Python and that
turned out not to be the case.

--
Oscar