Call julia from Python: which package?

[Martin Di Paola]

I played with Julia a few months ago. I was doing some data-science 
stuff with Pandas and the performance was terrible so I decided to give 
Julia a try.

My plan was to do the slow part in Julia and call it from Python.  
I tried juliacall (if I don't remember wrong) but I couldn't set up it.

It wasn't as smooth as it was advertised (but hey! you may have a better 
luck than me).

The other thing that I couldn't figure out is *how the data is shared 
between Python and Julia*. Basically numpy/pandas structures cannot be 
used by Julia own libraries as they are, they need to be copied at least 
and this can be a real performance hit.

I desisted the idea but you may still consider this as an real option.  
Just validate how much data you need to share (in my cases where quite 
large datasets, hence the issue).

Having said that, is Julia a real option? May be.

In my case the processing that I needed was quite basic and Julia did 
a really good job.

But I felt that the library is too fragmented. In Python you can relay 
on Pandas/numpy for processing and matplotlib/seaborn for plotting and 
you will 99% covered.

In Julia I need DataFrames.jl, Parquet.jl, CategoricalArrays.jl, 
StatsBase.jl, Statistics.jl and Peaks.jl

And I'm not including any plotting stuff.

This fragmentation is not just "inconvenient" because you need to 
install more packages but it is more difficult to code.

The API is not consistent so you need to be careful and double check 
that what you are calling is really doing what you think.

About the performance, Julia is not magic. It depends on how well it was 
coded each package.

In my case I had a good experience except with Parquet.jl which it 
didn't understand how to handle categories in the dataset and ended up 
loading a lot of duplicated strings and blew up the memory a few times.

I suggest you to write down what you need to speed up and see if it is 
implemented in Julia (do a proof of concept). Only then consider to do 
the switch.

Good luck and share your results!
Martin.


On Fri, Dec 17, 2021 at 07:12:22AM -0800, Dan Stromberg wrote:
>On Fri, Dec 17, 2021 at 7:02 AM Albert-Jan Roskam <sjeik_appie at hotmail.com>
>wrote:
>
>> Hi,
>>
>> I have a Python program that uses Tkinter for its GUI. It's rather slow so
>> I hope to replace many or all of the non-GUI parts by Julia code. Has
>> anybody experience with this? Any packages you can recommend? I found three
>> alternatives:
>>
>> * https://pyjulia.readthedocs.io/en/latest/usage.html#
>> * https://pypi.org/project/juliacall/
>> * https://github.com/JuliaPy/PyCall.jl
>>
>> Thanks in advance!
>>
>
>I have zero Julia experience.
>
>I thought I would share this though:
>https://stromberg.dnsalias.org/~strombrg/speeding-python/
>
>Even if you go the Julia route, it's probably still best to profile your
>Python code to identify the slow ("hot") spots, and rewrite only them.
>-- 
>https://mail.python.org/mailman/listinfo/python-list