From Perl to Python: restructuring a HPC workflow

[rusi]

On Mar 27, 4:29 pm, neurino <lelli.l... at googlemail.com> wrote:
> In the need for restructuring our daily workflow, i think it might be a
> good idea to ask the Python community and hopefully initiate a thread
> about pros and cons.
>
> We are a small group of people (approx. 10), working separetely on
> their own projects (each employee manages approx. 2-3 projects). We
> deal with high loads of data everyday.
>
> While the processing is accomplished with fortran and C programs mainly
> on three systems (one cluster, two standalone IBM HPCs,  8852 and p770,
> all managed by a grid-Engine), networking, pre/postprocessing, jobs
> queue administration and numerical analysis have been accomplished with
> Perl.
>
> This workflow has been flawless now for at least 15 years. New
> generations of employees have been given Perl scripts and they
> developed the tools further.
>
> If i think at the actual situation of Perl, i can't see a shiny time
> ahead. Perl 6 is far to be a reliable solution, the CPAN archive is
> slowing down. My idea is to persuade my colleagues to move toward
> Python-based solutions. But our concerns are that, in 3-4 years from
> now, the tools we are going to develop must be still scalable,
> mantainable, portable and of high-performance.
>
> We don't have any solid in-house know-how on Python. We just have to
> start everything from scracth. Where do you see advantages and
> drawbacks in switching from Perl to Python, given the work picture
> above?
>
> Thanks in advance for any opinions you might have.

Switching is always a con; see http://www.joelonsoftware.com/articles/fog0000000069.html
Assuming you have that under your belt
- if python is the way to go, asking on the scipy/numpy and ipython
lists may give you more specific answers.
- And if the 'rewrite-bug' has really got you, remember that if perl
is old, C/Fortran are older.
There are options today for rewriting the whole system, such as
haskell and julia http://julialang.org/

WARNING: If the Spolsky warning above for perl->python is X units,
take it 2X for Haskell and 4X for Julia!