From edd at theunixzoo.co.uk  Mon Feb  7 05:15:49 2022
From: edd at theunixzoo.co.uk (Edd Barrett)
Date: Mon, 7 Feb 2022 10:15:49 +0000
Subject: [python-uk] Research Assistant/Associate in PL/VMs,
 King's College London
Message-ID: <YgDxVZiFjTdX7vYO@arrakis.home>

Hi everyone,

The soft-dev team at King's has an open position for a Research Assistant or
Associate: apply on or before Feb 24th.

This position requires a mix of research and engineering to understand how to
migrate programs between different versions of a language, using Python 2 to
Python 3 as a concrete example. We will need to tackle the problem from
different angles, for example augmenting CPython 2.x to dynamically identify
idioms which will cause problems under Python 3.

Excellent programming skills (C and Python are particularly useful, but
excellence can be shown in any other programming language too e.g.
C++/Java/Rust) are required for this role as is the ability to work well with
others. The successful applicant will work in a friendly team with existing
Python and VM skills.

We welcome people with different backgrounds/routes into research: there's no
need to fit a standard profile (e.g. the successful applicant need not have a
PhD). We are also comfortable, and experienced, with long-term largely-remote
work within the UK.

More details can be found here:

  https://jobs.kcl.ac.uk/gb/en/job/036186/Research-Associate

Please contact Laurence Tratt <laurie at tratt.net> with any informal queries
you may have.

Thanks!

-- 
Best Regards
Edd Barrett

https://www.theunixzoo.co.uk

From ema.nicklas at techcitylabs.com  Tue Feb  8 11:35:40 2022
From: ema.nicklas at techcitylabs.com (Ema Nicklas)
Date: Tue, 8 Feb 2022 16:35:40 +0000
Subject: [python-uk] We are looking for highly-skilled Python back-end
 developers / data engineers.
Message-ID: <CAE7N79g52V4UGK=VvAfhGtM692Mp36eLDrEr+kLaNRm6k68v7Q@mail.gmail.com>

The job

Looking to join a friendly, developer-led company? We're Tech City Labs, a
growing data engineering and consulting company.

We're looking for talented data engineers to join our brilliant team of
engaging, experienced developers. Python is our language of choice, and
most of our work is doing data engineering in Python. You'll be working on
client engagements and our own startup-style internal projects, so expect
variety and the chance to work on a range of data-heavy projects.

We have several roles to fill, so we?d love to hear from you whatever your
level of experience.
Who Should Apply?

   - You?ll need 4+ years of experience working with Python and an interest
   in data engineering
   - Communication skills are vital: you'll need to be comfortable - and
   enjoy - working with clients and understanding complex requirements
   - As a remote-first business, excellent writing skills are essential:
   we're in Slack all-day
   - We?re strong believers in TDD (with Pytest) and you should be too
   - Experience working with AWS is a bonus, and being comfortable on the
   command line is a must
   - Must enjoy the challenge of getting stuck into architecture
   - You?ll enjoy working independently but can contribute to the small,
   agile teams in which we operate as needed
   - You?ll be fluent in English, and preferably based in the UK or have
   experience working in the UK


What will your day-to-day look like?

Varied!

If you're working on a client project, you'll be part of a small team,
responding to client requests and implementing them.

If you're running a new project for our data business, you'll be building
data systems from scratch, working closely with our Founder and assisting
with API and data pipeline development where required.

Some examples of recent projects include:

   - Building data transformations using dbt (http://getdbt.com)
   - Creating an API using the Serverless framework (
   https://www.serverless.com/) and deploying to AWS
   - Setting up the infrastructure to track changes in a database and
   deliver via web sockets in real-time
   - Writing data pipelines parse GIS data into PostGIS (
   https://postgis.net/)


Technologies we use

We understand that good developers can pick up any well-designed technology
quickly, so we don?t require specific experience other than Python.

If you join us, you can expect to be working with some of the following:

   - Postgres
   - MySQL
   - Airflow
   - dbt
   - A whole range of AWS services
   - Serverless
   - Flask
   - FastAPI

Who are Tech City Labs?

   - We?re a developer-led, data engineering and consulting company
   - We?re a friendly, collaborative and supportive team with a culture of
   development excellence
   - We value the differences our people bring
   - We are committed to remote working - we do get together at least
   quarterly so you should be prepared to travel occasionally to Reading or
   other exotic destinations (all expenses covered)
   - We're flexible, but you must be able to (mostly) keep UK working hours
   and attend morning standups
   - We practice and preach sound development practices. We hope you'll
   learn and grow as a developer working with us (and we'll learn from you,
   too)
   - We have a no-drama office policy. We value and cultivate enjoyable
   working relationships among team members.
   - If you apply, we guarantee that we will give you a response, yes or
   no; absolutely no ghosting!


What We Offer:

   - Flexible benefits - one size does not fit all. Prefer a beer
   membership to a gym card? That?s fine with us
   - Need somewhere to work that gets you out of your house? We?ll cover a
   co-working membership at a space convenient for you
   - Our quarterly meetups are all-expenses-paid and are enjoyable - really
   - Salary accordingly to experience, plus an annual bonus
   - 25 days holiday
   - A no-nonesense environment: we appreciate our team and do our best to
   show it.
   - Salary range: ?40-?60k per annum.


Application Process

1. We carefully review every CV we receive

2. For those that we shortlist, we'll ask you to do a short (less than five
minutes) VideoAsk interview that is about personality and what makes you
tick, not what?s on your CV

3. If we think your style will work with our team, and you could be happy
with us, we'll then have a 30min 'get to know each other? chat

4. Finally we may ask you to do a short coding test (1-2hr) in your own
time, which will be followed by an in-depth technical interview (45min) to
discuss the way you approached the coding task (if you did one), your
experience and your hard skills

5. If everyone is happy, we?ll discuss package details and getting you
onboard. If not, we?ll be honest about why we didn?t proceed to hire.

We respond to every application.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/python-uk/attachments/20220208/a1aa36d0/attachment-0001.html>

From narshad.380 at gmail.com  Wed Feb  9 03:42:45 2022
From: narshad.380 at gmail.com (Arshad Noman)
Date: Wed, 9 Feb 2022 13:42:45 +0500
Subject: [python-uk] PermissionError: [Errno 13] Permission denied:
 'Abc.xlsx'
In-Reply-To: <CA+0oV2gjf1crAZ1roxqbw3n5Bz6yYX-mD4QR=UrHC8_d4Sk0+Q@mail.gmail.com>
References: <CA+0oV2i+7L3B-Q1bKppCR5tDHkKL6v9iCw4PXieFR1=oR8RsGQ@mail.gmail.com>
 <CA+0oV2gjf1crAZ1roxqbw3n5Bz6yYX-mD4QR=UrHC8_d4Sk0+Q@mail.gmail.com>
Message-ID: <CA+0oV2jvW10yViY6Qhc910rvCF0FFxhqSuTMyt+Y6BwCFx2L4Q@mail.gmail.com>

When I enter data using Tkinter form in an Excel file when the excel file
is closed there is no error but when I enter data using Tkinter form when
the excel is already open following error comes:



Exception in Tkinter callback
Traceback (most recent call last):
File "C:\Users\Dani Brothers\Anaconda3\lib\tkinter\__init__.py", line 1705,
in __call__
return self.func(*args)
File "D:/Python/Book Bank/New folder/PyCharm/Final/Excel.py", line 61, in
SaveBook
workbook.save(filename="BookBank.xlsx")
File "C:\Users\Dani
Brothers\Anaconda3\lib\site-packages\openpyxl\workbook\workbook.py", line
392, in save
save_workbook(self, filename)
File "C:\Users\Dani
Brothers\Anaconda3\lib\site-packages\openpyxl\writer\excel.py", line 291,
in save_workbook
archive = ZipFile(filename, 'w', ZIP_DEFLATED, allowZip64=True)
File "C:\Users\Dani Brothers\Anaconda3\lib\zipfile.py", line 1207, in
__init__
self.fp = io.open(file, filemode)
PermissionError: [Errno 13] Permission denied: 'Abc.xlsx'



What to do to correct this error? I have already searched on google search
many times but no solution was found.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/python-uk/attachments/20220209/026a196e/attachment.html>

From walker_s at hotmail.co.uk  Wed Feb  9 04:58:30 2022
From: walker_s at hotmail.co.uk (SW)
Date: Wed, 9 Feb 2022 10:58:30 +0100
Subject: [python-uk] PermissionError: [Errno 13] Permission denied:
 'Abc.xlsx'
In-Reply-To: <CA+0oV2jvW10yViY6Qhc910rvCF0FFxhqSuTMyt+Y6BwCFx2L4Q@mail.gmail.com>
References: <CA+0oV2i+7L3B-Q1bKppCR5tDHkKL6v9iCw4PXieFR1=oR8RsGQ@mail.gmail.com>
 <CA+0oV2gjf1crAZ1roxqbw3n5Bz6yYX-mD4QR=UrHC8_d4Sk0+Q@mail.gmail.com>
 <CA+0oV2jvW10yViY6Qhc910rvCF0FFxhqSuTMyt+Y6BwCFx2L4Q@mail.gmail.com>
Message-ID: <HE1P190MB014057A3F3878818E6303CEBB82E9@HE1P190MB0140.EURP190.PROD.OUTLOOK.COM>

Responding to list in case this helps someone else as well:

When you open a file in Window ("when the excel is already open 
following error") there will generally (or maybe always, I don't recall 
precisely) be an exclusive lock taken on the file.
This means that while another application has a file open you'll get 
this sort of error when you try to access the file in any way.

Therefore, the solution is to make sure not to have the file open in 
another application while trying to run your script on it.

For more information you'll want to search for things like "Windows file 
locking".

Thanks,
S

On 09/02/2022 09:42, Arshad Noman wrote:
> When I enter data using Tkinter form in an Excel file when the excel 
> file is closed there is no error but when I enter data using Tkinter 
> form when the excel is already open following error comes:
>
>
>
> Exception in Tkinter callback
> Traceback (most recent call last):
> File "C:\Users\Dani Brothers\Anaconda3\lib\tkinter\__init__.py", line 
> 1705, in __call__
> return self.func(*args)
> File "D:/Python/Book Bank/New folder/PyCharm/Final/Excel.py", line 61, 
> in SaveBook
> workbook.save(filename="BookBank.xlsx")
> File "C:\Users\Dani 
> Brothers\Anaconda3\lib\site-packages\openpyxl\workbook\workbook.py", 
> line 392, in save
> save_workbook(self, filename)
> File "C:\Users\Dani 
> Brothers\Anaconda3\lib\site-packages\openpyxl\writer\excel.py", line 
> 291, in save_workbook
> archive = ZipFile(filename, 'w', ZIP_DEFLATED, allowZip64=True)
> File "C:\Users\Dani Brothers\Anaconda3\lib\zipfile.py", line 1207, in 
> __init__
> self.fp = io.open(file, filemode)
> PermissionError: [Errno 13] Permission denied: 'Abc.xlsx'
>
>
>
> What to do to correct this error? I have already searched on google 
> search many times but no solution was found.
>
> _______________________________________________
> python-uk mailing list
> python-uk at python.org
> https://mail.python.org/mailman/listinfo/python-uk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/python-uk/attachments/20220209/6ddcc744/attachment.html>

From tartley at tartley.com  Wed Feb  9 07:55:14 2022
From: tartley at tartley.com (Jonathan Hartley)
Date: Wed, 09 Feb 2022 06:55:14 -0600
Subject: [python-uk] PermissionError: [Errno 13] Permission denied:
 'Abc.xlsx'
In-Reply-To: <HE1P190MB014057A3F3878818E6303CEBB82E9@HE1P190MB0140.EURP190.PROD.OUTLOOK.COM>
References: <CA+0oV2i+7L3B-Q1bKppCR5tDHkKL6v9iCw4PXieFR1=oR8RsGQ@mail.gmail.com>
 <CA+0oV2gjf1crAZ1roxqbw3n5Bz6yYX-mD4QR=UrHC8_d4Sk0+Q@mail.gmail.com>
 <CA+0oV2jvW10yViY6Qhc910rvCF0FFxhqSuTMyt+Y6BwCFx2L4Q@mail.gmail.com>
 <HE1P190MB014057A3F3878818E6303CEBB82E9@HE1P190MB0140.EURP190.PROD.OUTLOOK.COM>
Message-ID: <a4f75a20-3049-4938-a450-f8fcfcdc6c3b@www.fastmail.com>

You might consider the idea that a script which modifies the spreadsheet should read from one file, and write to a different one. This might not fit your scenario, but if it does, then the write will never encounter the locking exception.

On Wed, Feb 9, 2022, at 03:58, SW wrote:
> Responding to list in case this helps someone else as well:
> 
> When you open a file in Window ("when the excel is already open following error") there will generally (or maybe always, I don't recall precisely) be an exclusive lock taken on the file.
> This means that while another application has a file open you'll get this sort of error when you try to access the file in any way.
> 
> Therefore, the solution is to make sure not to have the file open in another application while trying to run your script on it.
> 
> For more information you'll want to search for things like "Windows file locking".
> 
> Thanks,
> S
> On 09/02/2022 09:42, Arshad Noman wrote:
>> When I enter data using Tkinter form in an Excel file when the excel file is closed there is no error but when I enter data using Tkinter form when the excel is already open following error comes:
>> 
>> 
>> 
>> Exception in Tkinter callback
>> Traceback (most recent call last):
>> File "C:\Users\Dani Brothers\Anaconda3\lib\tkinter\__init__.py", line 1705, in __call__
>> return self.func(*args)
>> File "D:/Python/Book Bank/New folder/PyCharm/Final/Excel.py", line 61, in SaveBook
>> workbook.save(filename="BookBank.xlsx")
>> File "C:\Users\Dani Brothers\Anaconda3\lib\site-packages\openpyxl\workbook\workbook.py", line 392, in save
>> save_workbook(self, filename)
>> File "C:\Users\Dani Brothers\Anaconda3\lib\site-packages\openpyxl\writer\excel.py", line 291, in save_workbook
>> archive = ZipFile(filename, 'w', ZIP_DEFLATED, allowZip64=True)
>> File "C:\Users\Dani Brothers\Anaconda3\lib\zipfile.py", line 1207, in __init__
>> self.fp = io.open(file, filemode)
>> PermissionError: [Errno 13] Permission denied: 'Abc.xlsx'
>> 
>> 
>> 
>> What to do to correct this error? I have already searched on google search many times but no solution was found.
>> 
>> _______________________________________________
>> python-uk mailing list
>> python-uk at python.org
>> https://mail.python.org/mailman/listinfo/python-uk
>> 
> 
> _______________________________________________
> python-uk mailing list
> python-uk at python.org
> https://mail.python.org/mailman/listinfo/python-uk
> 

--
Jonathan Hartley  USA, Central(UTC-6)
twitter:@tartley  https://tartley.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/python-uk/attachments/20220209/776990e1/attachment-0001.html>

From daniele at vurt.org  Sat Feb 12 14:13:40 2022
From: daniele at vurt.org (Daniele Procida)
Date: Sat, 12 Feb 2022 19:13:40 +0000
Subject: [python-uk] Documentation workshops for researchers and research
 software engineers
Message-ID: <617084CA-9F67-42EF-874B-24F8B68B0A18@vurt.org>

Greetings Pythonistas.

As part of my fellowship of the Software Sustainability Institute, I am running a series of software documentation workshops.

The aim is to help improve research software and the reproducibility of research by introducing better, easier ways to document software, and also for me to learn how the work I?ve been doing in documentation in recent years can better serve the needs of researchers.

All the details:

https://docs.google.com/document/d/e/2PACX-1vQEZf-uZkDc1dOAWEUMWN1TaWBIQdED1bCcYGrOL4RP1HTw_juX7DgcG63LGRx3aSkTAtBmKtGRCc_r/pub

But don?t hesitate to get in touch with any questions.

Daniele

From robin at rtwilson.com  Thu Feb 17 09:34:28 2022
From: robin at rtwilson.com (Robin Wilson)
Date: Thu, 17 Feb 2022 14:34:28 +0000
Subject: [python-uk] Jobs: Python Full Stack Developer & Cloud Engineer with
 Anglo American
Message-ID: <etPan.620e5cf9.54dce0c.e3a9@rtwilson.com>

We are a new team at Anglo American (a large mining and minerals company), working on image data at scale. There are several problems with how things are currently, including: storage limited to local hard drives, compute limited to desktops, data silos and difficulties in finding and sharing image data. The solution that you will help build will use cloud and web technologies to store, search, visualize and run compute on global scale image archives (Terabyte to Petabyte).

Both jobs will be using Python extensively, and familiarity with the PyData stack (particularly XArray and Dask) and Python web development tools (eg. FastAPI) will be extremely useful. Experience with spatial data is also a massive plus.

Both jobs are remote, within UTC +/- 2 - although UK-based is ideal. Both will be contractors, at a competitive day rate.

Full stack developer:
Your focus will be full stack web development, building back-end APIs and front-end interfaces for users to easily access these large image archives. We will be building on open tools and standards written in Python (such as the Spatio-Temporal Asset Catalog, Titiler and Dask), and you will be extending and modifying these, as well as writing new serverless APIs in Python. Front-end development will be within the context of Anglo American frameworks, primarily using React, and will involve map visualization tools such as Leaflet and/or OpenLayers.

Full details:?https://drive.google.com/file/d/1nBnwC9JoVHmbNbYKtlpG8doRmA3Sg1JR/view

Cloud Engineer:
Your focus will be on using cloud technology to scale up capabilities (e.g. storage, search and compute). You will be building and orchestrating cloud services including serverless APIs, large databases, storage accounts, Kubernetes clusters and more, all using an Infrastructure as Code approach. We work on the Microsoft Azure cloud platform, and are building on top of open-source tools and open-standards such as the Spatio-Temporal Asset Catalog, webmap tiling services such as Titiler, and the Dask parallel processing framework.

Full details:?https://drive.google.com/file/d/1IXvmjrPQBfXG5RqeUQUhJh6JQm31YH8r/view

Thanks,

Robin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/python-uk/attachments/20220217/6e402d0d/attachment.html>

From jamesbtobin at gmail.com  Sun Feb 20 11:05:37 2022
From: jamesbtobin at gmail.com (James Tobin)
Date: Sun, 20 Feb 2022 16:05:37 +0000
Subject: [python-uk] JOB | Software Development Manager (Edinburgh, UK)
Message-ID: <CAMyPCTQSM1J_BC=zxa3YfNLtHyh0_v-rNvzUGVbXtWd_Kuutjw@mail.gmail.com>

Hello, I'm working with a software house (they build applications for
their financial clients) - that wants to establish an office in
Edinburgh and are therefore looking to hire a development manager -
someone to open and manage the office as well as code (Core Java,
Javascript, Python, React etc).  Consequently, I had hoped that some
members of this list may like to discuss with me further.  I can be
reached using "jamesbtobin (at) gmail (dot) com".  Kind regards, James

From jfine2358 at gmail.com  Thu Feb 24 09:54:01 2022
From: jfine2358 at gmail.com (Jonathan Fine)
Date: Thu, 24 Feb 2022 14:54:01 +0000
Subject: [python-uk] TeX Hour tonight: Better Technical Documentation: 18:30
 UTC time
In-Reply-To: <CALD=Yf-9wapt4jKZecGmkbNn5cmb3kd9A=jjBBur2uRt3HAGhg@mail.gmail.com>
References: <CALD=Yf-9wapt4jKZecGmkbNn5cmb3kd9A=jjBBur2uRt3HAGhg@mail.gmail.com>
Message-ID: <CALD=Yf_6grYEN-5YThzqP3E48O-V1PRkoROVw1AK1N2c8r0imQ@mail.gmail.com>

Hi

Tonight's TeX hour is from 6:30 to 7:30pm UK (and UTC) time. The UK time
now: https://time.is/UK.
The zoom URL is
https://us02web.zoom.us/j/78551255396?pwd=cHdJN0pTTXRlRCtSd1lCTHpuWmNIUT09

Last week I attended a "needs discovery" meeting organised by Daniele
Procida, who I've met several times at UK Python conferences. Daniele is
now one of several Directors of Engineering at Canonical, the  developers
of Ubuntu Linux, where he has special responsibility for improving both
documentation and the process by which it is created.

Daniele is also a current Fellow of the Software Sustainability Institute.
The meeting last week was to discover needs for a workshop he's holding
next month, to help Research Software Engineers improve their
documentation. There were over 20 people there.

I hope that at tonight's TeX Hour we will share our experience of writing,
rendering and using technical documentation, particularly  for software.
Daniele's approach is focussed on a quadrant of user needs.

The quadrant is

Top left: Tutorials (focussed on learning).
Top right: How-to guides (focussed on tasks).
Bottom left:  Explanation (focussed on understanding).
Bottom right: Reference (focussed on information).

The up-down axis is practice and theory. The left-right axis is study and
work.

Interested? Reflect on your own documentation experience, and visit:

https://www.software.ac.uk/about/fellows/daniele-procida
https://diataxis.fr/
https://mail.python.org/pipermail/python-uk/2022-February/004828.html

Tonight's TeX hour is from 6:30 to 7:30pm UK (and UTC) time. The UK time
now: https://time.is/UK.
The zoom URL is
https://us02web.zoom.us/j/78551255396?pwd=cHdJN0pTTXRlRCtSd1lCTHpuWmNIUT09

wishing you beautiful and effective documents

Jonathan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/python-uk/attachments/20220224/6aa24757/attachment.html>

From abdelkader.belahcene at enst.dz  Fri Feb 25 04:03:16 2022
From: abdelkader.belahcene at enst.dz (BELAHCENE Abdelkader)
Date: Fri, 25 Feb 2022 10:03:16 +0100
Subject: [python-uk] C is it faster than numpy
Message-ID: <CABcwF5m6u-NWhtnEwG+iopLFPZBJopO7hLP+f3uhwUdzn93wNw@mail.gmail.com>

Hi,
a lot of people think that C (or C++) is faster than python, yes I agree,
but I think that's not the case with numpy, I believe numpy is faster than
C, at least in some cases.


*Is there another explanation ?Or where can find  a doc speaking  about the
subject?*Thanks a lot
Regards
Numpy implements vectorization for arrays, or I'm wrong. Anyway here is an
example Let's look at the following case:
Here is the result on my laptop i3:

Labs$ *python3 tempsExe.py  50000*
  sum with Python: 1250025000 and NumPy 1250025000
      time used Python Sum: * 37.28 sec *
      time used  Numpy Sum:  *1.85 sec*

Labs$ *./tt    50000 *


*   CPU  time :7.521730    The value : 1250025000 *
--------------------------------------------

This is the Python3 program :

import timeit as it
import numpy as np
import sys
try :
n=eval(sys.argv[1])
except:
print ("needs integer as argument") ; exit()

a=range(1,n+1)
b=np.array(a)
def func1():     return sum(a)
def func2(): return np.sum(b)

print(f"sum with Python: {func1()} and NumPy {func2()} ")
tm1=it.timeit(stmt=func1, number=n)
print(f"time used Python Sum: {round(tm1,2)} sec")
tm2=it.timeit(stmt=func2, number=n)
print(f"time used  Numpy Sum: {round(tm2,2)} sec")

and Here the C program:
#include <time.h>
#include <stdio.h>
#include <stdlib.h>
long func1(int n){
         long  r=0;
        for (int  i=1; i<= n;i++) r+= i;
         return r;
}
int main(int argc, char* argv[]){
         clock_t c0, c1;
        long v,count; int n;
       if ( argc < 2) {
              printf("Please give an argument");
             return -1;
      }
    n=atoi(argv[1]);
    c0 = clock();
     *for (int j=0;j < n;j++) v=func1(n);*
     c1 = clock();
     printf ("\tCPU  time :%.2f sec", (float)(c1 - c0)/CLOCKS_PER_SEC);
     printf("\n\tThe value : %ld\n",  v);
}
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/python-uk/attachments/20220225/36f3e7c2/attachment.html>

From giorgio.zoppi at gmail.com  Fri Feb 25 04:41:50 2022
From: giorgio.zoppi at gmail.com (Giorgio Zoppi)
Date: Fri, 25 Feb 2022 09:41:50 +0000
Subject: [python-uk] C is it faster than numpy
In-Reply-To: <CABcwF5m6u-NWhtnEwG+iopLFPZBJopO7hLP+f3uhwUdzn93wNw@mail.gmail.com>
References: <CABcwF5m6u-NWhtnEwG+iopLFPZBJopO7hLP+f3uhwUdzn93wNw@mail.gmail.com>
Message-ID: <CAHW5HkoQ=9Bvcm7L3CejpMedFezUf+ni+7NDms_RLeyq-VSnzg@mail.gmail.com>

Well,
numpy is written in C :) Maybe your C is not the numpy equivalent?
Best Regards,
Giorgio

Il giorno ven 25 feb 2022 alle ore 09:03 BELAHCENE Abdelkader <
abdelkader.belahcene at enst.dz> ha scritto:

> Hi,
> a lot of people think that C (or C++) is faster than python, yes I agree,
> but I think that's not the case with numpy, I believe numpy is faster than
> C, at least in some cases.
>
>
> *Is there another explanation ?Or where can find  a doc speaking  about
> the subject?*Thanks a lot
> Regards
> Numpy implements vectorization for arrays, or I'm wrong. Anyway here is an
> example Let's look at the following case:
> Here is the result on my laptop i3:
>
> Labs$ *python3 tempsExe.py  50000*
>   sum with Python: 1250025000 and NumPy 1250025000
>       time used Python Sum: * 37.28 sec *
>       time used  Numpy Sum:  *1.85 sec*
>
> Labs$ *./tt    50000 *
>
>
> *   CPU  time :7.521730    The value : 1250025000 *
> --------------------------------------------
>
> This is the Python3 program :
>
> import timeit as it
> import numpy as np
> import sys
> try :
> n=eval(sys.argv[1])
> except:
> print ("needs integer as argument") ; exit()
>
> a=range(1,n+1)
> b=np.array(a)
> def func1():     return sum(a)
> def func2(): return np.sum(b)
>
> print(f"sum with Python: {func1()} and NumPy {func2()} ")
> tm1=it.timeit(stmt=func1, number=n)
> print(f"time used Python Sum: {round(tm1,2)} sec")
> tm2=it.timeit(stmt=func2, number=n)
> print(f"time used  Numpy Sum: {round(tm2,2)} sec")
>
> and Here the C program:
> #include <time.h>
> #include <stdio.h>
> #include <stdlib.h>
> long func1(int n){
>          long  r=0;
>         for (int  i=1; i<= n;i++) r+= i;
>          return r;
> }
> int main(int argc, char* argv[]){
>          clock_t c0, c1;
>         long v,count; int n;
>        if ( argc < 2) {
>               printf("Please give an argument");
>              return -1;
>       }
>     n=atoi(argv[1]);
>     c0 = clock();
>      *for (int j=0;j < n;j++) v=func1(n);*
>      c1 = clock();
>      printf ("\tCPU  time :%.2f sec", (float)(c1 - c0)/CLOCKS_PER_SEC);
>      printf("\n\tThe value : %ld\n",  v);
> }
> _______________________________________________
> python-uk mailing list
> python-uk at python.org
> https://mail.python.org/mailman/listinfo/python-uk
>


-- 
Life is a chess game - Anonymous.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/python-uk/attachments/20220225/696f2e15/attachment.html>

From walker_s at hotmail.co.uk  Fri Feb 25 04:53:20 2022
From: walker_s at hotmail.co.uk (SW)
Date: Fri, 25 Feb 2022 10:53:20 +0100
Subject: [python-uk] C is it faster than numpy
In-Reply-To: <CAHW5HkoQ=9Bvcm7L3CejpMedFezUf+ni+7NDms_RLeyq-VSnzg@mail.gmail.com>
References: <CABcwF5m6u-NWhtnEwG+iopLFPZBJopO7hLP+f3uhwUdzn93wNw@mail.gmail.com>
 <CAHW5HkoQ=9Bvcm7L3CejpMedFezUf+ni+7NDms_RLeyq-VSnzg@mail.gmail.com>
Message-ID: <AS8P193MB2013EBFD502337BA1DB2FD31B83E9@AS8P193MB2013.EURP193.PROD.OUTLOOK.COM>

I think it uses fortran (or used to?) but yes, still close to the metal.

Thanks,
S

On 25/02/2022 10:41, Giorgio Zoppi wrote:
> Well,
> numpy is written in C :) Maybe your C is not the numpy equivalent?
> Best Regards,
> Giorgio
>
> Il giorno ven 25 feb 2022 alle ore 09:03 BELAHCENE Abdelkader 
> <abdelkader.belahcene at enst.dz> ha scritto:
>
>     Hi,
>     a lot of people think that C (or C++) is faster than python, yes I
>     agree, but I think that's not the case with numpy, I believe numpy
>     is faster than C, at least in some cases.
>     *Is there another explanation ?
>     Or where can find ?a doc speaking ?about the subject?
>     *Thanks a lot
>     Regards
>     Numpy implements vectorization for arrays, or I'm wrong. Anyway
>     here is an example Let's look at the following case:
>     Here is the result on my laptop i3:
>
>     Labs$ *python3 tempsExe.py? 50000*
>     ? sum with Python: 1250025000 and NumPy 1250025000
>     ? ? ? time used Python Sum: *37.28 sec *
>     ? ? ? time used ?Numpy Sum: *1.85 sec*
>
>     Labs$ *./tt ?? 50000 *
>     *? CPU ?time :7.521730
>     ? ? The value : 1250025000
>     *--------------------------------------------
>
>     This is the Python3 program :
>
>     import timeit as it
>     import numpy as np
>     import sys
>     try :
>     n=eval(sys.argv[1])
>     except:
>     print ("needs integer as argument") ; exit()
>
>     a=range(1,n+1)
>     b=np.array(a)
>     def func1(): ? ? return sum(a)
>     def func2(): return np.sum(b)
>
>     print(f"sum with Python: {func1()} and NumPy {func2()} ")
>     tm1=it.timeit(stmt=func1, number=n)
>     print(f"time used Python Sum: {round(tm1,2)} sec")
>     tm2=it.timeit(stmt=func2, number=n)
>     print(f"time used ?Numpy Sum: {round(tm2,2)} sec")
>
>     and Here the C program:
>     #include <time.h>
>     #include <stdio.h>
>     #include <stdlib.h>
>     long func1(int n){
>     ???????? long ?r=0;
>     ??????? for (int ?i=1; i<= n;i++) r+= i;
>     ???????? return r;
>     }
>     int main(int argc, char* argv[]){
>     ???????? clock_t c0, c1;
>     ??????? long v,count; int n;
>     ?????? if ( argc < 2) {
>     ????????????? printf("Please give an argument");
>     ???????????? return -1;
>     ????? }
>     ??? n=atoi(argv[1]);
>     ??? c0 = clock();
>     *for (int j=0;j < n;j++) v=func1(n);*
>     ???? c1 = clock();
>     ???? printf ("\tCPU ?time :%.2f sec", (float)(c1 -
>     c0)/CLOCKS_PER_SEC);
>     ???? printf("\n\tThe value : %ld\n", ?v);
>     }
>     _______________________________________________
>     python-uk mailing list
>     python-uk at python.org
>     https://mail.python.org/mailman/listinfo/python-uk
>
>
>
> -- 
> Life is a chess game - Anonymous.
>
> _______________________________________________
> python-uk mailing list
> python-uk at python.org
> https://mail.python.org/mailman/listinfo/python-uk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/python-uk/attachments/20220225/0b21ddef/attachment-0001.html>

From visgean at gmail.com  Fri Feb 25 04:57:05 2022
From: visgean at gmail.com (Martin)
Date: Fri, 25 Feb 2022 09:57:05 +0000
Subject: [python-uk] C is it faster than numpy
In-Reply-To: <AS8P193MB2013EBFD502337BA1DB2FD31B83E9@AS8P193MB2013.EURP193.PROD.OUTLOOK.COM>
References: <CABcwF5m6u-NWhtnEwG+iopLFPZBJopO7hLP+f3uhwUdzn93wNw@mail.gmail.com>
 <CAHW5HkoQ=9Bvcm7L3CejpMedFezUf+ni+7NDms_RLeyq-VSnzg@mail.gmail.com>
 <AS8P193MB2013EBFD502337BA1DB2FD31B83E9@AS8P193MB2013.EURP193.PROD.OUTLOOK.COM>
Message-ID: <CAPy44ZUgQ=0__siSk9xhnbQy5APQJy3yXJUFr=CzO20QnXJPPA@mail.gmail.com>

Yeah it uses BLAS, you can read about it here
https://markus-beuckelmann.de/blog/boosting-numpy-blas.html

On Fri, 25 Feb 2022, 09:54 SW, <walker_s at hotmail.co.uk> wrote:

> I think it uses fortran (or used to?) but yes, still close to the metal.
>
> Thanks,
> S
>
> On 25/02/2022 10:41, Giorgio Zoppi wrote:
>
> Well,
> numpy is written in C :) Maybe your C is not the numpy equivalent?
> Best Regards,
> Giorgio
>
> Il giorno ven 25 feb 2022 alle ore 09:03 BELAHCENE Abdelkader <
> abdelkader.belahcene at enst.dz> ha scritto:
>
>> Hi,
>> a lot of people think that C (or C++) is faster than python, yes I agree,
>> but I think that's not the case with numpy, I believe numpy is faster than
>> C, at least in some cases.
>>
>>
>> *Is there another explanation ? Or where can find  a doc speaking  about
>> the subject? *Thanks a lot
>> Regards
>> Numpy implements vectorization for arrays, or I'm wrong. Anyway here is
>> an example Let's look at the following case:
>> Here is the result on my laptop i3:
>>
>> Labs$ *python3 tempsExe.py  50000*
>>   sum with Python: 1250025000 and NumPy 1250025000
>>       time used Python Sum: * 37.28 sec *
>>       time used  Numpy Sum:  *1.85 sec*
>>
>> Labs$ *./tt    50000 *
>>
>>
>> *   CPU  time :7.521730     The value : 1250025000 *
>> --------------------------------------------
>>
>> This is the Python3 program :
>>
>> import timeit as it
>> import numpy as np
>> import sys
>> try :
>> n=eval(sys.argv[1])
>> except:
>> print ("needs integer as argument") ; exit()
>>
>> a=range(1,n+1)
>> b=np.array(a)
>> def func1():     return sum(a)
>> def func2(): return np.sum(b)
>>
>> print(f"sum with Python: {func1()} and NumPy {func2()} ")
>> tm1=it.timeit(stmt=func1, number=n)
>> print(f"time used Python Sum: {round(tm1,2)} sec")
>> tm2=it.timeit(stmt=func2, number=n)
>> print(f"time used  Numpy Sum: {round(tm2,2)} sec")
>>
>> and Here the C program:
>> #include <time.h>
>> #include <stdio.h>
>> #include <stdlib.h>
>> long func1(int n){
>>          long  r=0;
>>         for (int  i=1; i<= n;i++) r+= i;
>>          return r;
>> }
>> int main(int argc, char* argv[]){
>>          clock_t c0, c1;
>>         long v,count; int n;
>>        if ( argc < 2) {
>>               printf("Please give an argument");
>>              return -1;
>>       }
>>     n=atoi(argv[1]);
>>     c0 = clock();
>>      *for (int j=0;j < n;j++) v=func1(n);*
>>      c1 = clock();
>>      printf ("\tCPU  time :%.2f sec", (float)(c1 - c0)/CLOCKS_PER_SEC);
>>      printf("\n\tThe value : %ld\n",  v);
>> }
>> _______________________________________________
>> python-uk mailing list
>> python-uk at python.org
>> https://mail.python.org/mailman/listinfo/python-uk
>>
>
>
> --
> Life is a chess game - Anonymous.
>
> _______________________________________________
> python-uk mailing listpython-uk at python.orghttps://mail.python.org/mailman/listinfo/python-uk
>
>
> _______________________________________________
> python-uk mailing list
> python-uk at python.org
> https://mail.python.org/mailman/listinfo/python-uk
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/python-uk/attachments/20220225/4368ed40/attachment.html>

From ed.hartley at gmail.com  Fri Feb 25 04:57:44 2022
From: ed.hartley at gmail.com (Edward Hartley)
Date: Fri, 25 Feb 2022 09:57:44 +0000
Subject: [python-uk] C is it faster than numpy
In-Reply-To: <CAHW5HkoQ=9Bvcm7L3CejpMedFezUf+ni+7NDms_RLeyq-VSnzg@mail.gmail.com>
References: <CAHW5HkoQ=9Bvcm7L3CejpMedFezUf+ni+7NDms_RLeyq-VSnzg@mail.gmail.com>
Message-ID: <C3857E5F-4A5F-4570-ABDC-A5E4EDBF6DE2@gmail.com>


Hi, 
Simply check the original Numpy aka Numerical Python docs where it?s comprehensively explained that the library is implemented in C with a thin Python wrapper. The docs were written circa ?98 by the original library author. The library was optimised over a long period before being released. I?ve seen similar results when replacing an optimised Fortran library with C/C++ the key is the effort given to optimisation, and care with avoiding unnecessary memory allocation. 
I?ll dig out the reference when I?ve tracked it down.
Best of Luck
Ed Hartley

> On 25 Feb 2022, at 09:42, Giorgio Zoppi <giorgio.zoppi at gmail.com> wrote:
> 
> ?
> Well,
> numpy is written in C :) Maybe your C is not the numpy equivalent?
> Best Regards,
> Giorgio
> 
>> Il giorno ven 25 feb 2022 alle ore 09:03 BELAHCENE Abdelkader <abdelkader.belahcene at enst.dz> ha scritto:
>> Hi,
>> a lot of people think that C (or C++) is faster than python, yes I agree, but I think that's not the case with numpy, I believe numpy is faster than C, at least in some cases.
>> Is there another explanation ?
>> Or where can find  a doc speaking  about the subject?
>> Thanks a lot 
>> Regards
>> Numpy implements vectorization for arrays, or I'm wrong. Anyway here is an example Let's look at the following case:
>> Here is the result on my laptop i3:
>> 
>> Labs$ python3 tempsExe.py  50000 
>>   sum with Python: 1250025000 and NumPy 1250025000
>>       time used Python Sum:  37.28 sec 
>>       time used  Numpy Sum:  1.85 sec
>> 
>> Labs$ ./tt    50000 
>>     CPU  time :7.521730
>>     The value : 1250025000 
>> --------------------------------------------
>> 
>> This is the Python3 program :
>> 
>> import timeit as it
>> import numpy as np
>> import sys
>> try :
>> n=eval(sys.argv[1])
>> except:
>> print ("needs integer as argument") ; exit()
>> 
>> a=range(1,n+1)
>> b=np.array(a)
>> def func1():     return sum(a)
>> def func2(): return np.sum(b)
>> 
>> print(f"sum with Python: {func1()} and NumPy {func2()} ")
>> tm1=it.timeit(stmt=func1, number=n)
>> print(f"time used Python Sum: {round(tm1,2)} sec")
>> tm2=it.timeit(stmt=func2, number=n)
>> print(f"time used  Numpy Sum: {round(tm2,2)} sec")
>> 
>> and Here the C program:
>> #include <time.h>
>> #include <stdio.h>
>> #include <stdlib.h>
>> long func1(int n){
>>          long  r=0;
>>         for (int  i=1; i<= n;i++) r+= i;
>>          return r;
>> }
>> int main(int argc, char* argv[]){
>>          clock_t c0, c1; 
>>         long v,count; int n;
>>        if ( argc < 2) {
>>               printf("Please give an argument");
>>              return -1;
>>       }
>>     n=atoi(argv[1]); 
>>     c0 = clock();
>>      for (int j=0;j < n;j++) v=func1(n);
>>      c1 = clock();
>>      printf ("\tCPU  time :%.2f sec", (float)(c1 - c0)/CLOCKS_PER_SEC);
>>      printf("\n\tThe value : %ld\n",  v);
>> }
>> _______________________________________________
>> python-uk mailing list
>> python-uk at python.org
>> https://mail.python.org/mailman/listinfo/python-uk
> 
> 
> -- 
> Life is a chess game - Anonymous.
> _______________________________________________
> python-uk mailing list
> python-uk at python.org
> https://mail.python.org/mailman/listinfo/python-uk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/python-uk/attachments/20220225/6cbf2e19/attachment.html>

From me at adamj.eu  Fri Feb 25 05:02:09 2022
From: me at adamj.eu (Adam Johnson)
Date: Fri, 25 Feb 2022 10:02:09 +0000
Subject: [python-uk] C is it faster than numpy
In-Reply-To: <C3857E5F-4A5F-4570-ABDC-A5E4EDBF6DE2@gmail.com>
References: <CAHW5HkoQ=9Bvcm7L3CejpMedFezUf+ni+7NDms_RLeyq-VSnzg@mail.gmail.com>
 <C3857E5F-4A5F-4570-ABDC-A5E4EDBF6DE2@gmail.com>
Message-ID: <CAMyDDM3U1jVoUiRdj02eoTDqm4k8Ugy22PJ3vckYuy8wkEMr6g@mail.gmail.com>

Also interestingly, you can see the language breakdown on the right hand
side of https://github.com/numpy/numpy/ :

Python 62.5%
C 35.3%
C++ 1.0%
Cython 0.9%
Shell 0.2%
Fortran 0.1%

Much of the Python code is tests.

On Fri, Feb 25, 2022 at 9:57 AM Edward Hartley <ed.hartley at gmail.com> wrote:

>
> Hi,
> Simply check the original Numpy aka Numerical Python docs where it?s
> comprehensively explained that the library is implemented in C with a thin
> Python wrapper. The docs were written circa ?98 by the original library
> author. The library was optimised over a long period before being released.
> I?ve seen similar results when replacing an optimised Fortran library with
> C/C++ the key is the effort given to optimisation, and care with avoiding
> unnecessary memory allocation.
> I?ll dig out the reference when I?ve tracked it down.
> Best of Luck
> Ed Hartley
>
> On 25 Feb 2022, at 09:42, Giorgio Zoppi <giorgio.zoppi at gmail.com> wrote:
>
> ?
> Well,
> numpy is written in C :) Maybe your C is not the numpy equivalent?
> Best Regards,
> Giorgio
>
> Il giorno ven 25 feb 2022 alle ore 09:03 BELAHCENE Abdelkader <
> abdelkader.belahcene at enst.dz> ha scritto:
>
>> Hi,
>> a lot of people think that C (or C++) is faster than python, yes I agree,
>> but I think that's not the case with numpy, I believe numpy is faster than
>> C, at least in some cases.
>>
>>
>> *Is there another explanation ?Or where can find  a doc speaking  about
>> the subject?*Thanks a lot
>> Regards
>> Numpy implements vectorization for arrays, or I'm wrong. Anyway here is
>> an example Let's look at the following case:
>> Here is the result on my laptop i3:
>>
>> Labs$ *python3 tempsExe.py  50000*
>>   sum with Python: 1250025000 and NumPy 1250025000
>>       time used Python Sum: * 37.28 sec *
>>       time used  Numpy Sum:  *1.85 sec*
>>
>> Labs$ *./tt    50000 *
>>
>>
>> *   CPU  time :7.521730    The value : 1250025000 *
>> --------------------------------------------
>>
>> This is the Python3 program :
>>
>> import timeit as it
>> import numpy as np
>> import sys
>> try :
>> n=eval(sys.argv[1])
>> except:
>> print ("needs integer as argument") ; exit()
>>
>> a=range(1,n+1)
>> b=np.array(a)
>> def func1():     return sum(a)
>> def func2(): return np.sum(b)
>>
>> print(f"sum with Python: {func1()} and NumPy {func2()} ")
>> tm1=it.timeit(stmt=func1, number=n)
>> print(f"time used Python Sum: {round(tm1,2)} sec")
>> tm2=it.timeit(stmt=func2, number=n)
>> print(f"time used  Numpy Sum: {round(tm2,2)} sec")
>>
>> and Here the C program:
>> #include <time.h>
>> #include <stdio.h>
>> #include <stdlib.h>
>> long func1(int n){
>>          long  r=0;
>>         for (int  i=1; i<= n;i++) r+= i;
>>          return r;
>> }
>> int main(int argc, char* argv[]){
>>          clock_t c0, c1;
>>         long v,count; int n;
>>        if ( argc < 2) {
>>               printf("Please give an argument");
>>              return -1;
>>       }
>>     n=atoi(argv[1]);
>>     c0 = clock();
>>      *for (int j=0;j < n;j++) v=func1(n);*
>>      c1 = clock();
>>      printf ("\tCPU  time :%.2f sec", (float)(c1 - c0)/CLOCKS_PER_SEC);
>>      printf("\n\tThe value : %ld\n",  v);
>> }
>> _______________________________________________
>> python-uk mailing list
>> python-uk at python.org
>> https://mail.python.org/mailman/listinfo/python-uk
>>
>
>
> --
> Life is a chess game - Anonymous.
> _______________________________________________
> python-uk mailing list
> python-uk at python.org
> https://mail.python.org/mailman/listinfo/python-uk
>
> _______________________________________________
> python-uk mailing list
> python-uk at python.org
> https://mail.python.org/mailman/listinfo/python-uk
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/python-uk/attachments/20220225/783dc513/attachment-0001.html>

From stestagg at gmail.com  Fri Feb 25 05:37:45 2022
From: stestagg at gmail.com (Stestagg)
Date: Fri, 25 Feb 2022 10:37:45 +0000
Subject: [python-uk] C is it faster than numpy
In-Reply-To: <CABcwF5m6u-NWhtnEwG+iopLFPZBJopO7hLP+f3uhwUdzn93wNw@mail.gmail.com>
References: <CABcwF5m6u-NWhtnEwG+iopLFPZBJopO7hLP+f3uhwUdzn93wNw@mail.gmail.com>
Message-ID: <CACFDE8YZUW2ZUEU+3FsRRjtdvP+0o36_1_Gh4ssm8FvLFNwrsg@mail.gmail.com>

This is a fascinating subject. Firstly:

a lot of people think that C (or C++) is faster than python, yes I agree,
>

if you look at raw execution speed, then this is absolutely correct.  The
reasons for this are many and complex, but people who use this as a
standalone reason to avoid Python almost always don't actually understand
the subject too well.

If you want to read more about this, I would recommend searching for
"python glue language"  for example:

https://numpy.org/devdocs/user/c-info.python-as-glue.html


> *Is there another explanation ?*
>

Yes, the people who wrote numpy implemented sum() 'better' than you did.
That's not a put-down or anything, just an interesting reason why Python as
a glue language works so well. Lots of very smart people spend a lot of
time making numpy fast, because lots of people rely on it being fast
(partly because raw Python is a bit too slow to do these sorts of things
natively).

In C/C++ it's easy to say: 'it's just a for-loop, let's write it the simple
way'.   9 times out of 10, the for loop will be much faster than you ever
need it to be in either language, but if you really need speed for whatever
reason, you're going to have to reach for a library.  Numpy's popularity
means that everyone knows to just use it for this sort of thing
(interestingly, if you have a GPU, using something like pytorch might be
much faster here, depending on data transfer overheads).  In C/C++ there
are many many libraries that can do this sort of thing, but none have the
same general appeal as Numpy (in my opinion).

This means that with Python, you write the simple bits in a nice dynamic,
easy to write language, and the hard bits get farmed out to libraries like
numpy, and you benefit from some really awesome optimized code that make
your code faster than the naive equivalent 'fast' C implementation.

For example the following are all (i believe) different array sum
implementations for different CPU vector features across different
architectures/extensions:
https://github.com/numpy/numpy/blob/b97e7d585de4b80ae8202c1028c0dc03f5dde4ef/numpy/core/src/common/simd/avx512/arithmetic.h#L353
https://github.com/numpy/numpy/blob/b97e7d585de4b80ae8202c1028c0dc03f5dde4ef/numpy/core/src/common/simd/sse/arithmetic.h#L327
https://github.com/numpy/numpy/blob/b97e7d585de4b80ae8202c1028c0dc03f5dde4ef/numpy/core/src/common/simd/neon/arithmetic.h#L283

It's absolutely possible to make C/C++ perform at the same speed as numpy,
but you will have to invest a lot of time/effort in learning about
performance programming to get there (this can be a really useful skill to
learn, if you're interested!)

Some anecdotes:

1. I was using a C data logger library for some data logger device that
performed awfully, so I re-wrote it, feature-for-feature in python, and it
was 100x faster.  Why?  Not because Python is faster than C, just that
somewhere the original implementation had some bug/issue that was causing
slowness (It was related to sqlite transaction handling, but the C code
made it very hard to spot the issue, whereas the python was much easier to
reason about).  Nothing about what this library was doing required
high-performance code, so using Python was not a speed issue at all.

2. I worked on a very large data processing application for a fortune 500
co in Python, one part of the input pipeline was never going to be fast
enough for our volumes in pure python, so we wrote a custom module in
Cython (A hybrid language that compiles to a C/C++ extension), this small
module allowed the entire system to perform up to spec, and (along with a
numpy-like library) meant we could implement a lot of custom business logic
in Python, rather than in less expressive languages, and generally
out-perform similar Java/other language systems.


Steve
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/python-uk/attachments/20220225/1e4dfbb5/attachment.html>

From abdelkader.belahcene at enst.dz  Fri Feb 25 11:22:22 2022
From: abdelkader.belahcene at enst.dz (BELAHCENE Abdelkader)
Date: Fri, 25 Feb 2022 17:22:22 +0100
Subject: [python-uk] C is it faster than numpy
In-Reply-To: <CAHW5HkoQ=9Bvcm7L3CejpMedFezUf+ni+7NDms_RLeyq-VSnzg@mail.gmail.com>
References: <CABcwF5m6u-NWhtnEwG+iopLFPZBJopO7hLP+f3uhwUdzn93wNw@mail.gmail.com>
 <CAHW5HkoQ=9Bvcm7L3CejpMedFezUf+ni+7NDms_RLeyq-VSnzg@mail.gmail.com>
Message-ID: <CABcwF5m284ON7nGNXGrFEdratBJawvMO4FWkataBWjPo-Oq61g@mail.gmail.com>

Hi,
What do you mean?
the python and C programs are not equivalent?


Le ven. 25 f?vr. 2022 ? 10:42, Giorgio Zoppi <giorgio.zoppi at gmail.com> a
?crit :

> Well,
> numpy is written in C :) Maybe your C is not the numpy equivalent?
> Best Regards,
> Giorgio
>
> Il giorno ven 25 feb 2022 alle ore 09:03 BELAHCENE Abdelkader <
> abdelkader.belahcene at enst.dz> ha scritto:
>
>> Hi,
>> a lot of people think that C (or C++) is faster than python, yes I agree,
>> but I think that's not the case with numpy, I believe numpy is faster than
>> C, at least in some cases.
>>
>>
>> *Is there another explanation ?Or where can find  a doc speaking  about
>> the subject?*Thanks a lot
>> Regards
>> Numpy implements vectorization for arrays, or I'm wrong. Anyway here is
>> an example Let's look at the following case:
>> Here is the result on my laptop i3:
>>
>> Labs$ *python3 tempsExe.py  50000*
>>   sum with Python: 1250025000 and NumPy 1250025000
>>       time used Python Sum: * 37.28 sec *
>>       time used  Numpy Sum:  *1.85 sec*
>>
>> Labs$ *./tt    50000 *
>>
>>
>> *   CPU  time :7.521730    The value : 1250025000 *
>> --------------------------------------------
>>
>> This is the Python3 program :
>>
>> import timeit as it
>> import numpy as np
>> import sys
>> try :
>> n=eval(sys.argv[1])
>> except:
>> print ("needs integer as argument") ; exit()
>>
>> a=range(1,n+1)
>> b=np.array(a)
>> def func1():     return sum(a)
>> def func2(): return np.sum(b)
>>
>> print(f"sum with Python: {func1()} and NumPy {func2()} ")
>> tm1=it.timeit(stmt=func1, number=n)
>> print(f"time used Python Sum: {round(tm1,2)} sec")
>> tm2=it.timeit(stmt=func2, number=n)
>> print(f"time used  Numpy Sum: {round(tm2,2)} sec")
>>
>> and Here the C program:
>> #include <time.h>
>> #include <stdio.h>
>> #include <stdlib.h>
>> long func1(int n){
>>          long  r=0;
>>         for (int  i=1; i<= n;i++) r+= i;
>>          return r;
>> }
>> int main(int argc, char* argv[]){
>>          clock_t c0, c1;
>>         long v,count; int n;
>>        if ( argc < 2) {
>>               printf("Please give an argument");
>>              return -1;
>>       }
>>     n=atoi(argv[1]);
>>     c0 = clock();
>>      *for (int j=0;j < n;j++) v=func1(n);*
>>      c1 = clock();
>>      printf ("\tCPU  time :%.2f sec", (float)(c1 - c0)/CLOCKS_PER_SEC);
>>      printf("\n\tThe value : %ld\n",  v);
>> }
>> _______________________________________________
>> python-uk mailing list
>> python-uk at python.org
>> https://mail.python.org/mailman/listinfo/python-uk
>>
>
>
> --
> Life is a chess game - Anonymous.
> _______________________________________________
> python-uk mailing list
> python-uk at python.org
> https://mail.python.org/mailman/listinfo/python-uk
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/python-uk/attachments/20220225/f9b2a0a7/attachment.html>

From abdelkader.belahcene at enst.dz  Fri Feb 25 11:23:49 2022
From: abdelkader.belahcene at enst.dz (BELAHCENE Abdelkader)
Date: Fri, 25 Feb 2022 17:23:49 +0100
Subject: [python-uk] C is it faster than numpy
In-Reply-To: <C3857E5F-4A5F-4570-ABDC-A5E4EDBF6DE2@gmail.com>
References: <CAHW5HkoQ=9Bvcm7L3CejpMedFezUf+ni+7NDms_RLeyq-VSnzg@mail.gmail.com>
 <C3857E5F-4A5F-4570-ABDC-A5E4EDBF6DE2@gmail.com>
Message-ID: <CABcwF5mEaEDo6yagjXLnAvVvO0k=PxH2Jnjq5wTvoL9Y-STGMg@mail.gmail.com>

Thanks,
But the different is very significant!!!
so?

Le ven. 25 f?vr. 2022 ? 10:58, Edward Hartley <ed.hartley at gmail.com> a
?crit :

>
> Hi,
> Simply check the original Numpy aka Numerical Python docs where it?s
> comprehensively explained that the library is implemented in C with a thin
> Python wrapper. The docs were written circa ?98 by the original library
> author. The library was optimised over a long period before being released.
> I?ve seen similar results when replacing an optimised Fortran library with
> C/C++ the key is the effort given to optimisation, and care with avoiding
> unnecessary memory allocation.
> I?ll dig out the reference when I?ve tracked it down.
> Best of Luck
> Ed Hartley
>
> On 25 Feb 2022, at 09:42, Giorgio Zoppi <giorgio.zoppi at gmail.com> wrote:
>
> ?
> Well,
> numpy is written in C :) Maybe your C is not the numpy equivalent?
> Best Regards,
> Giorgio
>
> Il giorno ven 25 feb 2022 alle ore 09:03 BELAHCENE Abdelkader <
> abdelkader.belahcene at enst.dz> ha scritto:
>
>> Hi,
>> a lot of people think that C (or C++) is faster than python, yes I agree,
>> but I think that's not the case with numpy, I believe numpy is faster than
>> C, at least in some cases.
>>
>>
>> *Is there another explanation ?Or where can find  a doc speaking  about
>> the subject?*Thanks a lot
>> Regards
>> Numpy implements vectorization for arrays, or I'm wrong. Anyway here is
>> an example Let's look at the following case:
>> Here is the result on my laptop i3:
>>
>> Labs$ *python3 tempsExe.py  50000*
>>   sum with Python: 1250025000 and NumPy 1250025000
>>       time used Python Sum: * 37.28 sec *
>>       time used  Numpy Sum:  *1.85 sec*
>>
>> Labs$ *./tt    50000 *
>>
>>
>> *   CPU  time :7.521730    The value : 1250025000 *
>> --------------------------------------------
>>
>> This is the Python3 program :
>>
>> import timeit as it
>> import numpy as np
>> import sys
>> try :
>> n=eval(sys.argv[1])
>> except:
>> print ("needs integer as argument") ; exit()
>>
>> a=range(1,n+1)
>> b=np.array(a)
>> def func1():     return sum(a)
>> def func2(): return np.sum(b)
>>
>> print(f"sum with Python: {func1()} and NumPy {func2()} ")
>> tm1=it.timeit(stmt=func1, number=n)
>> print(f"time used Python Sum: {round(tm1,2)} sec")
>> tm2=it.timeit(stmt=func2, number=n)
>> print(f"time used  Numpy Sum: {round(tm2,2)} sec")
>>
>> and Here the C program:
>> #include <time.h>
>> #include <stdio.h>
>> #include <stdlib.h>
>> long func1(int n){
>>          long  r=0;
>>         for (int  i=1; i<= n;i++) r+= i;
>>          return r;
>> }
>> int main(int argc, char* argv[]){
>>          clock_t c0, c1;
>>         long v,count; int n;
>>        if ( argc < 2) {
>>               printf("Please give an argument");
>>              return -1;
>>       }
>>     n=atoi(argv[1]);
>>     c0 = clock();
>>      *for (int j=0;j < n;j++) v=func1(n);*
>>      c1 = clock();
>>      printf ("\tCPU  time :%.2f sec", (float)(c1 - c0)/CLOCKS_PER_SEC);
>>      printf("\n\tThe value : %ld\n",  v);
>> }
>> _______________________________________________
>> python-uk mailing list
>> python-uk at python.org
>> https://mail.python.org/mailman/listinfo/python-uk
>>
>
>
> --
> Life is a chess game - Anonymous.
> _______________________________________________
> python-uk mailing list
> python-uk at python.org
> https://mail.python.org/mailman/listinfo/python-uk
>
> _______________________________________________
> python-uk mailing list
> python-uk at python.org
> https://mail.python.org/mailman/listinfo/python-uk
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/python-uk/attachments/20220225/deecdcc9/attachment-0001.html>

From sparks.m at gmail.com  Fri Feb 25 20:33:55 2022
From: sparks.m at gmail.com (Michael)
Date: Sat, 26 Feb 2022 01:33:55 +0000
Subject: [python-uk] C is it faster than numpy
In-Reply-To: <CABcwF5m284ON7nGNXGrFEdratBJawvMO4FWkataBWjPo-Oq61g@mail.gmail.com>
References: <CABcwF5m6u-NWhtnEwG+iopLFPZBJopO7hLP+f3uhwUdzn93wNw@mail.gmail.com>
 <CAHW5HkoQ=9Bvcm7L3CejpMedFezUf+ni+7NDms_RLeyq-VSnzg@mail.gmail.com>
 <CABcwF5m284ON7nGNXGrFEdratBJawvMO4FWkataBWjPo-Oq61g@mail.gmail.com>
Message-ID: <CAB+QZVBu0nfiQ39Zf_cftO7hYJfsHK3hYNqhYz4=TPzFAPoiiw@mail.gmail.com>

Hi,

On Fri, 25 Feb 2022 at 16:22, BELAHCENE Abdelkader <
abdelkader.belahcene at enst.dz> wrote:

> Hi,
> What do you mean?
> the python and C programs are not equivalent?
>
>
The python and C programs are NOT equivalent. (surface level they are: they
both calculate the triangular number of N, N times, inefficiently and
slowly, but the way they do it is radically different)

Let's specifically compare the num.py and the C versions. The pure python
version *should* always be slower, so it's irrelevant here. (NB, I show a
version below where the pure python version is quicker than both :-) )

The num.py version:

* It creates an array containing the numbers 1 to n. It then call's
num.py's sum function with that array n times.

The C version :
* Calls a function n times. That function has a tight loop using a long
that sums all the numbers from 1 to n.

These are *very* different operations. The former can be vectorised, and
while I don't use num.py, and while I'm not a betting person I would bet a
Mars bar that the num.py version will throw the array at a SIMD
implementation. A cursory look at the source to numpy does indeed show that
this is the case (fx: grabs a mars bar :) ), and not only that it's
optimised for a wide variety of CPU architectures.
https://github.com/numpy/numpy/tree/main/numpy/core/src/common/simd

If you don't know what that means, take a look at this --
https://en.wikipedia.org/wiki/Streaming_SIMD_Extensions numpy however
supports multiple versions of SIMD - including things like this:
https://en.wikipedia.org/wiki/AVX-512 - which is pretty neat.

Upshot - numpy will /essentially/ just throw the entire array at the CPU
and say "add these, give me the result". And it'll be done. (OK there's a
bit more to it, but that's the principle)

By contrast your naive C version has to repeatedly create stack frames,
push arguments onto it, pop arguments allocate memory on the stack, etc.
It's also single threaded, and has no means of telling the CPU "add all
these together, and I don't care how", so it literally is one after the
other. The actual work it's doing to get the same answer is simply much
greater and many many more clock cycles.

If you took the same approach as numpy it's *possible* you *might* get
something similarly fast. (But you'd have a steep learning curve and you'd
likely only optimise for one architecture...)

Aside: What size is the *value*  the C version is creating? Well it's just
(n * n+1 ) /2  - primary school triangular number stuff. So 50K
is 1250025000 - which is a 31 bit number. So the C version will fail on a
32 bit machine as soon as you try over 65536 as the argument... (On a
64bit machine admittedly the fall over number is admittedly higher... :-) )

C and C++ *are* faster than pure python. That's pretty much always going to
be the case ( *except* for a specialising python compiler that compiles a
subset of python to either C/C++/Rust or assembler). However, python +
highly optimised C/C++/etc libraries will likely outperform naive C/C++
code - as you've ably demonstrated here.

Why? Because while on the surface the two programs are vaguely similar -
calculate the triangular number of N, N times.  In practice, the way that
they do it is so different, you get dramatically different results.

As a bonus - you can get faster than both your num.py and C versions with
pure python, by several orders of magnitude. You can then squeeze out a few
more percent out of it.  Output from a pure python version of a piece of
code that performs the same operation - calculates the triangle number of
N, N times:

michael at conceptual:~$ ./triangles.py 15000000
time using pure python: 1.81 sec
time using pure python & memoisation: 1.7 sec

Note that's 15 million, not 50,000 - and it took the same time for my
machine (recent core i7) as numpy did for you on your machine for just
50,000. That's not because I have a faster machine (I expect I don't).

What's different? Well the pure python version is this - it recognises you
were calculating  the triangular number for N and just calculates that
instead :-)

def normaltriangle(n):
   return (n*(n+1))/2
... called like this ..
   tm1=it.timeit(stmt=lambda: normaltriangle(count), number=count)
   print(f"time using pure python: {round(tm1,2)} sec")

The (naive) memoisation version is this:

def memoise(f):
   cache = {}
   def g(n):
       if n in cache:
           return cache[n]
       else:
           X = f(n)
           cache[n] = X
           return X
   return g

@memoise
def triangle(n):
   return (n*(n+1))/2

... called like this ...
   tm2=it.timeit(stmt=lambda: triangle(count), number=count)
   print(f"time using pure python & memoisation: {round(tm2,2)} sec")


Original file containing both:

#!/usr/bin/python3

import timeit as it

def memoise(f):
   cache = {}
   def g(n):
       if n in cache:
           return cache[n]
       else:
           X = f(n)
           cache[n] = X
           return X
   return g

@memoise
def triangle(n):
   return (n*(n+1))/2


def normaltriangle(n):
   return (n*(n+1))/2


if __name__ == "__main__":
   import sys
   count = int(sys.argv[1])
   tm1=it.timeit(stmt=lambda: normaltriangle(count), number=count)
   print(f"time using pure python: {round(tm1,2)} sec")

   tm2=it.timeit(stmt=lambda: triangle(count), number=count)
   print(f"time using pure python & memoisation: {round(tm2,2)} sec")

I guess the point here is this: focussing on the language/tool C vs python
ignores how "well" the C code is written and how "well" the python code is
written.

However worrying about *that* rather than "is there a smarter way of doing
this" means you can sometimes miss the 300 fold speed up (vs numpy) or 6000
fold speed up (vs your python). I must admit as well, I was a little
surprised that the memoised version worked even this well with such simple
code.

But the point is when comparing C to numpy, you're really comparing *your*
C with *numpy's* C code, and as you can see from their code, your repeated
calls are very different from a vectorised call to CPU processor extensions
designed specifically for this sort of work...

Gets even sillier (in a fun way :) ) if you can throw the code at several
hundred GPU cores...

Hope this was vaguely interesting...


Michael.


On Fri, 25 Feb 2022 at 16:22, BELAHCENE Abdelkader <
abdelkader.belahcene at enst.dz> wrote:

> Hi,
> What do you mean?
> the python and C programs are not equivalent?
>
>
> Le ven. 25 f?vr. 2022 ? 10:42, Giorgio Zoppi <giorgio.zoppi at gmail.com> a
> ?crit :
>
>> Well,
>> numpy is written in C :) Maybe your C is not the numpy equivalent?
>> Best Regards,
>> Giorgio
>>
>> Il giorno ven 25 feb 2022 alle ore 09:03 BELAHCENE Abdelkader <
>> abdelkader.belahcene at enst.dz> ha scritto:
>>
>>> Hi,
>>> a lot of people think that C (or C++) is faster than python, yes I
>>> agree, but I think that's not the case with numpy, I believe numpy is
>>> faster than C, at least in some cases.
>>>
>>>
>>> *Is there another explanation ?Or where can find  a doc speaking  about
>>> the subject?*Thanks a lot
>>> Regards
>>> Numpy implements vectorization for arrays, or I'm wrong. Anyway here is
>>> an example Let's look at the following case:
>>> Here is the result on my laptop i3:
>>>
>>> Labs$ *python3 tempsExe.py  50000*
>>>   sum with Python: 1250025000 and NumPy 1250025000
>>>       time used Python Sum: * 37.28 sec *
>>>       time used  Numpy Sum:  *1.85 sec*
>>>
>>> Labs$ *./tt    50000 *
>>>
>>>
>>> *   CPU  time :7.521730    The value : 1250025000 *
>>> --------------------------------------------
>>>
>>> This is the Python3 program :
>>>
>>> import timeit as it
>>> import numpy as np
>>> import sys
>>> try :
>>> n=eval(sys.argv[1])
>>> except:
>>> print ("needs integer as argument") ; exit()
>>>
>>> a=range(1,n+1)
>>> b=np.array(a)
>>> def func1():     return sum(a)
>>> def func2(): return np.sum(b)
>>>
>>> print(f"sum with Python: {func1()} and NumPy {func2()} ")
>>> tm1=it.timeit(stmt=func1, number=n)
>>> print(f"time used Python Sum: {round(tm1,2)} sec")
>>> tm2=it.timeit(stmt=func2, number=n)
>>> print(f"time used  Numpy Sum: {round(tm2,2)} sec")
>>>
>>> and Here the C program:
>>> #include <time.h>
>>> #include <stdio.h>
>>> #include <stdlib.h>
>>> long func1(int n){
>>>          long  r=0;
>>>         for (int  i=1; i<= n;i++) r+= i;
>>>          return r;
>>> }
>>> int main(int argc, char* argv[]){
>>>          clock_t c0, c1;
>>>         long v,count; int n;
>>>        if ( argc < 2) {
>>>               printf("Please give an argument");
>>>              return -1;
>>>       }
>>>     n=atoi(argv[1]);
>>>     c0 = clock();
>>>      *for (int j=0;j < n;j++) v=func1(n);*
>>>      c1 = clock();
>>>      printf ("\tCPU  time :%.2f sec", (float)(c1 - c0)/CLOCKS_PER_SEC);
>>>      printf("\n\tThe value : %ld\n",  v);
>>> }
>>> _______________________________________________
>>> python-uk mailing list
>>> python-uk at python.org
>>> https://mail.python.org/mailman/listinfo/python-uk
>>>
>>
>>
>> --
>> Life is a chess game - Anonymous.
>> _______________________________________________
>> python-uk mailing list
>> python-uk at python.org
>> https://mail.python.org/mailman/listinfo/python-uk
>>
> _______________________________________________
> python-uk mailing list
> python-uk at python.org
> https://mail.python.org/mailman/listinfo/python-uk
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/python-uk/attachments/20220226/5cb3cce2/attachment-0001.html>

From peter.inglesby at gmail.com  Sun Feb 27 05:22:09 2022
From: peter.inglesby at gmail.com (Peter Inglesby)
Date: Sun, 27 Feb 2022 10:22:09 +0000
Subject: [python-uk] C is it faster than numpy
In-Reply-To: <CAB+QZVBu0nfiQ39Zf_cftO7hYJfsHK3hYNqhYz4=TPzFAPoiiw@mail.gmail.com>
References: <CABcwF5m6u-NWhtnEwG+iopLFPZBJopO7hLP+f3uhwUdzn93wNw@mail.gmail.com>
 <CAHW5HkoQ=9Bvcm7L3CejpMedFezUf+ni+7NDms_RLeyq-VSnzg@mail.gmail.com>
 <CABcwF5m284ON7nGNXGrFEdratBJawvMO4FWkataBWjPo-Oq61g@mail.gmail.com>
 <CAB+QZVBu0nfiQ39Zf_cftO7hYJfsHK3hYNqhYz4=TPzFAPoiiw@mail.gmail.com>
Message-ID: <CAENJrPmtggL3JCbJDUnJmjbfBDF6COnv7=3WditRuCxgzKws7w@mail.gmail.com>

>
> Hope this was vaguely interesting...
>

I certainly learnt something, thanks for the time you put into the
explanation Michael!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/python-uk/attachments/20220227/1aefa001/attachment.html>