Fwd: Re: Long running process - how to speed up?

[Alan Gauld]

On 19/02/2022 11:28, Shaozhong SHI wrote:

> I have a cvs file of 932956 row 

That's not a lot in modern computing terms.

> and have to have time.sleep in a Python
> script. 

Why? Is it a requirement by your customer? Your manager?
time.sleep() is not usually helpful if you want to do
things quickly.

> It takes a long time to process.

What is a "long time"? minutes? hours? days? weeks?

It should take a million times as long as it takes to
process one row. But you have given no clue what you
are doing in each row.
- reading a database?
- reading from the network? or the internet?
- writing to a database? or the internet?
- performing highly complex math operations?

Or perhaps the processing load is in analyzing the totality
of the data after reading it all? A very different type
of problem. But we just don't know.

All of these factors will affect performance.

> How can I speed up the processing? 

It all depends on the processing.
You could try profiling your code to see where the time is spent.

> Can I do multi-processing?

Of course. But there is no guarantee that will speed things
up if there is a bottleneck on a single resource somewhere.
But it might be possible to divide and conquer and get better
speed. It all depends on what you are doing. We can't tell.

We cannot answer such a vague question with any specific
solution.

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos