Long running process - how to speed up?

[Shaozhong SHI]

On Sat, 19 Feb 2022 at 18:51, Alan Gauld <learn2program at gmail.com> wrote:

> On 19/02/2022 11:28, Shaozhong SHI wrote:
>
> > I have a cvs file of 932956 row
>
> That's not a lot in modern computing terms.
>
> > and have to have time.sleep in a Python
> > script.
>
> Why? Is it a requirement by your customer? Your manager?
> time.sleep() is not usually helpful if you want to do
> things quickly.
>
> > It takes a long time to process.
>
> What is a "long time"? minutes? hours? days? weeks?
>
> It should take a million times as long as it takes to
> process one row. But you have given no clue what you
> are doing in each row.
> - reading a database?
> - reading from the network? or the internet?
> - writing to a database? or the internet?
> - performing highly complex math operations?
>
> Or perhaps the processing load is in analyzing the totality
> of the data after reading it all? A very different type
> of problem. But we just don't know.
>
> All of these factors will affect performance.
>
> > How can I speed up the processing?
>
> It all depends on the processing.
> You could try profiling your code to see where the time is spent.
>
> > Can I do multi-processing?
>
> Of course. But there is no guarantee that will speed things
> up if there is a bottleneck on a single resource somewhere.
> But it might be possible to divide and conquer and get better
> speed. It all depends on what you are doing. We can't tell.
>
> We cannot answer such a vague question with any specific
> solution.
>
> --
> Alan G
> Author of the Learn to Program web site
> http://www.alan-g.me.uk/
> http://www.amazon.com/author/alan_gauld
> Follow my photo-blog on Flickr at:
> http://www.flickr.com/photos/alangauldphotos
>
> --
> https://mail.python.org/mailman/listinfo/python-list


Do not know these answers yet.  Now, it appeared to hang/stop at a point
and does not move on.

Regards,

David