Thread Question

[Carl Banks]

Ritesh Raj Sarraf wrote:
> Simon Forman wrote:
> > > One other question I had,
> > > If my user passes the --zip option, download_from_web() internally (when the
> > > download is successful) zips the downloaded data to a zip file. Since in case
> > > of threading there'll be multiple threads, and say if one of the thread
> > > completes 2 seconds before others and is doing the zipping work:
> > > What will the other thread, at that moment do, if it completes while the
> > > previous thread is doing the zipping work ?
> >
> > The other threads will just take the next request from the Queue and
> > process it.  They won't "care" what the one thread is doing,
> > downloading, zipping, whatever.
> >
> >
>
> The thread will be marked as complete only when the function that it
> executed exits. Right ?
>
> download_from_web() internally calls a funtion to zip the file. So it
> doesn't return before zipping. Till then this thread is not complete
> and therefore is busy working (i.e. zipping).
>
> during the same time if another thread (which is also calling
> download_from_web) completes the download, the download function will
> again call the zip code. At that particular situtation, will it wait
> for the previous thread to complete the zipping and release the file so
> that it can zip more data to it or will it just panic and quit ?

If you have multiple threads trying to access the same ZIP file at the
same time, whether or not they use the same ZipFile object, you'll have
trouble.  You'd have to change download_from_web to protect against
simultaneous use.  A simple lock should suffice.  Create the lock in
the main thread, like so:

ziplock = threading.Lock()

Then change the zipping part of download_from_web to acquire and
release this lock; do zipfile operations only between them.

ziplock.acquire()
try:
    do_all_zipfile_stuff_here()
finally:
    ziplock.release()

If you can't change download_from_web, you might have no choice but to
download sequentially.

OTOH, if each thread uses a different ZIP file (and a different ZipFile
object), you wouldn't have to use a lock.  It doesn't sound like you're
doing that, though.

It shouldn't be a problem if one thread is zipping at the same time
another is downloading, unless there's some common data between them
for some reason.


Carl Banks