[SOLVED] Re: os walk() and threads problems (os.walk are thread safe?)
Marcus Alves Grando
marcus at sbh.eng.br
Wed Nov 14 09:30:42 EST 2007
Ok. I found the problem.
That's because in for i test "if EXIT" and break loop if it's true. In
main part i'm wait Queue to be empty and set EXIT after that, with this
subdirectories in for loop does not process and program exit.
Because that output are not same.
Removing "if EXIT" all works fine again.
Thanks all
Regards
Marcus Alves Grando wrote:
> I make one new version more equally to original version:
>
> --code--
> #!/usr/bin/python
>
> import os, sys, time
> import glob, random, Queue
> import threading
>
> EXIT = False
> BRANDS = {}
> LOCK=threading.Lock()
> EV=threading.Event()
> POOL=Queue.Queue(0)
> NRO_THREADS=20
>
> def walkerr(err):
> print err
>
> class Worker(threading.Thread):
> def run(self):
> EV.wait()
> while True:
> try:
> mydir=POOL.get(timeout=1)
> if mydir == None:
> continue
>
> for root, dirs, files in os.walk(mydir, onerror=walkerr):
> if EXIT:
> break
>
> terra_user = 'test'
> terra_brand = 'test'
> user_du = '0 a'
> user_total_files = 0
>
> LOCK.acquire()
> if not BRANDS.has_key(terra_brand):
> BRANDS[terra_brand] = {}
> BRANDS[terra_brand]['COUNT'] = 1
> BRANDS[terra_brand]['SIZE'] = int(user_du.split()[0])
> BRANDS[terra_brand]['FILES'] = user_total_files
> else:
> BRANDS[terra_brand]['COUNT'] = BRANDS[terra_brand]['COUNT'] + 1
> BRANDS[terra_brand]['SIZE'] = BRANDS[terra_brand]['SIZE'] +
> int(user_du.split()[0])
> BRANDS[terra_brand]['FILES'] = BRANDS[terra_brand]['FILES'] +
> user_total_files
> LOCK.release()
>
> except Queue.Empty:
> if EXIT:
> break
> else:
> continue
> except KeyboardInterrupt:
> break
> except Exception:
> print mydir
> raise
>
> if len(sys.argv) < 2:
> print 'Usage: %s dir...' % sys.argv[0]
> sys.exit(1)
>
> glob_dirs = []
> for i in sys.argv[1:]:
> glob_dirs = glob_dirs + glob.glob(i+'/[a-z_]*')
> random.shuffle(glob_dirs)
>
> for x in xrange(NRO_THREADS):
> Worker().start()
>
> try:
> for i in glob_dirs:
> POOL.put(i)
>
> EV.set()
> while not POOL.empty():
> time.sleep(1)
> EXIT = True
>
> while (threading.activeCount() > 1):
> time.sleep(1)
> except KeyboardInterrupt:
> EXIT=True
>
> for b in BRANDS:
> print '%s:%i:%i:%i' % (b, BRANDS[b]['SIZE'], BRANDS[b]['COUNT'],
> BRANDS[b]['FILES'])
> --code--
>
> And run in make servers:
>
> # uname -r
> 2.6.18-8.1.15.el5
> # python test.py /usr
> test:0:2267:0
> # python test.py /usr
> test:0:2224:0
> # python test.py /usr
> test:0:2380:0
> # python -V
> Python 2.4.3
>
> # uname -r
> 7.0-BETA2
> # python test.py /usr
> test:0:1706:0
> # python test.py /usr
> test:0:1492:0
> # python test.py /usr
> test:0:1524:0
> # python -V
> Python 2.5.1
>
> # uname -r
> 2.6.9-42.0.8.ELsmp
> # python test.py /usr
> test:0:1311:0
> # python test.py /usr
> test:0:1486:0
> # python test.py /usr
> test:0:1520:0
> # python -V
> Python 2.3.4
>
> I really don't know what's happen.
>
> Another ideia?
>
> Regards
>
> Chris Mellon wrote:
>> On Nov 13, 2007 1:06 PM, Marcus Alves Grando <marcus at sbh.eng.br> wrote:
>>> Diez B. Roggisch wrote:
>>>> Marcus Alves Grando wrote:
>>>>
>>>>> Diez B. Roggisch wrote:
>>>>>> Marcus Alves Grando wrote:
>>>>>>
>>>>>>> Hello list,
>>>>>>>
>>>>>>> I have a strange problem with os.walk and threads in python script. I
>>>>>>> have one script that create some threads and consume Queue. For every
>>>>>>> value in Queue this script run os.walk() and printing root dir. But if i
>>>>>>> increase number of threads the result are inconsistent compared with one
>>>>>>> thread.
>>>>>>>
>>>>>>> For example, run this code plus sort with one thread and after run again
>>>>>>> with ten threads and see diff(1).
>>>>>> I don't see any difference. I ran it with 1 and 10 workers + sorted the
>>>>>> output. No diff whatsoever.
>>>>> Do you test in one dir with many subdirs? like /usr or /usr/ports (in
>>>>> freebsd) for example?
>>>> Yes, over 1000 subdirs/files.
>>> Strange, because to me accurs every time.
>>>
>>>>>> And I don't know what you mean by diff(1) - was that supposed to be some
>>>>>> output?
>>>>> No. One thread produce one result and ten threads produce another result
>>>>> with less lines.
>>>>>
>>>>> Se example below:
>>>>>
>>>>> @@ -13774,8 +13782,6 @@
>>>>> /usr/compat/linux/proc/44
>>>>> /usr/compat/linux/proc/45
>>>>> /usr/compat/linux/proc/45318
>>>>> -/usr/compat/linux/proc/45484
>>>>> -/usr/compat/linux/proc/45532
>>>>> /usr/compat/linux/proc/45857
>>>>> /usr/compat/linux/proc/45903
>>>>> /usr/compat/linux/proc/46
>>>> I'm not sure what that directory is, but to me that looks like the
>>>> linux /proc dir, containing process ids. Which incidentially changes
>>>> between the two runs, as more threads will have process id aliases.
>>> My example are not good enough. I run this script in ports directory of
>>> freebsd and imap folders in my linux server, same thing.
>>>
>>> @@ -182,7 +220,6 @@
>>> /usr/ports/archivers/p5-POE-Filter-Bzip2
>>> /usr/ports/archivers/p5-POE-Filter-LZF
>>> /usr/ports/archivers/p5-POE-Filter-LZO
>>> -/usr/ports/archivers/p5-POE-Filter-LZW
>>> /usr/ports/archivers/p5-POE-Filter-Zlib
>>> /usr/ports/archivers/p5-PerlIO-gzip
>>> /usr/ports/archivers/p5-PerlIO-via-Bzip2
>>> @@ -234,7 +271,6 @@
>>> /usr/ports/archivers/star-devel
>>> /usr/ports/archivers/star-devel/files
>>> /usr/ports/archivers/star/files
>>> -/usr/ports/archivers/stuffit
>>> /usr/ports/archivers/szip
>>> /usr/ports/archivers/tardy
>>> /usr/ports/archivers/tardy/files
>>>
>>>
>> Are you just diffing the output? There's no guarantee that
>> os.path.walk() will always have the same order, or that your different
>> working threads will produce the same output in the same order. On my
>> system, for example, I get a different order of subdirectory output
>> when I run with 10 threads than with 1.
>>
>> walk() requires that stat() works for the next directory that will be
>> walked. It might be remotely possible that stat() is failing for some
>> reason and some directories are being lost (this is probably not going
>> to be reproducible). If you can reproduce it, trying using pdb to see
>> what's going on inside walk().
>
--
Marcus Alves Grando
marcus(at)sbh.eng.br | Personal
mnag(at)FreeBSD.org | FreeBSD.org
More information about the Python-list
mailing list