Cannot allocate memory when using os.spawn for moving files

[Piet van Oostrum]

>>>>> Andreas <lists at hilboll.de> (A) wrote:

>A> Hello there,
>A> I have a problem moving files from my local harddrive to a NFS share using
>A> a Python script.
>A> The script is used to run a model which produces large (~500MB) binary
>A> output files. The model itself is a Fortran program, and I call it from my
>A> script using the line

>A> os.spawnlp(os.P_WAIT, 'time', 'time', OPT.binary)

>A> where OPT.binary is the filename to be run. This works flawlessly.

>A> However, in order not to clutter up my harddrive, I want to move the
>A> generated files to a network location after each model run. To save time, I
>A> use os.P_NOWAIT here:

>A> os.spawnlp(os.P_NOWAIT, 'mv', 'mv', LOCALFILENAME, REMOTEFILENAME)

>A> where LOCALFILENAME is some string like '/home/andreas/model.bin' and
>A> REMOTEFILENAME is some string like '/home/nfs/model-output/'.

>A> Here I have the problem that after about every 10th model run, I get an
>A> error saying

>A> Traceback (most recent call last):
>A>   File "../../../../pyiup/b3dctm/run_b3dctm.py", line 281, in <module>
>A>     main()
>A>   File "../../../../pyiup/b3dctm/run_b3dctm.py", line 71, in main
>A>     copy_model_output()
>A>   File "../../../../pyiup/b3dctm/run_b3dctm.py", line 162, in
>A> copy_model_output
>A>     os.spawnlp(os.P_NOWAIT, 'mv', 'mv', get_base_filename(RUNDATE) +
>A> .pdg',
>A> os.path.join(OPT.datastoragepath,OPT.modelname,OPT.runname,RUNDATE.strftime('%Y/
>A> %m')))
>A>   File "/usr/lib64/python2.5/os.py", line 635, in spawnlp
>A>     return spawnvp(mode, file, args)
>A>   File "/usr/lib64/python2.5/os.py", line 584, in spawnvp
>A>     return _spawnvef(mode, file, args, None, execvp)
>A>   File "/usr/lib64/python2.5/os.py", line 530, in _spawnvef
>A>     pid = fork()
>A> OSError: [Errno 12] Cannot allocate memory

>A> And my Python script dies.

>A> Is there anyone who can help me with this problem? I'm on Ubuntu 8.04 64bit
>A> with Python 2.5. I am willing to accept that the moving of the files does
>A> not always work, but then it would be really helpful if I could somehow
>A> prevent my script from dying and just try the moving again.

On my Mac OS X system the fork man page says:

     [ENOMEM]           There is insufficient swap space for the new process.
This is errno 12.

So apparently you have too many processes running or you have too little
swap space.

Maybe you can have a `top' process running or run `ps' occasionally to
see how many processes there are and how big they are. Moving files from
a local file system to a network file system means copying and maybe
the mv tries to copy the files in very large chunks.

What you could do is to put the filenames in a work queue when your
Fortran program has finished and run a separate thread that moves the
files one by one.
-- 
Piet van Oostrum <piet at cs.uu.nl>
URL: http://pietvanoostrum.com [PGP 8DAE142BE17999C4]
Private email: piet at vanoostrum.org