Python script not letting go of files

[Weatherby,Gerard]

Does the script exit when complete?

If you’re running on a Linux based system and have root access you can use lsof to see what processes have with files open. (Or use the psutil Python package).

From: Python-list <python-list-bounces+gweatherby=uchc.edu at python.org> on behalf of Mike Dewhirst <miked at dewhirst.com.au>
Date: Tuesday, November 29, 2022 at 2:20 AM
To: python-list at python.org <python-list at python.org>
Subject: Python script not letting go of files
I have a script which fetches a production site directly from a
Subversion repo using svn export

It runs a bunch of commands by calling this little method ...

def trycmd(cmd, log):
     retcode = -1
     ret = f"Trying {cmd}"
     try:
         retcode = os.system(cmd)
         ret = f"\n{cmd} -ok-> {retcode}"
     except Exception as err:
         ret = f"\n{cmd} -fail-> {err}"
     log.write(remove_password(ret))
     return retcode

This is the fetching script (omitting variables at the top) which
appears to be keeping a finger on files which Apache wants.

with open(fetchlog, 'a') as log:
     ret = f"\n\nFetching {tag}"
     log.write(ret)
     cmd = f"sudo rm -Rf {site_root}"
     if trycmd(cmd, log) == 0:
         cmd = f"sudo svn export --force --username {usr} --password
{pw} {svn_repo} {site_root}"
         if trycmd(cmd, log) == 0:
             # get any new dependencies
             cmd = f"sudo -H pip install -r {reqfile}"
             if trycmd(cmd, log) == 0:
                 # run any migrations shipped from the repo
                 cmd = f"sudo python3 {site_root}/manage.py migrate
--noinput --settings={settings}"
                 if trycmd(cmd, log) == 0:
                     # shouldn't find anything
                     cmd = f"sudo python3 {site_root}/manage.py
makemigrations --noinput --settings={settings}"
                     if trycmd(cmd, log) == 0:
                         # should have been done already
                         cmd = f"sudo python3 {site_root}/manage.py
migrate --noinput --settings={settings}"
                         if trycmd(cmd, log) == 0:
                             # remove all static files from their Apache dir
                             cmd = f"sudo rm -Rf {static_root}/*"
                             if trycmd(cmd, log) == 0:
                                 # copy all static files to the Apache
location
                                 cmd = f"sudo python3
{site_root}/manage.py collectstatic --noinput --settings={settings}"
                                 if trycmd(cmd, log) == 0:
                                     # set all permissions
                                     cmd = f"sudo {scripts}/perms_{host}.sh"
                                     if trycmd(cmd, log) == 0:
                                         cmd = "sudo service apache2
restart"
                                         if trycmd(cmd, log) == 0:
                                             ret = f"\nFinish {tag}\n\n"
                                             log.write(ret)
                                         else:
                                             print("Apache didn't restart")
                                     else:
                                         print("Didn't set permissions")
                                 else:
                                     print("Didn't collectstaic")
                             else:
                                 print("Didn't delete static files")
                         else:
                             print("Didn't migrate 2")
                     else:
                         print("Didn't makemigration")
                 else:
                     print("Didn't migrate 1")
             else:
                 print("Didn't install requirements")
         else:
             print("Didn't get source")
     else:
         print("Didn't remove site")
exit()

The problem I'm trying to fix is that after an Apache reload Apache
seems hellbent on filling up its scoreboard and running out of
resources. The kind folk on the Apache mailing list say something is
hogging the workers.

You can see the last operation above is an Apache restart. It should be
an Apache reload. Reload however lets Apache run out of resources.

Do any of you Python folks see any blunders in the above code along the
lines of not letting go of py files or static assets?

mod_wsgi is configured with 100% defaults.
mod_mpm_event conf per advice from the Apache mailing list is ...

<IfModule mpm_event_module>
         ServerLimit                     32
         StartServers                    16
         MinSpareThreads                 400
         MaxSpareThreads                 800
         ThreadLimit                     64
         ThreadsPerChild                 50
         AsyncRequestWorkerFactor        2
         MaxRequestWorkers               800
         MaxConnectionsPerChild          0
</IfModule>

Server Version: Apache/2.4.52 (Ubuntu 2022.04) OpenSSL/3.0.2
mod_wsgi/4.9.0 Python/3.10
Server MPM: event
Server Built: 2022-09-30T04:09:50

DigitalOcean droplet 8BG RAM and lightly loaded.

Many thanks for any hints.

Cheers

Mike


--
Signed email is an absolute defence against phishing. This email has
been signed with my private key. If you import my public key you can
automatically decrypt my signature and be sure it came from me. Just
ask and I'll send it to you. Your email software can handle signing.