[issue36867] Make semaphore_tracker track other system resources

[Pierre Glaser]

Pierre Glaser <pierreglaser at msn.com> added the comment:

Shared memory segments are now tracked by the brand new resource_tracker!
Thanks Antoine for the review.

Does anyone have an opinion on introducing a public API for users to make the
resource_tracker track resources of their choice?

What We have in mind is:
- making public the existing resource_tracker.register/unregister
- adding a new function, resource_tracker.make_trackable(resource_type,
  cleanup_func), where:

    * resource_type is a string (an identifier that will be used each time a
      resource of the type resource_type needs tracking, via the call
      resource_tracker.register(resource_name, resource_type)
    * cleanup_func must be a callable taking a single string as argument (the
      name of the resource that needs tracking). This function will be called
      after the end of a process for reach resource of type resource_type the
      process did not clean properly

Under the hood, make_trackable simply populates resource_tracker._CLEANUP_FUNCS
with new items.

Here is a simple example:
	import os
	import resource_tracker
	import shutil
	from multiprocessing import util

	class ClassCreatingAFolder:
	    """Class where each instance creates a temporary folder.

	    Each temporary folder is supposed to exist for the duration of the instance
	    that created it.
	    """
	    def __init__(self, folder_name):
		self.folder_name = folder_name
		os.mkdir(folder_name)

		# any instance normally garbage-collected should remove its folder, and
		# notice the resource_tracker that its folder was correctly removed.
		util.Finalize(self, ClassCreatingAFolder.cleanup, args=(folder_name,))

		# If this session quits abruptly, the finalizer will not be called for
		# the instances of ClassCreatingAFolder that were still alive
		# before the shutdown. The resource_tracker comes into play, and removes
		# the folders associated to each of these resources.
		resource_tracker.register(
		    folder_name, # argument to shutil.rmtree
		    "ClassCreatingAFolder")

	    @staticmethod
	    def cleanup(folder_name):
		resource_tracker.unregister(folder_name, "ClassCreatingAFolder")
		shutil.rmtree(folder_name)

	# Tell the resource_tracker how to cleanup resources created by
	# ClassCreatingAFolder instances
	resource_tracker.make_trackable("ClassCreatingAFolder", shutil.rmtree)


Typical resources that can be made trackable include memmaped objects,
temporary folders. Our use-case is joblib that has its own mmap type that we
would like to track using the semaphore_tracker.

Any thoughts?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue36867>
_______________________________________