From snacktime at GMAIL.COM  Tue Feb  1 20:35:11 2005
From: snacktime at GMAIL.COM (snacktime)
Date: Tue, 1 Feb 2005 11:35:11 -0800
Subject: [PYTHON-CRYPTO] Memory leak in cherryssl.py
Message-ID: <1f060c4c0502011135519062b3@mail.gmail.com>

I was hoping that maybe someone could see an obvious reason why this
code causes a memory leak.  This is an ssl wrapper for cherrypy, a
small python webserver.  When I use it, every request consumes a bit
more memory and the process just grows and grows.  When using cherrypy
without the wrapper it works fine.  My first thought was that it might
be a problem with m2crypto since without the wrapper the problem goes
away, which is why I am sending this to the list as well as the author
or sslcherry.

The test code is very simple, it just returns a string.  You can see
the wrapper code (it's very small) at:

http://www.cherrypy.org/attachment/wiki/SslCherry/sslcherry.3.py

Following is the code I use to start cherrypy using sslcherry:
-----------------------------------------------------------
from cherrypy import cpg
import sslcherry

class OT:

  def process(self):

    return "OK"

  process.exposed = True

cpg.root = OT()
'''cpg.server.start()'''
sslcherry.start(configFile='ot.conf')




From michael at ELYT.COM  Fri Feb 25 04:47:45 2005
From: michael at ELYT.COM (Michael Dunstan)
Date: Fri, 25 Feb 2005 16:47:45 +1300
Subject: [PYTHON-CRYPTO] Yet another attempt to plug M2Crypto.SSL.Connection.Connection
Message-ID: <2f594a997a48121011f71034108a4b81@elyt.com>

Hi all,

Reading "M2Crypto woes"
http://www.artima.com/weblogs/viewpost.jsp?thread=95863 reminded me
that I should go back and see what happened with that patch that in
some fluke just happened to work for me.

Seems that this still exists even on the 0.15 branch. Okay - time to
take a closer look then. Attached is a trivial example demonstrating
the memory leak. When you run this it prints something like:

[<__main__.Leaky instance at 0x6de50>]

Meaning that we have an instance of the Leaky class that is now marked
as garbage and not collected. But the other two instances of non leaky
classes are not present. They have been garbage collected. A class with
the combination of assignment of a bound method and use of __del__
creates a cycle that can not be garbage collected. Well, that's how I'm
interpreting the behaviour I'm seeing. (With python 2.3.4.)

So what to do about this in the case of
M2Crypto.SSL.Connection.Connection? (Without resorting to wrapping
read/write with an if statement.) One way would be to provide a
different class for the different behaviour rather than use
setblocking(). For example:

class Connection:
     """A blocking SSL connection."""
     sendall = send = write = _write_bio
     recv = read = _read_bio

class NonBlockingConnection(Connection):
     """A non blocking SSL connection."""
     send = write = _write_nbio
     recv = read = _read_nbio

(I don't pretend to understand why sendall is not affected by the
different modes. Should it even exist for NonBlockingConnection?)

Not sure how that fits into the existing framework though.

Michael.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: leak.py
Type: application/octet-stream
Size: 1558 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-crypto/attachments/20050225/90782079/attachment.obj>

From gvanrossum at GMAIL.COM  Fri Feb 25 06:34:25 2005
From: gvanrossum at GMAIL.COM (Guido van Rossum)
Date: Thu, 24 Feb 2005 21:34:25 -0800
Subject: [PYTHON-CRYPTO] Yet another attempt to plug M2Crypto.SSL.Connection.Connection
In-Reply-To: <2f594a997a48121011f71034108a4b81@elyt.com>
References: <2f594a997a48121011f71034108a4b81@elyt.com>
Message-ID: <ca471dc205022421343adadc62@mail.gmail.com>

On Fri, 25 Feb 2005 16:47:45 +1300, Michael Dunstan <michael at elyt.com> wrote:
> Reading "M2Crypto woes"
> http://www.artima.com/weblogs/viewpost.jsp?thread=95863 reminded me
> that I should go back and see what happened with that patch that in
> some fluke just happened to work for me.
>
> Seems that this still exists even on the 0.15 branch. Okay - time to
> take a closer look then. Attached is a trivial example demonstrating
> the memory leak. When you run this it prints something like:
>
> [<__main__.Leaky instance at 0x6de50>]
>
> Meaning that we have an instance of the Leaky class that is now marked
> as garbage and not collected. But the other two instances of non leaky
> classes are not present. They have been garbage collected. A class with
> the combination of assignment of a bound method and use of __del__
> creates a cycle that can not be garbage collected. Well, that's how I'm
> interpreting the behaviour I'm seeing. (With python 2.3.4.)
>
> So what to do about this in the case of
> M2Crypto.SSL.Connection.Connection? (Without resorting to wrapping
> read/write with an if statement.) One way would be to provide a
> different class for the different behaviour rather than use
> setblocking(). For example:
>
> class Connection:
>      """A blocking SSL connection."""
>      sendall = send = write = _write_bio
>      recv = read = _read_bio
>
> class NonBlockingConnection(Connection):
>      """A non blocking SSL connection."""
>      send = write = _write_nbio
>      recv = read = _read_nbio
>
> (I don't pretend to understand why sendall is not affected by the
> different modes. Should it even exist for NonBlockingConnection?)
>
> Not sure how that fits into the existing framework though.

The best solution is usually not to use __del__. It is rarely
necessary. (About the only times where it's okay to have a __del__ is
when it's necessary is to delete an *external* resource such as an
integer file descriptor or a temporary file under all circumstances.)

In the case of a class whose instances are involved in cycles, __del__
will not be called by the collector, for very subtle reasons having to
do with not wanting to guess in which order the elements of the cycle
should be destroyed.

Assigning a bound method to an instance variable creates a cycle
because the bound method has a reference to the instance (its im_self
attribute).

Alas, it looks like the __del__ methods on Conection, Context and
Session are used to free the low-level BIO objects. So then the
solution may have to be to avoid cycles.

One tactic when dealing with something involved in a cycle that needs
to have a __del__ is to create a separate object that exists *just* to
have the __del__ method and is not involved in cycles. The collector
is smart enough to call the __del__ methods of objects that are not
involved in cycles but are kept alive by cycles. Example: A <-> B ->
C; when the cycle A <-> B is collected, C.__del__ is called, but not
A.__del__ or B.__del__.

BTW another leak in M2Crypto is the SSL Context created by
HTTPSConnection when none is passed in; SSL Context objects are only
deleted when their close() method is called, and nothing will ever
call close() on the ssl_ctx object contained in the HTTPSConnection
object.

There were some more leaks reported in the SWIG wrappers in the
followups to the above Blog url.

--
--Guido van Rossum (home page: http://www.python.org/~guido/)