From agaffney at gentoo.org Thu Jun 15 17:14:22 2006
From: agaffney at gentoo.org (Andrew Gaffney)
Date: Thu, 15 Jun 2006 10:14:22 -0500
Subject: [pyOpenSSL] pyopenssl and xmlrpclib
Message-ID: <4491794E.90405@gentoo.org>
I'm trying to integrate xmlrpclib and pyopenssl. I'm mostly there, but I'm
running into a problem:
Traceback (most recent call last):
File "./scirec.py", line 39, in ?
print client.say_hello()
File "/usr/lib/python2.4/xmlrpclib.py", line 1096, in __call__
return self.__send(self.__name, args)
File "/usr/lib/python2.4/xmlrpclib.py", line 1383, in __request
verbose=self.__verbose
File "/usr/lib/python2.4/xmlrpclib.py", line 1147, in request
return self._parse_response(h.getfile(), sock)
File "/usr/lib/python2.4/xmlrpclib.py", line 1276, in _parse_response
response = file.read(1024)
File "/usr/lib64/python2.4/socket.py", line 303, in read
data = self._sock.recv(recv_size)
OpenSSL.SSL.SysCallError: (9, 'Bad file descriptor')
My wrapper module code is below. Is there something I'm doing wrong?
import httplib
import xmlrpclib
import socket
from OpenSSL import SSL
class SecureXMLRPCClient(xmlrpclib.ServerProxy):
def __init__(self, host, port, client_cert, client_key, verify_cert_func=None):
xmlrpclib.ServerProxy.__init__(self, "https://" + host + ":" + str(port),
transport=SafeTransport(self.__host, client_cert, client_key, verify_cert_func),
encoding="utf-8", allow_none=True)
class SafeTransport(xmlrpclib.Transport):
def __init__(self, host, client_cert, client_key, verify_cert_func=None):
self.__host = host
self.__client_cert = client_cert
self.__client_key = client_key
self.__verify_cert_func = verify_cert_func
def make_connection(self, host):
host, extra_headers, x509 = self.get_host_info(host)
return HTTPS(host, self.__client_key, self.__client_cert, self.__verify_cert_func)
class HTTPS(httplib.HTTP):
def __init__(self, host='', key_file=None, cert_file=None, verify_cert_func=None):
self._setup(HTTPSConnection(host, key_file, cert_file, verify_cert_func))
# we never actually use these for anything, but we keep them
# here for compatibility with post-1.5.2 CVS.
self.key_file = key_file
self.cert_file = cert_file
class HTTPSConnection(httplib.HTTPConnection):
def __init__(self, host, key_file=None, cert_file=None, verify_cert_func=None):
httplib.HTTPConnection.__init__(self, host, None, None)
self.verify_cert_func = verify_cert_func
self.key_file = key_file
self.cert_file = cert_file
def connect(self):
# Initialize context
ctx = SSL.Context(SSL.SSLv23_METHOD)
if self.verify_cert_func:
ctx.set_verify(SSL.VERIFY_PEER, self.verify_cert_func) # Demand a certificate
ctx.use_privatekey_file(self.key_file)
ctx.use_certificate_file(self.cert_file)
# Set up client
# self.sock = SSL.Connection(ctx, socket.socket(socket.AF_INET,
socket.SOCK_STREAM))
real_sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
ssl_sock = SSL.Connection(ctx, real_sock)
ssl_sock.connect((self.host, self.port))
self.sock = SSLConnWrapper(ssl_sock)
print str(self.sock)
class SSLConnWrapper:
'''
Proxy class to provide makefile function on SSL Connection objects.
'''
def __init__(self, connection):
print "SSLConnWrapper.__init__()"
self.connection = connection
def __getattr__(self, function) :
return getattr(self.connection, function)
def makefile(self, mode, bufsize=0):
print "SSLConnWrapper.makefile()"
fo = socket._fileobject(self.connection) #, mode, bufsize)
return fo
def shutdown(self, _) :
return self.connection.shutdown()
--
Andrew Gaffney http://dev.gentoo.org/~agaffney/
Gentoo Linux Developer Installer Project
From dcbw at redhat.com Fri Jun 16 04:30:08 2006
From: dcbw at redhat.com (Dan Williams)
Date: Thu, 15 Jun 2006 22:30:08 -0400
Subject: [pyOpenSSL] pyopenssl and xmlrpclib
In-Reply-To: <4491794E.90405@gentoo.org>
References: <4491794E.90405@gentoo.org>
Message-ID: <1150425008.2506.5.camel@localhost.localdomain>
On Thu, 2006-06-15 at 10:14 -0500, Andrew Gaffney wrote:
> I'm trying to integrate xmlrpclib and pyopenssl. I'm mostly there, but I'm
> running into a problem:
If you need some example code of doing SSL and all things XMLRPC, take a
look at the files here:
http://cvs.fedora.redhat.com/viewcvs/extras-buildsys/common/?root=fedora
specifically SSLCommon.py and SSLConnection.py. Furthermore,
XMLRPCServerProxy.py, URLopener.py, FileTranser.py, FileDownloader.py,
and FileUploader.py might be of of interest as well.
These implementations use nonblocking sockets by default, which have a
few other complications, but you can safely turn that off by passing
timeout values of None, I think.
We've been running variations on this code for almost a year now in the
Fedora Extras build system, and it's been working pretty well.
Dan
> Traceback (most recent call last):
> File "./scirec.py", line 39, in ?
> print client.say_hello()
> File "/usr/lib/python2.4/xmlrpclib.py", line 1096, in __call__
> return self.__send(self.__name, args)
> File "/usr/lib/python2.4/xmlrpclib.py", line 1383, in __request
> verbose=self.__verbose
> File "/usr/lib/python2.4/xmlrpclib.py", line 1147, in request
> return self._parse_response(h.getfile(), sock)
> File "/usr/lib/python2.4/xmlrpclib.py", line 1276, in _parse_response
> response = file.read(1024)
> File "/usr/lib64/python2.4/socket.py", line 303, in read
> data = self._sock.recv(recv_size)
> OpenSSL.SSL.SysCallError: (9, 'Bad file descriptor')
>
>
> My wrapper module code is below. Is there something I'm doing wrong?
>
>
> import httplib
> import xmlrpclib
> import socket
> from OpenSSL import SSL
>
> class SecureXMLRPCClient(xmlrpclib.ServerProxy):
>
> def __init__(self, host, port, client_cert, client_key, verify_cert_func=None):
> xmlrpclib.ServerProxy.__init__(self, "https://" + host + ":" + str(port),
> transport=SafeTransport(self.__host, client_cert, client_key, verify_cert_func),
> encoding="utf-8", allow_none=True)
>
> class SafeTransport(xmlrpclib.Transport):
>
> def __init__(self, host, client_cert, client_key, verify_cert_func=None):
> self.__host = host
> self.__client_cert = client_cert
> self.__client_key = client_key
> self.__verify_cert_func = verify_cert_func
>
> def make_connection(self, host):
> host, extra_headers, x509 = self.get_host_info(host)
> return HTTPS(host, self.__client_key, self.__client_cert, self.__verify_cert_func)
>
> class HTTPS(httplib.HTTP):
>
> def __init__(self, host='', key_file=None, cert_file=None, verify_cert_func=None):
> self._setup(HTTPSConnection(host, key_file, cert_file, verify_cert_func))
>
> # we never actually use these for anything, but we keep them
> # here for compatibility with post-1.5.2 CVS.
> self.key_file = key_file
> self.cert_file = cert_file
>
> class HTTPSConnection(httplib.HTTPConnection):
>
> def __init__(self, host, key_file=None, cert_file=None, verify_cert_func=None):
> httplib.HTTPConnection.__init__(self, host, None, None)
> self.verify_cert_func = verify_cert_func
> self.key_file = key_file
> self.cert_file = cert_file
>
> def connect(self):
> # Initialize context
> ctx = SSL.Context(SSL.SSLv23_METHOD)
> if self.verify_cert_func:
> ctx.set_verify(SSL.VERIFY_PEER, self.verify_cert_func) # Demand a certificate
> ctx.use_privatekey_file(self.key_file)
> ctx.use_certificate_file(self.cert_file)
>
> # Set up client
> # self.sock = SSL.Connection(ctx, socket.socket(socket.AF_INET,
> socket.SOCK_STREAM))
> real_sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
> ssl_sock = SSL.Connection(ctx, real_sock)
> ssl_sock.connect((self.host, self.port))
> self.sock = SSLConnWrapper(ssl_sock)
> print str(self.sock)
>
> class SSLConnWrapper:
> '''
> Proxy class to provide makefile function on SSL Connection objects.
> '''
> def __init__(self, connection):
> print "SSLConnWrapper.__init__()"
> self.connection = connection
>
> def __getattr__(self, function) :
> return getattr(self.connection, function)
>
> def makefile(self, mode, bufsize=0):
> print "SSLConnWrapper.makefile()"
> fo = socket._fileobject(self.connection) #, mode, bufsize)
> return fo
>
> def shutdown(self, _) :
> return self.connection.shutdown()
>
From agaffney at gentoo.org Sun Jun 18 21:09:58 2006
From: agaffney at gentoo.org (Andrew Gaffney)
Date: Sun, 18 Jun 2006 14:09:58 -0500
Subject: [pyOpenSSL] pyopenssl and xmlrpclib
In-Reply-To: <1150425008.2506.5.camel@localhost.localdomain>
References: <4491794E.90405@gentoo.org>
<1150425008.2506.5.camel@localhost.localdomain>
Message-ID: <4495A506.1000607@gentoo.org>
Dan Williams wrote:
> On Thu, 2006-06-15 at 10:14 -0500, Andrew Gaffney wrote:
>> I'm trying to integrate xmlrpclib and pyopenssl. I'm mostly there, but I'm
>> running into a problem:
>
> If you need some example code of doing SSL and all things XMLRPC, take a
> look at the files here:
>
> http://cvs.fedora.redhat.com/viewcvs/extras-buildsys/common/?root=fedora
>
> specifically SSLCommon.py and SSLConnection.py. Furthermore,
> XMLRPCServerProxy.py, URLopener.py, FileTranser.py, FileDownloader.py,
> and FileUploader.py might be of of interest as well.
>
> These implementations use nonblocking sockets by default, which have a
> few other complications, but you can safely turn that off by passing
> timeout values of None, I think.
>
> We've been running variations on this code for almost a year now in the
> Fedora Extras build system, and it's been working pretty well.
>
> Dan
The RH implementation is close to what I had come up with, so I at least know I
was on the right track. That does get me a little farther, however, there's a
still a problem somewhere. I've modified my code to be as close to the RH code
and still have my own touches.
Currently, it looks like the request is sent, the headers are received back from
the server, and then the connection closes (but in a weird way). The 'closed'
var is getting set in the SSLConnection class, but stuff like select.select()
and socket.revc() don't seem to notice...they return "Bad file descriptor". I
added some code to SSLConnection's recv() to check self.closed and return None,
which gets me past that error. At that point, I get an exception from the XML
parsing code in xmlrpclib, apparently because no XML was ever sent back. The
following is a transcript of the session with the client code:
POST /RPC2 HTTP/1.0
Host: localhost:9876
User-Agent: xmlrpclib.py/1.0.1 (by www.pythonware.com)
Content-Type: text/xml
Content-Length: 103
say_hello
HTTP/1.0 200 OK
Server: BaseHTTP/0.3 Python/2.4.3
Date: Sun, 18 Jun 2006 18:16:09 GMT
Content-type: text/xml
Content-length: 131
and the socket is closed. This definitely seems to be a client-side problem. I
used 'openssl s_client' to make sure of this:
POST /RPC2 HTTP/1.0
Host: localhost:9876
User-Agent: xmlrpclib.py/1.0.1 (by www.pythonware.com)
Content-Type: text/xml
Content-Length: 103
say_hello
HTTP/1.0 200 OK
Server: BaseHTTP/0.3 Python/2.4.3
Date: Sun, 18 Jun 2006 18:28:16 GMT
Content-type: text/xml
Content-length: 131
hello
closed
The current version of my code can be found at
. If it makes any difference, I'm
using python-2.4.3, pyopenssl-0.6, and openssl-0.9.7j. Thanks for any help in
getting this working.
--
Andrew Gaffney http://dev.gentoo.org/~agaffney/
Gentoo Linux Developer Installer Project
From agaffney at gentoo.org Mon Jun 19 02:06:01 2006
From: agaffney at gentoo.org (Andrew Gaffney)
Date: Sun, 18 Jun 2006 19:06:01 -0500
Subject: [pyOpenSSL] pyopenssl and xmlrpclib
In-Reply-To: <4495A506.1000607@gentoo.org>
References: <4491794E.90405@gentoo.org> <1150425008.2506.5.camel@localhost.localdomain>
<4495A506.1000607@gentoo.org>
Message-ID: <4495EA69.2020405@gentoo.org>
Andrew Gaffney wrote:
> Dan Williams wrote:
>> On Thu, 2006-06-15 at 10:14 -0500, Andrew Gaffney wrote:
>>> I'm trying to integrate xmlrpclib and pyopenssl. I'm mostly there, but I'm
>>> running into a problem:
>> If you need some example code of doing SSL and all things XMLRPC, take a
>> look at the files here:
>>
>> http://cvs.fedora.redhat.com/viewcvs/extras-buildsys/common/?root=fedora
>>
>> specifically SSLCommon.py and SSLConnection.py. Furthermore,
>> XMLRPCServerProxy.py, URLopener.py, FileTranser.py, FileDownloader.py,
>> and FileUploader.py might be of of interest as well.
>>
>> These implementations use nonblocking sockets by default, which have a
>> few other complications, but you can safely turn that off by passing
>> timeout values of None, I think.
>>
>> We've been running variations on this code for almost a year now in the
>> Fedora Extras build system, and it's been working pretty well.
>>
>> Dan
>
> The RH implementation is close to what I had come up with, so I at least know I
> was on the right track. That does get me a little farther, however, there's a
> still a problem somewhere. I've modified my code to be as close to the RH code
> and still have my own touches.
>
> Currently, it looks like the request is sent, the headers are received back from
> the server, and then the connection closes (but in a weird way). The 'closed'
> var is getting set in the SSLConnection class, but stuff like select.select()
> and socket.revc() don't seem to notice...they return "Bad file descriptor". I
> added some code to SSLConnection's recv() to check self.closed and return None,
> which gets me past that error. At that point, I get an exception from the XML
> parsing code in xmlrpclib, apparently because no XML was ever sent back. The
> following is a transcript of the session with the client code:
Okay, I've gotten a little further in tracking down the problem. It appears to
be an issue with the close_refcount thing in SSLConnection. It increments when
makefile() is called and decrements when close() is called. Apparently, it looks
like when HTTPConnection.getresponse() closes the connection after instantiating
the response class, the response class never called makefile(), so the
close_refcount is decremented to 0 and closed, even though the response class is
using it. Any ideas on how to get around this?
--
Andrew Gaffney http://dev.gentoo.org/~agaffney/
Gentoo Linux Developer Installer Project
From agaffney at gentoo.org Mon Jun 19 02:24:17 2006
From: agaffney at gentoo.org (Andrew Gaffney)
Date: Sun, 18 Jun 2006 19:24:17 -0500
Subject: [pyOpenSSL] pyopenssl and xmlrpclib
In-Reply-To: <4495EA69.2020405@gentoo.org>
References: <4491794E.90405@gentoo.org> <1150425008.2506.5.camel@localhost.localdomain> <4495A506.1000607@gentoo.org>
<4495EA69.2020405@gentoo.org>
Message-ID: <4495EEB1.6020008@gentoo.org>
Andrew Gaffney wrote:
> Andrew Gaffney wrote:
>> Dan Williams wrote:
>>> On Thu, 2006-06-15 at 10:14 -0500, Andrew Gaffney wrote:
>>>> I'm trying to integrate xmlrpclib and pyopenssl. I'm mostly there, but I'm
>>>> running into a problem:
>>> If you need some example code of doing SSL and all things XMLRPC, take a
>>> look at the files here:
>>>
>>> http://cvs.fedora.redhat.com/viewcvs/extras-buildsys/common/?root=fedora
>>>
>>> specifically SSLCommon.py and SSLConnection.py. Furthermore,
>>> XMLRPCServerProxy.py, URLopener.py, FileTranser.py, FileDownloader.py,
>>> and FileUploader.py might be of of interest as well.
>>>
>>> These implementations use nonblocking sockets by default, which have a
>>> few other complications, but you can safely turn that off by passing
>>> timeout values of None, I think.
>>>
>>> We've been running variations on this code for almost a year now in the
>>> Fedora Extras build system, and it's been working pretty well.
>>>
>>> Dan
>> The RH implementation is close to what I had come up with, so I at least know I
>> was on the right track. That does get me a little farther, however, there's a
>> still a problem somewhere. I've modified my code to be as close to the RH code
>> and still have my own touches.
>>
>> Currently, it looks like the request is sent, the headers are received back from
>> the server, and then the connection closes (but in a weird way). The 'closed'
>> var is getting set in the SSLConnection class, but stuff like select.select()
>> and socket.revc() don't seem to notice...they return "Bad file descriptor". I
>> added some code to SSLConnection's recv() to check self.closed and return None,
>> which gets me past that error. At that point, I get an exception from the XML
>> parsing code in xmlrpclib, apparently because no XML was ever sent back. The
>> following is a transcript of the session with the client code:
>
>
>
> Okay, I've gotten a little further in tracking down the problem. It appears to
> be an issue with the close_refcount thing in SSLConnection. It increments when
> makefile() is called and decrements when close() is called. Apparently, it looks
> like when HTTPConnection.getresponse() closes the connection after instantiating
> the response class, the response class never called makefile(), so the
> close_refcount is decremented to 0 and closed, even though the response class is
> using it. Any ideas on how to get around this?
Okay, I finally have it "working". My above statement was actually wrong. It
looks like HTTPResponse *does* use makefile(), but HTTPConnection does *not*.
So, it calls close() which decrements the counter without ever calling
makefile() which increments the counter. The way I made it "work" was by
commenting all the calls to .close() and .shutdown(). This is technically
"wrong", but it gets around the faulty close_refcount logic. Can anyone suggest
a better way to do this?
--
Andrew Gaffney http://dev.gentoo.org/~agaffney/
Gentoo Linux Developer Installer Project
From misa at redhat.com Mon Jun 19 06:25:59 2006
From: misa at redhat.com (Mihai Ibanescu)
Date: Mon, 19 Jun 2006 00:25:59 -0400
Subject: [pyOpenSSL] pyopenssl and xmlrpclib
In-Reply-To: <4495EEB1.6020008@gentoo.org>
References: <4491794E.90405@gentoo.org>
<1150425008.2506.5.camel@localhost.localdomain>
<4495A506.1000607@gentoo.org> <4495EA69.2020405@gentoo.org>
<4495EEB1.6020008@gentoo.org>
Message-ID: <20060619042559.GA8251@abulafia.devel.redhat.com>
On Sun, Jun 18, 2006 at 07:24:17PM -0500, Andrew Gaffney wrote:
> Andrew Gaffney wrote:
>
> Okay, I finally have it "working". My above statement was actually wrong. It
> looks like HTTPResponse *does* use makefile(), but HTTPConnection does *not*.
> So, it calls close() which decrements the counter without ever calling
> makefile() which increments the counter. The way I made it "work" was by
> commenting all the calls to .close() and .shutdown(). This is technically
> "wrong", but it gets around the faulty close_refcount logic. Can anyone suggest
> a better way to do this?
The history behind that counter is (but this is all based on memory):
When the response object is created, httplib.HTTPConnection will use
makefile() to dup() the file descriptor it passes down to the HTTPResponse
object. This allows the code to be much cleaner, you can now close the main
connection object and just keep reading from the response one.
The problem with SSL is, it's not exposing real sockets. As such, there is no
dup() that works. The only way to achieve the same functionality is to play
the tricky counter thingie to keep the file descriptor open even after you
close the HTTPConnection. So, makefile() should increment the counter, and
close() should keep decrementing it (and do nothing else) until you hit zero,
at which point it should close the SSL connection.
For the same reason, select() works very unreliably on SSL sockets, because it
uses the underlying file descriptor/socket to check for incoming bytes. You
may (and I did) run into the situation where you get a couple of bytes on the
socket, but not enough to create input into the SSL buffer. So, select() would
return, but the read from the SSL socket would block.
Hope this helps.
Misa
From agaffney at gentoo.org Mon Jun 19 06:54:28 2006
From: agaffney at gentoo.org (Andrew Gaffney)
Date: Sun, 18 Jun 2006 23:54:28 -0500
Subject: [pyOpenSSL] pyopenssl and xmlrpclib
In-Reply-To: <20060619042559.GA8251@abulafia.devel.redhat.com>
References: <4491794E.90405@gentoo.org>
<1150425008.2506.5.camel@localhost.localdomain>
<4495A506.1000607@gentoo.org> <4495EA69.2020405@gentoo.org>
<4495EEB1.6020008@gentoo.org>
<20060619042559.GA8251@abulafia.devel.redhat.com>
Message-ID: <44962E04.2080804@gentoo.org>
Mihai Ibanescu wrote:
> On Sun, Jun 18, 2006 at 07:24:17PM -0500, Andrew Gaffney wrote:
>> Andrew Gaffney wrote:
>>
>> Okay, I finally have it "working". My above statement was actually wrong. It
>> looks like HTTPResponse *does* use makefile(), but HTTPConnection does *not*.
>> So, it calls close() which decrements the counter without ever calling
>> makefile() which increments the counter. The way I made it "work" was by
>> commenting all the calls to .close() and .shutdown(). This is technically
>> "wrong", but it gets around the faulty close_refcount logic. Can anyone suggest
>> a better way to do this?
>
> The history behind that counter is (but this is all based on memory):
> When the response object is created, httplib.HTTPConnection will use
> makefile() to dup() the file descriptor it passes down to the HTTPResponse
> object. This allows the code to be much cleaner, you can now close the main
> connection object and just keep reading from the response one.
>
> The problem with SSL is, it's not exposing real sockets. As such, there is no
> dup() that works. The only way to achieve the same functionality is to play
> the tricky counter thingie to keep the file descriptor open even after you
> close the HTTPConnection. So, makefile() should increment the counter, and
> close() should keep decrementing it (and do nothing else) until you hit zero,
> at which point it should close the SSL connection.
I completely understand the reason for the counter and the logic behind it. It's
just that it doesn't quite work as advertised :)
> For the same reason, select() works very unreliably on SSL sockets, because it
> uses the underlying file descriptor/socket to check for incoming bytes. You
> may (and I did) run into the situation where you get a couple of bytes on the
> socket, but not enough to create input into the SSL buffer. So, select() would
> return, but the read from the SSL socket would block.
I've removed all the select() and timeout stuff in my local code, since I don't
really care if it blocks (unlikely anyway with small xmlrpc requests).
--
Andrew Gaffney http://dev.gentoo.org/~agaffney/
Gentoo Linux Developer Installer Project