From exarkun at divmod.com  Wed Apr  1 22:26:43 2009
From: exarkun at divmod.com (Jean-Paul Calderone)
Date: Wed, 1 Apr 2009 15:26:43 -0500
Subject: [pyOpenSSL] Python core dumps when using pyOpenSSL 0.8.0 and
 threads
In-Reply-To: <49C96BAD.4070602@egenix.com>
Message-ID: <20090401202643.24697.1468755343.divmod.quotient.2307@henry.divmod.com>

On Wed, 25 Mar 2009 00:24:29 +0100, "M.-A. Lemburg" <mal at egenix.com> wrote:
>On 2009-03-22 17:32, Jean-Paul Calderone wrote:
>> Hello all,
>>
>> I just pushed a branch to launchpad which may address the 0.8 thread-related
>> crashes.
>>
>> The bug, along with a reproduction script, is described here:
>>
>>   https://bugs.launchpad.net/pyopenssl/+bug/344815
>>
>> The branch, along with checkout instructions, which fixes the problem is here:
>>
>>   https://code.launchpad.net/~exarkun/pyopenssl/thread-crash
>>
>> If you're not in to bzr, then you can get just the revision which includes
>> the fix here:
>>
>>   http://bazaar.launchpad.net/~exarkun/pyopenssl/thread-crash/revision/98
>

Hi Marc,

Thanks for looking into this.  Sorry for the delayed response, PyCon has
been keeping me busy.

>This doesn't appear to cover all cases, esp. not if you have an
>application that uses recursion in callbacks.
>
>The patch causes an already existing thread state to get overwritten
>by another one. If the application starts returning from a deeper
>nesting, this will cause a wrong thread state to get restored (actually,
>the last one will get restored multiple times).

I think this will work out okay.  The thread state is actually a TLS object
anyway.  Any given thread can only have one thread state object.  So
PyEval_SaveThread will always return the same PyInterpreterState*.  At
least, I think this is how it happens.

Going into a bit more detail...

Say someone calls Connection.send.  MY_BEGIN_ALLOW_THREADS is invoked and
PyEval_SaveThread swaps NULL in as the new PyInterpreterState and gives
back the existing PyInterpreterState (call it S1).  This gets saved with
pyOpenSSL's _pyOpenSSL_tstate_key.  Any key there already is deleted.

  Now say one of the pyOpenSSL global callbacks gets invoked.  It starts by
  using MY_END_ALLOW_THREADS.  This loads S1 from _pyOpenSSL_tstate_key and
  passes it to PyEval_RestoreThread which swaps S1 in as the new
  PyInterpreterState.  Then it calls into some arbitrary Python code.

    Now say this arbitrary Python code calls another Connection's send
    method.  MY_BEGIN_ALLOW_THREADS is invoked again.  This deletes the
    current value of _pyOpenSSL_tstate_key and invokes PyEval_SaveThread
    again which swaps in NULL as the new PyInterpreterState and then saves
    the existing PyInterpreterState in _pyOpenSSL_tstate_key.  Since the
    previous step saved S1 as the PyInterpreterState, this is S1 again.

    Eventually the send finishes, MY_END_ALLOW_THREADS is invoked.  This
    the value of _pyOpenSSL_tstate_key which is S1, as described in the
    previous step.  S1 is passed to PyEval_RestoreThread and then the send
    call is done.

  Now control returns to the global callback code which finishes up and
  gets ready to return back to OpenSSL.  It does this by invoking
  MY_BEGIN_ALLOW_THREADS which clears _pyOpenSSL_tstate_key and then saves
  the result of PyEval_SaveThread (S1) in it.  Then control returns to
  OpenSSL.

Now the very top Connection.send finishes and the implementation of that
method in pyOpenSSL gets ready to return to the calling Python code.  It
invokes MY_END_ALLOW_THREADS which loads S1 from _pyOpenSSL_tstate_key and
passes it to PyEval_RestoreThread.  Then it returns to the calling Python
code.

I wasn't actually very confident that the patch was correct, but now that
I've actually written all that up and traced through the various levels of
thread control APIs in CPython, I'm relatively confident that the patch is
doing the right thing.  I also learned that there are some APIs in CPython
which do much of what pyOpenSSL is doing.  They aren't available in Python
2.3 though, so I'm not quite ready to switch over to them.

How does all this sound?

Jean-Paul



From exarkun at divmod.com  Thu Apr  2 00:56:34 2009
From: exarkun at divmod.com (Jean-Paul Calderone)
Date: Wed, 1 Apr 2009 17:56:34 -0500
Subject: [pyOpenSSL] Python core dumps when using pyOpenSSL 0.8.0 and
 threads
In-Reply-To: <20090401202643.24697.1468755343.divmod.quotient.2307@henry.divmod.com>
Message-ID: <20090401225634.24697.912225037.divmod.quotient.2357@henry.divmod.com>

On Wed, 1 Apr 2009 15:26:43 -0500, Jean-Paul Calderone <exarkun at divmod.com> wrote:
>
> [snip]
>
>I wasn't actually very confident that the patch was correct, but now that
>I've actually written all that up and traced through the various levels of
>thread control APIs in CPython, I'm relatively confident that the patch is
>doing the right thing.  I also learned that there are some APIs in CPython
>which do much of what pyOpenSSL is doing.  They aren't available in Python
>2.3 though, so I'm not quite ready to switch over to them.

Shortly after sending this message, I realized that it is also a perfectly
valid explanation for why the current trunk code also works correctly and
does not segfault.  However, the current trunk code does segfault, so
something must be wrong. ;)

After a few hours with gdb, I found the problem.  Python threads identifiers
are only unique for the lifetime of the thread.  As threads are destroyed
and others created, the identifiers may be re-used.  Since thread identifiers
are used as TLS keys, this means that thread A may set a TLS key then exit
and a later thread B may then be able to see the value for that TLS key.
This means that without the delete, thread B won't save the correct new
PyThreadState*, thus causing the bug.

Jean-Paul



From adrian at coresecurity.com  Thu Apr  2 04:39:21 2009
From: adrian at coresecurity.com (Adrian Manrique)
Date: Wed, 01 Apr 2009 23:39:21 -0300
Subject: [pyOpenSSL] Python core dumps when using pyOpenSSL 0.8.0 and
 threads
In-Reply-To: <20090401225634.24697.912225037.divmod.quotient.2357@henry.divmod.com>
References: <20090401225634.24697.912225037.divmod.quotient.2357@henry.divmod.com>
Message-ID: <49D42559.7020206@coresecurity.com>

Jean-Paul Calderone wrote:
> On Wed, 1 Apr 2009 15:26:43 -0500, Jean-Paul Calderone <exarkun at divmod.com> wrote:
>   
>> [snip]
>>
>> I wasn't actually very confident that the patch was correct, but now that
>> I've actually written all that up and traced through the various levels of
>> thread control APIs in CPython, I'm relatively confident that the patch is
>> doing the right thing.  I also learned that there are some APIs in CPython
>> which do much of what pyOpenSSL is doing.  They aren't available in Python
>> 2.3 though, so I'm not quite ready to switch over to them.
>>     
>
> Shortly after sending this message, I realized that it is also a perfectly...
:))) what you were doing 3/18/09 ?? :)))

a/



From wgheath at gmail.com  Tue Apr  7 19:29:24 2009
From: wgheath at gmail.com (William Heath)
Date: Tue, 7 Apr 2009 10:29:24 -0700
Subject: [pyOpenSSL] Windows ssl socket closing due to timeout but client
	not notified until trying to send
Message-ID: <f08ddf990904071029v811f900v818e9c04efa126f3@mail.gmail.com>

Hello,
I am using openssl on windows with wxwidgets and python.  I download the
openssl libraries etc... to achieve this at:

http://gnuwin32.sourceforge.net/packages/openssl.htm<http://gnuwin32.sourceforge.net/packages/openssl.htm%3C/li%3E>
http://pyopenssl.sourceforge.net/<http://pyopenssl.sourceforge.net/%3C/li%3E>

My problem is that windows appears to close the ssl socket due to inactivity
but the client doesn't get that message until it attempts to send some data
through the socket.  Is this a known bug and is there an easy fix?

-Tim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pyopenssl-users/attachments/20090407/33723e11/attachment.html>

From patf at well.com  Fri Apr 17 19:18:14 2009
From: patf at well.com (patrick flaherty)
Date: Fri, 17 Apr 2009 10:18:14 -0700
Subject: [pyOpenSSL] Trying to build pyOpenSSL on windows (mingw vs MS
	compiler).
Message-ID: <49E8B9D6.9070502@well.com>

Hi,

Whether I use a cmd window (DOS) or a cygwin bash shell, I seem in both 
cases to get the MS compiler.

This compiles the first two modules:  crypto.obj and x509.obj.  But  it 
(the MS compiler [VS 2008]) chokes on x509name.c
with

x509name.c(16): error C2133: 'crypto_X509Name_methods' : unknown size.

My guess is that what I need that I need to be using mingw instead of 
the MS compiler.

But what I can't figure out is how to get setup.py to _use_ mingw 
instead of the VS 2008 compiler?

thanx - pat

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pyopenssl-users/attachments/20090417/9c99ee06/attachment.html>

From patf at well.com  Fri Apr 17 21:00:48 2009
From: patf at well.com (patrick flaherty)
Date: Fri, 17 Apr 2009 12:00:48 -0700
Subject: [pyOpenSSL] Trying to build pyOpenSSL on windows (mingw vs MS
 compiler).
In-Reply-To: <49E8B9D6.9070502@well.com>
References: <49E8B9D6.9070502@well.com>
Message-ID: <49E8D1E0.2060806@well.com>


Use the web - right?

This compiled all the modules (but with a number of warnings):

(from a cmd window)   setup.py build_ext -I c:\openssl\include -c mingw32

But then the gcc ( gcc -mno-cygwin ) link failed with:

gcc: c:\python25\libs\ssleay32.a: No such file or directory.

Actually I'd looked around the web already for ssleay32.a and found lots 
of references but didn't seem to find the library as such?

Sorry for all the fumbling around.

pat



patrick flaherty wrote:
> Hi,
>
> Whether I use a cmd window (DOS) or a cygwin bash shell, I seem in 
> both cases to get the MS compiler.
>
> This compiles the first two modules:  crypto.obj and x509.obj.  But  
> it (the MS compiler [VS 2008]) chokes on x509name.c
> with
>
> x509name.c(16): error C2133: 'crypto_X509Name_methods' : unknown size.
>
> My guess is that what I need that I need to be using mingw instead of 
> the MS compiler.
>
> But what I can't figure out is how to get setup.py to _use_ mingw 
> instead of the VS 2008 compiler?
>
> thanx - pat
>
> ------------------------------------------------------------------------
>
> ------------------------------------------------------------------------------
> Stay on top of everything new and different, both inside and 
> around Java (TM) technology - register by April 22, and save
> $200 on the JavaOne (SM) conference, June 2-5, 2009, San Francisco.
> 300 plus technical and hands-on sessions. Register today. 
> Use priority code J9JMT32. http://p.sf.net/sfu/p
> ------------------------------------------------------------------------
>
> _______________________________________________
> pyopenssl-list mailing list
> pyopenssl-list at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/pyopenssl-list
>   

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pyopenssl-users/attachments/20090417/6c223350/attachment.html>

From wgheath at gmail.com  Fri Apr 17 21:04:35 2009
From: wgheath at gmail.com (William Heath)
Date: Fri, 17 Apr 2009 12:04:35 -0700
Subject: [pyOpenSSL] Trying to build pyOpenSSL on windows (mingw vs MS
	compiler).
In-Reply-To: <49E8D1E0.2060806@well.com>
References: <49E8B9D6.9070502@well.com> <49E8D1E0.2060806@well.com>
Message-ID: <f08ddf990904171204qa0e7ccdla9151e7fe827ae14@mail.gmail.com>

I would love to know the answer to this too.  Keep it up!

On Fri, Apr 17, 2009 at 12:00 PM, patrick flaherty <patf at well.com> wrote:

>
> Use the web - right?
>
> This compiled all the modules (but with a number of warnings):
>
> (from a cmd window)   setup.py build_ext -I c:\openssl\include -c mingw32
>
> But then the gcc ( gcc -mno-cygwin ) link failed with:
>
> gcc: c:\python25\libs\ssleay32.a: No such file or directory.
>
> Actually I'd looked around the web already for ssleay32.a and found lots of
> references but didn't seem to find the library as such?
>
> Sorry for all the fumbling around.
>
> pat
>
>
>
> patrick flaherty wrote:
>
> Hi,
>
> Whether I use a cmd window (DOS) or a cygwin bash shell, I seem in both
> cases to get the MS compiler.
>
> This compiles the first two modules:  crypto.obj and x509.obj.  But  it
> (the MS compiler [VS 2008]) chokes on x509name.c
> with
>
> x509name.c(16): error C2133: 'crypto_X509Name_methods' : unknown size.
>
> My guess is that what I need that I need to be using mingw instead of the
> MS compiler.
>
> But what I can't figure out is how to get setup.py to *use* mingw instead
> of the VS 2008 compiler?
>
> thanx - pat
>
> ------------------------------
> ------------------------------------------------------------------------------
> Stay on top of everything new and different, both inside and
> around Java (TM) technology - register by April 22, and save
> $200 on the JavaOne (SM) conference, June 2-5, 2009, San Francisco.
> 300 plus technical and hands-on sessions. Register today.
> Use priority code J9JMT32. http://p.sf.net/sfu/p
>
> ------------------------------
> _______________________________________________
> pyopenssl-list mailing listpyopenssl-list at lists.sourceforge.nethttps://lists.sourceforge.net/lists/listinfo/pyopenssl-list
>
>
>
>
> ------------------------------------------------------------------------------
> Stay on top of everything new and different, both inside and
> around Java (TM) technology - register by April 22, and save
> $200 on the JavaOne (SM) conference, June 2-5, 2009, San Francisco.
> 300 plus technical and hands-on sessions. Register today.
> Use priority code J9JMT32. http://p.sf.net/sfu/p
> _______________________________________________
> pyopenssl-list mailing list
> pyopenssl-list at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/pyopenssl-list
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pyopenssl-users/attachments/20090417/fce95cad/attachment.html>

From patf at well.com  Sat Apr 18 01:53:54 2009
From: patf at well.com (patrick flaherty)
Date: Fri, 17 Apr 2009 16:53:54 -0700
Subject: [pyOpenSSL] Trying to build pyOpenSSL on windows (mingw vs MS
	compiler).
Message-ID: <49E91692.4090406@well.com>


OK - I've got it build and installed.

The library you need (and that I didn't have) is: ssleay32.a.  There's a 
MINGW subdirectory in the 32-bit
version of the OpenSSL kit here:

http://www.slproweb.com/products/Win32OpenSSL.html

_But the same library_ is not to be found in the 64 bit version.  I have 
a 64-bit machine and so first downloaded
the 64 bit version.  Eventually tried downloading the 32 bit version and 
found what I needed.

You put this into c:\pythonXX\libs.  The current setup.py expects this 
in c:\python25, but I have 2.6 so I needed
to modify the reference in setup.py appropriately.

build_ext then fully works and makes three .pyd files (dynamically 
loadable python modules/libraries I believe).

python setup.py build   (without the ext) then works.

And python install also works.

So then I turned my attention to the example programs.  I try to run the 
one to make the certificates and it crashes
upon startup because it can't import module tsafe.  tsafe = thread safe.

So where do I find this?  Am again looking around the web, but, so far, 
no joy.

pat

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pyopenssl-users/attachments/20090417/7c545c8b/attachment.html>

From exarkun at divmod.com  Mon Apr 20 15:51:02 2009
From: exarkun at divmod.com (Jean-Paul Calderone)
Date: Mon, 20 Apr 2009 09:51:02 -0400
Subject: [pyOpenSSL] Trying to build pyOpenSSL on windows (mingw vs MS
 compiler).
In-Reply-To: <49E91692.4090406@well.com>
Message-ID: <20090420135102.24697.617227132.divmod.quotient.9784@henry.divmod.com>

On Fri, 17 Apr 2009 16:53:54 -0700, patrick flaherty <patf at well.com> wrote:
>
>OK - I've got it build and installed.

Hooray.

> [snip]
>
>_But the same library_ is not to be found in the 64 bit version.  I have a 
>64-bit machine and so first downloaded
>the 64 bit version.  Eventually tried downloading the 32 bit version and 
>found what I needed.
>
>You put this into c:\pythonXX\libs.  The current setup.py expects this in 
>c:\python25, but I have 2.6 so I needed
>to modify the reference in setup.py appropriately.

I haven't done any testing of pyOpenSSL on 64 bit Windows.  So maybe this
works, but I have no idea. :)

> [snip]
>So then I turned my attention to the example programs.  I try to run the one 
>to make the certificates and it crashes
>upon startup because it can't import module tsafe.  tsafe = thread safe.
>
>So where do I find this?  Am again looking around the web, but, so far, no 
>joy.

What do you mean "it crashes"?  Do you have the tsafe module?  It should
have been installed with the rest of pyOpenSSL.

Jean-Paul



From patf at well.com  Mon Apr 20 22:39:47 2009
From: patf at well.com (patrick flaherty)
Date: Mon, 20 Apr 2009 13:39:47 -0700
Subject: [pyOpenSSL] DLL load failed.
Message-ID: <49ECDD93.1010606@well.com>


I was /assuming/ that the import problem was the last of the four 
modules - tsafe - since I could see that the other
3 pyd files were visibly there.  And as I looked for tsafe last Friday, 
I didn't find it.

But as I look more today, yes, tsafe.py/pyc is in place as well.

I seem to be having problems with finding any of these modules (upon 
import that is).  I've tried many variants (of reconfiguring things in 
various ways), but
I keep getting:

L:\MyID\Python\pyOpenSSL\examples>mk_simple_certs.py
Traceback (most recent call last):
  File "L:\pf\Python\pyOpenSSL\examples\mk_simple_certs.py", line 5, in 
<module>

    from OpenSSL import crypto
  File "c:\python26\Lib\site-packages\OpenSSL\__init__.py", line 8, in 
<module>
    import SSL, crypto, rand, tsafe
ImportError: DLL load failed: The specified module could not be found.

L:\MyID\Python\pyOpenSSL\examples>

Needless to say, I've googled the error message - get many hits.  Have 
tried several of the solutions that I thought might be candidates, but
mk_simple_certs.py so far refuses to cooperate.

pat
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pyopenssl-users/attachments/20090420/5d33ab69/attachment.html>

From exarkun at divmod.com  Mon Apr 20 22:46:09 2009
From: exarkun at divmod.com (Jean-Paul Calderone)
Date: Mon, 20 Apr 2009 16:46:09 -0400
Subject: [pyOpenSSL] DLL load failed.
In-Reply-To: <49ECDD93.1010606@well.com>
Message-ID: <20090420204609.24697.2122658136.divmod.quotient.9849@henry.divmod.com>

On Mon, 20 Apr 2009 13:39:47 -0700, patrick flaherty <patf at well.com> wrote:
>
>I was /assuming/ that the import problem was the last of the four modules - 
>tsafe - since I could see that the other
>3 pyd files were visibly there.  And as I looked for tsafe last Friday, I 
>didn't find it.
>
>But as I look more today, yes, tsafe.py/pyc is in place as well.
>
>I seem to be having problems with finding any of these modules (upon import 
>that is).  I've tried many variants (of reconfiguring things in various 
>ways), but
>I keep getting:
>
>L:\MyID\Python\pyOpenSSL\examples>mk_simple_certs.py
>Traceback (most recent call last):
>  File "L:\pf\Python\pyOpenSSL\examples\mk_simple_certs.py", line 5, in 
><module>
>
>    from OpenSSL import crypto
>  File "c:\python26\Lib\site-packages\OpenSSL\__init__.py", line 8, in 
><module>
>    import SSL, crypto, rand, tsafe
>ImportError: DLL load failed: The specified module could not be found.

Ah.  This is because of the OpenSSL library not being loadable.  If you
copy ssleay32.a into the Python library directory, C:\python26\Lib\ I
think, then Windows will find it and the error should be resolved.

I hope that the next release of pyOpenSSL will include a Windows installer
which deals with this automatically.

Let me know how it goes,

Jean-Paul



From patf at well.com  Mon Apr 20 23:57:00 2009
From: patf at well.com (patrick flaherty)
Date: Mon, 20 Apr 2009 14:57:00 -0700
Subject: [pyOpenSSL] DLL load failed.
Message-ID: <49ECEFAC.5090506@well.com>


Hello Jean-Paul

> Ah.  This is because of the OpenSSL library not being loadable.  If you
> copy ssleay32.a into the Python library directory, C:\python26\Lib\ I
> think, then Windows will find it and the error should be resolved.
>
> I hope that the next release of pyOpenSSL will include a Windows installer
> which deals with this automatically.
>
> Let me know how it goes,
> 
> Jean-Paul

No (alas) that didn't work.

So after I'd copied ssleay32.a to c:\python26\lib, I then had it both there
and in c:\python26\libs (where you need it for the build).

c:\python26\lib is, of course, in my Windows PATH.

thanx - pat






From patf at well.com  Tue Apr 21 19:58:41 2009
From: patf at well.com (patrick flaherty)
Date: Tue, 21 Apr 2009 10:58:41 -0700
Subject: [pyOpenSSL] DLL load failed.
Message-ID: <49EE0951.2010200@well.com>


well dumpbin recognizes, say, ssl.pyd as a to-at-least-some-extent
well-formed DLL.

 > C:\Python26\Lib\site-packages\OpenSSL>dumpbin ssl.pyd
 > Microsoft (R) COFF/PE Dumper Version 9.00.21022.08
 > Copyright (C) Microsoft Corporation.  All rights reserved.
 >
 >
 > Dump of file ssl.pyd
 >
 > File Type: DLL
 >
 > Summary
 >
 >       1000 .bss
 >       4000 .data
 >       1000 .edata
 >       2000 .idata
 >       2000 .rdata
 >       1000 .reloc
 >       4000 .text
 >
 > C:\Python26\Lib\site-packages\OpenSSL>

But even when I put the prebuilt ( the pyopenssl-win download )
module into place, the DLLs (not unsurprisingly) refused to import.

So how does this work and what's the problem?

thanx - pat








From exarkun at divmod.com  Tue Apr 21 20:27:31 2009
From: exarkun at divmod.com (Jean-Paul Calderone)
Date: Tue, 21 Apr 2009 14:27:31 -0400
Subject: [pyOpenSSL] DLL load failed.
In-Reply-To: <49EE0951.2010200@well.com>
Message-ID: <20090421182731.24697.83314322.divmod.quotient.10236@henry.divmod.com>

On Tue, 21 Apr 2009 10:58:41 -0700, patrick flaherty <patf at well.com> wrote:
>
>well dumpbin recognizes, say, ssl.pyd as a to-at-least-some-extent
>well-formed DLL.
>
> > C:\Python26\Lib\site-packages\OpenSSL>dumpbin ssl.pyd
> > Microsoft (R) COFF/PE Dumper Version 9.00.21022.08
> > Copyright (C) Microsoft Corporation.  All rights reserved.
> >
> >
> > Dump of file ssl.pyd
> >
> > File Type: DLL
> >
> > Summary
> >
> >       1000 .bss
> >       4000 .data
> >       1000 .edata
> >       2000 .idata
> >       2000 .rdata
> >       1000 .reloc
> >       4000 .text
> >
> > C:\Python26\Lib\site-packages\OpenSSL>
>
>But even when I put the prebuilt ( the pyopenssl-win download )
>module into place, the DLLs (not unsurprisingly) refused to import.
>
>So how does this work and what's the problem?

Windows is still largely a mystery to me, so I can't tell you how it works.
I do think that I understand the problem, which is that the OpenSSL library
isn't being found by Windows.  The pyOpenSSL library ssl.pyd is linked
against it.  Since you said you already have the OpenSSL libraries in the
Python libs directory, I'm not sure why it's not being found.

Jean-Paul



From patf at well.com  Tue Apr 21 21:53:31 2009
From: patf at well.com (patrick flaherty)
Date: Tue, 21 Apr 2009 12:53:31 -0700
Subject: [pyOpenSSL] DLL load failed.
Message-ID: <49EE243B.6050704@well.com>

Hello Jean-Paul

> Windows is still largely a mystery to me, so I can't tell you how it works.
> I do think that I understand the problem, which is that the OpenSSL library
> isn't being found by Windows.  The pyOpenSSL library ssl.pyd is linked
> against it.  Since you said you already have the OpenSSL libraries in the
> Python libs directory, I'm not sure why it's not being found.
>
> Jean-Paul

Here's a dumpbin of ssl.pyd's dependencies:

> Section contains the following imports: WS2_32.DLL 618CD30C Import 
> Address Table 618CD0A4 Import Name Table 0 time date stamp 0 Index of 
> first forwarder reference 1A WSAGetLastError KERNEL32.dll 618CD318 
> Import Address Table 618CD0B0 Import Name Table 0 time date stamp 0 
> Index of first forwarder reference 1 AddAtomA AF FindAtomA DC 
> GetAtomNameA msvcr71.dll 618CD32C Import Address Table 618CD0C4 Import 
> Name Table 0 time date stamp 0 Index of first forwarder reference 38 
> __dllonexit BD _errno 26B fflush 27A free 2AD malloc 2DD strncpy 
> msvcrt.dll 618CD34C Import Address Table 618CD0E4 Import Name Table 0 
> time date stamp 0 Index of first forwarder reference 6F _assert 215 
> abort 27A memset ssleay32.dll 618CD360 Import Address Table 618CD0F8 
> Import Name Table 0 time date stamp 0 Index of first forwarder 
> reference 9 ERR_load_SSL_strings 14 SSL_CTX_check_private_key 15 
> SSL_CTX_ctrl 17 SSL_CTX_free 18 SSL_CTX_get_cert_store 1F 
> SSL_CTX_get_timeout 21 SSL_CTX_get_verify_depth 22 
> SSL_CTX_get_verify_mode 23 SSL_CTX_load_verify_locations 24 
> SSL_CTX_new 2F SSL_CTX_set_cipher_list 30 SSL_CTX_set_client_CA_list 
> 34 SSL_CTX_set_default_passwd_cb 35 
> SSL_CTX_set_default_passwd_cb_userdata 37 SSL_CTX_set_ex_data 39 
> SSL_CTX_set_info_callback 3D SSL_CTX_set_session_id_context 3F 
> SSL_CTX_set_timeout 44 SSL_CTX_set_verify 45 SSL_CTX_set_verify_depth 
> 46 SSL_CTX_use_PrivateKey 48 SSL_CTX_use_PrivateKey_file 4C 
> SSL_CTX_use_certificate 4E SSL_CTX_use_certificate_chain_file 4F 
> SSL_CTX_use_certificate_file 6B SSL_ctrl 6C SSL_do_handshake 6F 
> SSL_free 73 SSL_get_cipher_list 7A SSL_get_error 7B SSL_get_ex_data 82 
> SSL_get_peer_certificate 8B SSL_get_shutdown 95 SSL_library_init 96 
> SSL_load_client_CA_file 98 SSL_new 9A SSL_pending 9B SSL_read 9C 
> SSL_renegotiate 9D SSL_renegotiate_pending A1 SSL_set_accept_state A5 
> SSL_set_connect_state A6 SSL_set_ex_data A7 SSL_set_fd B1 
> SSL_set_shutdown BB SSL_shutdown BE SSL_state_string_long C9 SSL_want 
> CA SSL_write CC SSLv23_method CF SSLv2_method D2 SSLv3_method D5 
> TLSv1_method libeay32.dll 618CD43C Import Address Table 618CD1D4 
> Import Name Table 0 time date stamp 0 Index of first forwarder 
> reference F8 BIO_free 10F BIO_new_file 274 DH_free 3C7 
> ERR_func_error_string 3C9 ERR_get_error 3D0 ERR_lib_error_string 3EE 
> ERR_peek_error 3F9 ERR_reason_error_string 606 PEM_read_bio_DHparams 
> 914 X509_STORE_CTX_get_current_cert 915 X509_STORE_CTX_get_error 916 
> X509_STORE_CTX_get_error_depth 917 X509_STORE_CTX_get_ex_data 923 
> X509_STORE_CTX_set_error 969 X509_dup 96D X509_free python25.dll 
> 618CD484 Import Address Table 618CD21C Import Name Table 0 time date 
> stamp 0 Index of first forwarder reference 7 PyArg_ParseTuple 1E 
> PyCObject_AsVoidPtr 1F PyCObject_FromVoidPtr 24 PyCObject_Type 27 
> PyCallable_Check 57 PyDict_GetItemString 5F PyDict_SetItemString 68 
> PyErr_Clear 6A PyErr_ExceptionMatches 6E PyErr_NewException 6F 
> PyErr_NoMemory 71 PyErr_Occurred 82 PyErr_SetNone 83 PyErr_SetObject 
> 84 PyErr_SetString 8F PyEval_CallObjectWithKeywords A1 
> PyEval_RestoreThread A2 PyEval_SaveThread A8 PyExc_AttributeError BC 
> PyExc_NotImplementedError C1 PyExc_RuntimeError CA PyExc_TypeError D2 
> PyExc_ValueError 10D PyImport_ImportModule 115 PyInt_AsLong 11A 
> PyInt_FromLong 120 PyInt_Type 128 PyList_Append 12E PyList_New 13E 
> PyLong_FromLong 162 PyModule_AddIntConstant 163 PyModule_AddObject 165 
> PyModule_GetDict 1A8 PyObject_AsFileDescriptor 1B9 PyObject_GC_Del 1BA 
> PyObject_GC_Track 1BB PyObject_GC_UnTrack 1BF PyObject_GetAttrString 
> 1C9 PyObject_IsTrue 227 PyString_AsString 232 PyString_FromString 233 
> PyString_FromStringAndSize 238 PyString_Size 239 PyString_Type 254 
> PyThread_create_key 259 PyThread_get_key_value 25E 
> PyThread_set_key_value 268 PyTuple_GetItem 271 PyType_IsSubtype 273 
> PyType_Type 2D6 Py_BuildValue 2E2 Py_FindMethod 2F4 Py_InitModule4 33C 
> _PyObject_GC_New 34D _PyString_Resize 377 _Py_NoneStruct 37E 
> _Py_TrueStruct 380 _Py_ZeroStruct

pat



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pyopenssl-users/attachments/20090421/88cbb5a9/attachment.html>

From patf at well.com  Tue Apr 21 22:05:12 2009
From: patf at well.com (patrick flaherty)
Date: Tue, 21 Apr 2009 13:05:12 -0700
Subject: [pyOpenSSL] DLL load failed.
Message-ID: <49EE26F8.30003@well.com>


well it lost all its formatting when it arrived here in the mailing list 
archive on
sourceforge

but I see that if one clicks on the 'Mail as HTML' part of Attachments, 
it will bring
up my message, properly formatted, in a browser window.

How does one actually respond here so as to maintain the same thread?

pat




From patf at well.com  Tue Apr 21 23:40:37 2009
From: patf at well.com (patrick flaherty)
Date: Tue, 21 Apr 2009 14:40:37 -0700
Subject: [pyOpenSSL] DLL load failed.
Message-ID: <49EE3D55.9070102@well.com>


Figured it out at least this far - it's the 64 bit-ness.

I installed the OpenSSL prebuilt module into a python 2.5 installation 
on a 32 bit windows machine.

mk_simple_certs.py ran successfully and produced the certs (in the 
simple subdirectory).

So I think it fair to say this all of this stuff will work on a 32-bit 
machine - but not a 64-bit machine.

I'm considering installing a 3.x version of python to see if this 
handles 64-bit windows better (in particular how python loads DLLs).

Note that on a 64-bit machine you have two versions of the system32 
directory (where Windows DLLs are often put):

c:\windows\system32

c:\windows\sysWOW64

Confusingly the first (which is the same as the old 32 bit name) is the 
64 bit location.  WOW64 is actually the 32 bit location.

pat




From exarkun at divmod.com  Tue Apr 28 14:28:37 2009
From: exarkun at divmod.com (Jean-Paul Calderone)
Date: Tue, 28 Apr 2009 08:28:37 -0400
Subject: [pyOpenSSL] [ANN] pyOpenSSL 0.9
In-Reply-To: 0
Message-ID: <20090428122837.24697.1129485049.divmod.quotient.12755@henry.divmod.com>

I'm happy to announce the release of pyOpenSSL 0.9. This release includes
several new features and a very important bug fix:

  * APIs have been introduced to get and set the version of an X509 request
  * Contexts now support loading CA certificates for verification from a
    specified directory or from the platform's default certificate store
  * DTLS-related constants, OP_NO_QUERY_MTU, OP_COOKIE_EXCHANGE, and
    OP_NO_TICKET are now exposed
  * X509Extension now has the `get_short_name? method for retrieving the
    short form of the extension type
  * It is now possible to create extensions which use any of the three
    possible OpenSSL implementation approaches
  * FILETYPE_TEXT can be used to dump keys, certificate, and CSRs in text
    format
  * Various compiler warnings have been fixed
  * A bug triggered by almost all uses of pyOpenSSL from multiple threads
    and leading to a crash has been fixed

Many thanks to numerous people who contributed patches to this release.

You can find pyOpenSSL 0.9 in the downloads area of the SourceForge project
page:

    https://sourceforge.net/project/showfiles.php?group_id=31249

Please use Launchpad to file any bug reports:

    https://bugs.launchpad.net/pyopenssl

Jean-Paul Calderone



From patf at well.com  Thu Apr 30 20:37:55 2009
From: patf at well.com (patrick flaherty)
Date: Thu, 30 Apr 2009 11:37:55 -0700
Subject: [pyOpenSSL] DLL load failed.
In-Reply-To: <20090420204609.24697.2122658136.divmod.quotient.9849@henry.divmod.com>
References: <20090420204609.24697.2122658136.divmod.quotient.9849@henry.divmod.com>
Message-ID: <49F9F003.2070902@well.com>


Hello Jean-Paul,

As you can see below I copied ssleay32.a to c:\python25\Lib, however the 
build (link/ld) fails.

>
> L:\pf\Python\pyOpenSSL>dir c:\python25\lib\ss*.a
>  Volume in drive C has no label.
>  Volume Serial Number is F818-DFCA
>
>  Directory of c:\python25\lib
>
> 03/27/2009  06:32 AM           171,812 ssleay32.a
>                1 File(s)        171,812 bytes
>                0 Dir(s)  103,286,464,512 bytes free
>
> L:\pf\Python\pyOpenSSL>
>
>
> L:\pf\Python\pyOpenSSL>setup.py build_ext -Ic:/openssl32/include -c 
> mingw32
> running build_ext
> building 'OpenSSL.crypto' extension
> writing build\temp.win32-2.5\Release\src\crypto\crypto.def
> x:\pf\cygwin\bin\gcc.exe -mno-cygwin -shared -s 
> build\temp.win32-2.5\Release\src
> \crypto\crypto.o build\temp.win32-2.5\Release\src\crypto\x509.o 
> build\temp.win32
> -2.5\Release\src\crypto\x509name.o 
> build\temp.win32-2.5\Release\src\crypto\pkey.
> o build\temp.win32-2.5\Release\src\crypto\x509store.o 
> build\temp.win32-2.5\Relea
> se\src\crypto\x509req.o 
> build\temp.win32-2.5\Release\src\crypto\x509ext.o build\
> temp.win32-2.5\Release\src\crypto\pkcs7.o 
> build\temp.win32-2.5\Release\src\crypt
> o\pkcs12.o build\temp.win32-2.5\Release\src\crypto\netscape_spki.o 
> build\temp.wi
> n32-2.5\Release\src\util.o c:\Python25\lib\ssleay32.a 
> build\temp.win32-2.5\Relea
> se\src\crypto\crypto.def -LC:\Python25\libs -LC:\Python25\PCBuild 
> -leay32 -lWs2_
> 32 -lpython25 -lmsvcr71 -o build\lib.win32-2.5\OpenSSL\crypto.pyd
> /usr/lib/gcc/i686-pc-mingw32/3.4.4/../../../../i686-pc-mingw32/bin/ld: 
> cannot fi
> nd -leay32
> collect2: ld returned 1 exit status
> error: command 'gcc' failed with exit status 1
>
>
I think the key part might

-leay32

I don't know gcc in detail but my guess is that this statement makes the 
gcc link/ld look for libeay32.dll - not ssleay32.a

pat





Jean-Paul Calderone wrote:
> On Mon, 20 Apr 2009 13:39:47 -0700, patrick flaherty <patf at well.com> wrote:
>   
>> I was /assuming/ that the import problem was the last of the four modules - 
>> tsafe - since I could see that the other
>> 3 pyd files were visibly there.  And as I looked for tsafe last Friday, I 
>> didn't find it.
>>
>> But as I look more today, yes, tsafe.py/pyc is in place as well.
>>
>> I seem to be having problems with finding any of these modules (upon import 
>> that is).  I've tried many variants (of reconfiguring things in various 
>> ways), but
>> I keep getting:
>>
>> L:\MyID\Python\pyOpenSSL\examples>mk_simple_certs.py
>> Traceback (most recent call last):
>>  File "L:\pf\Python\pyOpenSSL\examples\mk_simple_certs.py", line 5, in 
>> <module>
>>
>>    from OpenSSL import crypto
>>  File "c:\python26\Lib\site-packages\OpenSSL\__init__.py", line 8, in 
>> <module>
>>    import SSL, crypto, rand, tsafe
>> ImportError: DLL load failed: The specified module could not be found.
>>     
>
> Ah.  This is because of the OpenSSL library not being loadable.  If you
> copy ssleay32.a into the Python library directory, C:\python26\Lib\ I
> think, then Windows will find it and the error should be resolved.
>
> I hope that the next release of pyOpenSSL will include a Windows installer
> which deals with this automatically.
>
> Let me know how it goes,
>
> Jean-Paul
>
> ------------------------------------------------------------------------------
> Stay on top of everything new and different, both inside and 
> around Java (TM) technology - register by April 22, and save
> $200 on the JavaOne (SM) conference, June 2-5, 2009, San Francisco.
> 300 plus technical and hands-on sessions. Register today. 
> Use priority code J9JMT32. http://p.sf.net/sfu/p
> _______________________________________________
> pyopenssl-list mailing list
> pyopenssl-list at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/pyopenssl-list
>
>
>   

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pyopenssl-users/attachments/20090430/e03f9338/attachment.html>

From exarkun at divmod.com  Thu Apr 30 20:42:43 2009
From: exarkun at divmod.com (Jean-Paul Calderone)
Date: Thu, 30 Apr 2009 14:42:43 -0400
Subject: [pyOpenSSL] DLL load failed.
In-Reply-To: <49F9F003.2070902@well.com>
Message-ID: <20090430184243.24697.1562851226.divmod.quotient.14100@henry.divmod.com>

On Thu, 30 Apr 2009 11:37:55 -0700, patrick flaherty <patf at well.com> wrote:
>
>Hello Jean-Paul,
>
>As you can see below I copied ssleay32.a to c:\python25\Lib, however the 
>build (link/ld) fails.
>>
>>L:\pf\Python\pyOpenSSL>dir c:\python25\lib\ss*.a
>>  Volume in drive C has no label.
>>  Volume Serial Number is F818-DFCA
>>
>>  Directory of c:\python25\lib
>>
>>03/27/2009  06:32 AM           171,812 ssleay32.a
>>                1 File(s)        171,812 bytes
>>                0 Dir(s)  103,286,464,512 bytes free
>>
>>L:\pf\Python\pyOpenSSL>
>>
>>
>>L:\pf\Python\pyOpenSSL>setup.py build_ext -Ic:/openssl32/include -c mingw32
>>running build_ext
>>building 'OpenSSL.crypto' extension
>>writing build\temp.win32-2.5\Release\src\crypto\crypto.def
>>x:\pf\cygwin\bin\gcc.exe -mno-cygwin -shared -s 
>>build\temp.win32-2.5\Release\src
>>\crypto\crypto.o build\temp.win32-2.5\Release\src\crypto\x509.o 
>>build\temp.win32
>>-2.5\Release\src\crypto\x509name.o 
>>build\temp.win32-2.5\Release\src\crypto\pkey.
>>o build\temp.win32-2.5\Release\src\crypto\x509store.o 
>>build\temp.win32-2.5\Relea
>>se\src\crypto\x509req.o build\temp.win32-2.5\Release\src\crypto\x509ext.o 
>>build\
>>temp.win32-2.5\Release\src\crypto\pkcs7.o 
>>build\temp.win32-2.5\Release\src\crypt
>>o\pkcs12.o build\temp.win32-2.5\Release\src\crypto\netscape_spki.o 
>>build\temp.wi
>>n32-2.5\Release\src\util.o c:\Python25\lib\ssleay32.a 
>>build\temp.win32-2.5\Relea
>>se\src\crypto\crypto.def -LC:\Python25\libs -LC:\Python25\PCBuild -leay32 
>>-lWs2_
>>32 -lpython25 -lmsvcr71 -o build\lib.win32-2.5\OpenSSL\crypto.pyd
>>/usr/lib/gcc/i686-pc-mingw32/3.4.4/../../../../i686-pc-mingw32/bin/ld: 
>>cannot fi
>>nd -leay32
>>collect2: ld returned 1 exit status
>>error: command 'gcc' failed with exit status 1
>I think the key part might
>
>-leay32
>
>I don't know gcc in detail but my guess is that this statement makes the gcc 
>link/ld look for libeay32.dll - not ssleay32.a
>

It looks like gcc is being told to look in C:\Python25\libs - not C:\Python25\lib.  Perhaps that is causing the trouble?

By the way, with the release of pyOpenSSL 0.9, Windows binary eggs are
available for Python 2.3 and Python 2.5 (Python 2.4 and Python 2.6 are
giving me some problems).  You might want to check out
http://pypi.python.org/pypi/pyOpenSSL/0.9 and see if the pre-built version
works for you.

Jean-Paul