From dennis at illusions.com Wed Apr 3 01:33:26 2002 From: dennis at illusions.com (dennis at illusions.com) Date: Tue, 2 Apr 2002 16:33:26 -0700 (MST) Subject: userPassword Message-ID: Can someone give example code of setting a userPassword attribute to an encrypted password with python-ldap? -- --- Dennis Sacks dennis at illusions.com "An idiot with a computer is a faster, better idiot." - Rick Julius From Hans.Aschauer at Physik.uni-muenchen.de Wed Apr 3 14:56:57 2002 From: Hans.Aschauer at Physik.uni-muenchen.de (Hans Aschauer) Date: Wed, 3 Apr 2002 14:56:57 +0200 Subject: Schema support Message-ID: <20020403125657.909EF741@carrier.theorie.physik.uni-muenchen.de> I have finished a first version of schema support for python-ldap. There are four simple wrapper for the functions str2objectclass(), str2attributetype(), str2matchingrule(), and str2syntax() in the _ldap module (and thus in the ldap module --- I guess it would be a good idea to del() them from the ldap module in the ldap/__init__.py file...). These functions should not be used directly, since they are (probably) not thread safe. There exists a new module ldap.schema, which proviedes the following functions/classes: * Thread safe wrappers for the four C functions (str2xyz), which return lists of entries which correspond to the C data types (defined in ldap_schema.h). For thread safety, they simply use the _ldap_call() function (Michael, is this a good idea?). * Four simple classes (attributeType, objectClass, ldapSyntax, matchingRule). Their constructors take a string description of the schema, pass it to the wrapper functions and fill in the object attributes. See RFC2252 for details. * The class rootDSESchema (looking for a better name...), which takes an initialize()'d and simple_bind_s()'ed ldap object. The constructor queries the rootDSE and fills the attributeTypes, objectClasses, ldapSyntaxes, and matchingRules which are known by the ldap server into 8 dictionaries: for each of the four classes, there is one dictionary which takes the oid's as keys (these are those funny "1.2.3.4" things, which are supposed to be unique), and one dictionary takes the names as keys. The latter is necessary, since for example, in objectClasses, the allowed or required attributes are referred to by name rather than by oid. The names of the dictionaries are: objectClasses objectClassesByName attributeTypes attributeTypesByName ldapSyntaxes ldapSyntaxesByName matchingRules matchingRulesByName I've attached four files (a patch for setup.py, and the newly created files Modules/schema.c, Modules/schema.h, Lib/ldap/schema.py). Any comments? Does the user interface make sense? Could someone please try it with a _server_ different from OpenLDAP 2? As long as untested, I would say that it is only compatible with OpenLDAP --- there was some try/error involved in the RootDSE search... Hans -- Hans.Aschauer at Physik.uni-muenchen.de -------------- next part -------------- A non-text attachment was scrubbed... Name: setup.py.patch Type: text/x-diff Size: 240 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: schema.h Type: text/x-python Size: 244 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: schema.c Type: text/x-python Size: 8455 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: schema.py Type: text/x-python Size: 4698 bytes Desc: not available URL: From michael at stroeder.com Wed Apr 3 16:37:44 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Wed, 03 Apr 2002 16:37:44 +0200 Subject: Schema support References: <20020403125657.909EF741@carrier.theorie.physik.uni-muenchen.de> Message-ID: <3CAB13B7.6070407@stroeder.com> Hans Aschauer wrote: > I have finished a first version of schema support for python-ldap. Great! > * Thread safe wrappers for the four C functions (str2xyz), which return > lists of entries which correspond to the C data types (defined in > ldap_schema.h). For thread safety, they simply use the _ldap_call() > function (Michael, is this a good idea?). I don't know if it's good idea. But it was the best idea I came up with. So it's the right way to do it. > * Four simple classes (attributeType, objectClass, ldapSyntax, > matchingRule). Their constructors take a string description > of the schema, pass it to the wrapper functions and fill in the > object attributes. See RFC2252 for details. Ok. > * The class rootDSESchema (looking for a better name...), which takes > an initialize()'d and simple_bind_s()'ed ldap object. You should not do initialize() and simple_bind_s() because it could be a waste of resource and the root DSE and/or schema could be subject of access control. The application is responsible to make the connection and set up the right bind context => simply let the application pass in a instance of ldap.LDAPObject and solely do the search_s() calls within your class. > The constructor > queries the rootDSE and fills the attributeTypes, objectClasses, > ldapSyntaxes, and matchingRules which are known by the ldap server > into 8 dictionaries: for each of the four classes, there is one > dictionary which takes the oid's as keys (these are those funny > "1.2.3.4" things, which are supposed to be unique), and one > dictionary takes the names as keys. The latter is necessary, since > for example, in objectClasses, the allowed or required attributes are > referred to by name rather than by oid. I'd like to have a single OID registry since OIDs are really unique. There are no collisions. Simply map the OID in string notation to the schema element instance with a single dictionary. The reverse mapping can also be done by a single dictionary. I don't think that people are using the same names for attribute types and object classes. > there was some > try/error involved in the RootDSE search... He, he. BTW: Theoretically the sub schema sub entry can be different in each part of the DIT. Therefore an application has to query the subschemaSubentry attribute each time it uses the schema somewhere. Schema can be cached if the DN in subschemaSubentry attribute is the same. Maybe the OID registry could be a nested dictionary... Taken from web2ldap's module ldapsession (lines wrapped!): def getSubschemaEntryDN(self,dn): """Get DN of subschemaSubentry for dn""" # Search for DN of subschemaSubentry search_result = self.readEntry(dn,['subschemaSubentry']) if search_result: entry = search_result[0][1] return entry.get('subschemaSubentry',entry.get('subschemaSubentry',[None]))[0] else: return None Ciao, Michael. From michael at stroeder.com Wed Apr 3 09:48:26 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Wed, 03 Apr 2002 09:48:26 +0200 Subject: userPassword References: Message-ID: <3CAAB3CA.5030709@stroeder.com> dennis at illusions.com wrote: > Can someone give example code of setting a userPassword attribute to an > encrypted password with python-ldap? Concerning userPassword attribute it's handy to read and understand the OpenLDAP FAQ-O-MATIC: http://www.openldap.org/faq/data/cache/419.html Once I've started to implement a module for doing the whole thing on various password attributes with different syntaxes but that's probably overkill for you. Check the following files in tar.gz of http://www.web2ldap.de: pylib/ldaputil/passwd.py pylib/w2lapp/passwd.py Ciao, Michael. From Hans.Aschauer at Physik.uni-muenchen.de Wed Apr 3 19:42:36 2002 From: Hans.Aschauer at Physik.uni-muenchen.de (Hans Aschauer) Date: Wed, 3 Apr 2002 19:42:36 +0200 Subject: Schema support In-Reply-To: <3CAB13B7.6070407@stroeder.com> References: <20020403125657.909EF741@carrier.theorie.physik.uni-muenchen.de> <3CAB13B7.6070407@stroeder.com> Message-ID: <20020403174236.7C2647E7@carrier.theorie.physik.uni-muenchen.de> On Wednesday, 3. April 2002 16:37, you wrote: > Hans Aschauer wrote: > > (defined in ldap_schema.h). For thread safety, they simply use the > > _ldap_call() function (Michael, is this a good idea?). > > I don't know if it's good idea. But it was the best idea I came up > with. So it's the right way to do it. :-) [...] > > * The class rootDSESchema (looking for a better name...), which > > takes an initialize()'d and simple_bind_s()'ed ldap object. > > You should not do initialize() and simple_bind_s() because it could > be a waste of resource and the root DSE and/or schema could be > subject of access control. The application is responsible to make > the connection and set up the right bind context => simply let the > application pass in a instance of ldap.LDAPObject and solely do the > search_s() calls within your class. Ooops, that was exactly what I meant. Just pass an initialized and boud ldap object. > > The constructor > > queries the rootDSE and fills the attributeTypes, objectClasses, > > ldapSyntaxes, and matchingRules which are known by the ldap > > server into 8 dictionaries: for each of the four classes, there is > > one dictionary which takes the oid's as keys (these are those > > funny "1.2.3.4" things, which are supposed to be unique), and one > > dictionary takes the names as keys. The latter is necessary, since > > for example, in objectClasses, the allowed or required attributes > > are referred to by name rather than by oid. > > I'd like to have a single OID registry since OIDs are really unique. > There are no collisions. Simply map the OID in string notation to the > schema element instance with a single dictionary. The reverse mapping > can also be done by a single dictionary. I don't think that people > are using the same names for attribute types and object classes. You are right, this will really simplify things... [...] > BTW: Theoretically the sub schema sub entry can be different in each > part of the DIT. Therefore an application has to query the > subschemaSubentry attribute each time it uses the schema somewhere. > Schema can be cached if the DN in subschemaSubentry attribute is the > same. Maybe the OID registry could be a nested dictionary... You write of this possibility as a theoretical one. Do you know of any ldap server that implements this? Hmm, if I think about it, this is really usful as soon as referrals are being used. Are there other real-world situations where you (or someone else) have seen this? Maybe one should not automatically query the schema entries for every branch of the DIT, but only at "important" branches. If there exists no schema description at the desired level of the tree, the closest one is aquired (in Zopista speach) by climbing down the tree toward its root. It would then be up to the application writer (or some clever logics) to determine which branch is considered important. Would something like this make sense? -- Hans.Aschauer at Physik.uni-muenchen.de From Hans.Aschauer at Physik.uni-muenchen.de Wed Apr 3 10:57:39 2002 From: Hans.Aschauer at Physik.uni-muenchen.de (Hans Aschauer) Date: Wed, 3 Apr 2002 10:57:39 +0200 Subject: userPassword In-Reply-To: References: Message-ID: <20020403085739.91579744@carrier.theorie.physik.uni-muenchen.de> On Wednesday, 3. April 2002 01:33, dennis at illusions.com wrote: > Can someone give example code of setting a userPassword attribute to > an encrypted password with python-ldap? Hm, I dont know what exactly you want to do. Obviously, the encrypted password depends on the cryptographic hashing algorithmus you use (crypt, md5, sha1, etc). If you migrated your users from NIS, this is probably {crypt}. As far as I know, there exist different implementations of crypt, so that the result might be platform dependent. Here comes the code: import crypt passwd = 'mysecret' salt = "xy" userPassword = '{crypt}' + crypt.crypt(passwd,salt) And that's it. userPassword could then be used with the add() method of the ldap object. The salt string is a two character string and may be chosen at random (I'm not sure about this, anyway: sometimes it seems to be just the first two caracteres of the user name...). The salt is just the first two caracters of the encrypted password, so that it does not add security to the crypt algorithm. Diclaimer: dealing with passwords (plaintext _and_ encrypted) is dangerous. Just be sure that you know what you are doing! Don't use ldap connections which are not protected by other means! Hope it helps, Hans -- Hans.Aschauer at Physik.uni-muenchen.de From michael at stroeder.com Wed Apr 3 20:48:01 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Wed, 03 Apr 2002 20:48:01 +0200 Subject: Schema support References: <20020403125657.909EF741@carrier.theorie.physik.uni-muenchen.de> <3CAB13B7.6070407@stroeder.com> <20020403174236.7C2647E7@carrier.theorie.physik.uni-muenchen.de> Message-ID: <3CAB4E61.50101@stroeder.com> Hans Aschauer wrote: > On Wednesday, 3. April 2002 16:37, you wrote: > >>Hans Aschauer wrote: > >> > * The class rootDSESchema (looking for a better name...), which >> > takes an initialize()'d and simple_bind_s()'ed ldap object. >> >> [..] simply let the >>application pass in a instance of ldap.LDAPObject and solely do the >>search_s() calls within your class. > > Ooops, that was exactly what I meant. Just pass an initialized and boud > ldap object. Sorry, misread your phrase. >>BTW: Theoretically the sub schema sub entry can be different in each >>part of the DIT. > > You write of this possibility as a theoretical one. Do you know of any > ldap server that implements this? No I don't know one. Hmm, I have to check database configuration in Netscape and iPlanet DS again. But applications have to be prepared for it. > Maybe one should not automatically query the schema entries for every > branch of the DIT, but only at "important" branches. Now the really exciting questions is how you distinguish the "important" branches. ;-) IMHO efficient applications should request the DN of the sub schema sub entry by requesting attribute subschemaSubentry of current DN each and every time. If that points to a sub schema sub entry already parsed the cached schema instance can be used. > If there exists no > schema description at the desired level of the tree, the closest one is > aquired (in Zopista speach) by climbing down the tree toward its root. I'm pretty sure there are situations where tree climbing does not work. Don't try to be clever. Most times it does not work out. Anyway we should review RFC 2251, section 3.2.2 for that more closely. BTW: There could be also these attributes in sub schema sub entry: matchingRuleUse, dITStructureRules, dITContentRules and nameForms. I know OpenLDAP 2 server does not implement any of these. Does the schema parser in OpenLDAP 2 libs handle any of these? Ciao, Michael. From Hans.Aschauer at Physik.uni-muenchen.de Thu Apr 4 10:38:01 2002 From: Hans.Aschauer at Physik.uni-muenchen.de (Hans Aschauer) Date: Thu, 4 Apr 2002 10:38:01 +0200 Subject: Schema support In-Reply-To: <3CAB4E61.50101@stroeder.com> References: <20020403125657.909EF741@carrier.theorie.physik.uni-muenchen.de> <20020403174236.7C2647E7@carrier.theorie.physik.uni-muenchen.de> <3CAB4E61.50101@stroeder.com> Message-ID: <20020404083801.CAABF835@carrier.theorie.physik.uni-muenchen.de> On Wednesday, 3. April 2002 20:48, Michael Str?der wrote: > Hans Aschauer wrote: > > On Wednesday, 3. April 2002 16:37, you wrote: [about doing clever things in climbing down trees] > I'm pretty sure there are situations where tree climbing does not > work. Don't try to be clever. Most times it does not work out. Anyway > we should review RFC 2251, section 3.2.2 for that more closely. Ok, I went back to RFC 2251. That's how I understand it (please correct me when I'm wrong): Sec. 3.4: We can query the RootDSE for the subschemaSubentry attribute, which tells us which "subschema entries (or subentries) [are] known by this server": [aschauer at pygar]$ldapsearch -LLL -x -b "" -s base "(objectclass=*)" \ subschemaSubentry dn: subschemaSubentry: cn=Subschema Sec. 3.2.2: The result conforms to this section: "cn: this attribute MUST be used to form the RDN of the subschema entry". One thing I missed up to now: "Clients MUST only retrieve attributes from a subschema entry by requesting a base object search of the entry, where the search filter is '(objectClass=subschema)'" (up to now I used '(objectClass=*)'). Let's try it: [aschauer at pygar]$ldapsearch -LLL -x -b "cn=Subschema" -s base \ "(objectclass=subschema)" objectclasses | less Works! So, as far as I understand it, we should first query the rootDSE for the subschema (sub)entries, and use the results as search bases for querying the schema information. > BTW: There could be also these attributes in sub schema sub entry: > matchingRuleUse, dITStructureRules, dITContentRules and nameForms. > I know OpenLDAP 2 server does not implement any of these. Does the > schema parser in OpenLDAP 2 libs handle any of these? Nope. I just looked at the source, and the parsers seem to be kind of hard-coded. No generic BNF parser. But then, that's exactly how I would write such a parser ;-) So long, Hans -- Hans.Aschauer at Physik.uni-muenchen.de From michael at stroeder.com Thu Apr 4 23:51:54 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Thu, 04 Apr 2002 23:51:54 +0200 Subject: Schema support References: <20020403125657.909EF741@carrier.theorie.physik.uni-muenchen.de> <20020403174236.7C2647E7@carrier.theorie.physik.uni-muenchen.de> <3CAB4E61.50101@stroeder.com> <20020404083801.CAABF835@carrier.theorie.physik.uni-muenchen.de> Message-ID: <3CACCAFA.5020702@stroeder.com> Hans Aschauer wrote: > > Ok, I went back to RFC 2251. That's how I understand it (please correct > me when I'm wrong): > > Sec. 3.4: > We can query the RootDSE for the subschemaSubentry attribute, which > tells us which "subschema entries (or subentries) [are] known by this > server": > > [aschauer at pygar]$ldapsearch -LLL -x -b "" -s base "(objectclass=*)" \ > subschemaSubentry > dn: > subschemaSubentry: cn=Subschema Hmm, BTW it does not clarify how a client should deal with a multi-valued subschemaSubentry attribute. Should schemas be used exclusive-or or joined. That's another interesting topic to be discussed in IETF ldap-bis WG. > So, as far as I understand it, we should first query the rootDSE for > the subschema (sub)entries, and use the results as search bases for > querying the schema information. Take also a look at section 3.2.1.: --------------- snip ----------------- 3.2.1. Attributes of Entries [..] Entries MAY contain, among others, the following operational attributes, defined in [5]. These attributes are maintained automatically by the server and are not modifiable by clients: [..] - subschemaSubentry: the Distinguished Name of the subschema entry (or subentry) which controls the schema for this entry. --------------- snip ----------------- So my conclusion is that an application should first try to query operational attribute subschemaSubentry for a particular entry and fall-back to the subschemaSubentry attribute in the root DSE to find out the location of sub schema sub entry in the DIT. Puh... I guess I will start a discussion about that on the ldap-bis list. Ciao, Michael. From jens at zope.com Mon Apr 8 21:44:09 2002 From: jens at zope.com (Jens Vagelpohl) Date: Mon, 8 Apr 2002 15:44:09 -0400 Subject: non-ascii handling Message-ID: i have a product that uses python-ldap and i'm trying to make sure everything works when non-ascii characters are used in a DN. from what i have been reading about OpenLDAP it either wants pure ASCII passed to it (for search terms, DNs etc) or UTF-8-encoded unicode strings. my question is: does python-ldap do any automatic string conversions? i get search results just fine using a non-ascii search term when i do not convert the term myself and hand it to ldap.search_s, but i never get results if i convert the string by myself and then hand it to the search_s method. i'm trying to find out whether i am doing the wrong thing converting any and all non-ascii myself in my product before sending it over the line to the LDAP server using python-ldap. jens From michael at stroeder.com Wed Apr 10 19:17:03 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Wed, 10 Apr 2002 19:17:03 +0200 Subject: non-ascii handling References: Message-ID: <3CB4738F.9080309@stroeder.com> Jens, Sorry for answering that late. Jens Vagelpohl wrote: > i have a product that uses python-ldap and i'm trying to make sure > everything works when non-ascii characters are used in a DN. from what i > have been reading about OpenLDAP it either wants pure ASCII passed to it > (for search terms, DNs etc) or UTF-8-encoded unicode strings. Depends on the attribute. BTW: ASCII is a real subset of UTF-8. Or better said: The character entities encoded in ASCII are mapped to the very same encoding in UTF-8. > my question is: does python-ldap do any automatic string conversions? No! And I refused a patch which does. It cannot be done without applying knowledge about the schema (syntax of an attribute). Review the archives. > i > get search results just fine using a non-ascii search term when i do not > convert the term myself and hand it to ldap.search_s, but i never get > results if i convert the string by myself and then hand it to the > search_s method. If you have a Unicode object with a LDAP search filter than you have to encode that before calling method search_s(). Example (valid on my Linux console with ISO-8859-1): filter = unicode('cn=*Str?der*','iso-8859-1') l.search_s(search_root,ldap.SCOPE_SUB,filter.encode('utf-8')) Note that filter is a Unicode object created by passing a string and the known character set to the unicode() function. Ciao, Michael. From jens at zope.com Thu Apr 11 14:39:30 2002 From: jens at zope.com (Jens Vagelpohl) Date: Thu, 11 Apr 2002 08:39:30 -0400 Subject: non-ascii handling In-Reply-To: <3CB4738F.9080309@stroeder.com> Message-ID: <2ADC0496-4D49-11D6-9340-00039363690C@zope.com> michael, thanks for the answer, that helped a bit. in handling these various kinds of strings (both UTF-8 encoded unicode and latin-1 encoded unicode for web browser consumption) i always end up running into trouble at some point because in some situations strings get encoded more than once. does anyone know of a quick and fast test to determine whether a string is already encoded in a certain encoding? my knowledge of regular expressions (which i assume it would take for that) is extremely limited at best. jens On Wednesday, April 10, 2002, at 01:17 , Michael Str?der wrote: > Jens, > > Sorry for answering that late. > > Jens Vagelpohl wrote: >> i have a product that uses python-ldap and i'm trying to make sure >> everything works when non-ascii characters are used in a DN. from what i >> have been reading about OpenLDAP it either wants pure ASCII passed to it >> (for search terms, DNs etc) or UTF-8-encoded unicode strings. > > Depends on the attribute. BTW: ASCII is a real subset of UTF-8. Or better > said: The character entities encoded in ASCII are mapped to the very same > encoding in UTF-8. > >> my question is: does python-ldap do any automatic string conversions? > > No! And I refused a patch which does. It cannot be done without applying > knowledge about the schema (syntax of an attribute). Review the archives. > >> i get search results just fine using a non-ascii search term when i do >> not convert the term myself and hand it to ldap.search_s, but i never get >> results if i convert the string by myself and then hand it to the >> search_s method. > > If you have a Unicode object with a LDAP search filter than you have to > encode that before calling method search_s(). > > Example (valid on my Linux console with ISO-8859-1): > > filter = unicode('cn=*Str?der*','iso-8859-1') > l.search_s(search_root,ldap.SCOPE_SUB,filter.encode('utf-8')) > > Note that filter is a Unicode object created by passing a string and the > known character set to the unicode() function. > > Ciao, Michael. > From michael at stroeder.com Thu Apr 11 18:36:31 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Thu, 11 Apr 2002 18:36:31 +0200 Subject: non-ascii handling References: <2ADC0496-4D49-11D6-9340-00039363690C@zope.com> Message-ID: <3CB5BB8F.5070809@stroeder.com> Jens Vagelpohl wrote: > in handling these various kinds of strings (both UTF-8 encoded unicode > and latin-1 encoded unicode for web browser consumption) i always end up > running into trouble at some point because in some situations strings > get encoded more than once. Especially when implementing web applications you have to take great care to define the charset used. Use
or similar to also define the charset of the form input data. (This does not prevent e.g. StarOffice from sending ISO-8859-1 data.) Also set the charset of the output in the HTTP header *and* the section. > does anyone know of a quick and fast test to > determine whether a string is already encoded in a certain encoding? my > knowledge of regular expressions (which i assume it would take for that) > is extremely limited at best. Hmm, in some situations a try: unicode() except UnicodeError: might help. But I do not recommend such a solution (although sometimes used in web2ldap) and you have to apply specific knowledge about the character sets/encodings. From my personal experience it's much work to make an application Unicode-aware. But you should try to clean up your application design. It's worth the effort. Ciao, Michael. From sharmago at mail.nih.gov Fri Apr 12 06:23:19 2002 From: sharmago at mail.nih.gov (Sharma, Gopesh (NCI)) Date: Fri, 12 Apr 2002 00:23:19 -0400 Subject: Error importing LDAP module Message-ID: <76D006EB9224D411B12B0008C7C59D9607E5A26C@nihexchange9.nih.gov> Hello All: I am getting error while using 'import ldap' command. ************************************************************************** [root at tsdev2 bin]# ./python Python 2.1.2 (#1, Jan 25 2002, 13:17:56) [GCC 2.7.2.3] on linux2 Type "copyright", "credits" or "license" for more information. >>> import ldap Traceback (most recent call last): File "", line 1, in ? File "/usr/local/ZopePark/lib/python2.1/ldap.py", line 2, in ? from _ldap import __version__ ImportError: No module named _ldap >>> ************************************************************************** I am running Python2.1.2 from within Zope Framework. If I use python2.2 it works fine. Is there any Pythons 2.1.2 patch available so that I can install in Zope?? Please advise. Thanks, Gopesh From jens at zope.com Fri Apr 12 13:49:54 2002 From: jens at zope.com (Jens Vagelpohl) Date: Fri, 12 Apr 2002 07:49:54 -0400 Subject: Error importing LDAP module In-Reply-To: <76D006EB9224D411B12B0008C7C59D9607E5A26C@nihexchange9.nih.gov> Message-ID: <676655B4-4E0B-11D6-8898-00039363690C@zope.com> build python-ldap with the exact binary that is in zope. i assume you are using some binary distribution. setup.py build setup.py install that should do it. jens P.S.: a lot of peoples' problems could be solved if they used the source install of zope, which is very easy to install and build. then you can use any python module that is installed into your machine's python library path because your machine's python is used. On Friday, April 12, 2002, at 12:23 , Sharma, Gopesh (NCI) wrote: > Hello All: > > I am getting error while using 'import ldap' command. > > ************************************************************************* > * > > [root at tsdev2 bin]# ./python > Python 2.1.2 (#1, Jan 25 2002, 13:17:56) > [GCC 2.7.2.3] on linux2 > Type "copyright", "credits" or "license" for more information. >>>> import ldap > Traceback (most recent call last): > File "", line 1, in ? > File "/usr/local/ZopePark/lib/python2.1/ldap.py", line 2, in ? > from _ldap import __version__ > ImportError: No module named _ldap >>>> > > ************************************************************************* > * > I am running Python2.1.2 from within Zope Framework. If I use python2.2 it > works fine. > > Is there any Pythons 2.1.2 patch available so that I can install in Zope? > ? > > Please advise. > Thanks, > Gopesh > > From Hans.Aschauer at Physik.uni-muenchen.de Fri Apr 5 15:20:28 2002 From: Hans.Aschauer at Physik.uni-muenchen.de (Hans Aschauer) Date: Fri, 5 Apr 2002 15:20:28 +0200 Subject: Schema support In-Reply-To: <3CACCAFA.5020702@stroeder.com> References: <20020403125657.909EF741@carrier.theorie.physik.uni-muenchen.de> <20020404083801.CAABF835@carrier.theorie.physik.uni-muenchen.de> <3CACCAFA.5020702@stroeder.com> Message-ID: <20020405132029.0C2A0836@carrier.theorie.physik.uni-muenchen.de> On Thursday, 4. April 2002 23:51, Michael Str?der wrote: > Hmm, BTW it does not clarify how a client should deal with a > multi-valued subschemaSubentry attribute. Should schemas be used > exclusive-or or joined. That's another interesting topic to be > discussed in IETF ldap-bis WG. As I understand it, this issue has been addressed in draft-ietf-ldapbis-protocol-02.txt: """ B.8 Section 3.4 - Reworded text surrounding subschemaSubentry to reflect that it is a single-valued attribute that holds the schema for the root DSE. Also noted that if the server masters entries that use differing schema, each entry's subschemaSubentry attribute must be interrogated. This may change as further fine-tuning is done to the data model. """ [...] > So my conclusion is that an application should first try to query > operational attribute subschemaSubentry for a particular entry and > fall-back to the subschemaSubentry attribute in the root DSE to find > out the location of sub schema sub entry in the DIT. Puh... According to the draft, we should probably query the root DSE only if we are interested in the schema controlling the root DSE itself (which most applications are not, as I think). On the other hand, the RFC text is somwhat misleading (especially where it is changed by the draft), and I don't know how implementors of server software interpreted the text... Anyway, back to the python specific things :-) * If we implement per-entry schema information, caching is mandatory. And it can be easily done, since we we can create a mapping "subschema-entry RDN" => "schema information" (by doing it this way we would not need nested trees, and for servers which have only one subschema entry, one saves a lot of memory). To be specific, for my OpenLDAP installation, this mapping would look like {'cn=Subschema': } --- only one entry. Of course, the name of the class ldap.schema.rootDSESchema has to be changed... How about "schema"? * How should this cache be accessed? Logically, it belongs to the ldap connection (or to the ldap server, but there is not such python class, is it?). However, I don't know if it is too high-level to make it an attribute of the LDAPObject class. So long, Hans -- Hans.Aschauer at Physik.uni-muenchen.de From michael at stroeder.com Tue Apr 16 07:59:27 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Tue, 16 Apr 2002 07:59:27 +0200 Subject: Schema support References: <20020403125657.909EF741@carrier.theorie.physik.uni-muenchen.de> Message-ID: <3CBBBDBF.2020704@stroeder.com> Hans Aschauer wrote: > I have finished a first version of schema support for python-ldap. I tried your attachments. But to me it seems that there is lacking something. The build went fine but the functions do not appear in extension module _ldap. Maybe you also modified functions.c? Please re-send most recent version of your schema support and SASL patches. Ciao, Michael. From Hans.Aschauer at Physik.uni-muenchen.de Tue Apr 16 13:55:12 2002 From: Hans.Aschauer at Physik.uni-muenchen.de (Hans Aschauer) Date: Tue, 16 Apr 2002 13:55:12 +0200 Subject: Schema support Message-ID: <200204161355.12931.Hans.Aschauer@Physik.uni-muenchen.de> On Dienstag, 16. April 2002 07:59, Michael Str?der wrote: > I tried your attachments. But to me it seems that there is lacking > something. The build went fine but the functions do not appear in > extension module _ldap. Sorry, ashes on my head... I forgot to apped the patch for Modules/ldapmodule.c (which calls LDAPinit_schema()) -- see below. > Please re-send most recent version of your schema support and SASL > patches. Ok, here we go. The "grand unified patch" against cvs. The following files have beed added and do not appear in the patch. BTW: is there a way to include them using 'cvs diff'? Demo/sasl_bind.py Lib/ldap/sasl.py (new sasl module) Lib/ldap/schema.py (the schema module) Modules/schema.c Modules/schema.h __Opps__ I just saw that you have already applied the schema patches to CVS. The missing part in the C part is the following patch: Index: Modules/ldapmodule.c =================================================================== RCS file: /cvsroot/python-ldap/python-ldap/Modules/ldapmodule.c,v retrieving revision 1.3 diff -r1.3 ldapmodule.c 11a12 > #include "schema.h" 46a48 > LDAPinit_schema(d); The attached patch is a context sensitive diff in order to minimize the confusion of patches which are applied twice ;-) If you prefere a tar-ball of my sources, let me know. Maybe it is then easier to merge our trees (hint, hint :-). Hans -- Hans.Aschauer at Physik.uni-muenchen.de ------------------------------------------------------- -- Hans.Aschauer at Physik.uni-muenchen.de -------------- next part -------------- A non-text attachment was scrubbed... Name: sasl_bind.py Type: text/x-python Size: 412 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: sasl.py Type: text/x-python Size: 3068 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: schema.py Type: text/x-python Size: 4698 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: schema.c Type: text/x-csrc Size: 8455 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: schema.h Type: text/x-chdr Size: 244 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: sasl_schema1.diff Type: text/x-diff Size: 10415 bytes Desc: not available URL: From jmeile at hotmail.com Thu Apr 18 23:46:49 2002 From: jmeile at hotmail.com (Josef Meile) Date: Thu, 18 Apr 2002 23:46:49 +0200 Subject: Error message when running python-ldap under Zope and Suse Message-ID: Hi, I'm just trying to install python-ldap on suse 7.3, Zope 2.3.3 and Python 1.5.2 I also installed the LDAPUserManager 1.3 and the LDAPLoginAdapter 1.6. I followed all the instruccions: 1- Download and install the binary from suse 7.3 -> python-ldap-1.10alpha3-182.i386.rpm (I took it from ftp.suse.com) 2- Set the LD_LIBRARY_PATH where my liblber.so.2 and libldap.so.2 are: LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib export LD_LIBRARY_PATH 3- Set the PYTHONPATH where the python-ldap module is: PYTHONPATH=$PYTHONPATH:/usr/lib/python2.1/site-packages/python-ldap:/usr/lib /python2.1/site-packages export PYTHONPATH Here I also tried: PYTHONPATH=$PYTHONPATH:/usr/lib/python2.1/site-packages/python-ldap export PYTHONPATH and: PYTHONPATH=$PYTHONPATH:/usr/lib/python2.1/site-packages export PYTHONPATH But then I got one error more. These are the errors: ------ 2002-04-18T21:18:23 ERROR(200) Zope Couldn't import Products.LDAPLoginAdapter Traceback (innermost last): File /usr/local/Zope-2.3.0/lib/python/OFS/Application.py, line 528, in import_ products (Object: string) File /usr/local/Zope/lib/python/Products/LDAPLoginAdapter/__init__.py, line 13 , in ? File /usr/local/Zope/lib/python/Products/LDAPLoginAdapter/LDAPLoginAdapter.py, line 23, in ? File /usr/local/Zope/lib/python/Products/LDAPLoginAdapter/LDAPShared.py, line 19, in ? File /usr/lib/python2.1/site-packages/python-ldap/ldap.py, line 2, in ? ImportError: /usr/lib/python2.1/site-packages/_ldapmodule.so: undefined symbol: PyObject_Init ------ 2002-04-18T21:18:23 ERROR(200) Zope Couldn't import Products.LDAPUserManager Traceback (innermost last): File /usr/local/Zope-2.3.0/lib/python/OFS/Application.py, line 528, in import_ products (Object: string) File /usr/local/Zope/lib/python/Products/LDAPUserManager/__init__.py, line 13, in ? File /usr/local/Zope/lib/python/Products/LDAPUserManager/LDAPUserManager.py, l ine 16, in ? File /usr/local/Zope/lib/python/Products/LDAPUserManager/LDAPShared.py, line 2 3, in ? AttributeError: SCOPE_BASE ------ 2002-04-18T21:18:38 INFO(0) ZServer HTTP server started at Thu Apr 18 23:18:38 2 002 Hostname: localhost Port: 80 ------ 2002-04-18T21:18:38 INFO(0) ZServer FTP server started at Thu Apr 18 23:18:38 20 02 Hostname: localhost Port: 8021 ------ 2002-04-18T21:18:38 INFO(0) ZServer PCGI Server started at Thu Apr 18 23:18:38 2 002 Unix socket: /usr/local/Zope-2.3.0/var/pcgi.soc Does anybody know the mistake? Thanks in advanced, Josef. From jens at zope.com Thu Apr 18 23:55:35 2002 From: jens at zope.com (Jens Vagelpohl) Date: Thu, 18 Apr 2002 17:55:35 -0400 Subject: Error message when running python-ldap under Zope and Suse In-Reply-To: <002201c1e721$726938c0$5da88481@eco8> Message-ID: <0356EB9B-5317-11D6-9D93-00039363690C@zope.com> is your zope a binary version or the source version? apart from that you cannot expect python 1.5 (which is what your zope is running on) to find and use things in the python path of a python 2.1 installation. jens P.S.: never trust RPM packages for addons like this because they are always packaged with a lot of assumptions, like what python to use and where things are installed. On Thursday, April 18, 2002, at 05:38 , Josef Meile wrote: > Hi, > I'm just trying to install python-ldap on suse 7.3, Zope 2.3.3 and Python > 1.5.2 > I also installed the LDAPUserManager 1.3 and the LDAPLoginAdapter 1.6. > > I followed all the instruccions: > 1- Download and install the binary from suse 7.3 -> > python-ldap-1.10alpha3-182.i386.rpm (I took it from ftp.suse.com) > > 2- Set the LD_LIBRARY_PATH where my liblber.so.2 and libldap.so.2 are: > LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib > export LD_LIBRARY_PATH > > 3- Set the PYTHONPATH where the python-ldap module is: > PYTHONPATH=$PYTHONPATH:/usr/lib/python2.1/site-packages/python-ldap:/usr/ > lib > /python2.1/site-packages > export PYTHONPATH > > Here I also tried: > PYTHONPATH=$PYTHONPATH:/usr/lib/python2.1/site-packages/python-ldap > export PYTHONPATH > > and: > PYTHONPATH=$PYTHONPATH:/usr/lib/python2.1/site-packages > export PYTHONPATH > > But then I got one error more. > > These are the errors: > > ------ > 2002-04-18T21:18:23 ERROR(200) Zope Couldn't import > Products.LDAPLoginAdapter > Traceback (innermost last): > File /usr/local/Zope-2.3.0/lib/python/OFS/Application.py, line 528, in > import_ > products > (Object: string) > File /usr/local/Zope/lib/python/Products/LDAPLoginAdapter/__init__.py, > line 13 > , in ? > File > /usr/local/Zope/lib/python/Products/LDAPLoginAdapter/LDAPLoginAdapter.py, > line 23, in ? > File /usr/local/Zope/lib/python/Products/LDAPLoginAdapter/LDAPShared.py, > line > 19, in ? > File /usr/lib/python2.1/site-packages/python-ldap/ldap.py, line 2, in ? > ImportError: /usr/lib/python2.1/site-packages/_ldapmodule.so: undefined > symbol: > PyObject_Init > > > ------ > 2002-04-18T21:18:23 ERROR(200) Zope Couldn't import > Products.LDAPUserManager > Traceback (innermost last): > File /usr/local/Zope-2.3.0/lib/python/OFS/Application.py, line 528, in > import_ > products > (Object: string) > File /usr/local/Zope/lib/python/Products/LDAPUserManager/__init__.py, > line > 13, > in ? > File > /usr/local/Zope/lib/python/Products/LDAPUserManager/LDAPUserManager.py, l > ine 16, in ? > File /usr/local/Zope/lib/python/Products/LDAPUserManager/LDAPShared.py, > line 2 > 3, in ? > AttributeError: SCOPE_BASE > > > ------ > 2002-04-18T21:18:38 INFO(0) ZServer HTTP server started at Thu Apr 18 > 23:18:38 2 > 002 > Hostname: localhost > Port: 80 > ------ > 2002-04-18T21:18:38 INFO(0) ZServer FTP server started at Thu Apr 18 > 23:18:38 20 > 02 > Hostname: localhost > Port: 8021 > ------ > 2002-04-18T21:18:38 INFO(0) ZServer PCGI Server started at Thu Apr 18 > 23:18:38 2 > 002 > Unix socket: /usr/local/Zope-2.3.0/var/pcgi.soc > > > Does anybody know the mistake? > > Thanks in advanced, > Josef. > > > From jmeile at hotmail.com Fri Apr 19 12:54:08 2002 From: jmeile at hotmail.com (Josef Meile) Date: Fri, 19 Apr 2002 12:54:08 +0200 Subject: Error message when running python-ldap under Zope and Suse References: <0356EB9B-5317-11D6-9D93-00039363690C@zope.com> Message-ID: > is your zope a binary version or the source version? Actually, I don't know it because it was installed when I started with it. How can I now it? > apart from that you cannot expect python 1.5 (which is what your zope is > running on) to find and use things in the python path of a python 2.1 > installation. I know it, but there isn't a suse version on the python-ldap's sourceforge page. So, where should I get it? Or can I perhaps change the python version? > P.S.: never trust RPM packages for addons like this because they are always > packaged with a lot of assumptions, like what python to use and where > things are installed. Which one should I download? Thanks for your reply, Josef. From jens at zope.com Fri Apr 19 13:49:01 2002 From: jens at zope.com (Jens Vagelpohl) Date: Fri, 19 Apr 2002 07:49:01 -0400 Subject: Error message when running python-ldap under Zope and Suse In-Reply-To: Message-ID: <70BECF18-538B-11D6-AB1A-00039363690C@zope.com> >> is your zope a binary version or the source version? > Actually, I don't know it because it was installed when I started with it. > How can I now it? if there is a python binary anywhere underneath the zope directory then you most likely have a binary distribution, which presents problems for newbies trying to install additional python modules. > >> apart from that you cannot expect python 1.5 (which is what your zope is >> running on) to find and use things in the python path of a python 2.1 >> installation. > I know it, but there isn't a suse version on the python-ldap's sourceforge > page. So, > where should I get it? Or can I perhaps change the python version? zope 2.3.x runs on python 1.5.2. you will have to get the python-ldap 1.10alpha3 source and build it i suppose. > >> P.S.: never trust RPM packages for addons like this because they are > always >> packaged with a lot of assumptions, like what python to use and where >> things are installed. > Which one should I download? get the source jens From jlittle at open-it.org Fri Apr 19 19:26:46 2002 From: jlittle at open-it.org (Joe Little) Date: Fri, 19 Apr 2002 10:26:46 -0700 Subject: Schema support In-Reply-To: <3CBBBDBF.2020704@stroeder.com> Message-ID: Now that some of this has been commited, any interest in making another -pre release? On 4/15/02 10:59 PM, "Michael Str?der" wrote: > Hans Aschauer wrote: >> I have finished a first version of schema support for python-ldap. > > I tried your attachments. But to me it seems that there is lacking > something. The build went fine but the functions do not appear in extension > module _ldap. > > Maybe you also modified functions.c? > > Please re-send most recent version of your schema support and SASL patches. > > Ciao, Michael. > > > From michael at stroeder.com Fri Apr 19 19:43:25 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Fri, 19 Apr 2002 19:43:25 +0200 Subject: Schema support References: Message-ID: <3CC0573D.7020506@stroeder.com> Joe Little wrote: > Now that some of this has been commited, any interest in making another -pre > release? Far too early to ask for a pre-release. Ciao, Michael. From michael at stroeder.com Sat Apr 20 20:11:41 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Sat, 20 Apr 2002 20:11:41 +0200 Subject: trouble installing 2.0.0pre04 References: <20020420131700.I18505-100000@willow.id-vault.com> <3CC1AD2A.1080809@stroeder.com> Message-ID: <3CC1AF5D.9050809@stroeder.com> Michael Str?der wrote: > Michael Schwartz wrote: > >> >> -L/usr/local/lib -llber -lldap -lresolv -o >> build/lib.freebsd-4.4-RELEASE-i386-2.1/_ldap.so >> /usr/libexec/elf/ld: cannot find -lresolv >> error: command 'cc' failed with exit status 1 > > > [..] > >> libs = lber ldap resolv > > > Should be on BSD: > > libs = lber ldap > > It works on your FreeBSD machine. Look into setup.cfg my home dir. BTW: There's also a FreeBSD port for Python2.2 and python-ldap (and even web2ldap). Ciao, Michael. From michael at stroeder.com Sat Apr 20 20:02:18 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Sat, 20 Apr 2002 20:02:18 +0200 Subject: trouble installing 2.0.0pre04 References: <20020420131700.I18505-100000@willow.id-vault.com> Message-ID: <3CC1AD2A.1080809@stroeder.com> Michael Schwartz wrote: > > -L/usr/local/lib -llber -lldap -lresolv -o > build/lib.freebsd-4.4-RELEASE-i386-2.1/_ldap.so > /usr/libexec/elf/ld: cannot find -lresolv > error: command 'cc' failed with exit status 1 > [..] > libs = lber ldap resolv Should be on BSD: libs = lber ldap It works on your FreeBSD machine. Look into setup.cfg my home dir. Ciao, Michael. From senux at senux.com Tue Apr 23 10:32:38 2002 From: senux at senux.com (Brian Lee) Date: Tue, 23 Apr 2002 17:32:38 +0900 Subject: docs of python-ldap in pdf formatted file Message-ID: <20020423083238.GA18043@jupiter.senux.com> Thanks all! I downloaded and tried to print out pdf version of python-ldap document but i couldn't it. I think page 6 is broken and my windows say can't print it. Can this be real? Others? - Brian,. From spereira at soleri.com Fri Apr 26 10:15:46 2002 From: spereira at soleri.com (Sylvain Pereira) Date: Fri, 26 Apr 2002 10:15:46 +0200 (MEST) Subject: Ldap over SSL Message-ID: <1019808946.3cc90cb24ddc9@imp.pro.proxad.net> Hello, I am developping a python cgi application to administrate an ldap server for a customer. I can't use any of the existing products, they do not fit. I can connect, add, modify, delete to the LDAP server. As I wrote it before, it is an administration application, so the ldap requests should be encrypted thanks to SSL. I cannot find any documentation. My python is compiled with openssl and ldap support. Any idea ? From spereira at soleri.com Fri Apr 26 11:24:33 2002 From: spereira at soleri.com (Sylvain Pereira) Date: Fri, 26 Apr 2002 11:24:33 +0200 (MEST) Subject: Ldap over SSL In-Reply-To: <3CC9110F.8030104@stroeder.com> References: <1019808946.3cc90cb24ddc9@imp.pro.proxad.net> <3CC9110F.8030104@stroeder.com> Message-ID: <1019813073.3cc91cd195a95@imp.pro.proxad.net> > > > > I am developping a python cgi application to administrate an ldap > server for > > a customer. > > What's the LDAP server product and OS? Well it's iPlanet Directory Server 5.1 on SunOS 8, and I already checked the SSL connection with a Java program (I am new to python). The python cgi runs on an Irix 6.5 > See Demo/initialize.py for various examples using LDAP over SSL (LDAPS) > or > StartTLS extended operation (RFC2830). Thanks, I also took a look a the list archives ("TLS Context"). Still, I get the following error when testing with the python interpreter : ############################################ >>> import ldap >>> print ldap.__version__ 2.0.0pre04 >>> ldap.set_option (ldap.OPT_X_TLS_CACERTFILE,'/usr/freeware/apache/conf/cert7.db') Traceback (most recent call last): File "", line 1, in ? File "/usr/freeware/lib/python2.1/site-packages/ldap/functions.py", line 101, in set_option _ldap_call(_ldap.set_option,option,invalue) File "/usr/freeware/lib/python2.1/site-packages/ldap/__init__.py", line 31, in _ldap_call result = apply(func,args,kwargs) ldap.LDAPError: {'errnum': -1} ############################################ Any idea ? From michael at stroeder.com Fri Apr 26 11:58:20 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Fri, 26 Apr 2002 11:58:20 +0200 Subject: Ldap over SSL References: <1019808946.3cc90cb24ddc9@imp.pro.proxad.net> <3CC9110F.8030104@stroeder.com> <1019813073.3cc91cd195a95@imp.pro.proxad.net> Message-ID: <3CC924BC.10200@stroeder.com> Sylvain Pereira wrote: >>>I am developping a python cgi application to administrate an ldap >> >>server for >> >>>a customer. >> >>What's the LDAP server product and OS? > > Well it's iPlanet Directory Server 5.1 on SunOS 8, and I already checked > the SSL connection with a Java program (I am new to python). > The python cgi runs on an Irix 6.5 If it's running on a Unix platform iDS 5.1 should also support StartTLS extended operation (not available with Win32 version) which is the standard track (see RFC2830). >>>>import ldap >>>>print ldap.__version__ >>> > 2.0.0pre04 > >>>>ldap.set_option >>> > (ldap.OPT_X_TLS_CACERTFILE,'/usr/freeware/apache/conf/cert7.db') That's a Netscape certificate DB for Netscape's NSS lib. Something completely different. OpenLDAP2 uses OpenSSL as SSL lib. Therefore just use a "PEM file" containing the certificate. The same for the cert/key when using client certs for authentication. Ciao, Michael. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 2922 bytes Desc: S/MIME Cryptographic Signature URL: From michael at stroeder.com Fri Apr 26 10:34:23 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Fri, 26 Apr 2002 10:34:23 +0200 Subject: Ldap over SSL References: <1019808946.3cc90cb24ddc9@imp.pro.proxad.net> Message-ID: <3CC9110F.8030104@stroeder.com> Sylvain Pereira wrote: > > I am developping a python cgi application to administrate an ldap server for > a customer. What's the LDAP server product and OS? > so the ldap requests should be encrypted thanks to SSL. See Demo/initialize.py for various examples using LDAP over SSL (LDAPS) or StartTLS extended operation (RFC2830). Note that there are also options to set for validating the server's cert against a trusted root cert and doing client cert/key for strong authentication: ldap.set_option(ldap.OPT_X_TLS_CACERTFILE,tls_cacert_full_pathname) ldap.set_option(ldap.OPT_X_TLS_CERTFILE,tls_client_cert_full_pathname) ldap.set_option(ldap.OPT_X_TLS_KEYFILE,tls_client_key_full_pathname) Ciao, Michael. From jens at zope.com Fri Apr 26 13:53:06 2002 From: jens at zope.com (Jens Vagelpohl) Date: Fri, 26 Apr 2002 07:53:06 -0400 Subject: Ldap over SSL In-Reply-To: <3CC924BC.10200@stroeder.com> Message-ID: <2BEB050C-590C-11D6-BEE8-00039363690C@zope.com> if the LDAP server listens on a specific SSL-enabled port (meaning a port that uses the ldaps protocol, the standard port for that being 636) then that is much easier to work with then StartTLS. conn = ldap.initialize( 'ldaps://my.ldap.server' ) etc. granted, this is not the "standard track", but it is very easy to use and works reliably. i have once or twice tried to use StartTLS but all i ever got were error messages that don't say anything about the actual error, and my need to use StartTLS instead of ldaps was never great enough for me to investigate further. jens On Friday, April 26, 2002, at 05:58 , Michael Str?der wrote: > Sylvain Pereira wrote: >>>> I am developping a python cgi application to administrate an ldap >>> >>> server for >>> >>>> a customer. >>> >>> What's the LDAP server product and OS? >> Well it's iPlanet Directory Server 5.1 on SunOS 8, and I already checked >> the SSL connection with a Java program (I am new to python). >> The python cgi runs on an Irix 6.5 > > If it's running on a Unix platform iDS 5.1 should also support StartTLS > extended operation (not available with Win32 version) which is the > standard track (see RFC2830). > >>>>> import ldap >>>>> print ldap.__version__ >>>> >> 2.0.0pre04 >>>>> ldap.set_option >>>> >> (ldap.OPT_X_TLS_CACERTFILE,'/usr/freeware/apache/conf/cert7.db') > > That's a Netscape certificate DB for Netscape's NSS lib. Something > completely different. > > OpenLDAP2 uses OpenSSL as SSL lib. Therefore just use a "PEM file" > containing the certificate. The same for the cert/key when using client > certs for authentication. > > Ciao, Michael. > From jmeile at hotmail.com Sat Apr 27 16:22:26 2002 From: jmeile at hotmail.com (Josef Meile) Date: Sat, 27 Apr 2002 16:22:26 +0200 Subject: Installing source of python-ldap in suse... References: Message-ID: > > is python runnable from the command line? ie is it a problem with you PATH? > Yes, but I have to go to the directory where it is. From jmeile at hotmail.com Sat Apr 27 17:31:24 2002 From: jmeile at hotmail.com (Josef Meile) Date: Sat, 27 Apr 2002 17:31:24 +0200 Subject: Installing source of python-ldap in suse... References: Message-ID: > um, then add the directory that contains python to your $PATH. > you should be able to invoke python from anywhere then, and the scripts > that you first referred to should now work. I just did it but this error is still there: make[1]: Entering directory `/usr/local/python/2.1.3/python-ldap-1.10alpha3/Modules' make[1]: *** No rule to make target `/usr/local/lib/python2.1/config/Makefile', needed by `sedscript'. Stop. make[1]: Leaving directory `/usr/local/python/2.1.3/python-ldap-1.10alpha3/Modules' make: *** [boot] Error 2 It is looking for Makefile in /usr/local/lib/python2.1/config/, but that's not the directory where my python is installed. This line is also wrong: checking python version... 2.1 I have python 2.1.3 and not 2.1. From jmeile at hotmail.com Sat Apr 27 19:48:54 2002 From: jmeile at hotmail.com (Josef Meile) Date: Sat, 27 Apr 2002 19:48:54 +0200 Subject: Installing source of python-ldap in suse... References: Message-ID: Finally I could do it. The LDAPUserFolder could be correctly imported into Zope. I just got the following warning: WARNING: Python C API version mismatch for module _ldap: This Python has API version 1010, module _ldap has version 1007. But I could find some users on my LDAP server. Could this warning represent problems in the future? My last question: Will the LDAPUserFolder catch automatically every user from the LDAP server who try to log in? or do I have to seach and I add them by myself? Thanks for your patience, Josef. From jens at zope.com Sat Apr 27 21:31:29 2002 From: jens at zope.com (Jens Vagelpohl) Date: Sat, 27 Apr 2002 15:31:29 -0400 Subject: Installing source of python-ldap in suse... In-Reply-To: Message-ID: <5F74E844-5A15-11D6-A8BA-00039363690C@zope.com> josef, first of all, i urge you to read more documentation in order to familiarize yourself better with basic concepts of LDAP, python and systems maintenance. the warning you get means that the ldap module has been compiled with an *older* version of python than the python that runs your zope installation. that means you did not point the python-ldap configuration/compilation machinery to the correct python binary. even if it appears to be working correctly i would try and recompile the module with the correct python binary. questions about the LDAPUserFolder are off-topic for this mailing list, which concerns itself with questions about the python-ldap module but not really with software that is built on top of python-ldap. there is no concept like the LDAPUserFolder "catching" a user from the LDAP server. all users that can be found in the LDAP branch that you configured in the user folder with the users base/users search scope settings will be usable for authentication, provided the users also have some roles attached to them in the group entries that can be found in the LDAP branch you configured in the user folder using the groups base/groups search scope settings. jens On Saturday, April 27, 2002, at 01:48 , Josef Meile wrote: > Finally I could do it. The LDAPUserFolder could be correctly imported > into Zope. I just got the following warning: > > WARNING: Python C API version mismatch for module _ldap: > This Python has API version 1010, module _ldap has version 1007. > > But I could find some users on my LDAP server. Could this warning > represent problems in the future? > > My last question: > Will the LDAPUserFolder catch automatically every user from the > LDAP server who try to log in? or do I have to seach and > I add them by myself? > > Thanks for your patience, > Josef. > > From idzikowski at atsolute.com Tue Apr 30 17:33:40 2002 From: idzikowski at atsolute.com (Jan Idzikowski) Date: Tue, 30 Apr 2002 17:33:40 +0200 Subject: problems ldap-module on AIX Message-ID: <20020430173340.240d956e.idzikowski@atsolute.com> hallo, i have a problem with the ldap-module on AIX. I use Python-2.1.3 installed thread-safe with the IBM C-Compiler cc_r and the option -qmaxmem=4000. With the normaly C-Compiler cc i could not start Zope and some other python application. (but with the the normaly C-Compiler cc python install the ldap-module is working) I have try to install the ldap-module with python setup.py build and get the warnings thet i have to compile it with the option MAXMEM > 2048. I have do this and the compile was fine. But i can't get the ldap-module working. I import ldap and than i try l = ldap.initialize("ldap://my.ldapserver:389") and get a Segmentation fault (core dumped). i think the problem is the thread-safe installation of python, and the ibm c-compiler, but i have to use this compiler and can't use a other, like gcc. have anybody a idee? thanks jan idzikowski at atsolute.com From jens at zope.com Tue Apr 30 21:01:06 2002 From: jens at zope.com (Jens Vagelpohl) Date: Tue, 30 Apr 2002 15:01:06 -0400 Subject: opening a connection with more than a single server Message-ID: <9FEC733E-5C6C-11D6-AE90-003065C7DEAE@zope.com> hi everyone, i had no idea that this ever worked, but apparently (i verified it with python-ldap 1.10alpha3) ldap.open used to accept a hostname string as argument with more than one LDAP host, separated by whitespace. this worked in a failover fashion where if one became unavailable the next one would be used to execute queries. i am noticing that this no longer works on 2.0. is this (now that i know about it...) useful feature going to be available again at some point? (either in ldap.open or through ldap.initialize) jens From michael at stroeder.com Tue Apr 30 20:12:37 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Tue, 30 Apr 2002 20:12:37 +0200 Subject: problems ldap-module on AIX References: <20020430173340.240d956e.idzikowski@atsolute.com> Message-ID: <3CCEDE95.8010103@stroeder.com> Jan Idzikowski wrote: > > i have a problem with the ldap-module on AIX. Disclaimer: I have no personal experience on AIX and IBM's C compiler. > I use Python-2.1.3 installed thread-safe with the IBM > C-Compiler cc_r and the option -qmaxmem=4000. I also have no clue what "Python-2.1.3 installed thread-safe" means. But please check the options you chose when compiling Python by issuing the following command in Python's source tree. $ ./configure --help Some candidates for further examination might be: --without-gcc never use gcc --with-signal-module disable/enable signal module --with(out)-threads[=DIRECTORY] disable/enable thread support --with(out)-thread[=DIRECTORY] deprecated; use --with(out)-threads --with-pth use GNU pth threading libraries --with(out)-cycle-gc disable/enable garbage collection --with(out)-pymalloc disable/enable specialized mallocs --with-wctype-functions use wctype.h functions --with-dl-dld=DL_DIR,DLD_DIR GNU dynamic linking --with-fpectl enable SIGFPE catching At least on Linux I had to stay away of using cycle-gc and pymalloc. (Check list archive!) Ciao, Michael. From jens at zope.com Tue Apr 30 21:08:08 2002 From: jens at zope.com (Jens Vagelpohl) Date: Tue, 30 Apr 2002 15:08:08 -0400 Subject: problems ldap-module on AIX In-Reply-To: <3CCEDE95.8010103@stroeder.com> Message-ID: <9B816122-5C6D-11D6-AE90-003065C7DEAE@zope.com> the cycle-gc option should be safe again in python-2.1.3. it was rooted out after enough people complained about their Zope servers segfaulting, cycle-gc had a bug in it that was consequently fixed for 2.1.3 jens On Tuesday, April 30, 2002, at 02:12 , Michael Str?der wrote: > Jan Idzikowski wrote: > > > > i have a problem with the ldap-module on AIX. > > Disclaimer: I have no personal experience on AIX and > IBM's C compiler. > > > I use Python-2.1.3 installed thread-safe with the IBM > > C-Compiler cc_r and the option -qmaxmem=4000. > > I also have no clue what "Python-2.1.3 installed thread-safe" means. > > But please check the options you chose when compiling Python by issuing > the following command in Python's source tree. > > $ ./configure --help > > Some candidates for further examination might be: > > --without-gcc never use gcc > --with-signal-module disable/enable signal module > --with(out)-threads[=DIRECTORY] disable/enable thread support > --with(out)-thread[=DIRECTORY] deprecated; use > --with(out)-threads > --with-pth use GNU pth threading libraries > --with(out)-cycle-gc disable/enable garbage collection > --with(out)-pymalloc disable/enable specialized mallocs > --with-wctype-functions use wctype.h functions > --with-dl-dld=DL_DIR,DLD_DIR GNU dynamic linking > --with-fpectl enable SIGFPE catching > > At least on Linux I had to stay away of using cycle-gc and pymalloc. > (Check list archive!) > > Ciao, Michael. > > > From michael at stroeder.com Tue Apr 30 23:01:11 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Tue, 30 Apr 2002 23:01:11 +0200 Subject: opening a connection with more than a single server References: <9FEC733E-5C6C-11D6-AE90-003065C7DEAE@zope.com> Message-ID: <3CCF0617.6040503@stroeder.com> Jens Vagelpohl wrote: > > i had no idea that this ever worked, but apparently (i verified it with > python-ldap 1.10alpha3) ldap.open used to accept a hostname string as > argument with more than one LDAP host, separated by whitespace. this > worked in a failover fashion where if one became unavailable the next > one would be used to execute queries. 1. You have to decide what you want. You requested the old "parameter signature" for ldap.open(). You remember? Think of defining the port number... 2. I'd recommend you drop support for old python-ldap and use ldap.initialize() directly. That's far more powerful anyway. Hmm, I would have to remove/modify the is_ldap_url(uri) in the wrapper function for handling the space separated list. But that's least of a problem. 3. IMHO it can't be used for fail-over. AFAIK simply the connection is established to the first host reachable. You could argue that it works kinda fail-over-like if you re-open the LDAP connection for the requests all the time. But if you have a persistent connection you have to implement fail-over by catching ldap.SERVER_DOWN. (I'm currently thinking about sub-classing ldapobject.LDAPObject for this purpose. Note that fail-over means repeating the whole connection-establishment sequence after connecting to the replica). Ciao, Michael. From jens at zope.com Wed May 1 19:21:55 2002 From: jens at zope.com (Jens Vagelpohl) Date: Wed, 1 May 2002 13:21:55 -0400 Subject: opening a connection with more than a single server In-Reply-To: <3CCF0617.6040503@stroeder.com> Message-ID: >> i had no idea that this ever worked, but apparently (i verified it with >> python-ldap 1.10alpha3) ldap.open used to accept a hostname string as >> argument with more than one LDAP host, separated by whitespace. this >> worked in a failover fashion where if one became unavailable the next >> one would be used to execute queries. > > 1. You have to decide what you want. You requested the old "parameter > signature" for ldap.open(). You remember? Think of defining the port > number... this actually doesn't have anything to do with my earlier complaints about ldap.open. i'm just saying that apparently back in the 1.10 days ldap.open never checked the parameter and passed it straight to the ldap libraries, which would interpret it as "if the first server is dead just use the second". > 2. I'd recommend you drop support for old python-ldap and use > ldap.initialize() directly. That's far more powerful anyway. Hmm, I would > have to remove/modify the is_ldap_url(uri) in the wrapper function for > handling the space separated list. But that's least of a problem. my software is written so that it attempts to determine what version of python-ldap is in use and behave accordingly. anything that is not version 2 and above uses ldap.open, 2.x uses ldap.initialize. simply dropping support for ldap.open is not an option, most everyone i know still uses 1. 10alpha3. but apart from that' that's not really my point. i'm just wondering if this "server fallback" behavior could find its way back into python-ldap2, regardless of connection building method used, because it seems it is supported by the OpenLDAP libraries and i am thinking it is a useful behavior. that's all. > 3. IMHO it can't be used for fail-over. AFAIK simply the connection is > established to the first host reachable. You could argue that it works > kinda fail-over-like if you re-open the LDAP connection for the requests > all the time. But if you have a persistent connection you have to > implement fail-over by catching ldap.SERVER_DOWN. (I'm currently thinking > about sub-classing ldapobject.LDAPObject for this purpose. Note that > fail-over means repeating the whole connection-establishment sequence > after connecting to the replica). it's true that the first available connection is used, and my software actually does re-open the connection every single time. so for this use case it does work like failover. if i remember correctly i had at one time tried to come up with a way to test whether a given LDAP connection object was still good for use, but could not find anything that would not already involve making LDAP queries. so i assumed the difference in overhead was not that large. let me know if i am way off-base here, i could be talking out of my rear end. i have never experimented with long-lived connections. i am all for improving performance, if you can show me a good way to test existing connection objects and tell me that it is indeed faster to do it, i'm all ears. jens From michael at stroeder.com Wed May 1 19:36:31 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Wed, 01 May 2002 19:36:31 +0200 Subject: opening a connection with more than a single server References: Message-ID: <3CD0279F.4070002@stroeder.com> Jens Vagelpohl wrote: >>> i had no idea that this ever worked, but apparently (i verified it >>> with python-ldap 1.10alpha3) ldap.open used to accept a hostname >>> string as argument with more than one LDAP host, separated by >>> whitespace. this worked in a failover fashion where if one became >>> unavailable the next one would be used to execute queries. >> >> >> 1. You have to decide what you want. You requested the old "parameter >> signature" for ldap.open(). You remember? Think of defining the port >> number... > > this actually doesn't have anything to do with my earlier complaints > about ldap.open. It has. > i'm just saying that apparently back in the 1.10 days > ldap.open never checked the parameter and passed it straight to the ldap > libraries, which would interpret it as "if the first server is dead just > use the second". Well, we can implement it like this. But it contradicts your inquiry for a parameter signature like described in the documentation. --------------------- snip ----------------------- open(host [, port=PORT]) Opens a new connection with an LDAP server, and return an LDAP object (see 1.1.4) used to perform operations on that server. host is a string containing solely the host name. port is an integer specifying the port where the LDAP server is listening (default is 389). Note: Using this function is deprecated. --------------------- snip ----------------------- We can either 1. change the docs and pass string-parameter _host_ to OpenLDAP's ldap_open() function and drop parameter _port_ or 2. comply to the docs which means not supporting the multiple-hosts-in-space-separated-list parameter. If you don't get the difference I'm pretty sure that you've never used to connect to a LDAP server running on "non-standard" port... >> 2. I'd recommend you drop support for old python-ldap and use >> ldap.initialize() directly. > > my software is written so that it attempts to determine what version of > python-ldap is in use and behave accordingly. I did that in former versions of web2ldap but dropped support for python-ldap 1.x because it's getting too complex if you want to do that completely (referrals, etc.). Also python-ldap does not support LDAPv3 which has some serious implications if you take RFC2251 ff. seriously (e.g. character set T.61 for LDAPv2 instead of UTF-8). > i'm just > wondering if this "server fallback" behavior could find its way back > into python-ldap2, regardless of connection building method used, > because it seems it is supported by the OpenLDAP libraries and i am > thinking it is a useful behavior. that's all. Someone has to decide on that. See above. > if i remember correctly i had at one time tried to come up with a way to > test whether a given LDAP connection object was still good for use, but > could not find anything that would not already involve making LDAP queries. Yes, you have to try a LDAPRequest and catch ldap.SERVER_DOWN. > i have > never experimented with long-lived connections. i am all for improving > performance, if you can show me a good way to test existing connection > objects and tell me that it is indeed faster to do it, i'm all ears. Well, if you're after performance you should work with persistent connections. Especially for web2ldap it was a great performance boost since web2ldap does a lot of special stuff after establishing the connection. Doing this for each request is a performance nightmare. Think of negotiating LDAP protocol version, evaluating RootDSE attributes, more complex bind operations, etc. Ciao, Michael. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 2922 bytes Desc: S/MIME Cryptographic Signature URL: From idzikowski at atsolute.com Thu May 2 13:27:23 2002 From: idzikowski at atsolute.com (Jan Idzikowski) Date: Thu, 2 May 2002 13:27:23 +0200 Subject: problems ldap-module on AIX In-Reply-To: <3CCEDE95.8010103@stroeder.com> References: <20020430173340.240d956e.idzikowski@atsolute.com> <3CCEDE95.8010103@stroeder.com> Message-ID: <20020502132723.0a649e13.idzikowski@atsolute.com> On Tue, 30 Apr 2002 20:12:37 +0200 Michael Str?der wrote: > Jan Idzikowski wrote: > > > > i have a problem with the ldap-module on AIX. > > Disclaimer: I have no personal experience on AIX and > IBM's C compiler. > > > I use Python-2.1.3 installed thread-safe with the IBM > > C-Compiler cc_r and the option -qmaxmem=4000. > > I also have no clue what "Python-2.1.3 installed thread-safe" means. > > But please check the options you chose when compiling Python by > issuing the following command in Python's source tree. > > $ ./configure --help > > Some candidates for further examination might be: > > --without-gcc never use gcc > --with-signal-module disable/enable signal module > --with(out)-threads[=DIRECTORY] disable/enable thread support > --with(out)-thread[=DIRECTORY] deprecated; use > --with(out)-threads > --with-pth use GNU pth threading libraries > --with(out)-cycle-gc disable/enable garbage collection > --with(out)-pymalloc disable/enable specialized mallocs > --with-wctype-functions use wctype.h functions > --with-dl-dld=DL_DIR,DLD_DIR GNU dynamic linking > --with-fpectl enable SIGFPE catching > i have python install with --without-gcc --with-pymalloc --with-cycle-gc and make CC="cc_r" OPT="-O -qmaxmem=4000" and all looks fine, but the python-ldap-modul get a Segmentation fault, i have use the dbx (aix debugger) and get the output: Segmentation fault in ldap_search_ext at 0xd10860a8 ($t1) 0xd10860a8 (ldap_search_ext+0x218) 800c0000 lwz r0,0x0(r12) greetings jan From michael at stroeder.com Thu May 2 12:37:01 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Thu, 02 May 2002 12:37:01 +0200 Subject: problems ldap-module on AIX References: <20020430173340.240d956e.idzikowski@atsolute.com> <3CCEDE95.8010103@stroeder.com> <20020502132723.0a649e13.idzikowski@atsolute.com> Message-ID: <3CD116CD.50405@stroeder.com> Jan Idzikowski wrote: > i have python install with --without-gcc --with-pymalloc --with-cycle-gc > and make CC="cc_r" OPT="-O -qmaxmem=4000" Try --without-pymalloc. Ciao, Michael. From idzikowski at atsolute.com Thu May 2 18:24:33 2002 From: idzikowski at atsolute.com (Jan Idzikowski) Date: Thu, 2 May 2002 18:24:33 +0200 Subject: problems ldap-module on AIX In-Reply-To: <3CD116CD.50405@stroeder.com> References: <20020430173340.240d956e.idzikowski@atsolute.com> <3CCEDE95.8010103@stroeder.com> <20020502132723.0a649e13.idzikowski@atsolute.com> <3CD116CD.50405@stroeder.com> Message-ID: <20020502182433.6d7fb603.idzikowski@atsolute.com> On Thu, 02 May 2002 12:37:01 +0200 Michael Str?der wrote: > Jan Idzikowski wrote: > > i have python install with --without-gcc --with-pymalloc > --with-cycle-gc > > and make CC="cc_r" OPT="-O -qmaxmem=4000" > > Try --without-pymalloc. i have install python with: ./configure --prefix=/home/ji/python --without-gcc --with-thread --without-pymalloc --with-cycle-gc with the same problem: import ldap l = ldap.initialize("ldap://myldapserver:389") Segmentation fault (core dumped) and in the core dump: reading symbolic information ... [using memory image in core] Segmentation fault in ldap_search_ext at 0xd228a0a8 ($t1) 0xd228a0a8 (ldap_search_ext+0x218) 800c0000 lwz r0,0x0(r12) do you have any idee? greeting jan From michael at stroeder.com Thu May 2 17:21:52 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Thu, 02 May 2002 17:21:52 +0200 Subject: problems ldap-module on AIX References: <20020430173340.240d956e.idzikowski@atsolute.com> <3CCEDE95.8010103@stroeder.com> <20020502132723.0a649e13.idzikowski@atsolute.com> <3CD116CD.50405@stroeder.com> <20020502182433.6d7fb603.idzikowski@atsolute.com> Message-ID: <3CD15990.1080002@stroeder.com> Jan Idzikowski wrote: > i have install python with: > ./configure --prefix=/home/ji/python --without-gcc --with-thread --without-pymalloc --with-cycle-gc 1. Check output of ./configure --help. Option --with-thread is deprecated. Use --with-threads. 2. Continue trying --without-cycle-gc as stated in my first reply. Make sure to re-compile *all* additional extension modules each time you change build options. Ciao, Michael. From jens at zope.com Thu May 2 14:32:46 2002 From: jens at zope.com (Jens Vagelpohl) Date: Thu, 2 May 2002 08:32:46 -0400 Subject: opening a connection with more than a single server In-Reply-To: <3CD0279F.4070002@stroeder.com> Message-ID: > 1. change the docs and pass string-parameter _host_ to OpenLDAP's > ldap_open() function and drop parameter _port_ or the space-separated host list was only used in place of the first argument, "host". i am not sure if you were still able to pass a "port" parameter or what would happen if you did. > 2. comply to the docs which means not supporting the > multiple-hosts-in-space-separated-list parameter. comply to which docs? the docs that came with python-ldap 1.10? i have a feeling that the author might not have realized that you can send space-separated host names in the "host" portion of the argument list. > If you don't get the difference I'm pretty sure that you've never used to > connect to a LDAP server running on "non-standard" port... well of course i have ;) all my products allow selecting nonstandard ports, too. > I did that in former versions of web2ldap but dropped support for > python-ldap 1.x because it's getting too complex if you want to do that > completely (referrals, etc.). Also python-ldap does not support LDAPv3 > which has some serious implications if you take RFC2251 ff. seriously (e. > g. character set T.61 for LDAPv2 instead of UTF-8). well, my products lack the complexity to really be worried about that. no one has ever come to me and said they couldn't get things to work using referrals :) and i don't think i'll try to provide that ability. >> if i remember correctly i had at one time tried to come up with a way to >> test whether a given LDAP connection object was still good for use, but >> could not find anything that would not already involve making LDAP >> queries. > > Yes, you have to try a LDAPRequest and catch ldap.SERVER_DOWN. is there a suitable and simple LDAP request that would work on most servers without knowing what the tree looks like? > > i have >> never experimented with long-lived connections. i am all for improving >> performance, if you can show me a good way to test existing connection >> objects and tell me that it is indeed faster to do it, i'm all ears. > > Well, if you're after performance you should work with persistent > connections. Especially for web2ldap it was a great performance boost > since web2ldap does a lot of special stuff after establishing the > connection. Doing this for each request is a performance nightmare. Think > of negotiating LDAP protocol version, evaluating RootDSE attributes, more > complex bind operations, etc. well, again, my products lack that kind of complexity... i connect, i bind, i might rebind as someone else, i read and i write. then i disconnect. nothing exotic really. jens From michael at stroeder.com Thu May 2 17:53:21 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Thu, 02 May 2002 17:53:21 +0200 Subject: opening a connection with more than a single server References: Message-ID: <3CD160F1.5030900@stroeder.com> Jens Vagelpohl wrote: >> 1. change the docs and pass string-parameter _host_ to OpenLDAP's >> ldap_open() function and drop parameter _port_ or > > the space-separated host list was only used in place of the first argument, > "host". i am not sure if you were still able to pass a "port" parameter > or what would happen if you did. Jens, can you please tell us exactly how the declaration of ldap.open() should look like and how the behaviour should be? I've changed the declaration and behaviour according to your inquiry because you complained about an incompatible change (review thread "signature of ldap.open changed?" started by you). > comply to which docs? the docs that came with python-ldap 1.10? The docs weren't changed since 1.x for ldap.open(). > i have a > feeling that the author might not have realized that you can send > space-separated host names in the "host" portion of the argument list. It's very likely that David did not know about the space-separated multiple host feature. I'm not even sure if it was available in OpenLDAP 1.x or Netscape 3 libs. But he defined the ldap.open() function's interface and old code might be using it. As I understood your former inquiry you relied on this old behaviour of ldap.open(). Now you're requesting a change. >> I did that in former versions of web2ldap but dropped support for >> python-ldap 1.x because it's getting too complex if you want to do >> that completely (referrals, etc.). Also python-ldap does not support >> LDAPv3 which has some serious implications if you take RFC2251 ff. >> seriously (e. >> g. character set T.61 for LDAPv2 instead of UTF-8). > > well, my products lack the complexity to really be worried about that. Not caring about the complexity does not mean the complexity isn't there... >> Yes, you have to try a LDAPRequest and catch ldap.SERVER_DOWN. > > is there a suitable and simple LDAP request that would work on most > servers without knowing what the tree looks like? Read RootDSE on LDAPv3 servers. Well, that means you have to know the protocol version. Also the node of the search root has to be existent. But IMHO you're thinking the wrong way: Just try the LDAPRequest, catch ldap.SERVER_DOWN, re-connect/-bind and re-try same LDAPRequest if necessary. >>> never experimented with long-lived connections. i am all for >>> improving performance, >> >> Well, if you're after performance you should work with persistent >> connections. Especially for web2ldap it was a great performance boost >> since web2ldap does a lot of special stuff after establishing the >> connection. Doing this for each request is a performance nightmare. >> Think of negotiating LDAP protocol version, evaluating RootDSE >> attributes, more complex bind operations, etc. > > well, again, my products lack that kind of complexity... Are you sure? ;-) > i connect, i bind, > i might rebind as someone else, i read and i write. then i disconnect. > nothing exotic really. Not exotical but too many LDAPRequests for being a light-weight authenticating process. Also think of the costs when doing a connect, especially SSL/TLS connection establishment. Ciao, Michael. From jens at zope.com Thu May 2 18:31:33 2002 From: jens at zope.com (Jens Vagelpohl) Date: Thu, 2 May 2002 12:31:33 -0400 Subject: opening a connection with more than a single server In-Reply-To: <3CD160F1.5030900@stroeder.com> Message-ID: <10601E23-5DEA-11D6-AB42-003065C7DEAE@zope.com> > Jens, can you please tell us exactly how the declaration of ldap.open() > should look like and how the behaviour should be? > > I've changed the declaration and behaviour according to your inquiry > because you complained about an incompatible change (review thread > "signature of ldap.open changed?" started by you). the declaration of ldap.open doesn't have anything to do with this as far as i know. actually, it now even seems to be supporting two ways to get a connection: ldap.open( 'server:port' ) ldap.open( 'server', port ) the only item (for this discussion) that i am looking it is the string 'host'. i know that under 1.10 i could do.. ldap.open( 'host1 host2 host3' ) assuming they all run on the default port. i'm not interested in non-default ports here. if i execute the same on python2.1 and python-ldap 2.0pre4 i am getting a strange error: ldap.LDAPError: ( 2, 'No such file or directory' ). don't get me wrong, i am not trying to warm over some ldap.open declaration semantics. i'm not even trying to badger anyone into bringing back this type of behavior, i'm just curious if that's something that would be interesting for others and the package as a whole. >> It's very likely that David did not know about the space-separated >> multiple host feature. I'm not even sure if it was available in OpenLDAP >> 1.x or Netscape 3 libs. But he defined the ldap.open() function's >> interface and old code might be using it. > > As I understood your former inquiry you relied on this old behaviour of > ldap.open(). Now you're requesting a change. this whole deal with the space-separated hosts is nothing i ever relied on (because i did not know it even worked at all). it was brought to my attention by someone else who had been doing it previously and it failed after he upgraded python-ldap. i appreciate your comments regarding long-lived connections. i'll start using it when i have time to do more tinkering. jens From michael at stroeder.com Thu May 2 18:41:52 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Thu, 02 May 2002 18:41:52 +0200 Subject: opening a connection with more than a single server References: <10601E23-5DEA-11D6-AB42-003065C7DEAE@zope.com> Message-ID: <3CD16C50.1030605@stroeder.com> Jens Vagelpohl wrote: > ldap.open doesn't have anything to do with this as > far as i know. It has. Please think again. > actually, it now even seems to be supporting two ways to > get a connection: > > ldap.open( 'server:port' ) > ldap.open( 'server', port ) Note: The exact declaration is currently: def open(host,port=389,trace_level=0,trace_file=sys.stdout): > the only item (for this discussion) that i am looking it is the > string 'host'. i know that under 1.10 i could do.. > > ldap.open( 'host1 host2 host3' ) Which is not possible with separate key-word parameter _port_ in function parameter list of ldap.open(). > assuming they all run on the default port. i'm not interested > in non-default ports here. Are you kidding? > don't get me wrong, i am not trying to warm over some ldap.open > declaration semantics. If it's just an interesting thing go and use ldap.initialize() and we'll close this thread. Ciao, Michael. From michael at stroeder.com Sat May 4 20:51:37 2002 From: michael at stroeder.com (=?ISO-8859-15?Q?Michael_Str=F6der?=) Date: Sat, 04 May 2002 20:51:37 +0200 Subject: Finally added Hans' SASL and schema patches Message-ID: <3CD42DB9.9040507@stroeder.com> HI! Finally I've added the schema and SASL patches contributed by Hans a couple of weeks ago. I did not make noteable changes so far. !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Note that some parts of the API needs additional work. Nobody should rely on any parts of this implementation for now. Any complaints about "incompatible" changes made are redirected to /dev/null. !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interested developers are encouraged to sync their local CVS tree, look at the API and comment. Ciao, Michael. From Godwin.Vaz at eng.monash.edu.au Mon May 6 06:03:55 2002 From: Godwin.Vaz at eng.monash.edu.au (Godwin Vaz) Date: Mon, 06 May 2002 14:03:55 +1000 Subject: Zope Ldap Message-ID: <3CD600AB.F22E3F8B@eng.monash.edu.au> Hi All I am new to Zope and am trying to install Ldap under Win2000. I have downloaded the .dll anf python files as instructed on Barrys page. I get the following errors 2002-05-06T03:36:26 PROBLEM(100) Init Ambiguous name for method of Products.LDAP Adapter.LDAPAdapter.LDAPAdapter: "manage_main" != "manage" ------ 2002-05-06T03:36:28 PROBLEM(100) Init Ambiguous name for method of Products.Pyth onMethod.PythonMethod.PythonMethod: "manage" != "manage_main" D:\zope\lib\python\Products\UserLDAP\UserLDAP.py:11: DeprecationWarning: the reg ex module is deprecated; please use the re module import Globals, App.Undo, socket, regex ------ 2002-05-06T03:36:29 PROBLEM(100) Init Ambiguous name for method of Products.User LDAP.UserLDAP.UserLDAP: "_mainUser" != "manage" ------ 2002-05-06T03:36:29 PROBLEM(100) Init Ambiguous name for method of Products.User LDAP.UserLDAP.UserLDAP: "_mainUser" != "manage_main" ------ 2002-05-06T03:36:31 PROBLEM(100) Init Ambiguous name for method of Products.ZLDA PConnection.Entry.ZopeEntry: "manage_attributes" != "manage_main" Your help is appreciated. Thanks once again Regards Godwin -------------- next part -------------- A non-text attachment was scrubbed... Name: Godwin.Vaz.vcf Type: text/x-vcard Size: 372 bytes Desc: Card for Godwin Vaz URL: From jens at zope.com Mon May 6 06:22:17 2002 From: jens at zope.com (Jens Vagelpohl) Date: Mon, 6 May 2002 00:22:17 -0400 Subject: Zope Ldap In-Reply-To: <3CD600AB.F22E3F8B@eng.monash.edu.au> Message-ID: godwin, these errors have nothing to do with python-ldap and are off-topic on this list. however, knowing some of the zope products listed i can give you some more or less useful information: all the product that give you these error messages (UserLDAP, LDAPAdapter, ZLAPConnection) have not been updated in over a year (LDAPAdapter) and over 2 years (UserLDAP, ZLDAPConnection), respectively. i'm not sure where you got them or who told you to download them, i would not even be able to find them anymore. if your goal is to authenticate users in zope using records from a LDAP database i suggest you look at the LDAPUserFolder: http://www.dataflake.org/software/ldapuserfolder jens On Monday, May 6, 2002, at 12:03 , Godwin Vaz wrote: > Hi All > > I am new to Zope and am trying to install Ldap under Win2000. I have > downloaded the .dll anf python files as instructed on Barrys page. I get > the following errors > > 2002-05-06T03:36:26 PROBLEM(100) Init Ambiguous name for method of > Products.LDAP > Adapter.LDAPAdapter.LDAPAdapter: "manage_main" != "manage" > ------ > 2002-05-06T03:36:28 PROBLEM(100) Init Ambiguous name for method of > Products.Pyth > onMethod.PythonMethod.PythonMethod: "manage" != "manage_main" > D:\zope\lib\python\Products\UserLDAP\UserLDAP.py:11: DeprecationWarning: > the reg > ex module is deprecated; please use the re module > import Globals, App.Undo, socket, regex > ------ > 2002-05-06T03:36:29 PROBLEM(100) Init Ambiguous name for method of > Products.User > LDAP.UserLDAP.UserLDAP: "_mainUser" != "manage" > ------ > 2002-05-06T03:36:29 PROBLEM(100) Init Ambiguous name for method of > Products.User > LDAP.UserLDAP.UserLDAP: "_mainUser" != "manage_main" > ------ > 2002-05-06T03:36:31 PROBLEM(100) Init Ambiguous name for method of > Products.ZLDA > PConnection.Entry.ZopeEntry: "manage_attributes" != "manage_main" > > Your help is appreciated. > > Thanks once again > > Regards > Godwin > From holbert.13 at osu.edu Thu May 9 21:01:05 2002 From: holbert.13 at osu.edu (Richard Holbert) Date: Thu, 09 May 2002 15:01:05 -0400 Subject: Type Problem with Exception Handler Error Messages In-Reply-To: <3CA38D25.5090208@stroeder.com> References: <200203281927.OAA08090@mail4.uts.ohio-state.edu> <3CA38D25.5090208@stroeder.com> Message-ID: <0GVU006O2Y5KB0@mail-mta5.service.ohio-state.edu> Dear Michael, I've ran into a problem with the way the exception handler error messages are returned by python-ldap. I'm trying to print exception handler error messages using the following: try: l.simple_bind_s(login_dn, login_pw) except ldap.LDAPError, e: print "BINDFAILED", e sys.exit(1) But it doesn't work as expected. Here's what I get: BINDFAILED {'desc': "Can't contact LDAP server"} This is because your exception handler returns e is an instance of a dictionary, so I have to do the following: try: l.simple_bind_s(login_dn, login_pw) except ldap.LDAPError, e: print "BINDFAILED", e.args[0]['desc'] sys.exit(1) To get: BINDFAILED Can't contact LDAP server The exception handler in IO is a bit more straight forward. This example: try: input = open("config.txt","r") except IOError, e: print "CONFIGFILEERROR", e sys.exit(1) yields the following: CONFIGFILEERROR [Errno 2] No such file or directory: 'config.txt' Thanks! Rick From michael at stroeder.com Thu May 9 23:13:02 2002 From: michael at stroeder.com (Michael =?iso-8859-1?Q?Str=F6der?=) Date: Thu, 09 May 2002 23:13:02 +0200 Subject: Type Problem with Exception Handler Error Messages References: <200203281927.OAA08090@mail4.uts.ohio-state.edu> <3CA38D25.5090208@stroeder.com> <0GVU006O2Y5KB0@mail-mta5.service.ohio-state.edu> Message-ID: <3CDAE65E.88BF2B19@stroeder.com> Richard Holbert schrieb: > > I've ran into a problem with the way the exception handler error messages are > returned by python-ldap. > [..] > BINDFAILED {'desc': "Can't contact LDAP server"} > > This is because your exception handler returns e is an instance of a > dictionary, so I have to do the following: > [..] > The exception handler in IO is a bit more straight forward. Feel free to contribute an implementation of a __str__() method for the LDAPError class. Should be in Modules/error.c. Ciao, Michael. From idzikowski at atsolute.com Mon May 6 15:36:56 2002 From: idzikowski at atsolute.com (Jan Idzikowski) Date: Mon, 6 May 2002 15:36:56 +0200 Subject: probleme with escape-encoding Message-ID: <20020506153656.275dd546.idzikowski@atsolute.com> hallo, i have a problem with special german characters like "?", "?", "?", "?". in ldap it is '\xc3\xbc' for '?', but when i to get the user from ldapuserfolder, i always get: Error Type: UnicodeError Error Value: ASCII decoding error: ordinal not in range(128) in utils.py line 70 'return unicode( st )' the problem is, that my entries in utf-8 and when i try unicode(st, 'utf-8') it works have anybody a same problem? From jens at zope.com Mon May 6 14:16:32 2002 From: jens at zope.com (Jens Vagelpohl) Date: Mon, 6 May 2002 08:16:32 -0400 Subject: probleme with escape-encoding In-Reply-To: <20020506153656.275dd546.idzikowski@atsolute.com> Message-ID: <19F2F1CA-60EB-11D6-8249-00039363690C@zope.com> jan, this list is for the python-ldap module, not for products built on top of it. if you have a problem with the LDAPUserFolder please file a tracker issue: http://www.dataflake.org/software/tracker jens On Monday, May 6, 2002, at 09:36 , Jan Idzikowski wrote: > hallo, > > i have a problem with special german characters like "?", "?", "?", "?". > in ldap it is '\xc3\xbc' for '?', but when i to get the user from > ldapuserfolder, > i always get: > Error Type: UnicodeError > Error Value: ASCII decoding error: ordinal not in range(128) > > in utils.py line 70 'return unicode( st )' > > the problem is, that my entries in utf-8 and when i try unicode(st, > 'utf-8') it works > > have anybody a same problem? > > From t.kloppenburg at billiton.de Mon May 13 13:33:53 2002 From: t.kloppenburg at billiton.de (Tjabo Kloppenburg) Date: Mon, 13 May 2002 13:33:53 +0200 Subject: modify method Message-ID: <200205131333.53089.t.kloppenburg@billiton.de> hi, I'm doing some tests with python ldap. bind and search work well, but now I'ld like to take a search result, modify a single field, and modify the "data row" in the ldap database. ldap.modify() takes a dn and a modifyList, but that seems to be complicated. Is there a way to use the search result to generate the modify list? tk. + Tired of coding perl+ldap. + From michael at stroeder.com Mon May 13 17:29:03 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Mon, 13 May 2002 17:29:03 +0200 Subject: modify method References: <200205131333.53089.t.kloppenburg@billiton.de> Message-ID: <3CDFDBBF.3020705@stroeder.com> Tjabo Kloppenburg wrote: > > I'm doing some tests with python ldap. > bind and search work well, but now I'ld like to take a search result, modify a single field, > and modify the "data row" in the ldap database. > > ldap.modify() takes a dn and a modifyList, but that seems to be complicated. > Is there a way to use the search result to generate the modify list? If you only want to modify a single attribute it's probably overkill to look into (undocumented) sub module ldap.modlist. Ciao, Michael. From t.kloppenburg at billiton.de Thu May 16 15:54:29 2002 From: t.kloppenburg at billiton.de (Tjabo Kloppenburg) Date: Thu, 16 May 2002 15:54:29 +0200 Subject: Fwd: Re: python-ldap-2.0.0pre04 and _ldap Message-ID: <200205161554.29488.t.kloppenburg@billiton.de> hi jelske :), > But when I want to use the installed python-ldap, > /usr/local/lib/python2.2/site-packages/ldap/__init__.py > wants to import from _ldap > and that isn't anywhere. > in site-packages is a _ldap.so. that's allright. On my suse linux box it is the same - and it runs. the "from _ldap import *" loads _ldap.so -- I think it is the lib form of a package. or a extension library for python, written in C. this is the output of "rpm -ql python-ldap-2.0.0pre04-45" on my suse box: /usr/lib/python2.2/site-packages/_ldap.so /usr/lib/python2.2/site-packages/ldap /usr/lib/python2.2/site-packages/ldap/__init__.py /usr/lib/python2.2/site-packages/ldap/__init__.pyc /usr/lib/python2.2/site-packages/ldap/async.py /usr/lib/python2.2/site-packages/ldap/async.pyc /usr/lib/python2.2/site-packages/ldap/functions.py /usr/lib/python2.2/site-packages/ldap/functions.pyc /usr/lib/python2.2/site-packages/ldap/ldapobject.py /usr/lib/python2.2/site-packages/ldap/ldapobject.pyc /usr/lib/python2.2/site-packages/ldap/modlist.py /usr/lib/python2.2/site-packages/ldap/modlist.pyc /usr/lib/python2.2/site-packages/ldapurl.py /usr/lib/python2.2/site-packages/ldapurl.pyc /usr/lib/python2.2/site-packages/ldif.py /usr/lib/python2.2/site-packages/ldif.pyc /usr/share/doc/packages/python-ldap /usr/share/doc/packages/python-ldap/README tk. -- Mit freundlichem Gru? Tjabo Kloppenburg - billiton internet services GmbH Entwicklung+Technik From michael at stroeder.com Fri May 17 12:43:18 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Fri, 17 May 2002 12:43:18 +0200 Subject: python-ldap-2.0.0pre04 References: <3791A9B1-6967-11D6-AE35-0030657031EA@Bi.FhG.de> Message-ID: <3CE4DEC6.5020402@stroeder.com> Jelske.Kloppenburg at Bi.FhG.de wrote: > > I just tried it once more. First I cleared the site-packages folder. > Then I began with python-ldap-2.0.0pre04.tar.gz. > After having installed I still get that > ImportError: Failure linking new module I have no idea about Mac OSX. Under Linux I would try ldd to check if the shared libs are found. Ciao, Michael. From t.kloppenburg at billiton.de Fri May 17 11:45:53 2002 From: t.kloppenburg at billiton.de (Tjabo Kloppenburg) Date: Fri, 17 May 2002 11:45:53 +0200 Subject: fetching the structure / obejctClass defs Message-ID: <200205171145.53395.t.kloppenburg@billiton.de> hi, is it possible to fetch a list of all supported objectClasses and their attribs from an openLDAP server using python-ldap? how? thanks in advance, Tjabo Kloppenburg - billiton internet services GmbH Entwicklung+Technik From stroh at scan-plus.de Fri May 17 12:15:37 2002 From: stroh at scan-plus.de (Igor Stroh) Date: 17 May 2002 12:15:37 +0200 Subject: fetching the structure / obejctClass defs In-Reply-To: <200205171145.53395.t.kloppenburg@billiton.de> References: <200205171145.53395.t.kloppenburg@billiton.de> Message-ID: <1021630537.1723.8.camel@neo> Heya, On Fre, 2002-05-17 at 11:45, Tjabo Kloppenburg wrote: > is it possible to fetch a list of all supported objectClasses and their attribs > from an openLDAP server using python-ldap? > > how? actually it is, but the representation type of attributeTypes and objectClasses is 'string', so you get a list of strings that you first have to parse somehow if you want to utilize it... see [1], it works for me, but I'm pretty sure I left some schema attributes (like STRUCTURAL etc.) out, so I might just as well fail to return correct values for your own objectClass... it parses 'person' pretty neat though... HTH, Igor [1]: #!/usr/bin/python2.1 import ldap import string import re # bind credentials ADMIN_DN = "" ADMIN_PW = "" # which objectClass(es) should we look for O_CLASSES = ['person'] # some global RE objects search_oid = re.compile("^\\( (?P\d.*\d) .*") search_name = re.compile(".* NAME '(?P[a-zA-Z0-9_]+)' .*") search_desc = re.compile(".* DESC '(?P.*)' .*") search_must_attr = re.compile(".*\s?MUST\s?\\(\s?(?P[a-zA-Z0-9_$\ ]+)\s?\\)\s?.*") search_may_attr = re.compile(".*\s?MAY\s?\\(\s?(?P[a-zA-Z0-9_$ ]+)\s?\\)\s?.*") def init_all(): filter = ".*NAME '%s'.*" % O_CLASSES[0] del O_CLASSES[0] for cl in O_CLASSES: filter += "|.*NAME '%s'.*" % O_CLASSES[0] ls = re.compile(filter, re.I) con = ldap.open("212.75.33.146") con.simple_bind_s(ADMIN_DN,ADMIN_PW) entries = con.search_s("cn=Subschema", ldap.SCOPE_BASE, \ "objectclass=subschema", ['objectclasses']) ocs = entries[0][1]['objectClasses'] return [oc for oc in ocs if ls.search(oc)] def parseIt(): ocs = init_all() ret = {} for i in ocs: key = search_name.sub("\g", i) ret[key] = {} ret[key]['OID'] = search_oid.sub("\g", i) if search_desc.search(i): ret[key]['DESC'] = search_desc.sub("\g", i) else: ret[key]['DESC'] = "" if search_must_attr.search(i): ret[key]['MUST'] = [attr.strip() for attr in \ string.split(search_must_attr.sub("\g", i), "$")] else: ret[key]['MUST'] = [] if search_may_attr.search(i): ret[key]['MAY'] = [attr.strip() for attr in \ string.split(search_may_attr.sub("\g", i), "$")] else: ret[key]['MAY'] = [] return ret def getAllAttributes(): attrs = [] ocs = parseIt() for key in ocs.keys(): attrs = attrs + ocs[key]['MUST'] + ocs[key]['MAY'] return attrs -- ScanPlus GmbH NOC Ulm - Germany - Griesbadgasse 7-13 - D 89073 Ulm TEL +49 731 920 13 100 - FAX +49 731 920 13 290 eMail: support at scan-plus.de Amtsgericht Ulm - HRB3220 - Geschaeftsfuehrer: Juergen Hoermann -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 232 bytes Desc: This is a digitally signed message part URL: From michael at stroeder.com Fri May 17 20:37:04 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Fri, 17 May 2002 20:37:04 +0200 Subject: fetching the structure / obejctClass defs References: <200205171145.53395.t.kloppenburg@billiton.de> <1021630537.1723.8.camel@neo> Message-ID: <3CE54DD0.5080900@stroeder.com> Igor Stroh wrote: > > actually it is, but the representation type of attributeTypes and > objectClasses is 'string', so you get a list of strings that you first > have to parse somehow if you want to utilize it... see [1], it works for > me, but I'm pretty sure I left some schema attributes (like STRUCTURAL > etc.) out, so I might just as well fail to return correct values for > your own objectClass... it parses 'person' pretty neat though... I've recently checked in the patches contributed by Hans Aschauer which wraps the schema parser function in OpenLDAP 2 libs and provide a new sub module ldap.schema. I also made some changes to the Python parts which I will check in this weekend. Well, full LDAPv3 schema support goes far beyond just knowing which attribute types are required/allowed. Especially it would be very interesting to implement a set of syntaxes with their matching rules. This would allow to check the syntax of data input and display the data with more a-priori knowledge. Ciao, Michael. From davidma at eskimo.com Thu May 16 22:06:21 2002 From: davidma at eskimo.com (David Margrave) Date: Thu, 16 May 2002 13:06:21 -0700 (PDT) Subject: start TLS Message-ID: Hello, I'm interested in working on TLS support in python-ldap. I have been meaning look at this for some time, but only recently I finally downloaded and worked a bit with openldap 2.0.23. My understanding of client TLS support (i.e. command line tools like ldapsearch, or apps that use libldap) is the following: 1) it enforces the requirement that the subject DN in the certificate contain the FQDN of the hostname you supplied, 2) if the FQDN does not match the cn in the subject DN, it will look in the subjectAltName extension for a match. This is helpful for load balancers scenarios where the FQDN would not match the subject DN, 3) no CA certificate checking is done. Supposedly steps 1 and 2 are to guard against man-in-the-middle attacks, but I can't find any reference anywhere for how to configure a client with a local store of 'trusted root CA certificates'. This means that a man-in-the-middle attack is still possible. Can anyone provide a bit of insight? Maybe the python-ldap module could be made a bit more flexible than client apps like ldapsearch in this regard, say by giving the developer the option of providing certificate verification callbacks, etc. Thanks, Dave From michael at stroeder.com Sun May 26 12:32:29 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Sun, 26 May 2002 12:32:29 +0200 Subject: start TLS References: Message-ID: <3CF0B9BD.2050107@stroeder.com> David Margrave wrote: > > but I can't find any reference anywhere for how to configure a client with > a local store of 'trusted root CA certificates'. This means that a > man-in-the-middle attack is still possible. You can set it globally. ldap.set_option(ldap.OPT_X_TLS_CACERTFILE,'/etc/ssl.crt/cacerts.pem') or ldap.set_option(ldap.OPT_X_TLS_CACERTDIR,'/etc/ssl.crt') I didn't succeed setting it for a specific LDAP connection but I may have done something wrong. > Can anyone provide a bit of insight? Maybe the python-ldap module could > be made a bit more flexible than client apps like ldapsearch in this > regard, say by giving the developer the option of providing certificate > verification callbacks, etc. IMHO we have to rely on what the OpenLDAP 2 libs are exposing. Certificate validation is IMHO incomplete at the moment e.g. CRL checking is not supported. In theory OpenSSL's SSL context structure would allow to set such a call-back function. But i doubt we can access that safely through OpenLDAP's API. Off course you're welcome to dig into this and find a better solution. Ciao, Michael. From stroh at scan-plus.de Mon May 27 13:39:47 2002 From: stroh at scan-plus.de (Igor Stroh) Date: 27 May 2002 13:39:47 +0200 Subject: Size limit exceeded Message-ID: <1022499587.25945.6.camel@neo> Hi there, I've got a little problem here with big result sets returned by the search_s method... my slapd has a sizelimit set to 500 (default value)... now the thing is, I have about 20.000 entries returned by a search query and I really need _all_ of them... so is there a possibility to say 'give me just result[start:offset] records for this search'? I mean I could increase the sizelimit for the ldap server but that would affect other applications then my python apps (e.g. GQ) either which I definitely want to avoid... Also, I could run the query asynchronously using search() instead of search_s(), but that would require to rewrite a whole bunch of my apps... Any suggestions? TIA, Igor -- ScanPlus GmbH NOC Ulm - Germany - Griesbadgasse 7-13 - D 89073 Ulm TEL +49 731 920 13 100 - FAX +49 731 920 13 290 eMail: support at scan-plus.de Amtsgericht Ulm - HRB3220 - Geschaeftsfuehrer: Juergen Hoermann -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 232 bytes Desc: This is a digitally signed message part URL: From michael at stroeder.com Mon May 27 14:01:33 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Mon, 27 May 2002 14:01:33 +0200 Subject: Size limit exceeded References: <1022499587.25945.6.camel@neo> Message-ID: <3CF2201D.2060304@stroeder.com> Igor Stroh wrote: > > I've got a little problem here with big result sets returned by the > search_s method... my slapd has a sizelimit set to 500 (default > value)... now the thing is, I have about 20.000 entries returned by a > search query and I really need _all_ of them... Several possibilities: 1. Ask the admin to increase the size limit. 2. Do several searches. > so is there a > possibility to say 'give me just result[start:offset] records for this > search'? No. You can just limit search by everything you can mention in a LDAP search filter. A general LDAP forum is more appropriate to discuss this. > I mean I could increase the sizelimit for the ldap server but > that would affect other applications then my python apps Some LDAP servers can impose different size limits for different bind DNs. Usually the Manager DN is not affected by size limits and ACLs. Ciao, Michael. From t.kloppenburg at billiton.de Tue May 21 09:05:58 2002 From: t.kloppenburg at billiton.de (Tjabo Kloppenburg) Date: Tue, 21 May 2002 09:05:58 +0200 Subject: fetching the structure / obejctClass defs In-Reply-To: <3CE54DD0.5080900@stroeder.com> References: <200205171145.53395.t.kloppenburg@billiton.de> <1021630537.1723.8.camel@neo> <3CE54DD0.5080900@stroeder.com> Message-ID: <200205210905.58139.t.kloppenburg@billiton.de> hi, > I've recently checked in the patches contributed by Hans Aschauer > which wraps the schema parser function in OpenLDAP 2 libs and > provide a new sub module ldap.schema... so this is part of python-ldap? tk. -- Mit freundlichem Gru? Tjabo Kloppenburg - billiton internet services GmbH Entwicklung+Technik From michael at stroeder.com Sun May 26 12:32:29 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Sun, 26 May 2002 12:32:29 +0200 Subject: start TLS References: Message-ID: <3CF0B9BD.2050107@stroeder.com> David Margrave wrote: > > but I can't find any reference anywhere for how to configure a client with > a local store of 'trusted root CA certificates'. This means that a > man-in-the-middle attack is still possible. You can set it globally. ldap.set_option(ldap.OPT_X_TLS_CACERTFILE,'/etc/ssl.crt/cacerts.pem') or ldap.set_option(ldap.OPT_X_TLS_CACERTDIR,'/etc/ssl.crt') I didn't succeed setting it for a specific LDAP connection but I may have done something wrong. > Can anyone provide a bit of insight? Maybe the python-ldap module could > be made a bit more flexible than client apps like ldapsearch in this > regard, say by giving the developer the option of providing certificate > verification callbacks, etc. IMHO we have to rely on what the OpenLDAP 2 libs are exposing. Certificate validation is IMHO incomplete at the moment e.g. CRL checking is not supported. In theory OpenSSL's SSL context structure would allow to set such a call-back function. But i doubt we can access that safely through OpenLDAP's API. Off course you're welcome to dig into this and find a better solution. Ciao, Michael. From astreib at indiana.edu Wed May 22 23:42:55 2002 From: astreib at indiana.edu (Allan Streib) Date: Wed, 22 May 2002 16:42:55 -0500 (EST) Subject: problem with modify using python-ldap Message-ID: Using python-ldap-python-ldap-2.0.0pre04 with Python 2.2.1, and OpenLDAP 2.0.23 on RedHat Linux. Deleting and adding an attribute is taking 9 seconds. The same actions using ldapmodify(1) on the command line complete virtually instantly. Oddly, the behavior does not manifest if log level 1 (trace function calls) is enabled. Here is a log snip at level 6 (heavy trace debugging plus debug packet handling). This first snip is with ldapmodify(1) as the client: do_modify: dn (iuEduUPID=9900000004,ou=people,dc=iu,dc=edu) modifications: delete: iuEduMailDeliveryTo add: iuEduMailDeliveryTo ldbm_back_modify: ldbm_modify_internal: delete removing entire attribute iuEduMailDeliveryTo ldbm_modify_internal: add ldbm_modify_internal: replace ldbm_modify_internal: replace send_ldap_result: 0:: Now with python-ldap's modify_s operation as the client: do_modify: dn (iuEduUPID=9900000004,ou=people,dc=iu,dc=edu) modifications: delete: iuEduMailDeliveryTo add: iuEduMailDeliveryTo ldbm_back_modify: ldbm_modify_internal: delete removing entire attribute iuEduMailDeliveryTo ldbm_modify_internal: add ldbm_modify_internal: replace ldbm_modify_internal: replace ldap_read: want=1 error=Resource temporarily unavailable **** hangs for about 9 seconds here **** send_ldap_result: 0:: Ultimately the modifications succeed. The difference I see is that "Resource temporarily unavailable" message -- what is that resource? Any ideas for further troubleshooting? Thanks, Allan From hyc at highlandsun.com Thu May 23 00:16:16 2002 From: hyc at highlandsun.com (Howard Chu) Date: Wed, 22 May 2002 15:16:16 -0700 Subject: problem with modify using python-ldap In-Reply-To: Message-ID: The "Resource Unavailable" message means a system call failed with errno = EAGAIN. In this case, ldap_read was trying to read from the connection and there was no data available. Sounds like the python library isn't flushing the entire LDAP request out to the network. The fact that the behavior changes with debug level set to 1 is disturbing. It sounds like some kind of race condition is involved, because there should not be any ldap_read's occurring between the modify completing and sending its result. You didn't specify which version of RedHat Linux you're using, but I'd guess you're on 7.0 or 7.1 with a buggy threads library. -- Howard Chu Chief Architect, Symas Corp. Director, Highland Sun http://www.symas.com http://highlandsun.com/hyc Symas: Premier OpenSource Development and Support > -----Original Message----- > From: owner-openldap-software at OpenLDAP.org > [mailto:owner-openldap-software at OpenLDAP.org]On Behalf Of Allan Streib > Using python-ldap-python-ldap-2.0.0pre04 with Python 2.2.1, and OpenLDAP > 2.0.23 on RedHat Linux. > > Deleting and adding an attribute is taking 9 seconds. The same actions > using ldapmodify(1) on the command line complete virtually instantly. > > Oddly, the behavior does not manifest if log level 1 (trace function > calls) is enabled. Here is a log snip at level 6 (heavy trace debugging > plus debug packet handling). This first snip is with ldapmodify(1) as the > client: > > do_modify: dn (iuEduUPID=9900000004,ou=people,dc=iu,dc=edu) > modifications: > delete: iuEduMailDeliveryTo > add: iuEduMailDeliveryTo > ldbm_back_modify: > ldbm_modify_internal: delete > removing entire attribute iuEduMailDeliveryTo > ldbm_modify_internal: add > ldbm_modify_internal: replace > ldbm_modify_internal: replace > send_ldap_result: 0:: > > Now with python-ldap's modify_s operation as the client: > > do_modify: dn (iuEduUPID=9900000004,ou=people,dc=iu,dc=edu) > modifications: > delete: iuEduMailDeliveryTo > add: iuEduMailDeliveryTo > ldbm_back_modify: > ldbm_modify_internal: delete > removing entire attribute iuEduMailDeliveryTo > ldbm_modify_internal: add > ldbm_modify_internal: replace > ldbm_modify_internal: replace > ldap_read: want=1 error=Resource temporarily unavailable > **** hangs for about 9 seconds here **** > send_ldap_result: 0:: > > Ultimately the modifications succeed. The difference I see is that > "Resource temporarily unavailable" message -- what is that resource? > > Any ideas for further troubleshooting? > > Thanks, > > Allan > From astreib at indiana.edu Thu May 23 00:38:12 2002 From: astreib at indiana.edu (Allan Streib) Date: Wed, 22 May 2002 17:38:12 -0500 (EST) Subject: problem with modify using python-ldap In-Reply-To: Message-ID: On Wed, 22 May 2002, Howard Chu wrote: > The "Resource Unavailable" message means a system call failed with errno > = EAGAIN. In this case, ldap_read was trying to read from the connection > and there was no data available. Sounds like the python library isn't > flushing the entire LDAP request out to the network. > > The fact that the behavior changes with debug level set to 1 is > disturbing. It sounds like some kind of race condition is involved, > because there should not be any ldap_read's occurring between the modify > completing and sending its result. You didn't specify which version of > RedHat Linux you're using, but I'd guess you're on 7.0 or 7.1 with a > buggy threads library. It is RedHat 7.2. I am not seeing this anymore, the last hour or so. Have not changed anything, but start and stop slapd a bunch of times with different debugging levels. It's back to my normal 256 at this time, and everything is working. ??? Allan From michael at stroeder.com Mon May 27 20:36:11 2002 From: michael at stroeder.com (=?ISO-8859-15?Q?Michael_Str=F6der?=) Date: Mon, 27 May 2002 20:36:11 +0200 Subject: fetching the structure / obejctClass defs References: <200205171145.53395.t.kloppenburg@billiton.de> <1021630537.1723.8.camel@neo> <3CE54DD0.5080900@stroeder.com> <200205210905.58139.t.kloppenburg@billiton.de> Message-ID: <3CF27C9B.8000309@stroeder.com> Tjabo Kloppenburg wrote: > >>I've recently checked in the patches contributed by Hans Aschauer >>which wraps the schema parser function in OpenLDAP 2 libs and >>provide a new sub module ldap.schema... > > so this is part of python-ldap? It *will be* part of python-ldap. It's not ready to be released yet. BTW: Just fetching objectClass definition is the easy part of the show. We have to think about how to implement and register known LDAP syntaxes for displaying, matching and input handling. Ciao, Michael. From michael at stroeder.com Mon May 27 21:12:17 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Mon, 27 May 2002 21:12:17 +0200 Subject: problem with modify using python-ldap References: Message-ID: <3CF28511.3060404@stroeder.com> Howard Chu wrote: >> do_modify: dn (iuEduUPID=9900000004,ou=people,dc=iu,dc=edu) >> modifications: >> delete: iuEduMailDeliveryTo >> add: iuEduMailDeliveryTo >> ldbm_back_modify: >> ldbm_modify_internal: delete >> removing entire attribute iuEduMailDeliveryTo >> ldbm_modify_internal: add >> ldbm_modify_internal: replace >> ldbm_modify_internal: replace >> ldap_read: want=1 error=Resource temporarily unavailable >> **** hangs for about 9 seconds here **** >> send_ldap_result: 0:: > > The "Resource Unavailable" message means a system call failed with errno = > EAGAIN. In this case, ldap_read was trying to read from the connection and > there was no data available. Sounds like the python library isn't flushing > the entire LDAP request out to the network. Hmm, I have no clue why that happens. I've never had this problem except with some very strange LDAP servers like bigfoot's. Howard, how to make sure the LDAP request is flushed? Allan, if this problem occurs again can you post the relevant snippets of your Python code and can you also use trace_level=1 when calling ldap.open() or ldap.initialize() which writes a trace log containing all the python-ldap API calls (maybe solely on python-ldap-dev). Ciao, Michael. From michael at stroeder.com Wed May 29 13:40:14 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Wed, 29 May 2002 13:40:14 +0200 Subject: fetching the structure / obejctClass defs References: <200205171145.53395.t.kloppenburg@billiton.de> <1021630537.1723.8.camel@neo> <3CE54DD0.5080900@stroeder.com> <200205210905.58139.t.kloppenburg@billiton.de> <3CF27C9B.8000309@stroeder.com> <20020528122026.GA28263@neo.ulm.scan-plus.de> Message-ID: <3CF4BE1E.1020409@stroeder.com> jenner at neo.ulm.scan-plus.de wrote: > On Mon, May 27, 2002 at 08:36:11PM +0200, Michael Str?der wrote: > >>BTW: Just fetching objectClass definition is the easy part of the >>show. We have to think about how to implement and register known >>LDAP syntaxes for displaying, matching and input handling. > > What do you mean by 'matching'? Matching different values in reversed > polish notation according to their syntaxes and matching rules? Is that > not supposed to be done by the LDAP server? Yes it's done by the LDAP server when processing search and compare requests. But it's also handy for having it at the client-side. E.g. in recent web2ldap I'm checking for characteristic attributes (attributes used for forming RDN) and disallow modifying it in the normal modify input form since weird things can happen on various LDAP servers. Now I can only do case-insensitive string matching which is most times right. But in a completely correct implementation the matching should be done according to the matching rules defined for these attributes used. > And what about 'input > handling' -- client-side classchecking? Syntax checking of attribute values in user's input. > I guess I'm totally wrong, but I > have no idea how schema parsing might be usefull except displaying > available classes and attributetypes... Think of displaying attribute values in a UI. It helps if you have knowledge about the syntax used: normal UTF-8 strings vs. ASCII strings, E-mail addresses as mailto:-link or URLs as active links or... Ciao, Michael. From michael at stroeder.com Fri May 31 00:32:43 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Fri, 31 May 2002 00:32:43 +0200 Subject: FYI: Linuxtag References: <3CF6A7B3.5030202@stroeder.com> Message-ID: <3CF6A88B.5020000@stroeder.com> HI! BTW: Anyone who wants to ask me personally about python-ldap programming or web2ldap can do so at the Crypto booth at Linuxtag 2002 in Karlsruhe, Germany from 2002-06-06 until 2002-06-09. http://www.linuxtag.de Ciao, Michael. From Hans.Aschauer at Physik.uni-muenchen.de Fri May 31 11:15:22 2002 From: Hans.Aschauer at Physik.uni-muenchen.de (Hans Aschauer) Date: Fri, 31 May 2002 11:15:22 +0200 Subject: fetching the structure / obejctClass defs In-Reply-To: <3CF4BE1E.1020409@stroeder.com> References: <200205171145.53395.t.kloppenburg@billiton.de> <20020528122026.GA28263@neo.ulm.scan-plus.de> <3CF4BE1E.1020409@stroeder.com> Message-ID: <200205311115.22878.Hans.Aschauer@Physik.uni-muenchen.de> On Mittwoch, 29. Mai 2002 13:40, Michael Str?der wrote: [...] > Now I can only do case-insensitive string matching which is most > times right. But in a completely correct implementation the > matching should be done according to the matching rules defined > for these attributes used. Michael, what do you think, would it be worth the effort digging into the openldap sources in order to pythonize the server-side syntax checking and attribute matching facilities for client use? [One hour later after scanning the openldap sources] I think I could do it, but for this I would need to teach myself "something" about BER. Does anybody know of some high-level overview (i.e. tutorial style, not RFC's) on this basic-encoding-rules thing (and, if possible, how it is used in (open)ldap)? The disadvantage of this borrow-code-from-openldap-approach is that we will face some licensing fun, and that we will get more entangled with the openldap server (i.e. syntax checking is only available for syntaxes supported by the openldap server, etc.) Hans -- Hans.Aschauer at Physik.uni-muenchen.de From michael at stroeder.com Sun Jun 2 20:31:07 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Sun, 02 Jun 2002 20:31:07 +0200 Subject: fetching the structure / obejctClass defs References: <200205171145.53395.t.kloppenburg@billiton.de> <20020528122026.GA28263@neo.ulm.scan-plus.de> <3CF4BE1E.1020409@stroeder.com> <200205311115.22878.Hans.Aschauer@Physik.uni-muenchen.de> Message-ID: <3CFA646B.6020707@stroeder.com> Hans Aschauer wrote: > > Michael, what do you think, would it be worth the effort > digging into > the openldap sources in order to pythonize the server-side > syntax checking and attribute matching facilities for client > use? Only if the API is not subject to continous changes. We have to ask that on OpenLDAP-software mailing list. I suspect the answer of Kurt is "don't rely on that API". > [One hour later after scanning the openldap sources] I think I > could do it, but for this I would need to teach myself > "something" about BER. Hmm, if someone is digging into ber.h we're very near to an approach where we re-implement parts of the LDAP protocol engine. > Does anybody know of some high-level overview (i.e. tutorial > style, not RFC's) on this basic-encoding-rules thing (and, if > possible, how it is used in (open)ldap)? There's something called "A Layman's Introduction to BER, DER and ASN.1" on RSA's web site. > The disadvantage of this borrow-code-from-openldap-approach is > that we will face some licensing fun, and that we will get more > entangled with > the openldap server (i.e. syntax checking is only available for > syntaxes supported by the openldap server, etc.) Hmm, then it does not make sense to me. Also I believe that we can come up with a better Pythonic design if we define thoroughly thought Python classes. Ciao, Michael. From Hans.Aschauer at Physik.uni-muenchen.de Mon Jun 3 13:08:11 2002 From: Hans.Aschauer at Physik.uni-muenchen.de (Hans Aschauer) Date: Mon, 3 Jun 2002 13:08:11 +0200 Subject: fetching the structure / obejctClass defs In-Reply-To: <3CFA646B.6020707@stroeder.com> References: <200205171145.53395.t.kloppenburg@billiton.de> <200205311115.22878.Hans.Aschauer@Physik.uni-muenchen.de> <3CFA646B.6020707@stroeder.com> Message-ID: <200206031308.11671.Hans.Aschauer@Physik.uni-muenchen.de> On Sonntag, 2. Juni 2002 20:31, Michael Str?der wrote: > Hans Aschauer wrote: [...] > > Does anybody know of some high-level overview (i.e. tutorial > > style, not RFC's) on this basic-encoding-rules thing (and, if > > possible, how it is used in (open)ldap)? > > There's something called "A Layman's Introduction to BER, DER and > ASN.1" on RSA's web site. Thank you! > Hmm, then it does not make sense to me. Also I believe that we can > come up with a better Pythonic design if we define thoroughly > thought Python classes. Ok, I think you are right. I just took a deeper look at the sources, and the code seems to look like hand-coded regular expressions. Rewriting this in python does really make sense... BTW: Do you know of a more or less complete list of syntax descriptions? I know sites like http://www.alvestrand.no/objectid/top.html, and for many syntaxes there are BNF definitions (which is just fine), but sometimes they reference material which I cannont find online (e.g. the syntax for telefon numbers, 1.3.6.1.4.1.1466.115.121.1.50, is described in E.123). Hans -- Hans.Aschauer at Physik.uni-muenchen.de From michael at stroeder.com Mon Jun 3 13:23:28 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Mon, 03 Jun 2002 13:23:28 +0200 Subject: fetching the structure / obejctClass defs References: <200205171145.53395.t.kloppenburg@billiton.de> <200205311115.22878.Hans.Aschauer@Physik.uni-muenchen.de> <3CFA646B.6020707@stroeder.com> <200206031308.11671.Hans.Aschauer@Physik.uni-muenchen.de> Message-ID: <3CFB51B0.9040807@stroeder.com> Hans Aschauer wrote: > I just took a deeper look at the sources, and > the code seems to look like hand-coded regular expressions. Uurgs! I'd like to see a base class for a LDAP syntax with a method validate() which is called when setting the attribute value. More or less like I'm doing form parameter validation in PyWebLib. Also one could emulate comparison of values by implementing __cmp__(). With such a vehicle it would be easy to implement new syntaxes by just implementing these two methods. > BTW: Do you know of a more or less complete list of syntax descriptions? > [..] > many syntaxes there are BNF definitions (which is just fine), but > sometimes they reference material which I cannont find online (e.g. the > syntax for telefon numbers, 1.3.6.1.4.1.1466.115.121.1.50, is described > in E.123). You hit one of the famous problems in schema handling. The good news is that it seems you understood the problem... ;-) > I know sites like http://www.alvestrand.no/objectid/top.html That's the best I'd come up with. Ciao, Michael. From mcicogni at siosistemi.it Wed Jun 5 14:59:19 2002 From: mcicogni at siosistemi.it (Mauro Cicognini) Date: Wed, 05 Jun 2002 14:59:19 +0200 Subject: Strange behavior in Python-LDAP 2.0.0pre04 Message-ID: <3CFE0B27.80900@siosistemi.it> Hi all, I'd wish to bring to your attention a bizarre thing that I noticed yesterday. The documentation, at chapter 1.1.5.1 "LDAP operations", in the description of the result() method, states that [...] The result() method returns a tuple of the form |(/result-type/, /result-data/)|. [...] Now, I found that this isn't always true in the current (2.0.0pre04) implementation: in fact, when result() is called with the 'all' parameter set to non-zero (which BTW is the default), it returns just the /result-data/ part. In other words, in this case the method returns just a list instead of a (string, list) tuple. This breaks all of my code that uses asynchronous methods, like for instance this snippet: l = ldap.open(LDAPServer) msgid = l.simple_bind(username, password) status,res = l.result(msgid, all=1, timeout=60) if status != 'RES_BIND': return -1 will always raise a UnpackTuple exception, since the return value of result() will be [] instead of ('RES_BIND', None) as I expected. Someone may argue that the 'all' parameter is only significant in search() responses; this is true, but it still shouldn't cause undocumented behavior in the called function, IMHO. Anyway, this snippet breaks too: msgid = l.search("o=myCompany.com",ldap.SCOPE_SUBTREE,'uid=a*',['cn','mail']) status,res = l.result(msgid, all=1, timeout=60) if status != 'RES_SEARCH_RESULT': return -1 I tried to figure out if something was wrong on my part, or if I was calling some other method in the wrong way, but it appears everything is correct and that in fact it's the result() implementation within ldapobject.py that does this: the underlying C calls are correct, and they return the right data structure. The code in the result() implementation, if returning all data, simply appears to throw away the result type; it builds a list containing all results, and returns just that. I'd like to point out again that calling result() with all=0 would return a tuple as documented, instead. Moreover, the 1.x version of PythonLDAP result() worked 'correctly', i.e. exactly as documented in all cases. I think it's strange that nobody noticed this so far, so I do think I'm doing something wrong, or that I missed the discussion on this list about changing the result() behavior. May anyone shed light on this subject? Thank you in advance Mauro Cicognini P.S. This happens because I've finally managed to compile python-ldap under native Windows 32 (no Cygwin) and I was doing a bit of testing. This was the only problem I noticed so far, but it doesn't look to be related to the C part at all... -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 2926 bytes Desc: S/MIME Cryptographic Signature URL: From michael at stroeder.com Wed Jun 5 15:55:41 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Wed, 05 Jun 2002 15:55:41 +0200 Subject: Strange behavior in Python-LDAP 2.0.0pre04 References: <3CFE0B27.80900@siosistemi.it> Message-ID: <3CFE185D.3050205@stroeder.com> Mauro Cicognini wrote: > I'd wish to bring to your attention a bizarre thing that I noticed > yesterday. > > The documentation, at chapter 1.1.5.1 "LDAP operations", in the > description of the result() method, states that > > [...] > The result() method returns a tuple of the form |(/result-type/, > /result-data/)|. > [...] > > Now, I found that this isn't always true in the current (2.0.0pre04) > implementation: in fact, when result() is called with the 'all' > parameter set to non-zero (which BTW is the default), it returns just > the /result-data/ part. You're absolutely right. > I'd like to point out again that calling result() with all=0 would > return a tuple as documented, instead. > Moreover, the 1.x version of PythonLDAP result() worked 'correctly', > i.e. exactly as documented in all cases. > > I think it's strange that nobody noticed this so far, I noticed it and wanted to change it long ago but did not discuss that here. Thanks for reminding me about this open issue. I guess others didn't notice it since nobody calls result() with all=1. Instead others are using e.g. search_s() instead I guess. I've implemented LDAPObject.result() in ldap.ldapobject which is mainly a wrapper around the internal result() method. The original result() is never called with all=1. Instead LDAPObject.result() handles that internally (also the timeout parameter). The goal was to turn all sync API calls into async API calls to avoid having to use the module-wide lock too long. (OpenLDAP 2 libs are not thread-safe.) The code is somewhat clunky and everybody is invited to dig into that. Ciao, Michael. From michael at stroeder.com Wed Jun 5 15:55:19 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Wed, 05 Jun 2002 15:55:19 +0200 Subject: Win32 support (was: Strange behavior in Python-LDAP 2.0.0pre04) References: <3CFE0B27.80900@siosistemi.it> Message-ID: <3CFE1847.7040100@stroeder.com> Mauro Cicognini wrote: > P.S. This happens because I've finally managed to compile > python-ldap under native Windows 32 (no Cygwin) That's good news! Can you package some Win32 binaries? Maybe we can incorporate the necessary changes if any into python-ldap? Ciao, Michael. From mcicogni at siosistemi.it Wed Jun 5 16:14:50 2002 From: mcicogni at siosistemi.it (Mauro Cicognini) Date: Wed, 05 Jun 2002 16:14:50 +0200 Subject: Strange behavior in Python-LDAP 2.0.0pre04 References: <3CFE0B27.80900@siosistemi.it> <3CFE185D.3050205@stroeder.com> Message-ID: <3CFE1CDA.8040001@siosistemi.it> Michael Str?der wrote: > I noticed it and wanted to change it long ago but did not discuss that > here. Thanks for reminding me about this open issue. I guess others > didn't notice it since nobody calls result() with all=1. Instead > others are using e.g. search_s() instead I guess. This is a welcome reassurance. I'm not going crazy then ;-) > I've implemented LDAPObject.result() in ldap.ldapobject which is > mainly a wrapper around the internal result() method. The original > result() is never called with all=1. Instead LDAPObject.result() > handles that internally (also the timeout parameter). The goal was to > turn all sync API calls into async API calls to avoid having to use > the module-wide lock too long. (OpenLDAP 2 libs are not thread-safe.) > The code is somewhat clunky and everybody is invited to dig into that. I have always used the async versions of the calls for the same reason. In fact I've never resorted to calling result() with all=0 so far, just because my result sets are (usually) on the smaller side. So it's OK if I change the result() implementation within ldapobject.py? My idea would be to make result() always return what's expected in the docos, i.e. always a tuple, emulating the 1.x behavior (and avoiding the breakage in my existing code). Suggestions, anyone? Mauro -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 2926 bytes Desc: S/MIME Cryptographic Signature URL: From michael at stroeder.com Wed Jun 5 16:24:28 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Wed, 05 Jun 2002 16:24:28 +0200 Subject: Strange behavior in Python-LDAP 2.0.0pre04 References: <3CFE0B27.80900@siosistemi.it> <3CFE185D.3050205@stroeder.com> <3CFE1CDA.8040001@siosistemi.it> Message-ID: <3CFE1F1C.5020502@stroeder.com> Mauro Cicognini wrote: >> The goal was to >> turn all sync API calls into async API calls to avoid having to use >> the module-wide lock too long. (OpenLDAP 2 libs are not thread-safe.) >> The code is somewhat clunky and everybody is invited to dig into that. > > I have always used the async versions of the calls for the same reason. > In fact I've never resorted to calling result() with all=0 so far, just > because my result sets are (usually) on the smaller side. Just use the search_s() or search_st() methods which are solely implemented in ldap.ldapobject.LDAPObject. I completely removed them from the C module part. Set trace_level=1 when calling ldap.initialize() to track all the python-ldap API calls. > So it's OK if I change the result() implementation within ldapobject.py? > My idea would be to make result() always return what's expected in the > docos, i.e. always a tuple, emulating the 1.x behavior (and avoiding the > breakage in my existing code). You're welcome to dig into that and find the bug I've made. But please keep in mind as requirement that the C implementation should never be called with all=1. Note that ldap.ldapobject.LDAPObject.*_s() and ldap.ldapobject.LDAPObject.search_st() will also need some tweaking after fixing the bug in ldap.ldapobject.LDAPObject.result(). Ciao, Michael. From mcicogni at siosistemi.it Wed Jun 5 18:42:59 2002 From: mcicogni at siosistemi.it (Mauro Cicognini) Date: Wed, 05 Jun 2002 18:42:59 +0200 Subject: Win32 support (was: Strange behavior in Python-LDAP 2.0.0pre04) References: <3CFE0B27.80900@siosistemi.it> <3CFE1847.7040100@stroeder.com> Message-ID: <3CFE3F93.4020904@siosistemi.it> Michael Str?der wrote: > Mauro Cicognini wrote: > > P.S. This happens because I've finally managed to compile > > python-ldap under native Windows 32 (no Cygwin) > > That's good news! > > Can you package some Win32 binaries? > Maybe we can incorporate the necessary changes if any into python-ldap? > I had to write a new setup.cfg; I don't know if one could just insert a win32 section in the existing one and make it work automagically. Maybe so, since setup.py is custom too. That's not the end of the story, though. I need some help with distutils, and here's why. On one hand, the actual LDAP (and LBER) libraries are made static; moreover, I was able to build them so that they link the correct shared objects (MSVCRT.DLL, i.e. the same that Python needs). This way I could fold them into the _ldap shared object (_ldap.pyd) and not have bizarre runtime errors. On the other hand, the SASL library gets built as a DLL; I can make it into a static library, although I don't really know if it would work; but the real problem is that the LDAP and LBER code looks for SASL support in a shared object. I don't have time to delve into this and find out if OpenLDAP could be tweaked to support a statically linked SASL library, and, besides, the build procedure is already complex enough (and officially experimental: your mileage may vary). So I just put my LIBSASL.DLL into the Windows system directory; it works :-) However, to replicate this arrangement on other machines I'd have to tell distutils that a) the binary distribution needs to include LIBSASL.DLL, too; and b) when installing it, it needs to go somewhere on the PATH. I have no idea how. Any suggestions? Thanks, Mauro -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 2926 bytes Desc: S/MIME Cryptographic Signature URL: From michael at stroeder.com Thu Jun 6 02:19:09 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Thu, 06 Jun 2002 02:19:09 +0200 Subject: Win32 support (was: Strange behavior in Python-LDAP 2.0.0pre04) References: <3CFE0B27.80900@siosistemi.it> <3CFE1847.7040100@stroeder.com> <3CFE3F93.4020904@siosistemi.it> Message-ID: <3CFEAA7D.6080708@stroeder.com> Mauro Cicognini wrote: > > That's not the end of the story, though. I need some help with > distutils, and here's why. > [..] > However, to replicate this arrangement on other machines I'd have to > tell distutils that a) the binary distribution needs to include > LIBSASL.DLL, too; and b) when installing it, it needs to go somewhere on > the PATH. I have no idea how. I'd recommend to just make this a installation requirement. Exactly like the OpenLDAP and SASL libs have to be in place in a Unix environment *before* you build python-ldap. Ciao, Michael. From anders.karlsson at cendio.se Mon Jun 10 09:59:42 2002 From: anders.karlsson at cendio.se (Anders Karlsson) Date: Mon, 10 Jun 2002 09:59:42 +0200 (CEST) Subject: Speed difference, which version to use? Message-ID: I recently started to use Python-LDAP in a project. Since I first found the 1.10alpha3-version I used that. Everything worked fine and fast. Since it was built for Python 1.x I upgraded to Python-LDAP 2.0.0pre04 which worked fine, but considerable slower that the old version. A python program that changed some posts in the LDAP database and did two searches took about five seconds with 2.0.0pre04, compared to less than a second with 1.10alpha3. Why is there such big speed difference? Have I done something wrong or is it a known matter? Which version do you recomend me to use? Regards, From Hans.Aschauer at Physik.uni-muenchen.de Mon Jun 3 13:55:07 2002 From: Hans.Aschauer at Physik.uni-muenchen.de (Hans Aschauer) Date: Mon, 3 Jun 2002 13:55:07 +0200 Subject: fetching the structure / obejctClass defs In-Reply-To: <3CFB51B0.9040807@stroeder.com> References: <200205171145.53395.t.kloppenburg@billiton.de> <200206031308.11671.Hans.Aschauer@Physik.uni-muenchen.de> <3CFB51B0.9040807@stroeder.com> Message-ID: <200206031355.07170.Hans.Aschauer@Physik.uni-muenchen.de> On Montag, 3. Juni 2002 13:23, Michael Str?der wrote: > I'd like to see a base class for a LDAP syntax with a method > validate() which is called when setting the attribute value. More > or less like I'm doing form parameter validation in PyWebLib. Also > one could emulate comparison of values by implementing __cmp__(). > With such a vehicle it would be easy to implement new syntaxes by > just implementing these two methods. So this would mean that we change the implementation from "syntaxes are instances of the syntax class" to "syntaxes are subclasses an (abstract) syntax base class". Which objects would then instanciate the syntax classes? Maybe some kind of attribute class instances, through multiple inheritance? This is btw. not the only place where we have to decide if we want to use inheritance or instanciation for schema information. The same thing holds for objectClasses, etc! For objectClasses, inheritance seems to make sense, since they inherit from each other, and I think it would be possible (in python), to build a class hirachy from server information "on the fly". However, this is quite contrary to what I am used to (and I think many other programmers, too): Usually, I consider the class hirachy as part of the API, which is static. > You hit one of the famous problems in schema handling. The good > news is that it seems you understood the problem... ;-) Too bad. Seems to be a lot of work, then! -- Hans.Aschauer at Physik.uni-muenchen.de From michael at stroeder.com Fri Jun 14 17:31:49 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Fri, 14 Jun 2002 17:31:49 +0200 Subject: Speed difference, which version to use? References: Message-ID: <3D0A0C65.90101@stroeder.com> Anders Karlsson wrote: > > Since it was built for Python 1.x I upgraded to Python-LDAP 2.0.0pre04 > which worked fine, but considerable slower that the old version. A python > program that changed some posts in the LDAP database and did two searches > took about five seconds with 2.0.0pre04, compared to less than a second > with 1.10alpha3. > > Why is there such big speed difference? Hmm, there might be different issues here: 1. I changed the implementation of the synchronous methods *_s() to avoid getting blocked in OpenLDAP 2 _s() functions. Especially there's a reimplementation of result() in ldap.ldapobject.LDAPObject which does the timeout handling itself. The main reason is that the OpenLDAP libs are not thread-safe and therefore a module-wide lock is used to serialize all calls into the OpenLDAP libs. (If someone manages to use OpenLDAP 2.1's libldap_r instead of libldap this might help to reduce some of the locking.) 2. You might also have upgraded the LDAP server to OpenLDAP 2 (which does stricter checking) or changed the indexing configuration (which is a performance penalty when writing)? 3. The OpenLDAP 2.x libs are slower. (python-ldap 1.x was linked against OpenLDAP 1.x libs.) Now for 1. issue: You could try a very simple result() implementation (tweak Lib/ldap/ldapobject.py): def result(self,msgid=_ldap.RES_ANY,all=1,timeout=-1): return self._ldap_call(self._l.result,msgid,all,timeout) I did some testing with this method implementation for measuring the overhead of my non-blocking version. I read the RootDSE of Netscape DS 4.16SP1 running on the same box. (Solely searching the RootDSE is an appropriate method to eliminate the influence of database backends and such.) On average it seems to be approx. 65% faster to use this simple method above. This is something to consider. Hmm, but you are experiencing a performance difference which is much higher (5 times slower as I understand your posting). I'd really appreciate if you could send more information about your environment and *all* the changes you did. Performance measurement numbers done under really the same conditions are also appreciated off course. Ciao, Michael. From mcicogni at siosistemi.it Fri Jun 14 18:59:07 2002 From: mcicogni at siosistemi.it (Mauro Cicognini) Date: Fri, 14 Jun 2002 18:59:07 +0200 Subject: Speed difference, which version to use? References: <3D0A0C65.90101@stroeder.com> Message-ID: <3D0A20DB.6000500@siosistemi.it> I have experienced significant delays, too, using the new version. Since I have recently managed to compile the whole stuff under Win32, I am doing some experimenting before releasing the binaries, and this is actually blocking me a bit. However, I have seen that the searches don't show significant differences with the old 1.x version. The single call that takes a lot is the bind step: when I ask a Python script to bind it just seems to sit there and wait, more or less for 5 seconds (strange coincidence). Could all this be related to the SASL libraries? Mauro -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 2926 bytes Desc: S/MIME Cryptographic Signature URL: From michael at stroeder.com Fri Jun 14 19:48:44 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Fri, 14 Jun 2002 19:48:44 +0200 Subject: Speed difference, which version to use? References: <3D0A0C65.90101@stroeder.com> <3D0A20DB.6000500@siosistemi.it> Message-ID: <3D0A2C7C.6080108@stroeder.com> Mauro Cicognini wrote: > > However, I have seen that the searches don't show significant > differences with the old 1.x version. Can you provide numbers? > The single call that takes a lot is the bind step: when I ask a Python > script to bind it just seems to sit there and wait, more or less for 5 > seconds (strange coincidence). > > Could all this be related to the SASL libraries? Hmm, no idea. I guess you are you talking about a simple bind. Ciao, Michael. From michael at stroeder.com Fri Jun 14 19:47:08 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Fri, 14 Jun 2002 19:47:08 +0200 Subject: Speed difference, which version to use? References: <3D0A0C65.90101@stroeder.com> Message-ID: <3D0A2C1C.3030101@stroeder.com> Michael Str?der wrote: > Anders Karlsson wrote: > > You could try a very simple result() implementation (tweak > Lib/ldap/ldapobject.py): > > def result(self,msgid=_ldap.RES_ANY,all=1,timeout=-1): > return self._ldap_call(self._l.result,msgid,all,timeout) > Just a warning: This is not meant as serious replacement. Just for measuring the performance. Rest of LDAPObject class would have to be tweaked too. Ciao, Michael. From anders.karlsson at cendio.se Mon Jun 17 10:39:18 2002 From: anders.karlsson at cendio.se (Anders Karlsson) Date: Mon, 17 Jun 2002 10:39:18 +0200 (CEST) Subject: Speed difference, which version to use? In-Reply-To: <3D0A0C65.90101@stroeder.com> Message-ID: On Fri, 14 Jun 2002, Michael Str?der wrote: > 2. You might also have upgraded the LDAP server to OpenLDAP 2 > (which does stricter checking) or changed the indexing > configuration (which is a performance penalty when writing)? Yes, it's linked against OpenLDAP 2. But this can't be the problem alone since the bundled ldap-tools (ldapadd, ldapsearch..) works fast. Thanks a lot for your answers and ideas! I will try to look deeper into my environment and see if I can find the bottlenecks. Regards, From mcicogni at siosistemi.it Mon Jun 17 12:18:06 2002 From: mcicogni at siosistemi.it (mcicogni at siosistemi.it) Date: Mon, 17 Jun 2002 12:18:06 +0200 Subject: Speed difference, which version to use? Message-ID: <671d1e7b.1e7b671d@siosistemi.it> Michael Str?der writes: > Mauro Cicognini wrote: > > > > However, I have seen that the searches don't show significant > > differences with the old 1.x version. > > Can you provide numbers? Hmmm, not really. It's just my feeling trying to do some simple searches from the Python command line: I type the instruction, hit return and '>>>' comes up immediately (i.e. no visibile delay). I could do some timing, but I don't have the time now to set up the environment now... sorry about this. I'd be glad to supply sources and binaries to anyone interested, though. > > The single call that takes a lot is the bind step: when I ask a > Python > > script to bind it just seems to sit there and wait, more or less > for 5 > > seconds (strange coincidence). > > > > Could all this be related to the SASL libraries? > > Hmm, no idea. I guess you are you talking about a simple bind. Yes, it's a simple bind: username and password. I guess it's also the only one supported by Win32 without certificates or the whole Kerberos shebang. I really hope someone with SASL experience comes to our rescue. Mauro From mcicogni at siosistemi.it Mon Jun 24 10:01:06 2002 From: mcicogni at siosistemi.it (Mauro Cicognini) Date: Mon, 24 Jun 2002 10:01:06 +0200 Subject: Win32 version working OK [was Re: Speed difference, which version to use?] References: <671d1e7b.1e7b671d@siosistemi.it> Message-ID: <3D16D1C2.3000806@siosistemi.it> I'm glad to announce that I've found the limiting factor for the 5-odd seconds delay during binds, which I was annoyed by, and that I was able to remove it. Therefore, it appears that the native Win32 (i.e., non-Cygwin) version of PythonLDAP 2.0.0pre04 (linked against OpenLDAP 2.0 libs) is indeed a workable replacement for PythonLDAP 1.x (under Windows, that is). If anyone's interested, here's what I found. By sniffing network traffic I saw that it wasn't LDAP's fault per se. In fact, for some reason the current libraries (as opposed to the old UMich libs that I used in PythonLDAP 1.x) do a reverse-resolution on the server's IP address before attempting to bind, on both DNS _and_ NetBIOS (remember I'm dealing with Windows machines here). Note that the I passed the LDAP server's address as a DNS name, and that the IP address was correctly resolved by my DNS server. Anyhow, the client always tries to find the NetBIOS name of the server machine, and this was what caused the delay, since my LDAP server is behind a firewall which is configured to disallow NetBIOS queries (the client tries 3 times the query, then gives up). Once I let NetBIOS-ns through (UDP port 137) the delay disappeared. If anyone has a clue to what is causing this rather bizarre behavior in the client, please let me know: I do think this is a misfeature, and I'd like to #ifdef it away at least in my version of the libs. However, I can't really tell who's doing this: it might be happening within SASL, or within OpenLDAP, or in any of some minor different components that need to be there when compiling. I can tell that it isn't Windows fault, at least: I'm sure of this because I compiled, on the same machine (WinXP Pro using MSVC++ 6.0sp4), both versions of PythonLDAP (old 1.x style linked with UMich libs, and new 2.x linked with OpenLDAP libs); and the former isn't trying reverse resolution, whereas the latter is. If anyone's interested, I can post the compiled Win32 binaries somewhere so that others may test them more thoroughly than myself. Thanks everybody Mauro -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 2926 bytes Desc: S/MIME Cryptographic Signature URL: From michael at stroeder.com Mon Jun 24 11:19:47 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Mon, 24 Jun 2002 11:19:47 +0200 Subject: Win32 version working OK [was Re: Speed difference, which version to use?] References: <671d1e7b.1e7b671d@siosistemi.it> <3D16D1C2.3000806@siosistemi.it> Message-ID: <3D16E433.9060708@stroeder.com> Mauro Cicognini wrote: > I'm glad to announce that I've found the limiting factor for the 5-odd > seconds delay during binds, Note that with python-ldap 2.0.0pre04 ldap.initialize() is used in any case which wraps ldap_initialize in the OpenLDAP libs. ldap_initialize() has a slightly different behaviour than ldap_open(). AFAIK it just initializes the LDAP connection context but does not open the LDAP connection. The LDAP connection is opened when doing the first LDAPRequest - no matter which one. Having said this I'd like to see your test script. You might wanna check if the delay really happens during BindRequest or any first LDAPRequest. Note that LDAPv3 does not require you to send a BindRequest prior to other LDAPRequests. > By sniffing network traffic I saw that it wasn't LDAP's fault per se. In > fact, for some reason the current libraries (as opposed to the old UMich > libs that I used in PythonLDAP 1.x) do a reverse-resolution on the > server's IP address before attempting to bind, on both DNS _and_ NetBIOS > (remember I'm dealing with Windows machines here). Hmm, reverse lookups might make sense when using LDAP over SSL or LDAP with StartTLS to cross-check the server's name with the CN attribute in the subject DN of the server certificate. Since the OpenLDAP 1.x libs did not have any support for SSL/TLS this might be an issue with OpenLDAP 2.x libs. Just thoughts, not sure though... > Note that the I passed the LDAP server's address as a DNS name, and that > the IP address was correctly resolved by my DNS server. Can you please try to use the IP address directly and check if the same behaviour happens? > Anyhow, the > client always tries to find the NetBIOS name of the server machine, and > this was what caused the delay, since my LDAP server is behind a > firewall which is configured to disallow NetBIOS queries (the client > tries 3 times the query, then gives up). Once I let NetBIOS-ns through > (UDP port 137) the delay disappeared. > [..] > I can tell that it isn't Windows fault, at least: I'm not sure if that conclusion is right. 1. I remember reverse lookup problems with various software on Windows. (Therefore your observation is very interesting for other things too.) 2. I can't imagine why the OpenLDAP 2 libs should explicitly do reverse NETBIOS lookups other than using a default parameter somewhere which causes that. Now how's the behaviour on Windows with the normal OpenLDAP tools ldapsearch, etc.? Ciao, Michael. From dman at dman.ddts.net Tue Jun 25 04:04:47 2002 From: dman at dman.ddts.net (Derrick 'dman' Hudson) Date: Mon, 24 Jun 2002 21:04:47 -0500 Subject: defect Message-ID: <20020625020447.GA23462@dman.ddts.net> I tried using python-ldap today (1.9.999.pre04-1, python 2.1.3-3), but it is way too inefficient. A simple search that results in 2 entries returned takes 30 seconds. Watching with top shows nearly 100% CPU usage for the 30 seconds, on an otherwise idle Athlon XP 1800+. OpenLDAP (2.0) is running on that same machine, however using ldapsearch or exim yields immediate results. $ cat maillist.py #!/usr/bin/python2.1 import sys import ldap INIT = "ldap://localhost/" BASE = "ou=People,o=International Teams" TIMEOUT = 40 listname = sys.argv[1] ldapconn = ldap.initialize( INIT ) results = ldapconn.search_st( BASE , ldap.SCOPE_ONELEVEL , '(mailGroupLocalPart=%s)' % listname , attrlist = ("uid",) , timeout = TIMEOUT ) names = [ attrs['uid'][0] for dn , attrs in results] print ' , '.join( names ) $ time ./maillist.py AITP dann , derrick real 0m36.555s user 0m32.990s sys 0m3.490s $ time ldapsearch -x -h localhost -b "ou=People,o=International Teams" -s one "(mailGroupLocalPart=aitp)" "uid" | grep uid: uid: dann uid: derrick real 0m0.033s user 0m0.000s sys 0m0.010s I think this rather clearly points to python-ldap as the culprit. I need to do some integration of LDAP and some web-based programs, and would like to work with python, but this sort of performance hit just won't be usable. I'm willing to help with the code, if you point me to the interesting parts (and help me learn the C API of python and openldap as I go). I did a quick glance over the archives and read this thread : http://www.geocrawler.com/lists/3/SourceForge/1568/0/8932688/ -D -- How great is the love the Father has lavished on us, that we should be called children of God! 1 John 3:1 http://dman.ddts.net/~dman/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 240 bytes Desc: not available URL: From jens at zope.com Tue Jun 25 14:28:15 2002 From: jens at zope.com (Jens Vagelpohl) Date: Tue, 25 Jun 2002 08:28:15 -0400 Subject: defect In-Reply-To: <20020625020447.GA23462@dman.ddts.net> Message-ID: <05EC71D8-8837-11D6-8BA3-00039363690C@zope.com> i have seen similar strange slowdowns **if i connect to localhost**. connecting to OpenLDAP on another host across the network was much faster (meaning: it worked at the expected speed). the slowdown never occurred on python-ldap 1.10.3. both were compiled against OpenLDAP2-libraries. so far i tended to assume that the problem might be a local configuration issue on my machine, though i haven't been able to find it yet. jens On Monday, June 24, 2002, at 10:04 , Derrick 'dman' Hudson wrote: > > I tried using python-ldap today (1.9.999.pre04-1, python 2.1.3-3), but > it is way too inefficient. A simple search that results in 2 entries > returned takes 30 seconds. Watching with top shows nearly 100% CPU > usage for the 30 seconds, on an otherwise idle Athlon XP 1800+. > OpenLDAP (2.0) is running on that same machine, however using > ldapsearch or exim yields immediate results. > > $ cat maillist.py > #!/usr/bin/python2.1 > > import sys > import ldap > > INIT = "ldap://localhost/" > BASE = "ou=People,o=International Teams" > TIMEOUT = 40 > > listname = sys.argv[1] > ldapconn = ldap.initialize( INIT ) > results = ldapconn.search_st( BASE , ldap.SCOPE_ONELEVEL , > '(mailGroupLocalPart=%s)' % listname , > attrlist = ("uid",) , timeout = TIMEOUT > ) > names = [ attrs['uid'][0] for dn , attrs in results] > print ' , '.join( names ) > > > $ time ./maillist.py AITP > dann , derrick > > real 0m36.555s > user 0m32.990s > sys 0m3.490s > > $ time ldapsearch -x -h localhost -b "ou=People,o=International Teams" > -s one "(mailGroupLocalPart=aitp)" "uid" | grep uid: > uid: dann > uid: derrick > > real 0m0.033s > user 0m0.000s > sys 0m0.010s > > > > I think this rather clearly points to python-ldap as the culprit. I > need to do some integration of LDAP and some web-based programs, and > would like to work with python, but this sort of performance hit just > won't be usable. I'm willing to help with the code, if you point me > to the interesting parts (and help me learn the C API of python and > openldap as I go). > > I did a quick glance over the archives and read this thread : > http://www.geocrawler.com/lists/3/SourceForge/1568/0/8932688/ > > -D > From dman at dman.ddts.net Tue Jun 25 19:31:21 2002 From: dman at dman.ddts.net (Derrick 'dman' Hudson) Date: Tue, 25 Jun 2002 12:31:21 -0500 Subject: defect In-Reply-To: <05EC71D8-8837-11D6-8BA3-00039363690C@zope.com> References: <20020625020447.GA23462@dman.ddts.net> <05EC71D8-8837-11D6-8BA3-00039363690C@zope.com> Message-ID: <20020625173121.GA32715@dman.ddts.net> On Tue, Jun 25, 2002 at 08:28:15AM -0400, Jens Vagelpohl wrote: | i have seen similar strange slowdowns **if i connect to localhost**. | connecting to OpenLDAP on another host across the network was much faster | (meaning: it worked at the expected speed). Actually, you're right. I never tried from a different host until you said this. I presumed using the loopback interface would have the least amount of delay due to eliminating network hardware from the equation. That certainly gives some (initial) improvement possibilities. -D -- A)bort, R)etry, B)ang it with a large hammer http://dman.ddts.net/~dman/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 240 bytes Desc: not available URL: From leif.hedstrom at propel.com Tue Jun 25 21:00:32 2002 From: leif.hedstrom at propel.com (Leif Hedstrom) Date: Tue, 25 Jun 2002 12:00:32 -0700 Subject: defect Message-ID: Well, I don't know if this is the same problem I had with Python LDAP v1.x, we haven't tested v2.x yet. But, the result() function in Python LDAP can go into a very tight poll loop, with extreme effects if the Python process is running on the same machines as the LDAP server. Python will almost completely starve slapd for any CPU time ... Adding a short sleep() in the polling loop of ldapobject.result() helps, a lot. Like while all: while ldap_result[0] is None: if (timeout>=0) and (time.time()-start_time>timeout): self._ldap_call(self._l.abandon,msgid) raise _ldap.TIMELIMIT_EXCEEDED( "LDAP time limit (%d secs) exceeded." % (timeout) ) ldap_result = self._ldap_call(self._l.result,msgid,0,0) if ldap_result[0] is None: time.sleep(.01) if ldap_result[1] is None: break (note the time.sleep() call if there was no result). Alternatively, adding an (arbitrarily) long timeout in the call to ldap_result() also accomplishes the same thing, like: ldap_result = self._ldap_call(self._l.result,msgid,0,15 * 60) I haven't dug deep into this problem yet, to figure out if this is an OpenLDAP library problem, or a Python LDAP problem. I just know that preventing ldapobject.result() from going into the tight polling loop solved our problems. :-) Cheers, -- Leif From jens at zope.com Tue Jun 25 22:02:04 2002 From: jens at zope.com (Jens Vagelpohl) Date: Tue, 25 Jun 2002 16:02:04 -0400 Subject: defect In-Reply-To: Message-ID: <6BA87784-8876-11D6-8C86-003065C7DEAE@zope.com> leif, i will be darned :) your little hack does indeed work on my setup, too. a test script that took 30 seconds to run now finishes in 2 seconds... jens On Tuesday, June 25, 2002, at 03:00 , Leif Hedstrom wrote: > Well, I don't know if this is the same problem I had with Python LDAP v1. > x, > we haven't tested v2.x yet. But, the result() function in Python LDAP can > go > into a very tight poll loop, with extreme effects if the Python process is > running on the same machines as the LDAP server. Python will almost > completely starve slapd for any CPU time ... > > Adding a short sleep() in the polling loop of ldapobject.result() helps, a > lot. Like > > while all: > while ldap_result[0] is None: > if (timeout>=0) and (time.time()-start_time>timeout): > self._ldap_call(self._l.abandon,msgid) > raise _ldap.TIMELIMIT_EXCEEDED( > "LDAP time limit (%d secs) exceeded." % (timeout) > ) > ldap_result = self._ldap_call(self._l.result,msgid,0,0) > if ldap_result[0] is None: > time.sleep(.01) > if ldap_result[1] is None: > break > > (note the time.sleep() call if there was no result). Alternatively, adding > an (arbitrarily) long timeout in the call to ldap_result() also > accomplishes > the same thing, like: > > ldap_result = self._ldap_call(self._l.result,msgid,0,15 * 60) > > I haven't dug deep into this problem yet, to figure out if this is an > OpenLDAP library problem, or a Python LDAP problem. I just know that > preventing ldapobject.result() from going into the tight polling loop > solved > our problems. :-) > > Cheers, > > -- Leif > From michael at stroeder.com Wed Jun 26 01:28:34 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Wed, 26 Jun 2002 01:28:34 +0200 Subject: defect References: <05EC71D8-8837-11D6-8BA3-00039363690C@zope.com> Message-ID: <3D18FCA2.4030806@stroeder.com> Jens Vagelpohl wrote: > i have seen similar strange slowdowns **if i connect to localhost**. > connecting to OpenLDAP on another host across the network was much > faster (meaning: it worked at the expected speed). the slowdown never > occurred on python-ldap 1.10.3. both were compiled against > OpenLDAP2-libraries. > > so far i tended to assume that the problem might be a local > configuration issue on my machine, though i haven't been able to find it > yet. Please, be more precise: Which OS? Which LDAP server and version? Which kind of queries? Logging? Ciao, Michael. From michael at stroeder.com Wed Jun 26 01:26:41 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Wed, 26 Jun 2002 01:26:41 +0200 Subject: defect References: <20020625020447.GA23462@dman.ddts.net> Message-ID: <3D18FC31.1030005@stroeder.com> Derrick 'dman' Hudson wrote: > I tried using python-ldap today (1.9.999.pre04-1, python 2.1.3-3), but > it is way too inefficient. A simple search that results in 2 entries > returned takes 30 seconds. If that would be normal I would not use python-ldap. Let's see. > Watching with top shows nearly 100% CPU > usage for the 30 seconds, on an otherwise idle Athlon XP 1800+. > OpenLDAP (2.0) is running on that same machine, however using > ldapsearch or exim yields immediate results. Frankly this is a very unprecise performance measurement. As I wrote in one of my former postings there might be performance issues due to the nature of the async reimplementation of LDAPObject.result(). And it might look bad that your CPU meter shows 100% load. But I also suspect a few other effects if you just say "30 seconds vs. immediate results". > '(mailGroupLocalPart=%s)' % listname , Is attribute mailGroupLocalPart indexed? An equality index should be sufficient here. Also note that OpenLDAP 2's ldbm backend implements server-side DB caching. > $ time ./maillist.py AITP > dann , derrick > > real 0m36.555s > user 0m32.990s > sys 0m3.490s > > $ time ldapsearch -x -h localhost -b "ou=People,o=International Teams" > -s one "(mailGroupLocalPart=aitp)" "uid" | grep uid: > uid: dann > uid: derrick > > real 0m0.033s > user 0m0.000s > sys 0m0.010s > > I think this rather clearly points to python-ldap as the culprit. Hmm, I have some doubts. Let's see. > I > need to do some integration of LDAP and some web-based programs, and > would like to work with python, but this sort of performance hit just > won't be usable. As you might have noticed I'm doing web programming with python-ldap. ;-) I'm using web2ldap for maintaining and searching my personal address book and it's pretty responsive when using a fast browser. I'm also browsing very large data sets (>150000). Just to give you a clue what I'm actually doing with python-ldap in a commercial pilot project: I'm scanning 170000 entries in far less than an hour (mainly just reading the uid attribute). I'm doing diffing whole entries at a rate of 50 entries/second (some other work with a SQL DB is involved here). The process runs on a P-III 450 Mhz box against a 4-CPU, 1GB RAM server running iPlanet Directory Server 5.1. > I'm willing to help with the code, if you point me > to the interesting parts (and help me learn the C API of python and > openldap as I go). To find out the interesting parts one has to do proper performance measuring. And I would be really glad to see some *real* numbers. Please take this advice to produce numbers I can take serious: 1. Eliminate all disk access => turn off all logging. 2. Eliminate caching issues => do many searches, throw away first result. Example: During my former expirements with large group entries (>60000 member attribute values on OpenLDAP and 200000 on NS DS 4.16) I experienced a 30+ seconds interval when accessing the group entry for the first time. Later the CompareRequest was done much faster. 3. Eliminate DB backend issues => only search RootDSE. (This hint by Kurt Zeilenga.) 4. Maximize performance impact of python-ldap => use faster LDAP server. I took some numbers on my P-III laptop against a locally installed Netscape Directory Server 4.16SP1 which is much faster than recent OpenLDAP. Test script is attached. There are three test cases especially for the guys who are blaming python-ldap for bad performance but are reconnecting to the LDAP server for each query. ;-) *** Read the RootDSE on same connection 279.630273 searches/second *** Read the RootDSE on newly created connection without extra simple bind 171.760053 searches/second *** Read the RootDSE on newly created connection with an extra simple bind 144.559940 searches/second Now without the async implementation of LDAPObject.result() directly wrapping the built-in C implementation: *** Read the RootDSE on same connection 464.693726 searches/second *** Read the RootDSE on newly created connection without extra simple bind 248.286182 searches/second *** Read the RootDSE on newly created connection with an extra simple bind 202.098080 searches/second The rates are obviously much higher (+40%..+65%) but note that we did everything here to make python-ldap look really bad. Under real-world conditions with real DB backend activity, server logging on disk and whatever the percentage of the higher rate looks surely different. => I have yet to see some serious numbers proving the "30 seconds vs. immediate results". Ciao, Michael. -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: performance.py URL: From jens at zope.com Wed Jun 26 15:57:37 2002 From: jens at zope.com (Jens Vagelpohl) Date: Wed, 26 Jun 2002 09:57:37 -0400 Subject: defect In-Reply-To: <3D18FCA2.4030806@stroeder.com> Message-ID: michael, - Athlon 1 Ghz - 784 MB RAM - RedHat 7.2 - OpenLDAP 2.0.23 compiled from source (no logging) - Python 2.1.3 compiled from source - python-ldap 2.0.0pre4 compiled from source (using distutils) the query i used for my own testing is simple: - bind anonymously once - search with scope SCOPE_BASE for objectclass=* at a known existing searchbase. now i took your test script (thanks for providing that!) and here are my results: - without leif's "fix": *** Read the RootDSE on same connection 8.462220 searches/second *** Read the RootDSE on newly created connection without extra simple bind 6.347031 searches/second *** Read the RootDSE on newly created connection with an extra simple bind 3.198623 searches/second - with leif's "fix": *** Read the RootDSE on same connection 82.275864 searches/second *** Read the RootDSE on newly created connection without extra simple bind 85.599961 searches/second *** Read the RootDSE on newly created connection with an extra simple bind 47.726886 searches/second this seems to point to the "tight loop that prevents slapd from answering requests" theory leif had. jens On Tuesday, June 25, 2002, at 07:28 , Michael Str?der wrote: > Jens Vagelpohl wrote: >> i have seen similar strange slowdowns **if i connect to localhost**. >> connecting to OpenLDAP on another host across the network was much >> faster (meaning: it worked at the expected speed). the slowdown never >> occurred on python-ldap 1.10.3. both were compiled against >> OpenLDAP2-libraries. >> so far i tended to assume that the problem might be a local >> configuration issue on my machine, though i haven't been able to find it >> yet. > > Please, be more precise: > Which OS? > Which LDAP server and version? > Which kind of queries? > Logging? > > Ciao, Michael. > > > > From dman at dman.ddts.net Wed Jun 26 20:32:07 2002 From: dman at dman.ddts.net (Derrick 'dman' Hudson) Date: Wed, 26 Jun 2002 13:32:07 -0500 Subject: defect In-Reply-To: <3D18FC31.1030005@stroeder.com> References: <20020625020447.GA23462@dman.ddts.net> <3D18FC31.1030005@stroeder.com> Message-ID: <20020626183207.GA13037@dman.ddts.net> On Wed, Jun 26, 2002 at 01:26:41AM +0200, Michael Str?der wrote: | Derrick 'dman' Hudson wrote: | >I tried using python-ldap today (1.9.999.pre04-1, python 2.1.3-3), but | >it is way too inefficient. A simple search that results in 2 entries | >returned takes 30 seconds. | | If that would be normal I would not use python-ldap. Let's see. :-). | >Watching with top shows nearly 100% CPU | >usage for the 30 seconds, on an otherwise idle Athlon XP 1800+. | >OpenLDAP (2.0) is running on that same machine, however using | >ldapsearch or exim yields immediate results. | | Frankly this is a very unprecise performance measurement. True, but the ability to measure the time by my watch is rather, umm, noticeable :-). | > '(mailGroupLocalPart=%s)' % listname , | | Is attribute mailGroupLocalPart indexed? No (AFAIK). | An equality index should be sufficient here. | > I need to do some integration of LDAP and some web-based programs, | > and would like to work with python, but this sort of performance | > hit just won't be usable. | | As you might have noticed I'm doing web programming with | python-ldap. ;-) I'm using web2ldap for maintaining and searching | my personal address book and it's pretty responsive when using a | fast browser. I'm also browsing very large data sets (>150000). That's good. I installed web2ldap and ldapexplorer yesterday to evaluate them. web2ldap didn't look very useful -- modification/timestamp attributes are shown for any entry. Actually, trying it again today, but using a different host as the LDAP server (not localhost), it looks much more useful (it shows all the attributes). I wonder why that is. | Just to give you a clue what I'm actually doing with python-ldap | in a commercial pilot project: I'm scanning 170000 entries in far | less than an hour (mainly just reading the uid attribute). I'm | doing diffing whole entries at a rate of 50 entries/second (some | other work with a SQL DB is involved here). Interesting. The "People" node on our tree has 500 child nodes, each of which has no children. | The process runs on a P-III 450 Mhz box against a 4-CPU, 1GB RAM | server running iPlanet Directory Server 5.1. I notice that your ldap server is not on the same machine as python-ldap. As Jens mentioned, and I subsequently discovered for myself, running python-ldap on a separate host from the ldap server doesn't have the performance problem. I only experience the problem with python-ldap and slapd on the same machine. | > I'm willing to help with the code, if you point me | >to the interesting parts (and help me learn the C API of python and | >openldap as I go). | | To find out the interesting parts one has to do proper performance | measuring. :-). | And I would be really glad to see some *real* numbers. Please take | this advice to produce numbers I can take serious: Ok, here we go. | 1. Eliminate all disk access => turn off all logging. Logging is (and was) off. | 2. Eliminate caching issues => do many searches, throw away first result. Right. My earlier, crude, measurements were repeatable every time. If it was a caching issue I would expect the first to be slow but not the latter ones. | 3. Eliminate DB backend issues => only search RootDSE. | (This hint by Kurt Zeilenga.) DSE ... I don't think I've run across this TLA before. | 4. Maximize performance impact of python-ldap => use faster LDAP server. By "server" are you referring to hardware or software? | I took some numbers on my P-III laptop against a locally installed | Netscape Directory Server 4.16SP1 which is much faster than recent | OpenLDAP. Can I get Netscape Directory Server for Debian? Is it Free? If not then there is very little possibility of using it. | Test script is attached. Thanks! | There are three test cases especially for the guys who are blaming | python-ldap for bad performance but are reconnecting to the LDAP | server for each query. ;-) Is it allowed to reconnect for each query if each query is run from a separate process at disparate times? ;-) Here is what I get : Server : Hardware : Athlon XP 1800+ (1.5 GHz clock, I think) 256 MB DDR RAM IDE disk (I don't know much on the specs, but it is relatively new and fairly quick) Software : OpenLDAP 2.0.23 , ldbm backend Debian woody/sid Linux 2.4.18 Other Load : light Client 1 : Hardware : same Software (same) OpenLDAP client library (v 2.0.23) python 2.1.3 python-ldap 1.9.999.pre04 Other Load : same Client 2 : Hardware : Duron 750 (750 MHz clock) 256 MB PC133 SDRAM Software (same) OpenLDAP client library (v 2.0.23) python 2.1.3 python-ldap 1.9.999.pre04 Other Load : light-moderate Test 1 : Client 1 *** Read the RootDSE on same connection 1719.493984 searches/second *** Read the RootDSE on newly created connection without extra simple bind 30.879468 searches/second *** Read the RootDSE on newly created connection with an extra simple bind 26.940053 searches/second Watching with top shows the system working as hard as it could, with the client eating most of the CPU and slapd using very very little. Test 2 : Client 2 *** Read the RootDSE on same connection 895.992294 searches/second *** Read the RootDSE on newly created connection without extra simple bind 161.131824 searches/second *** Read the RootDSE on newly created connection with an extra simple bind 146.020832 searches/second Watching both systems with top showed (CPU wise) the server not even sweating while the client got a decent workout. Test 3 : Client 1 , without the async implementation of LDAPObject.result(), directly wrapping the built-in C implementation: *** Read the RootDSE on same connection 2039.669716 searches/second *** Read the RootDSE on newly created connection without extra simple bind 279.827530 searches/second *** Read the RootDSE on newly created connection with an extra simple bind 262.486823 searches/second Watching with top shows _both_ python-ldap and slapd getting a fair amount of CPU time, and the CPU was only running at ~50% of its capacity (good things, IMO :-)). Test 4 : same as test 3 but with Client 2 *** Read the RootDSE on same connection 1069.199140 searches/second *** Read the RootDSE on newly created connection without extra simple bind 163.548164 searches/second *** Read the RootDSE on newly created connection with an extra simple bind 151.169328 searches/second Similar observations in top, except that the server wasn't working nearly as hard as in test3. This seems to show that the difference in the result() method is more significant/noticeable on a fast system than on a slow one. | => I have yet to see some serious numbers proving the "30 seconds | vs. immediate results". If I add index mailGroupLocalPart eq to slapd.conf and restart the daemon, my script runs really fast on "client1", but (incorrectly) doesn't return any results. Using that script on "client2" yields these results (just for comparison) : real 0m0.147s user 0m0.120s sys 0m0.020s This sort of time is quite acceptable :-). Does this help? Oh, BTW, some of the docs are missing (404) on the web site. For example, go to http://python-ldap.sourceforge.net/pydoc/ldap.html and click on the "functions" link. -D -- The way of a fool seems right to him, but a wise man listens to advice. Proverbs 12:15 http://dman.ddts.net/~dman/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 240 bytes Desc: not available URL: From michael at stroeder.com Wed Jun 26 22:58:24 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Wed, 26 Jun 2002 22:58:24 +0200 Subject: defect References: Message-ID: <3D1A2AF0.9090802@stroeder.com> Leif Hedstrom wrote: > Well, I don't know if this is the same problem I had with Python LDAP v1.x, > we haven't tested v2.x yet. But, the result() function in Python LDAP can go > into a very tight poll loop, with extreme effects if the Python process is > running on the same machines as the LDAP server. Python will almost > completely starve slapd for any CPU time ... > > Adding a short sleep() in the polling loop of ldapobject.result() helps, a > lot. Yes, you're right. A time.sleep(0.0001) right before the inner result() call makes python-ldap hand over the CPU to the OS. > while all: > while ldap_result[0] is None: > if (timeout>=0) and (time.time()-start_time>timeout): > self._ldap_call(self._l.abandon,msgid) > raise _ldap.TIMELIMIT_EXCEEDED( > "LDAP time limit (%d secs) exceeded." % (timeout) > ) > ldap_result = self._ldap_call(self._l.result,msgid,0,0) > if ldap_result[0] is None: > time.sleep(.01) > if ldap_result[1] is None: > break It's possible to make it somewhat simpler since we have a first result() call before the while loops. while all: while ldap_result[0] is None: if (timeout>=0) and (time.time()-start_time>timeout): self._ldap_call(self._l.abandon,msgid) raise _ldap.TIMELIMIT_EXCEEDED( "LDAP time limit (%d secs) exceeded." % (timeout) ) time.sleep(0.0001) ldap_result = self._ldap_call(self._l.result,msgid,0,0) if ldap_result[1] is None: break all_results.extend(ldap_result[1]) ldap_result = None,None return all_results > Alternatively, adding > an (arbitrarily) long timeout in the call to ldap_result() also accomplishes > the same thing, like: > > ldap_result = self._ldap_call(self._l.result,msgid,0,15 * 60) Which is a bad idea in a multi-threaded environment. > I haven't dug deep into this problem yet, to figure out if this is an > OpenLDAP library problem, or a Python LDAP problem. The main problem here is that the OpenLDAP libs are not thread-safe. Therefore a module-wide lock is needed to serialize all calls into OpenLDAP libs. But just wrapping ldap_result() with locking around it would lead to blocking threads if one thread is waiting for large search results. That's why I implemented LDAPObject.result() in Python like it is today. Unfortunately we cannot deal with waiting for data at the socket level which leads to higher-level polling loop. Everybody is encouraged to try the time.sleep(0.0001) hack and look how it "feels" now. BTW: it makes the simple benchmark I posted yesterday slower. It always depends what you wanna optimize... ;-) I suspect this still might not fully explain the 30s-vs.-immediate-results reports. There may be more issues with other effects like Mauro described. I guess the best solution would be if somebody with some spare cycles digs into OpenLDAP's libldap_r to check if it's ready for use with python-ldap. This would make it possible to do a finer grained locking on LDAP connections instead of module-wide locking allowing different threads with different LDAPObject instances to run without blocking each other. Ciao, Michael. From michael at stroeder.com Wed Jun 26 22:35:54 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Wed, 26 Jun 2002 22:35:54 +0200 Subject: defect References: Message-ID: <3D1A25AA.6080506@stroeder.com> Jens Vagelpohl wrote: > > - with leif's "fix": > > *** Read the RootDSE on same connection > 82.275864 searches/second > *** Read the RootDSE on newly created connection without extra simple bind > 85.599961 searches/second > *** Read the RootDSE on newly created connection with an extra simple bind > 47.726886 searches/second Strange that you get more performance in the second case with a new connection established each time. Is that always the same when you repeat the test? Ciao, Michael. From michael at stroeder.com Wed Jun 26 20:19:02 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Wed, 26 Jun 2002 20:19:02 +0200 Subject: defect References: <20020625020447.GA23462@dman.ddts.net> <05EC71D8-8837-11D6-8BA3-00039363690C@zope.com> <20020625173121.GA32715@dman.ddts.net> Message-ID: <3D1A0596.70902@stroeder.com> Derrick 'dman' Hudson wrote: > On Tue, Jun 25, 2002 at 08:28:15AM -0400, Jens Vagelpohl wrote: > | i have seen similar strange slowdowns **if i connect to localhost**. > | connecting to OpenLDAP on another host across the network was much faster > | (meaning: it worked at the expected speed). > > Actually, you're right. I never tried from a different host until you > said this. I presumed using the loopback interface would have the > least amount of delay due to eliminating network hardware from the > equation. Derrick, Jens, did you read the posting sent by Mauro about the reverse NETBIOS name lookups? Are you by any chance in a Win32 DNS/WINS environment? Ciao, Michael. From michael at stroeder.com Wed Jun 26 20:29:13 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Wed, 26 Jun 2002 20:29:13 +0200 Subject: python-ldap and OpenLDAP 2.1.x (was: web2ldap and Solaris running problem) References: <3D197607.9030906@bcv.it> Message-ID: <3D1A07F9.6070706@stroeder.com> Franco Spinelli wrote: > I have installed openldap 2.1.2 fron openldap site > [..] > from _ldap import * >> ImportError: ld.so.1: /usr/local/bin/python: fatal: relocation error: >> file /usr/local/lib/python2.2/site-packages/_ldap.so: >> symbol ldap_url_search: referenced symbol not found Hmm, it seems that ldap_url_search is no longer available in OpenLDAP 2.1. So far I've never tested python-ldap with OpenLDAP 2.1. You have two options: * Use OpenLDAP 2.0.x to build python-ldap against that (see parameters library_dirs and include_dirs in setup.cfg to specify a separate OpenLDAP 2.0.x directory you used for OpenLDAP 2.0.x's ./configure --prefix) * Remove any reference to ldap_url_search in Modules/LDAPObject.c. web2ldap does not use this function. Ciao, Michael. From jens at zope.com Thu Jun 27 13:45:34 2002 From: jens at zope.com (Jens Vagelpohl) Date: Thu, 27 Jun 2002 07:45:34 -0400 Subject: defect In-Reply-To: <3D1A2AF0.9090802@stroeder.com> Message-ID: <640F23F6-89C3-11D6-9B0D-00039363690C@zope.com> On Wednesday, June 26, 2002, at 04:58 , Michael Str?der wrote: > It's possible to make it somewhat simpler since we have a first result() > call before the while loops. > > while all: > while ldap_result[0] is None: > if (timeout>=0) and (time.time()-start_time>timeout): > self._ldap_call(self._l.abandon,msgid) > raise _ldap.TIMELIMIT_EXCEEDED( > "LDAP time limit (%d secs) exceeded." % (timeout) > ) > time.sleep(0.0001) > ldap_result = self._ldap_call(self._l.result,msgid,0,0) > if ldap_result[1] is None: > break > all_results.extend(ldap_result[1]) > ldap_result = None,None > return all_results > this simplified version seems to slow down my setup. all of a sudden i get only 50-70% of my previous speed. here is a result set with the result method changed to the format shown above:: *** Read the RootDSE on same connection 50.432497 searches/second *** Read the RootDSE on newly created connection without extra simple bind 49.699850 searches/second *** Read the RootDSE on newly created connection with an extra simple bind 31.885130 searches/second here's a result set with leif's version:: *** Read the RootDSE on same connection 100.785835 searches/second *** Read the RootDSE on newly created connection without extra simple bind 95.549340 searches/second *** Read the RootDSE on newly created connection with an extra simple bind 49.825837 searches/second jens From jens at zope.com Thu Jun 27 13:42:20 2002 From: jens at zope.com (Jens Vagelpohl) Date: Thu, 27 Jun 2002 07:42:20 -0400 Subject: defect In-Reply-To: <3D1A25AA.6080506@stroeder.com> Message-ID: michael, you're right, that was my mistake. i was transcribing the values from the screen on a second machine, it was not done via copy/paste. here's a current copy/paste result:: *** Read the RootDSE on same connection 100.785835 searches/second *** Read the RootDSE on newly created connection without extra simple bind 95.549340 searches/second *** Read the RootDSE on newly created connection with an extra simple bind 49.825837 searches/second jens On Wednesday, June 26, 2002, at 04:35 , Michael Str?der wrote: > Jens Vagelpohl wrote: >> - with leif's "fix": >> *** Read the RootDSE on same connection >> 82.275864 searches/second >> *** Read the RootDSE on newly created connection without extra simple bind >> 85.599961 searches/second >> *** Read the RootDSE on newly created connection with an extra simple bind >> 47.726886 searches/second > > Strange that you get more performance in the second case with a new > connection established each time. Is that always the same when you repeat > the test? > > Ciao, Michael. > From leif.hedstrom at propel.com Thu Jun 27 18:58:42 2002 From: leif.hedstrom at propel.com (Leif Hedstrom) Date: Thu, 27 Jun 2002 09:58:42 -0700 Subject: defect Message-ID: I think you want to do the sleep only if ldap_result[0] is None, otherwise you'll cause a sleep every loop, regardless if data is available or not. That's why I originally put it after the call to result() inside the polling loop. -- Leif From michael at stroeder.com Thu Jun 27 19:47:17 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Thu, 27 Jun 2002 19:47:17 +0200 Subject: defect References: <640F23F6-89C3-11D6-9B0D-00039363690C@zope.com> Message-ID: <3D1B4FA5.6060904@stroeder.com> Jens Vagelpohl wrote: > > On Wednesday, June 26, 2002, at 04:58 , Michael Str?der wrote: > >> It's possible to make it somewhat simpler since we have a first >> result() call before the while loops. >> >> while all: > > this simplified version seems to slow down my setup. Yes. I wrote that when posting the code snippet. > here's a result set with leif's version:: Leif's version sets non-zero timeout. Therefore it's faster since OpenLDAP's ldap_result() can use select() to determine just-in-time when data is ready to be read. But it blocks which is a bad thing because of ldap._ldap_lock serializing *all* calls... Just adding the time.sleep() hands over the CPU to the OS. Off course the while-loop is not just-in-time there if received data is ready. Again, the problem is that the OpenLDAP libs are not thread-safe... Ciao, Michael. From michael at stroeder.com Fri Jun 28 09:22:26 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Fri, 28 Jun 2002 09:22:26 +0200 Subject: defect References: Message-ID: <3D1C0EB2.2080702@stroeder.com> Leif Hedstrom wrote: > I think you want to do the sleep only if ldap_result[0] is None, otherwise > you'll cause a sleep every loop, regardless if data is available or not. ??? while ldap_result[0] is None: if (timeout>=0) and (time.time()-start_time>timeout): self._ldap_call(self._l.abandon,msgid) raise _ldap.TIMELIMIT_EXCEEDED( "LDAP time limit (%d secs) exceeded." % (timeout) ) time.sleep(0.0001) ldap_result = self._ldap_call(self._l.result,msgid,0,0) > That's why I originally put it after the call to result() inside the polling > loop. Note that there is a first call to self._l.result() outside both while-loops and the time.sleep() is in the inner loop (while ldap_result[0] is None:). IMHO it should be equivalent. Ciao, Michael. From michael at stroeder.com Thu Jun 27 19:41:16 2002 From: michael at stroeder.com (=?ISO-8859-15?Q?Michael_Str=F6der?=) Date: Thu, 27 Jun 2002 19:41:16 +0200 Subject: Bug in LDAPObject.c, l_ldap_result() Message-ID: <3D1B4E3C.3040609@stroeder.com> HI! While digging into the performance issues with ldap.ldapobject.LDAPObject.result() I noticed that the original C module's result() method does not return results if called with all=1. See the test script attached. Here's the output: *** Using LDAPObject.result() *** [('', {'objectClass': ['top', 'OpenLDAProotDSE']})] *** Using C wrapper method directly *** ('RES_SEARCH_RESULT', None) The RES_SEARCH_RESULT is ok but not None as second tuple item. I guess nobody noticed that since everybody is solely using the synchronous methods *_s(). I noticed that quite a while ago but forgot this unsolved issue. Since I'm not a C programmer and David is not available anymore for python-ldap I'd like to ask if someone is willing to dig into LDAPObject.c's function l_ldap_result(). I guess the if-else-statement in question is (keep in mind the search references!): if (res_type == LDAP_RES_SEARCH_ENTRY || res_type == LDAP_RES_SEARCH_REFERENCE ) pmsg = LDAPmessage_to_python( self->ldap, msg ); else { int result; char **refs = NULL; LDAP_BEGIN_ALLOW_THREADS( self ); ldap_parse_result( self->ldap, msg, &result, NULL, NULL, &refs, NULL, 0 ); LDAP_END_ALLOW_THREADS( self ); if (result != LDAP_SUCCESS) { /* result error */ char *e, err[1024]; if (result == LDAP_REFERRAL && refs && refs[0]) { snprintf(err, sizeof(err), "Referral:\n%s", refs[0]); e = err; } else e = "ldap_parse_result"; return LDAPerror( self->ldap, e ); } pmsg = Py_None; } Hope to find a contributor with C knowledge who has some spare cycles to track that down... Ciao, Michael. -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: result.py URL: From michael at stroeder.com Fri Jun 28 16:44:54 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Fri, 28 Jun 2002 16:44:54 +0200 Subject: ldap_url_search ? References: <3D197607.9030906@bcv.it> <3D1A07F9.6070706@stroeder.com> Message-ID: <3D1C7666.9070700@stroeder.com> Michael Str?der wrote: > Franco Spinelli wrote: > >> I have installed openldap 2.1.2 fron openldap site > > [..] >>> symbol ldap_url_search: referenced symbol not found >> > > Hmm, it seems that ldap_url_search is no longer available in > OpenLDAP > 2.1. So far I've never tested python-ldap with OpenLDAP 2.1. > > [..] > * Remove any reference to ldap_url_search in > Modules/LDAPObject.c. web2ldap does not use this function. Anyone having any objections against removing methods url_search*()? Anyone really using it? If yes, we could easily reimplement it later with help of module ldapurl. The only issue would be that ldapurl needs Python 2.x (not compatible with Python 1.5.x). Ciao, Michael. From jens at zope.com Fri Jun 28 17:11:55 2002 From: jens at zope.com (Jens Vagelpohl) Date: Fri, 28 Jun 2002 11:11:55 -0400 Subject: ldap_url_search ? In-Reply-To: <3D1C7666.9070700@stroeder.com> Message-ID: <62194861-8AA9-11D6-BF24-003065C7DEAE@zope.com> > Anyone having any objections against removing methods url_search*()? > > Anyone really using it? If yes, we could easily reimplement it > later with help of module ldapurl. The only issue would be that > ldapurl needs Python 2.x (not compatible with Python 1.5.x). > no objections. never used it. jens From jlittle at open-it.org Fri Jun 28 18:39:11 2002 From: jlittle at open-it.org (Joe Little) Date: Fri, 28 Jun 2002 09:39:11 -0700 Subject: defect In-Reply-To: <3D1B4FA5.6060904@stroeder.com> Message-ID: <9359F543-8AB5-11D6-99EE-0003930B38C0@open-it.org> Michael. We beat this one up a bit before... But I asked Luke Howard (who is generally in the know on this stuff) again about OpenLDAP thread-safety, and he said that it is thread safe. Perhaps Leif can chime in here on where it stands. On Thursday, June 27, 2002, at 10:47 AM, Michael Str?der wrote: > Jens Vagelpohl wrote: >> On Wednesday, June 26, 2002, at 04:58 , Michael Str?der wrote: >>> It's possible to make it somewhat simpler since we have a first result() call before the while loops. >>> >>> while all: >> this simplified version seems to slow down my setup. > > Yes. I wrote that when posting the code snippet. > >> here's a result set with leif's version:: > > Leif's version sets non-zero timeout. Therefore it's faster since OpenLDAP's ldap_result() can use select() to determine just-in-time when data is ready to be read. But it blocks which is a bad thing because of ldap._ldap_lock serializing *all* calls... > > Just adding the time.sleep() hands over the CPU to the OS. Off course the while-loop is not just-in-time there if received data is ready. > > Again, the problem is that the OpenLDAP libs are not thread-safe... > > Ciao, Michael. > > > > From michael at stroeder.com Fri Jun 28 18:43:42 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Fri, 28 Jun 2002 18:43:42 +0200 Subject: defect References: <9359F543-8AB5-11D6-99EE-0003930B38C0@open-it.org> Message-ID: <3D1C923E.1030302@stroeder.com> Joe Little wrote: > > We beat this one up a bit before... But I asked Luke Howard (who is > generally in the know on this stuff) again about OpenLDAP thread-safety, > and he said that it is thread safe. Since when? Which version? Which lib? libldap_r is supposed to be reentrant but that's not what python-ldap is built against. Furthermore it was not possible to get a positive statement from Kurt Zeilenga last time I asked. Ciao, Michael. From jlittle at open-it.org Fri Jun 28 18:49:14 2002 From: jlittle at open-it.org (Joe Little) Date: Fri, 28 Jun 2002 09:49:14 -0700 Subject: defect In-Reply-To: <3D1C923E.1030302@stroeder.com> Message-ID: two snippets from Luke when speaking about 2.1 (and whether it was finally thread safe: lukeh: On what is thread safe? The client library? The server? I thought it was always pretty much thread safe. I think they've shaken down some issues with the transactional (bdb) backend. When I mentioned complaints here about thread safety, his response: I haven't seen those complaints. The OpenLDAP client library is guaranteed to be thread safe as long as you (a) link against libldap_r and (b) do not permit concurrent access to a single LDAP session (ie. use one per thread or a mutex around it). So, libldap_r is one requirement... but the other seems to be already set by our use of internal thread locking to that single session. On Friday, June 28, 2002, at 09:43 AM, Michael Str?der wrote: > Joe Little wrote: >> We beat this one up a bit before... But I asked Luke Howard (who is generally in the know on this stuff) again about OpenLDAP thread-safety, and he said that it is thread safe. > > Since when? Which version? Which lib? libldap_r is supposed to be reentrant but that's not what python-ldap is built against. Furthermore it was not possible to get a positive statement from Kurt Zeilenga last time I asked. > > Ciao, Michael. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/enriched Size: 1321 bytes Desc: not available URL: From leif.hedstrom at propel.com Fri Jun 28 19:34:22 2002 From: leif.hedstrom at propel.com (Leif Hedstrom) Date: Fri, 28 Jun 2002 10:34:22 -0700 Subject: defect Message-ID: > Note that there is a first call to self._l.result() outside both > while-loops and the time.sleep() is in the inner loop (while > ldap_result[0] is None:). IMHO it should be equivalent. Yeah, my bad, you are absolutely correct. I still find it kinda kludgy to have to put a sleep() statement in the loop like this, but it works for us... ;-) ciao, -- leif From michael at stroeder.com Fri Jun 28 21:00:19 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Fri, 28 Jun 2002 21:00:19 +0200 Subject: defect References: Message-ID: <3D1CB243.8010108@stroeder.com> Joe Little wrote: > two snippets from Luke when speaking about 2.1 Note that most people still build against OpenLDAP 2.0.x. I don't want to mandate OpenLDAP 2.1.x at the moment because there will surely be more complaints about this than about bad performance. I don't like #ifdef's either. Lesson learned so far: I'm not a C programmer and the guys adding the #ifdef's won't be available for maintaining the C code later. > When I mentioned complaints here about thread safety, his response: > > I haven't seen those complaints. The OpenLDAP client library is guaranteed > to be thread safe as long as you (a) link against libldap_r and (b) do > not permit concurrent access to a single LDAP session (ie. use one per > thread or a mutex around it). Ok, not really new information. That was also my understanding from Kurt's postings quite a while ago. Caveats: 1. libldap_r has not been tested with python-ldap yet. 2. libldap_r is not fully implemented in OpenLDAP 2.0.x. AFAIK in 2.0 it's just a hack to make slurpd work. 3. We can't get completely rid of locking. We can do finer grained locking per LDAPObject instance. AFAIK global functions have still to be protected by module-wide locking. > So, libldap_r is one requirement... I'd be glad if someone dives into the gory details. > but the other seems to be already > set by our use of internal thread locking to that single session. At the moment there's a single module-wide lock serializing all calls into libldap since this is required with OpenLDAP 2.0.x. This can be easily changed off course. Ciao, Michael. From dman at dman.ddts.net Fri Jun 28 21:39:23 2002 From: dman at dman.ddts.net (Derrick 'dman' Hudson) Date: Fri, 28 Jun 2002 14:39:23 -0500 Subject: ldap_url_search ? In-Reply-To: <3D1C7666.9070700@stroeder.com> References: <3D197607.9030906@bcv.it> <3D1A07F9.6070706@stroeder.com> <3D1C7666.9070700@stroeder.com> Message-ID: <20020628193923.GA8082@dman.ddts.net> On Fri, Jun 28, 2002 at 04:44:54PM +0200, Michael Str?der wrote: | Michael Str?der wrote: | > Franco Spinelli wrote: | > | >> I have installed openldap 2.1.2 fron openldap site | > > [..] | >>> symbol ldap_url_search: referenced symbol not found | > | > Hmm, it seems that ldap_url_search is no longer available in | > OpenLDAP | > 2.1. So far I've never tested python-ldap with OpenLDAP 2.1. | > | > [..] | > * Remove any reference to ldap_url_search in | > Modules/LDAPObject.c. web2ldap does not use this function. | | Anyone having any objections against removing methods url_search*()? | | Anyone really using it? The only objection I have is that it really helped me to learn how to work with an LDAP URL. I needed to use URLs to configure exim to pull data from LDAP, and I was having trouble using 'ldapsearch' -- it didn't like the URL no matter what I gave it. For me it was a valuable tool. Though if OpenLDAP 2.1 doesn't have the function, then that's a real strong argument for removing those methods (at the very least when compiled against OpenLDAP 2.1). | If yes, we could easily reimplement it later with help of module | ldapurl. The only issue would be that ldapurl needs Python 2.x (not | compatible with Python 1.5.x). Personally I don't consider compatibility with python 1.5.2 a significant feature. 2.0 was released a long time ago, by internet standards. -D -- Microsoft DNS service terminates abnormally when it receives a response to a dns query that was never made. Fix information: run your DNS service on a different platform. -- bugtraq http://dman.ddts.net/~dman/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 240 bytes Desc: not available URL: From michael at stroeder.com Fri Jun 28 21:36:26 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Fri, 28 Jun 2002 21:36:26 +0200 Subject: ldap_url_search ? References: <3D197607.9030906@bcv.it> <3D1A07F9.6070706@stroeder.com> <3D1C7666.9070700@stroeder.com> <20020628193923.GA8082@dman.ddts.net> Message-ID: <3D1CBABA.5040402@stroeder.com> Derrick 'dman' Hudson wrote: > On Fri, Jun 28, 2002 at 04:44:54PM +0200, Michael Str?der wrote: > | Michael Str?der wrote: > | > | Anyone having any objections against removing methods url_search*()? > | > | Anyone really using it? > > The only objection I have is that it really helped me to learn how to > work with an LDAP URL. How about practicing with the module ldapurl? It has more features anyway. And it can construct LDAP URLs from search parameters with method unparse(). > Though if OpenLDAP 2.1 doesn't have the function, then that's a real > strong argument for removing those methods (at the very least when > compiled against OpenLDAP 2.1). Well, another strong argument is that there's no one actively maintaining the C module part under Modules/. Therefore I'd be glad to keep it as short as possible. Ciao, Michael. From jlittle at open-it.org Fri Jun 28 13:26:04 2002 From: jlittle at open-it.org (Joe Little) Date: Fri, 28 Jun 2002 04:26:04 -0700 Subject: defect In-Reply-To: <3D1CB243.8010108@stroeder.com> Message-ID: On Friday, June 28, 2002, at 12:00 PM, Michael Str?der wrote: > Joe Little wrote: >> two snippets from Luke when speaking about 2.1 > > Note that most people still build against OpenLDAP 2.0.x. I don't want > to mandate OpenLDAP 2.1.x at the moment because there will surely be > more complaints about this than about bad performance. Agreed. I talked with Luke about 2.1's improvements in this area, and he replied that 2.0 already was thread safe. I'll be diving into 2.1, but 2.0 should be sufficient for coding thread safe python-ldap once it uses libldap_r. > > I don't like #ifdef's either. Lesson learned so far: I'm not a C > programmer and the guys adding the #ifdef's won't be available for > maintaining the C code later. > >> When I mentioned complaints here about thread safety, his response: >> I haven't seen those complaints. The OpenLDAP client library is >> guaranteed >> to be thread safe as long as you (a) link against libldap_r and (b) do >> not permit concurrent access to a single LDAP session (ie. use one per >> thread or a mutex around it). > > Ok, not really new information. That was also my understanding from > Kurt's postings quite a while ago. > > Caveats: > 1. libldap_r has not been tested with python-ldap yet. > 2. libldap_r is not fully implemented in OpenLDAP 2.0.x. AFAIK in 2.0 > it's just a hack to make slurpd work. What I understood from Luke is that it has been completed in 2.x, with bugs worked out along the way up to 2.0.25. > 3. We can't get completely rid of locking. We can do finer grained > locking per LDAPObject instance. AFAIK global functions have still to > be protected by module-wide locking. > >> So, libldap_r is one requirement... > > I'd be glad if someone dives into the gory details. > >> but the other seems to be already set by our use of internal thread >> locking to that single session. > > At the moment there's a single module-wide lock serializing all calls > into libldap since this is required with OpenLDAP 2.0.x. This can be > easily changed off course. > Well, I hope this small discussion shows a way forward. > Ciao, Michael. > > > > From dman at dman.ddts.net Fri Jun 28 23:13:46 2002 From: dman at dman.ddts.net (Derrick 'dman' Hudson) Date: Fri, 28 Jun 2002 16:13:46 -0500 Subject: ldap_url_search ? In-Reply-To: <3D1CBABA.5040402@stroeder.com> References: <3D197607.9030906@bcv.it> <3D1A07F9.6070706@stroeder.com> <3D1C7666.9070700@stroeder.com> <20020628193923.GA8082@dman.ddts.net> <3D1CBABA.5040402@stroeder.com> Message-ID: <20020628211346.GC8082@dman.ddts.net> On Fri, Jun 28, 2002 at 09:36:26PM +0200, Michael Str?der wrote: | Derrick 'dman' Hudson wrote: | >On Fri, Jun 28, 2002 at 04:44:54PM +0200, Michael Str?der wrote: | >| Michael Str?der wrote: | >| | >| Anyone having any objections against removing methods url_search*()? | >| | >| Anyone really using it? | > | >The only objection I have is that it really helped me to learn how to | >work with an LDAP URL. | | How about practicing with the module ldapurl? How? The documentation is pretty sparse. (I guess I could read the source, though, and then I would say "Ok") http://python-ldap.sourceforge.net/doc/python-ldap/module-ldapurl.html http://python-ldap.sourceforge.net/pydoc/ldapurl.html If the ldapurl module does everything that the current url_search methods do, then there is no compelling reason to keep the latter around. -D -- "...In the UNIX world, people tend to interpret `non-technical user' as meaning someone who's only ever written one device driver." --Daniel Pead http://dman.ddts.net/~dman/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 240 bytes Desc: not available URL: From michael at stroeder.com Sat Jun 29 13:02:12 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Sat, 29 Jun 2002 13:02:12 +0200 Subject: Module ldapurl (was: ldap_url_search ?) References: <3D197607.9030906@bcv.it> <3D1A07F9.6070706@stroeder.com> <3D1C7666.9070700@stroeder.com> <20020628193923.GA8082@dman.ddts.net> <3D1CBABA.5040402@stroeder.com> <20020628211346.GC8082@dman.ddts.net> Message-ID: <3D1D93B4.5050408@stroeder.com> Derrick 'dman' Hudson wrote: > On Fri, Jun 28, 2002 at 09:36:26PM +0200, Michael Str?der wrote: > | Derrick 'dman' Hudson wrote: > | >On Fri, Jun 28, 2002 at 04:44:54PM +0200, Michael Str?der wrote: > | >| Michael Str?der wrote: > | >| > | >| Anyone having any objections against removing methods url_search*()? > | >| > | >| Anyone really using it? > | > > | >The only objection I have is that it really helped me to learn how to > | >work with an LDAP URL. > | > | How about practicing with the module ldapurl? > > How? The documentation is pretty sparse. > [..] > If the ldapurl module does everything that the current url_search > methods do, then there is no compelling reason to keep the latter > around. It parses and unparses LDAP URLs and is designed to be completely independent from module ldap. Some sample code for parsing a LDAP URL: >>> import ldapurl >>> ldap_url = ldapurl.LDAPUrl('ldap://localhost:1389/dc=stroeder,dc=com?cn???bindname=cn=Michael%2cdc=stroeder%2cdc=com,X-BINDPW=secret') >>> # Using the parsed LDAP URL by reading the class attributes >>> ldap_url.dn u'dc=stroeder,dc=com' >>> ldap_url.hostport 'localhost:1389' >>> ldap_url.attrs ['cn'] >>> ldap_url.filterstr u'(objectclass=*)' >>> ldap_url.who 'cn=Michael,dc=stroeder,dc=com' >>> ldap_url.cred 'secret' >>> ldap_url.scope 0 Some sample code for the other way round: >>> ldap_url = ldapurl.LDAPUrl(hostport='localhost:1389',dn=u'dc=stroeder,dc=com',attrs=['cn','mail'],who='cn=Michael,dc=stroeder,dc=com',cred='secret') >>> ldap_url.unparse() u'ldap://localhost:1389/dc=stroeder,dc=com?cn,mail?base?(objectclass=*)?bindname=cn=Michael%2Cdc=stroeder%2Cdc=com,X-BINDPW=secret' That's pretty much all it does. Note that you don't have to specify all the class attributes. There are reasonable defaults assumed. And who and cred maps to None if not specified. See the tests at the bottom of ldapurl.py for more examples. Advice: For security reasons you shouldn't specify passwords in LDAP URLs unless you really know what you're doing. Ciao, Michael. From michael at stroeder.com Sat Jun 29 13:49:48 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Sat, 29 Jun 2002 13:49:48 +0200 Subject: Building python-ldap with libldap_r (was: defect) References: Message-ID: <3D1D9EDC.9020402@stroeder.com> Joe Little wrote: > > On Friday, June 28, 2002, at 12:00 PM, Michael Str?der wrote: > >> Note that most people still build against OpenLDAP 2.0.x. I don't want >> to mandate OpenLDAP 2.1.x at the moment because there will surely be >> more complaints about this than about bad performance. > > Agreed. I talked with Luke about 2.1's improvements in this area, and he > replied that 2.0 already was thread safe. I'll be diving into 2.1, but > 2.0 should be sufficient for coding thread safe python-ldap once it uses > libldap_r. > >> Caveats: >> 1. libldap_r has not been tested with python-ldap yet. >> 2. libldap_r is not fully implemented in OpenLDAP 2.0.x. AFAIK in 2.0 >> it's just a hack to make slurpd work. > > What I understood from Luke is that it has been completed in 2.x, with > bugs worked out along the way up to 2.0.25. I tried to build python-ldap against libldap_r from OpenLDAP's REL_ENG_2 branch by simply modifying setup.cfg: libs = ldap_r lber sasl The build went fine. ldd shows output like expected: $ ldd build/lib.linux-i686-2.2/_ldap.so libldap_r.so.2 => /usr/lib/libldap_r.so.2 (0x4001f000) liblber.so.2 => /usr/lib/liblber.so.2 (0x4004b000) libsasl.so.7 => /usr/lib/libsasl.so.7 (0x40055000) [..] But import does not work. $ python -c "import ldap" Traceback (most recent call last): File "", line 1, in ? File "/usr/lib/python2.2/site-packages/ldap/__init__.py", line 5, in ? from _ldap import * ImportError: /usr/lib/python2.2/site-packages/_ldap.so: undefined symbol: ldap_first_reference Anyone willing to dig into this? Ciao, Michael. From dman at dman.ddts.net Sat Jun 29 01:10:14 2002 From: dman at dman.ddts.net (Derrick 'dman' Hudson) Date: Fri, 28 Jun 2002 18:10:14 -0500 Subject: defect In-Reply-To: References: <3D1C923E.1030302@stroeder.com> Message-ID: <20020628231014.GA14145@dman.ddts.net> On Fri, Jun 28, 2002 at 09:49:14AM -0700, Joe Little wrote: | I haven't seen those complaints. The OpenLDAP client library is guaranteed | to be thread safe as long as you | do not permit concurrent access to a single LDAP session (ie. use | one per thread or a mutex around it). (FWIW) This sounds perfectly acceptable to me. Making a separate connection for each thread is likely to reduce contention anyways (improves performance). I don't see my patch (other thread) on the list, but don't use it -- it breaks web2ldap :-(. I need to work on it some more. -D -- The Consultant's Curse: When the customer has beaten upon you long enough, give him what he asks for, instead of what he needs. This is very strong medicine, and is normally only required once. http://dman.ddts.net/~dman/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 240 bytes Desc: not available URL: From michael at stroeder.com Sat Jun 29 14:00:48 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Sat, 29 Jun 2002 14:00:48 +0200 Subject: Connection pooling (was: defect) References: <3D1C923E.1030302@stroeder.com> <20020628231014.GA14145@dman.ddts.net> Message-ID: <3D1DA170.50909@stroeder.com> Derrick 'dman' Hudson wrote: > On Fri, Jun 28, 2002 at 09:49:14AM -0700, Joe Little wrote: > > | do not permit concurrent access to a single LDAP session (ie. use > | one per thread or a mutex around it). > > (FWIW) > This sounds perfectly acceptable to me. > Making a separate connection for each thread is likely to reduce > contention anyways (improves performance). This remembers me of an old idea I had to implement persistent LDAP connection pooling with Queue.Queue found in Python's standard lib. E.g. if a HTTP server serves HTTP requests with 100 active threads you're not really keen to open a LDAP connection per thread because 1. your system ressources might get exhausted (e.g. file handles) 2. opening a connection is very slow (e.g. when initializing attributes read from RootDSE, SSL negotiation, SASL binds, etc.). Ciao, Michael. From michael at stroeder.com Sat Jun 29 14:29:25 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Sat, 29 Jun 2002 14:29:25 +0200 Subject: ldap_url_search ? References: <3D197607.9030906@bcv.it> <3D1A07F9.6070706@stroeder.com> <3D1C7666.9070700@stroeder.com> Message-ID: <3D1DA825.7050906@stroeder.com> Michael Str?der wrote: > > Hmm, it seems that ldap_url_search is no longer available in > > OpenLDAP > > 2.1. > > [..] > Anyone having any objections against removing methods url_search*()? I removed method url_search from LDAPObject.c. We can reimplement that in the Python wrapper class ldap.ldapobject.LDAPObject if there's request. Ciao, Michael. From dman at dman.ddts.net Fri Jun 28 23:07:24 2002 From: dman at dman.ddts.net (Derrick 'dman' Hudson) Date: Fri, 28 Jun 2002 16:07:24 -0500 Subject: Bug in LDAPObject.c, l_ldap_result() In-Reply-To: <3D1B4E3C.3040609@stroeder.com> References: <3D1B4E3C.3040609@stroeder.com> Message-ID: <20020628210724.GB8082@dman.ddts.net> On Thu, Jun 27, 2002 at 07:41:16PM +0200, Michael Str?der wrote: | While digging into the performance issues with | ldap.ldapobject.LDAPObject.result() I noticed that the original C | module's result() method does not return results if called with | all=1. See the test script attached. | Hope to find a contributor with C knowledge who has some spare | cycles to track that down... --- LDAPObject.c.orig Fri Jun 28 15:59:11 2002 +++ LDAPObject.c Fri Jun 28 15:59:37 2002 @@ -818,7 +818,7 @@ e = "ldap_parse_result"; return LDAPerror( self->ldap, e ); } - pmsg = Py_None; + pmsg = LDAPmessage_to_python( self->ldap, msg ); } result_str = LDAPconstant( res_type ); It really does help to not ignore the data that's returned :-). I haven't thoroughly tested it (actually, I haven't tested any more than the script you supplied tests it, except that I explicitly added the 'all' and 'timeout' arguments), but it looks correct and seems to work. (the patch is against 2.0.0pre04, but is easy enough to do manually if CVS has different line numbers) -D -- "Wipe Info uses hexadecimal values to wipe files. This provides more security than wiping with decimal values." -- Norton SystemWorks 2002 Manual http://dman.ddts.net/~dman/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 240 bytes Desc: not available URL: From michael at stroeder.com Sat Jun 29 16:32:52 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Sat, 29 Jun 2002 16:32:52 +0200 Subject: Bug in LDAPObject.c, l_ldap_result() References: <3D1B4E3C.3040609@stroeder.com> <20020628210724.GB8082@dman.ddts.net> Message-ID: <3D1DC514.1080208@stroeder.com> Derrick 'dman' Hudson wrote: > > --- LDAPObject.c.orig Fri Jun 28 15:59:11 2002 > +++ LDAPObject.c Fri Jun 28 15:59:37 2002 > @@ -818,7 +818,7 @@ > e = "ldap_parse_result"; > return LDAPerror( self->ldap, e ); > } > - pmsg = Py_None; > + pmsg = LDAPmessage_to_python( self->ldap, msg ); > } > > result_str = LDAPconstant( res_type ); > > > It really does help to not ignore the data that's returned :-). Oh, yes. > I haven't thoroughly tested it You already noticed the issues with web2ldap. One has to fix two other parts of python-ldap for this patch to work with web2ldap. I've checked in Modules/LDAPObject.c Lib/ldap/ldapobject.py Lib/ldap/async.py. Now the C module's result() is directly wrapped and LDAPObject.c contains the fix above. I guess I will add a derived class for the non-blocking version of LDAPObject.result(). So everyone will be free to choose. Ciao, Michael. From dman at dman.ddts.net Sat Jun 29 19:52:44 2002 From: dman at dman.ddts.net (Derrick 'dman' Hudson) Date: Sat, 29 Jun 2002 12:52:44 -0500 Subject: Bug in LDAPObject.c, l_ldap_result() In-Reply-To: <3D1DC514.1080208@stroeder.com> References: <3D1B4E3C.3040609@stroeder.com> <20020628210724.GB8082@dman.ddts.net> <3D1DC514.1080208@stroeder.com> Message-ID: <20020629175244.GA23123@dman.ddts.net> On Sat, Jun 29, 2002 at 04:32:52PM +0200, Michael Str?der wrote: | Derrick 'dman' Hudson wrote: | >- pmsg = Py_None; | >+ pmsg = LDAPmessage_to_python( self->ldap, msg ); | >It really does help to not ignore the data that's returned :-). | | Oh, yes. | | >I haven't thoroughly tested it | | You already noticed the issues with web2ldap. Yep. I was beating myself over the head last night trying to figure out the right type to return. | One has to fix two other parts of python-ldap for this patch to work | with web2ldap. Ahh. | I've checked in Modules/LDAPObject.c Lib/ldap/ldapobject.py | Lib/ldap/async.py. Now the C module's result() is directly wrapped | and LDAPObject.c contains the fix above. I see now : def search_st(...) - return self.result(msgid,all=1,timeout=timeout) + return self.result(msgid,all=1,timeout=timeout)[1] That's precisely what I kept running into -- the line ldap_result[0][1] _sometimes_ wasn't getting the right element. I thought the result function wasn't returning the right value (it needed another layer of list nesting), but when I would add the extra list some other parts wouldn't work! That certainly explains it :-). | I guess I will add a derived class for the non-blocking version of | LDAPObject.result(). So everyone will be free to choose. Isn't that what 'timeout=0' is for? I also think the C function needs to be modified to treat Py_None as NULL to allow python code to specify a timeout that is indefinitely long. -D -- "GUIs normally make it simple to accomplish simple actions and impossible to accomplish complex actions." --Doug Gwyn (22/Jun/91 in comp.unix.wizards) http://dman.ddts.net/~dman/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 240 bytes Desc: not available URL: From michael at stroeder.com Sat Jun 29 19:54:12 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Sat, 29 Jun 2002 19:54:12 +0200 Subject: Bug in LDAPObject.c, l_ldap_result() References: <3D1B4E3C.3040609@stroeder.com> <20020628210724.GB8082@dman.ddts.net> <3D1DC514.1080208@stroeder.com> <20020629175244.GA23123@dman.ddts.net> Message-ID: <3D1DF444.10108@stroeder.com> Derrick 'dman' Hudson wrote: > > I see now : > > def search_st(...) > - return self.result(msgid,all=1,timeout=timeout) > + return self.result(msgid,all=1,timeout=timeout)[1] > > > That's precisely what I kept running into Please test it! This whole stuff with result() is somewhat incomplete anyway since the result type > | I guess I will add a derived class for the non-blocking version of > | LDAPObject.result(). So everyone will be free to choose. > > Isn't that what 'timeout=0' is for? Yes. But then the application has to handle that. The non-blocking version of LDAPObject.result() is the one which needed the time.sleep() hack. I will add it to ldapobject.py in a separate class. > I also think the C function needs > to be modified to treat Py_None as NULL to allow python code to > specify a timeout that is indefinitely long. IMHO that's done with timeout=-1. Looking at l_ldap_result() in LDAPObject.c it seems to be implemented correctly (although reading C makes my eyes hurt): if (timeout >= 0) { tvp = &tv; set_timeval_from_double( tvp, timeout ); } else { tvp = NULL; } Ciao, Michael. From dman at dman.ddts.net Sat Jun 29 22:29:58 2002 From: dman at dman.ddts.net (Derrick 'dman' Hudson) Date: Sat, 29 Jun 2002 15:29:58 -0500 Subject: Bug in LDAPObject.c, l_ldap_result() In-Reply-To: <3D1DF444.10108@stroeder.com> References: <3D1B4E3C.3040609@stroeder.com> <20020628210724.GB8082@dman.ddts.net> <3D1DC514.1080208@stroeder.com> <20020629175244.GA23123@dman.ddts.net> <3D1DF444.10108@stroeder.com> Message-ID: <20020629202958.GA26301@dman.ddts.net> On Sat, Jun 29, 2002 at 07:54:12PM +0200, Michael Str?der wrote: | Derrick 'dman' Hudson wrote: | > | >I see now : | > | > def search_st(...) | >- return self.result(msgid,all=1,timeout=timeout) | >+ return self.result(msgid,all=1,timeout=timeout)[1] | > | > | >That's precisely what I kept running into | | Please test it! Now I have (some more, at least). I copied async.py from CVS and backported the patches to LDAPObject.c and ldapobject.py to version 2.0.0pre04 and now web2ldap works great (and faster!). Re-timing the maillist.py script I originally posted yields this : real 0m0.206s user 0m0.070s sys 0m0.040s As crude as time(1) may be, that is clearly better and isn't long enough to measure with my watch or to show up in top :-). | This whole stuff with result() is somewhat | incomplete anyway since the result type Looks like your sentence was cut off. The CVS version of python-ldap doesn't work for me, though. I had to change some of the sasl stuff in LDAPObject.c for it to compile, and I get a SEGV with web2ldap. (I also changed a couple other minor details that gcc complained about) I'll work on that problem later. | >| I guess I will add a derived class for the non-blocking version of | >| LDAPObject.result(). So everyone will be free to choose. | > | >Isn't that what 'timeout=0' is for? | | Yes. But then the application has to handle that. Isn't that the purpose of not blocking in the first place? | The non-blocking version of LDAPObject.result() is the one which | needed the time.sleep() hack. The version that was in release 2.0.0pre04 was blocking if all was true. It just did the blocking in python/python-ldap instead of in C/libldap2. | I will add it to ldapobject.py in a separate class. I have no complaints with that. | >I also think the C function needs | >to be modified to treat Py_None as NULL to allow python code to | >specify a timeout that is indefinitely long. | | IMHO that's done with timeout=-1. Looking at l_ldap_result() in | LDAPObject.c it seems to be implemented correctly Ahh, yeah, that is one suitable convention. Doesn't timeout=None sound amusingly accurate though? (in english at least) -D -- > SELECT * FROM users WHERE clue > 0 0 rows returned (http://www.thinkgeek.com/images/products/zoom/no-clue.jpg) http://dman.ddts.net/~dman/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 240 bytes Desc: not available URL: From michael at stroeder.com Sun Jun 30 16:05:45 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Sun, 30 Jun 2002 16:05:45 +0200 Subject: Bug in LDAPObject.c, l_ldap_result() References: <3D1B4E3C.3040609@stroeder.com> <20020628210724.GB8082@dman.ddts.net> <3D1DC514.1080208@stroeder.com> <20020629175244.GA23123@dman.ddts.net> <3D1DF444.10108@stroeder.com> <20020629202958.GA26301@dman.ddts.net> Message-ID: <3D1F1039.30505@stroeder.com> Derrick 'dman' Hudson wrote: > On Sat, Jun 29, 2002 at 07:54:12PM +0200, Michael Str?der wrote: > | This whole stuff with result() is somewhat > | incomplete anyway since the result type > > Looks like your sentence was cut off. The result type is not returned in search results when using search_s() which is somewhat incomplete. Search results may contain real entries and search continuations. One has to distinguish that by looking at the type (tuple vs. string). But that's not nice. > The CVS version of python-ldap doesn't work for me, though. I had to > change some of the sasl stuff in LDAPObject.c for it to compile, Hmm, which version of OpenLDAP? I usually compile against CVS version checked out from REL_ENG_2 branch. That works smoothly. Can you post your SASL-related changes? > and I get a SEGV with web2ldap. Did you check http://www.web2ldap.de/faq.html ? -------------------------- snip --------------------------- web2ldap crashs a lot of with segmentation faults. This is not a web2ldap issue. Please check how your Python interpreter was compiled. It has to be build with configure --without-pymalloc. You probably have to rebuild all C extension modules involved afterwards. -------------------------- snip --------------------------- > (I also changed a couple other minor > details that gcc complained about) Again: Post these changes for open discussion. > | Yes. But then the application has to handle that. > > Isn't that the purpose of not blocking in the first place? > > | The non-blocking version of LDAPObject.result() is the one which > | needed the time.sleep() hack. > > The version that was in release 2.0.0pre04 was blocking if all was > true. It just did the blocking in python/python-ldap instead of in > C/libldap2. Yes. My wording was rather misleading. Blocking yes, but not locking all the time. Therefore other threads calling into the LDAP libs are allowed to run although one thread is blocked. > | >I also think the C function needs > | >to be modified to treat Py_None as NULL to allow python code to > | >specify a timeout that is indefinitely long. > | > | IMHO that's done with timeout=-1. Looking at l_ldap_result() in > | LDAPObject.c it seems to be implemented correctly > > Ahh, yeah, that is one suitable convention. Doesn't > timeout=None > sound amusingly accurate though? (in english at least) Well, I have no clue why David chose -1 instead of None. Ciao, Michael. From steffen.ries at sympatico.ca Sun Jun 30 00:08:59 2002 From: steffen.ries at sympatico.ca (Steffen Ries) Date: Sat, 29 Jun 2002 18:08:59 -0400 Subject: set_rebind_proc Message-ID: <20020629220859.GYSS1505.tomts21-srv.bellnexxia.net@gondolin.beleriand> Hi, I recently ran into a problem with a replicated iPlanet directory server. My client was setup to write to the slave directory, which answered with a REFERRAL. The client was trying to follow the referral but could not authenticate with the master directory. After looking into the code I found that the "set_rebind_proc" method has been disabled "until made OpenLDAP2 compatible". Since I need this functionality, I have implemented an openLDAP2 compatible version of set_rebind_proc. The old implementation had an IMHO very bad design flaw: only one rebind_proc was allowed for the whole module. In other words, if you try to open more than one directory connection, the rebind_proc could only be set to one directory, rebinds to the other directory would have fatal consequences (either python-fatal error or a segfault). Since I need to support multiple connections to different directories as well, I implemented a solution for this problem. My solution is not ideal, but I think it is a reasonable compromise. Ideally I would extend the LDAP object and store the rebind callback in the extended version, but LDAP is an opaque object handled by the OpenLDAP library. My alternative is to keep track of the allocated LDAPObjects and map the LDAP object to the corresponding LDAPObject. For this purpose I'm storing the LDAPObjects in a linked list. I don't expect a big number of LDAPObjects to be allocated, so the overhead of doing a linear search should be acceptable. I kept the interface for the callback the same as it was for the old implementation, i.e. the callback has to accept an LDAP object and return a tuple (DN, CRED, METHOD). The second part of the patch is for the python wrapper. I added an automatic rebind_procedure, which is installed when you call a version of "bind()". This way referrals should normally be transparent to the client. I am not sure whether this is a good idea or not (?). /steffen -- steffen.ries at sympatico.ca <> Gravity is a myth -- the Earth sucks! -------------- next part -------------- A non-text attachment was scrubbed... Name: python-ldap.diff Type: text/x-patch Size: 6766 bytes Desc: not available URL: From michael at stroeder.com Sun Jun 30 17:08:15 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Sun, 30 Jun 2002 17:08:15 +0200 Subject: set_rebind_proc References: <20020629220859.GYSS1505.tomts21-srv.bellnexxia.net@gondolin.beleriand> Message-ID: <3D1F1EDF.1000108@stroeder.com> Steffen Ries wrote: > > I recently ran into a problem with a replicated iPlanet directory > server. My client was setup to write to the slave directory, which > answered with a REFERRAL. The client was trying to follow the > referral but could not authenticate with the master directory. Steffen, I can confirm that you already hit many of the issues with following referrals. > After looking into the code I found that the "set_rebind_proc" method > has been disabled "until made OpenLDAP2 compatible". Since I need this > functionality, I have implemented an openLDAP2 compatible version of > set_rebind_proc. AFAIR there was no decision made yet in the OpenLDAP project how to implement set_rebind_proc. But I have to check recent status. > The old implementation had an IMHO very bad design flaw: only one > rebind_proc was allowed for the whole module. Which is not the only design flaw of set_rebind_proc. It's rather flawed by design... > Since I need to support multiple connections to different directories > as well, I implemented a solution for this problem. I have to admit that I'm scared about adding such functionality into the C part of python-ldap. I'd like to encourage you to find a Python-only solution which is easier to maintain. BTW: That's exactly one of the benefits of class ldap.ldapobject.LDAPObject: It can be sub-classed for extending its functionality. > I kept the interface for the callback the same as it was for the old > implementation, i.e. the callback has to accept an LDAP object and > return a tuple (DN, CRED, METHOD). Please check whether this complies to what's done in OpenLDAP 2 today. > The second part of the patch is for the python wrapper. I added an > automatic rebind_procedure, which is installed when you call a version > of "bind()". This way referrals should normally be transparent to the > client. I am not sure whether this is a good idea or not (?). My own investigations of this problem lead to not add any generic referral mechanism to python-ldap because one can't find a generic way without making too many assumptions. Several issues with referrals and - a similar problem - LDAP connection failover. Connection handling: An application has to decide whether it follows the referral for a single operation (e.g. a single write operations) or drop the current LDAP connection completely and switch over to the referred server for all following operations. An interesting question in e.g. a one master, many slaves scenario. Another issue is whether there are more operations involved to properly establish a new LDAP connection, e.g. calling start_tls_s(). Bind sequence: Currently most implementations solely use simple bind. But e.g. SASL BindRequests are rather more complicated (Kerberos via GSS-API, SSL with client certificates). Also the target server mentioned in the referral might have other SASL capabilities. Another issue is setting the right LDAP protocol version (negotiated vs. set by a-priori knowledge). Security issues: I have no clue how to model all the different bind mechanisms securely. Also one would have to store credentials (not only passwords)... I'd rather like to see a framework which helps the application developer to implement generic exception handlers for e.g. ldap.SERVER_DOWN or ldap.REFERRAL and decide what to do in his/her specific environment. We could store the last operation tried in a class attribute by just extending LDAPObject._ldap_call(). Hmm, adding the parameters of the LDAP operation which raised a LDAPError exception to the LDAPError exception instance itself would be really nice... Ciao, Michael. From steffen.ries at sympatico.ca Sun Jun 30 18:37:22 2002 From: steffen.ries at sympatico.ca (Steffen Ries) Date: Sun, 30 Jun 2002 12:37:22 -0400 Subject: set_rebind_proc In-Reply-To: Michael =?iso-8859-1?q?Str=F6der's?= message of "Sun, 30 Jun 2002 17:08:15 +0200" References: <20020629220859.GYSS1505.tomts21-srv.bellnexxia.net@gondolin.beleriand> <3D1F1EDF.1000108@stroeder.com> Message-ID: <20020630163722.MEEO18351.tomts7-srv.bellnexxia.net@gondolin.beleriand> Michael Str?der writes: [...] > > After looking into the code I found that the "set_rebind_proc" method > > has been disabled "until made OpenLDAP2 compatible". Since I need this > > functionality, I have implemented an openLDAP2 compatible version of > > set_rebind_proc. > > AFAIR there was no decision made yet in the OpenLDAP project how to > implement set_rebind_proc. But I have to check recent status. I guess I should have done more research and less reverse engineering... :-) I looked through the OpenLDAP mailing list archives and found only a couple of messages relating to this issue. I can't say whether the mechanism is finalized, but I've seen concerns that it should not be implemented the way it is. [...] > I have to admit that I'm scared about adding such functionality into > the C part of python-ldap. I'd like to encourage you to find a > Python-only solution which is easier to maintain. BTW: That's exactly > one of the benefits of class ldap.ldapobject.LDAPObject: It can be > sub-classed for extending its functionality. Hmm. Implementing a Python-only solution would require to catch the ldap.REFERRAL (or ldap.PARTIAL_RESULT) for each of the operations, decide how to chase the referral and then retry the operation. I can try to come up with a proposal... > > I kept the interface for the callback the same as it was for the old > > implementation, i.e. the callback has to accept an LDAP object and > > return a tuple (DN, CRED, METHOD). > > Please check whether this complies to what's done in OpenLDAP 2 today. It does not. The new API call passes URL, request type and message-id. I ignored these values (mostly because I don't have any particular use for them...). [...] > I'd rather like to see a framework which helps the application > developer to implement generic exception handlers for > e.g. ldap.SERVER_DOWN or ldap.REFERRAL and decide what to do in > his/her specific environment. > > We could store the last operation tried in a class attribute by just > extending LDAPObject._ldap_call(). Hmm, adding the parameters of the > LDAP operation which raised a LDAPError exception to the LDAPError > exception instance itself would be really nice... Are you thinking of registering error specific exception handlers or pushing the functionality to the application? I would like to add some functionality to the library (call it a framework), rather than burden the application to wrap every search/add/modify with an exception handler for chasing referrals. I have this kind of code in some places for fail over, which makes it pretty hard to maintain. /steffen -- steffen.ries at sympatico.ca <> Gravity is a myth -- the Earth sucks! From michael at stroeder.com Sun Jun 30 19:09:36 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Sun, 30 Jun 2002 19:09:36 +0200 Subject: set_rebind_proc References: <20020629220859.GYSS1505.tomts21-srv.bellnexxia.net@gondolin.beleriand> <3D1F1EDF.1000108@stroeder.com> <20020630163722.MEEO18351.tomts7-srv.bellnexxia.net@gondolin.beleriand> Message-ID: <3D1F3B50.4070802@stroeder.com> Steffen Ries wrote: > >>I have to admit that I'm scared about adding such functionality into >>the C part of python-ldap. I'd like to encourage you to find a >>Python-only solution which is easier to maintain. BTW: That's exactly >>one of the benefits of class ldap.ldapobject.LDAPObject: It can be >>sub-classed for extending its functionality. > > Hmm. Implementing a Python-only solution would require to catch the > ldap.REFERRAL (or ldap.PARTIAL_RESULT) for each of the operations, > decide how to chase the referral and then retry the operation. I can > try to come up with a proposal... > [..] >>I'd rather like to see a framework which helps the application >>developer to implement generic exception handlers for >>e.g. ldap.SERVER_DOWN or ldap.REFERRAL and decide what to do in >>his/her specific environment. > > Are you thinking of registering error specific exception handlers or > pushing the functionality to the application? LDAPObject._ldap_call could be the point from where registered exception handlers are called. E.g. a new method LDAPObject.ErrorHandler(e,func,args,kwargs) could check whether a generic handler method is registered for a specific LDAPError class and call that handler with these parameters. Now the application would sub-class LDAPObject and implement such handlers applying the specific knowledge about the environment. Note that we have the advantage of having OO-oriented API and exceptions. Therefore we aren't limited to such crude mechanisms common in C like call-back functions. IMHO we should think in this direction. > I would like to add some functionality to the library (call it a > framework), rather than burden the application to wrap every > search/add/modify with an exception handler for chasing referrals. I > have this kind of code in some places for fail over, which makes it > pretty hard to maintain. Well, I'd like to have something in the python-ldap library too. But although it sounds pretty easy I guess it's not. There are some caveats I'm sure (e.g. some kind of nested error handling in exception handlers...). Ciao, Michael. From michael at stroeder.com Sun Jun 30 23:17:30 2002 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Sun, 30 Jun 2002 23:17:30 +0200 Subject: Module ldapurl (was: ldap_url_search ?) References: <3D197607.9030906@bcv.it> <3D1A07F9.6070706@stroeder.com> <3D1C7666.9070700@stroeder.com> <20020628193923.GA8082@dman.ddts.net> <3D1CBABA.5040402@stroeder.com> <20020628211346.GC8082@dman.ddts.net> <3D1D93B4.5050408@stroeder.com> Message-ID: <3D1F756A.6070404@stroeder.com> Michael Str?der wrote: > Derrick 'dman' Hudson wrote: > >> On Fri, Jun 28, 2002 at 09:36:26PM +0200, Michael Str?der wrote: >> | | How about practicing with the module ldapurl? >> >> How? The documentation is pretty sparse. BTW: There's Demo/Lib/ldapurl/urlsearch.py since ages... ;-) Ciao, Michael. From michael at stroeder.com Sun Jun 30 23:38:10 2002 From: michael at stroeder.com (=?ISO-8859-15?Q?Michael_Str=F6der?=) Date: Sun, 30 Jun 2002 23:38:10 +0200 Subject: FYI: python-ldap with MDS() Message-ID: <3D1F7A42.90008@stroeder.com> HI! Here's what people are doing with recently added SASL support: http://www-itg.lbl.gov/gtg/projects/misc/mds_python-ldap.html Ciao, Michael.