From best at univention.de Thu Jan 5 13:07:23 2017 From: best at univention.de (Florian Best) Date: Thu, 5 Jan 2017 19:07:23 +0100 Subject: [python-ldap] AttributeError: ReconnectLDAPObject has no attribute '_l' Message-ID: Hello :) As there doesn't seem to be a bug reporting system, I will post to this mailing list. I got the following traceback: res = self.lo.search_ext_s(base, ldap_scope, filter, attr, serverctrls=serverctrls, clientctrls=None, timeout=timeout, sizelimit=sizelimit) File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 918, in search_ext_s return self._apply_method_s(SimpleLDAPObject.search_ext_s,*args,**kwargs) File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 865, in _apply_method_s self.reconnect(self._uri,retry_max=self._retry_max,retry_delay=self._retry_delay) File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 843, in reconnect SimpleLDAPObject.unbind_s(self) File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 600, in unbind_s return self.unbind_ext_s(None,None) File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 591, in unbind_ext_s msgid = self.unbind_ext(serverctrls,clientctrls) File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 588, in unbind_ext return self._ldap_call(self._l.unbind_ext,RequestControlTuples(serverctrls),RequestControlTuples(clientctrls)) File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 136, in __getattr__ self.__class__.__name__,repr(name) AttributeError: ReconnectLDAPObject has no attribute '_l' I have two assumtions about this, but I am not completely sure as this is only a traceback reported by one of our customers and I couldn't properly reproduce it. Maybe you have another explanation. The first, more probable reason is because unbind_s() is called twice: During the reconnection SimpleLDAPObject.unbind_s() is called first in _apply_method_s() and second in reconnect() while the server is still down. The second call would cause that SimpleLDAPObject.unbind_s() calls a method which uses self._l while reconnect() already removed self._l. (The second thing I could imagine would be If two threads use the same object, this can happen during the reconnection?) I attached a patch which maybe prevent this, as _l is not removed anymore from the namespace of that object. The expected behavior would be that just ldap.SERVER_DOWN would be raised (and not AttributeError which can't be really handled without parsing the exception message and wrapping every ldap call). Best regards Florian -- Florian Best Open Source Software Engineer Univention GmbH be open Mary-Somerville-Str.1 28359 Bremen Tel.: +49 421 22232-0 Fax : +49 421 22232-99 best at univention.de http://www.univention.de Gesch?ftsf?hrer: Peter H. Ganten HRB 20755 Amtsgericht Bremen Steuer-Nr.: 71-597-02876 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: attribute_error.patch Type: text/x-diff Size: 1751 bytes Desc: not available URL: From riteshn at gmail.com Thu Jan 5 13:03:51 2017 From: riteshn at gmail.com (Ritesh Nadhani) Date: Thu, 5 Jan 2017 10:03:51 -0800 Subject: [python-ldap] LDAP pagination cookie not working across AD server restart Message-ID: Hello I am writing a script that would poll an AD with 250K user information. It looks something like (from which I took code inspiration): https://gist.github.com/mattfahrner/c228ead9c516fc322d3a To be resilient of server restarts and my app crashing, I am writing the cookie into a file and reading it from there in the next iteration. Everything works and even my script crashes, on restart it picks up the right cookie file from storage and starts from the correct offset (ignoring users that have already been fetched). The problem seems to be that, while the operation is going on if I restart the server, I get SERVER_DOWN exception, on which case I just re-setup the connection again after the server is back and continue on the loop. Unfortunately, in this case, using the older cookie gives me: UNAVAILABLE_CRITICAL_EXTENSION: {'info': '00000057: LdapErr: DSID-0C0907B8, comment: Error processing control, data 0, v2580', 'desc': 'Critical extension is unavailable'} .. and I cannot figure out the issue. The cookie way works brilliant if my app and the network IO behaves correctly. If my app crashes, I can restart and reuse the cookie with a new connection object. But if I restart the AD server itself, reconnecting and using the cookie does not work. Is it not possible? -- Ritesh From michael at stroeder.com Thu Jan 5 13:35:28 2017 From: michael at stroeder.com (=?UTF-8?Q?Michael_Str=c3=b6der?=) Date: Thu, 5 Jan 2017 19:35:28 +0100 Subject: [python-ldap] AttributeError: ReconnectLDAPObject has no attribute '_l' In-Reply-To: References: Message-ID: Florian Best wrote: > As there doesn't seem to be a bug reporting system, I will post to this > mailing list. Right! > I got the following traceback: > [..] > File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 136, in > __getattr__ > self.__class__.__name__,repr(name) > AttributeError: ReconnectLDAPObject has no attribute '_l' > > I have two assumtions about this, but I am not completely sure as this is > only a traceback reported by one of our customers and I couldn't properly > reproduce it. Maybe you have another explanation. Which python-ldap version is this? In recent python-ldap release the unbind_ext() method looks like this (long lines wrapped): def unbind_ext(self,serverctrls=None,clientctrls=None): """ [..] """ res = self._ldap_call(self._l.unbind_ext,RequestControlTuples(serverctrls),RequestControlTuples(clientctrls)) try: del self._l except AttributeError: pass return res Not sure when this try-except-block was added (probably by me) though. > The first, more probable reason is because unbind_s() is called twice: During > the reconnection SimpleLDAPObject.unbind_s() is called first in > _apply_method_s() and second in reconnect() while the server is still down. > The second call would cause that SimpleLDAPObject.unbind_s() calls a method > which uses self._l while reconnect() already removed self._l. There have been numerous fixes to ReconnectLDAPObject for various subtle issues. I vaguely remember that the one above was one of them. In short: Use recent release 2.4.28. Probably relevant for you: The Debian folks added 2.4.28 to Debian testing. > (The second thing I could imagine would be If two threads use the same object, > this can happen during the reconnection?) Should not happen because locks are used. But I'd recommend to use separate ReconnectLDAPObject instance per thread anyway. > I attached a patch which maybe prevent this, as _l is not removed anymore from > the namespace of that object. Which is definitely the wrong approach because LDAPObject._l is not usable at that time anymore. Ciao, Michael. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3829 bytes Desc: S/MIME Cryptographic Signature URL: From michael at stroeder.com Thu Jan 5 13:56:23 2017 From: michael at stroeder.com (=?UTF-8?Q?Michael_Str=c3=b6der?=) Date: Thu, 5 Jan 2017 19:56:23 +0100 Subject: [python-ldap] LDAP pagination cookie not working across AD server restart In-Reply-To: References: Message-ID: Ritesh Nadhani wrote: > I am writing a script that would poll an AD with 250K user > information. It looks something like (from which I took code > inspiration): > > https://gist.github.com/mattfahrner/c228ead9c516fc322d3a So you're using the Simple Paged Control (see RFC 2696): https://tools.ietf.org/html/rfc2696 > To be resilient of server restarts and my app crashing, I am writing > the cookie into a file and reading it from there in the next > iteration. Hmm... > Everything works and even my script crashes, on restart it picks up > the right cookie file from storage and starts from the correct offset > (ignoring users that have already been fetched). I wonder why this works in your case. In general I doubt that it's guaranteed to work. > The problem seems to be that, while the operation is going on if I > restart the server, I get SERVER_DOWN exception, on which case I just > re-setup the connection again after the server is back and continue on > the loop. Unfortunately, in this case, using the older cookie gives > me: > > UNAVAILABLE_CRITICAL_EXTENSION: {'info': '00000057: LdapErr: > DSID-0C0907B8, comment: Error processing control, data 0, v2580', > 'desc': 'Critical extension is unavailable'} > > .. > > and I cannot figure out the issue. The cookie way works brilliant if > my app and the network IO behaves correctly. If my app crashes, I can > restart and reuse the cookie with a new connection object. > > But if I restart the AD server itself, reconnecting and using the > cookie does not work. In general I would never expect this (both cases) to work since the server can throw away any context of your former stale LDAP connection. The first case may work with AD but likely does not work with other LDAP servers. AFAICS there is no text in RFC 2696 clarifying this. I wonder why AD returns UNAVAILABLE_CRITICAL_EXTENSION though. But hey, RFC 2696 uses pretty blurry text like "it SHOULD return the appropriate error". Ciao, Michael. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3829 bytes Desc: S/MIME Cryptographic Signature URL: From riteshn at gmail.com Thu Jan 5 14:07:35 2017 From: riteshn at gmail.com (Ritesh Nadhani) Date: Thu, 5 Jan 2017 11:07:35 -0800 Subject: [python-ldap] LDAP pagination cookie not working across AD server restart In-Reply-To: References: Message-ID: On Thu, Jan 5, 2017 at 10:56 AM, Michael Str?der wrote: > Ritesh Nadhani wrote: >> I am writing a script that would poll an AD with 250K user >> information. It looks something like (from which I took code >> inspiration): >> >> https://gist.github.com/mattfahrner/c228ead9c516fc322d3a > > So you're using the Simple Paged Control (see RFC 2696): > > https://tools.ietf.org/html/rfc2696 > Yes. >> To be resilient of server restarts and my app crashing, I am writing >> the cookie into a file and reading it from there in the next >> iteration. > > Hmm... > >> Everything works and even my script crashes, on restart it picks up >> the right cookie file from storage and starts from the correct offset >> (ignoring users that have already been fetched). > > I wonder why this works in your case. > In general I doubt that it's guaranteed to work. Well, I will be honest and this is not something that I found over the docs or anything. I just wrote the implementation and during my tests, it seemed to work so I just assumed. My bad. > >> The problem seems to be that, while the operation is going on if I >> restart the server, I get SERVER_DOWN exception, on which case I just >> re-setup the connection again after the server is back and continue on >> the loop. Unfortunately, in this case, using the older cookie gives >> me: >> >> UNAVAILABLE_CRITICAL_EXTENSION: {'info': '00000057: LdapErr: >> DSID-0C0907B8, comment: Error processing control, data 0, v2580', >> 'desc': 'Critical extension is unavailable'} >> >> .. >> >> and I cannot figure out the issue. The cookie way works brilliant if >> my app and the network IO behaves correctly. If my app crashes, I can >> restart and reuse the cookie with a new connection object. >> >> But if I restart the AD server itself, reconnecting and using the >> cookie does not work. > > In general I would never expect this (both cases) to work since the server can > throw away any context of your former stale LDAP connection. The first case may > work with AD but likely does not work with other LDAP servers. AFAICS there is > no text in RFC 2696 clarifying this. > > I wonder why AD returns UNAVAILABLE_CRITICAL_EXTENSION though. But hey, RFC 2696 > uses pretty blurry text like "it SHOULD return the appropriate error". > So I guess my question becomes: If i have to efficiently pull all user information from a directory service where the network IO can be a bit sketchy, whats the best to query when I want to ignore already fetched users information? As far as I can see, there is no way to specifiy ordering, offseting during the query option. Or I am just out of luck? > Ciao, Michael. > > -- Ritesh From stephen.butler at gmail.com Thu Jan 5 14:21:19 2017 From: stephen.butler at gmail.com (Stephen J. Butler) Date: Thu, 5 Jan 2017 13:21:19 -0600 Subject: [python-ldap] LDAP pagination cookie not working across AD server restart In-Reply-To: References: Message-ID: VLV lets you sort and specify offsets. I wouldn't expect the paged results cookie to survive reconnects or server restarts. The MS docs don't say what causes it to be invalidated, but they do specify that clients treat it as a blob and should restart queries where it fails: https://technet.microsoft.com/en-us/windows-server-docs/identity/ad-ds/manage/how-ldap-server-cookies-are-handled I don't see any guarantees mentioned for when the cookie will succeed. Which to me means your strategy should be a rare case (where retry is acceptable) and not a standard path. On Thu, Jan 5, 2017 at 1:07 PM, Ritesh Nadhani wrote: > On Thu, Jan 5, 2017 at 10:56 AM, Michael Str?der > wrote: > > Ritesh Nadhani wrote: > >> I am writing a script that would poll an AD with 250K user > >> information. It looks something like (from which I took code > >> inspiration): > >> > >> https://gist.github.com/mattfahrner/c228ead9c516fc322d3a > > > > So you're using the Simple Paged Control (see RFC 2696): > > > > https://tools.ietf.org/html/rfc2696 > > > > Yes. > > >> To be resilient of server restarts and my app crashing, I am writing > >> the cookie into a file and reading it from there in the next > >> iteration. > > > > Hmm... > > > >> Everything works and even my script crashes, on restart it picks up > >> the right cookie file from storage and starts from the correct offset > >> (ignoring users that have already been fetched). > > > > I wonder why this works in your case. > > In general I doubt that it's guaranteed to work. > > Well, I will be honest and this is not something that I found over the > docs or anything. I just wrote the implementation and during my tests, > it seemed to work so I just assumed. My bad. > > > > >> The problem seems to be that, while the operation is going on if I > >> restart the server, I get SERVER_DOWN exception, on which case I just > >> re-setup the connection again after the server is back and continue on > >> the loop. Unfortunately, in this case, using the older cookie gives > >> me: > >> > >> UNAVAILABLE_CRITICAL_EXTENSION: {'info': '00000057: LdapErr: > >> DSID-0C0907B8, comment: Error processing control, data 0, v2580', > >> 'desc': 'Critical extension is unavailable'} > >> > >> .. > >> > >> and I cannot figure out the issue. The cookie way works brilliant if > >> my app and the network IO behaves correctly. If my app crashes, I can > >> restart and reuse the cookie with a new connection object. > >> > >> But if I restart the AD server itself, reconnecting and using the > >> cookie does not work. > > > > In general I would never expect this (both cases) to work since the > server can > > throw away any context of your former stale LDAP connection. The first > case may > > work with AD but likely does not work with other LDAP servers. AFAICS > there is > > no text in RFC 2696 clarifying this. > > > > I wonder why AD returns UNAVAILABLE_CRITICAL_EXTENSION though. But hey, > RFC 2696 > > uses pretty blurry text like "it SHOULD return the appropriate error". > > > > So I guess my question becomes: > > If i have to efficiently pull all user information from a directory > service where the network IO can be a bit sketchy, whats the best to > query when I want to ignore already fetched users information? > > As far as I can see, there is no way to specifiy ordering, offseting > during the query option. > > Or I am just out of luck? > > > > Ciao, Michael. > > > > > > > > -- > Ritesh > _______________________________________________ > python-ldap mailing list > python-ldap at python.org > https://mail.python.org/mailman/listinfo/python-ldap > -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael at stroeder.com Thu Jan 5 16:27:05 2017 From: michael at stroeder.com (=?UTF-8?Q?Michael_Str=c3=b6der?=) Date: Thu, 5 Jan 2017 22:27:05 +0100 Subject: [python-ldap] LDAP pagination cookie not working across AD server restart In-Reply-To: References: Message-ID: <3024b6a8-c1ff-1bf1-cfa1-1c3b814614cf@stroeder.com> Ritesh Nadhani wrote: > On Thu, Jan 5, 2017 at 10:56 AM, Michael Str?der wrote: >> In general I would never expect this (both cases) to work since the server >> can throw away any context of your former stale LDAP connection. The first >> case may work with AD but likely does not work with other LDAP servers. >> AFAICS there is no text in RFC 2696 clarifying this. > > So I guess my question becomes: > > If i have to efficiently pull all user information from a directory > service where the network IO can be a bit sketchy, whats the best to > query when I want to ignore already fetched users information? Hmm, up to now you mentioned your client crashing or AD getting restarted. Unstable network link is a different problem field. > As far as I can see, there is no way to specifiy ordering, offseting > during the query option. If you want to sync many AD entries then you should better read about how to correctly use USN values to retrieve recently changed entries in a robust manner (store highest committed USN as sync state). Another alternative is to use the DirSync control. Ciao, Michael. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3829 bytes Desc: S/MIME Cryptographic Signature URL: From bbaetz at google.com Thu Jan 5 18:09:33 2017 From: bbaetz at google.com (Bradley Baetz) Date: Thu, 05 Jan 2017 23:09:33 +0000 Subject: [python-ldap] [PATCH] Fix check for an empty error message Message-ID: When trying to get the server's error message, check if the string is empty rather than checking the pointer against '\0'. (This is already NULL-checked in the previous line) Thanks, Bradley Index: Modules/errors.c ============================== ===================================== RCS file: /cvsroot/python-ldap/python-ldap/Modules/errors.c,v retrieving revision 1.25 diff -u -r1.25 errors.c --- Modules/errors.c 23 Jun 2015 09:45:09 -0000 1.25 +++ Modules/errors.c 5 Jan 2017 00:22:35 -0000 @@ -98,7 +98,7 @@ Py_XDECREF(str); } else if (ldap_get_option(l, LDAP_OPT_ERROR_STRING, &error) >= 0 && error != NULL) { - if (error != '\0') { + if (*error != '\0') { str = PyString_FromString(error); if (str) PyDict_SetItemString( info, "info", str ); -- Bradley Baetz | Site Reliability Manager | bbaetz at google.com | +61-416-080841 -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael at stroeder.com Fri Jan 6 10:11:29 2017 From: michael at stroeder.com (=?UTF-8?Q?Michael_Str=c3=b6der?=) Date: Fri, 6 Jan 2017 16:11:29 +0100 Subject: [python-ldap] [PATCH] Fix check for an empty error message In-Reply-To: References: Message-ID: <30694e8c-ec1c-a6cc-0be0-a5f310f5f8e7@stroeder.com> Bradley Baetz via python-ldap wrote: > When trying to get the server's error message, check if the string is empty > rather than checking the pointer against '\0'. (This is already NULL-checked > in the previous line) Thanks for your fix. Your patch got badly formatted in the HTML mail. But I've committed this to CVS HEAD (see below). Could you please elaborate a bit on how this affected your python-ldap usage. Such a note in file CHANGES might be helpful for others. Ciao, Michael. P.S.: Please subscribe to the low-traffic python-ldap mailing list. So I don't have to approve all your postings and you'll get all responses. Index: Modules/errors.c =================================================================== RCS file: /cvsroot/python-ldap/python-ldap/Modules/errors.c,v retrieving revision 1.25 diff -u -r1.25 errors.c --- Modules/errors.c 23 Jun 2015 09:45:09 -0000 1.25 +++ Modules/errors.c 6 Jan 2017 15:00:24 -0000 @@ -98,7 +98,7 @@ Py_XDECREF(str); } else if (ldap_get_option(l, LDAP_OPT_ERROR_STRING, &error) >= 0 && error != NULL) { - if (error != '\0') { + if (*error != '\0') { str = PyString_FromString(error); if (str) PyDict_SetItemString( info, "info", str ); -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3829 bytes Desc: S/MIME Cryptographic Signature URL: From bbaetz at google.com Sun Jan 8 19:12:47 2017 From: bbaetz at google.com (Bradley Baetz) Date: Mon, 09 Jan 2017 00:12:47 +0000 Subject: [python-ldap] [PATCH] Fix check for an empty error message In-Reply-To: <30694e8c-ec1c-a6cc-0be0-a5f310f5f8e7@stroeder.com> References: <30694e8c-ec1c-a6cc-0be0-a5f310f5f8e7@stroeder.com> Message-ID: On Sat, 7 Jan 2017 at 02:11 Michael Str?der wrote: > Bradley Baetz via python-ldap wrote: > > When trying to get the server's error message, check if the string is > empty > > rather than checking the pointer against '\0'. (This is already > NULL-checked > > in the previous line) > > Thanks for your fix. > > Your patch got badly formatted in the HTML mail. > But I've committed this to CVS HEAD (see below). > > Could you please elaborate a bit on how this affected your python-ldap > usage. > Such a note in file CHANGES might be helpful for others. > This was noticed via some compiler static analysis/warnings for common typos/errors, rather than a specific bug. Bradley > > Ciao, Michael. > > P.S.: Please subscribe to the low-traffic python-ldap mailing list. So I > don't > have to approve all your postings and you'll get all responses. > > Index: Modules/errors.c > =================================================================== > RCS file: /cvsroot/python-ldap/python-ldap/Modules/errors.c,v > retrieving revision 1.25 > diff -u -r1.25 errors.c > --- Modules/errors.c 23 Jun 2015 09:45:09 -0000 1.25 > +++ Modules/errors.c 6 Jan 2017 15:00:24 -0000 > @@ -98,7 +98,7 @@ > Py_XDECREF(str); > } else if (ldap_get_option(l, LDAP_OPT_ERROR_STRING, &error) >= 0 > && error != NULL) { > - if (error != '\0') { > + if (*error != '\0') { > str = PyString_FromString(error); > if (str) > PyDict_SetItemString( info, "info", str ); > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pviktori at redhat.com Mon Jan 9 11:17:43 2017 From: pviktori at redhat.com (Petr Viktorin) Date: Mon, 9 Jan 2017 17:17:43 +0100 Subject: [python-ldap] Tests run by hand Message-ID: <2613548c-6300-3205-77cd-aa5f2115741d@redhat.com> Hello, These tests: Tests/Lib/ldap/schema/test_tokenizer.py Tests/Lib/ldap/test_modlist.py Tests/Lib/test_ldapurl.py only signal failure by printing to stdout. This makes them cumbersome to run automatically. Is there a reason for this behavior? Would you accept a patch that converts them to unittest and adds them to runtests.sh? -- Petr Viktorin From michael at stroeder.com Mon Jan 9 11:32:39 2017 From: michael at stroeder.com (=?UTF-8?Q?Michael_Str=c3=b6der?=) Date: Mon, 9 Jan 2017 17:32:39 +0100 Subject: [python-ldap] Tests run by hand In-Reply-To: <2613548c-6300-3205-77cd-aa5f2115741d@redhat.com> References: <2613548c-6300-3205-77cd-aa5f2115741d@redhat.com> Message-ID: <422c7ca2-f4b9-a459-bebb-b541173068f4@stroeder.com> Petr Viktorin wrote: > These tests: > > Tests/Lib/ldap/schema/test_tokenizer.py > Tests/Lib/ldap/test_modlist.py > Tests/Lib/test_ldapurl.py > > only signal failure by printing to stdout. This makes them cumbersome to run > automatically. > Is there a reason for this behavior? Ancient historic artefacts before automated testing got normal. ;-) > Would you accept a patch that converts them to unittest and adds them to > runtests.sh? Yes. Ciao, Michael. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3829 bytes Desc: S/MIME Cryptographic Signature URL: From michael at stroeder.com Wed Jan 25 15:05:03 2017 From: michael at stroeder.com (=?UTF-8?Q?Michael_Str=c3=b6der?=) Date: Wed, 25 Jan 2017 21:05:03 +0100 Subject: [python-ldap] ANN: python-ldap 2.4.29 Message-ID: <960a8df2-912d-b817-aaf5-224f10b55cdf@stroeder.com> Find a new release of python-ldap: https://pypi.python.org/pypi/python-ldap/2.4.29 python-ldap provides an object-oriented API to access LDAP directory servers from Python programs. It mainly wraps the OpenLDAP 2.x libs for that purpose. Additionally it contains modules for other LDAP-related stuff (e.g. processing LDIF, LDAP URLs and LDAPv3 schema). Project's web site: http://www.python-ldap.org/ Ciao, Michael. ---------------------------------------------------------------- Released 2.4.29 2017-01-25 Changes since 2.4.28: Modules/ * Fixed checking for empty server error message (thanks to Bradley Baetz) * Fixed releasing GIL when calling ldap_start_tls_s() (thanks to Lars Munch) -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3829 bytes Desc: S/MIME Cryptographic Signature URL: From pviktori at redhat.com Wed Feb 1 10:46:24 2017 From: pviktori at redhat.com (Petr Viktorin) Date: Wed, 1 Feb 2017 16:46:24 +0100 Subject: [python-ldap] Tests run by hand In-Reply-To: <422c7ca2-f4b9-a459-bebb-b541173068f4@stroeder.com> References: <2613548c-6300-3205-77cd-aa5f2115741d@redhat.com> <422c7ca2-f4b9-a459-bebb-b541173068f4@stroeder.com> Message-ID: On 01/09/2017 05:32 PM, Michael Str?der wrote: > Petr Viktorin wrote: >> These tests: >> >> Tests/Lib/ldap/schema/test_tokenizer.py >> Tests/Lib/ldap/test_modlist.py >> Tests/Lib/test_ldapurl.py >> >> only signal failure by printing to stdout. This makes them cumbersome to run >> automatically. >> Is there a reason for this behavior? > > Ancient historic artefacts before automated testing got normal. ;-) > >> Would you accept a patch that converts them to unittest and adds them to >> runtests.sh? It's low priority, other projects are taking up a lot of my time, but I'm slowly working on this. I noticed that the last two test cases in Tests/Lib/ldap/schema/test_tokenizer.py (marked "for Oracle") fail. Unfortunately LDAP tokenization is beyond my experience, I can't tell what the behavior should be here. Can you shed some light on it? -- Petr Viktorin From pviktori at redhat.com Wed Feb 1 10:41:32 2017 From: pviktori at redhat.com (Petr Viktorin) Date: Wed, 1 Feb 2017 16:41:32 +0100 Subject: [python-ldap] Tests run by hand In-Reply-To: <422c7ca2-f4b9-a459-bebb-b541173068f4@stroeder.com> References: <2613548c-6300-3205-77cd-aa5f2115741d@redhat.com> <422c7ca2-f4b9-a459-bebb-b541173068f4@stroeder.com> Message-ID: <68acd679-9f39-631a-178b-d42d50fa98ae@redhat.com> On 01/09/2017 05:32 PM, Michael Str?der wrote: > Petr Viktorin wrote: >> These tests: >> >> Tests/Lib/ldap/schema/test_tokenizer.py >> Tests/Lib/ldap/test_modlist.py >> Tests/Lib/test_ldapurl.py >> >> only signal failure by printing to stdout. This makes them cumbersome to run >> automatically. >> Is there a reason for this behavior? > > Ancient historic artefacts before automated testing got normal. ;-) > >> Would you accept a patch that converts them to unittest and adds them to >> runtests.sh? It's low priority, other projects are taking up a lot of my time, but I'm slowly working on this. I noticed that the last two test cases in Tests/Lib/ldap/schema/test_tokenizer.py (marked "for Oracle") fail. Unfortunately LDAP tokenization is beyond my experience, I can't tell what the behavior should be here. Can you shed some light on it? -- Petr Viktorin From michael at stroeder.com Wed Feb 1 13:43:26 2017 From: michael at stroeder.com (=?UTF-8?Q?Michael_Str=c3=b6der?=) Date: Wed, 1 Feb 2017 19:43:26 +0100 Subject: [python-ldap] Tests run by hand In-Reply-To: References: <2613548c-6300-3205-77cd-aa5f2115741d@redhat.com> <422c7ca2-f4b9-a459-bebb-b541173068f4@stroeder.com> Message-ID: <33e398a8-d0ac-e18a-41d2-82e428ea475e@stroeder.com> Petr Viktorin wrote: > On 01/09/2017 05:32 PM, Michael Str?der wrote: >> Petr Viktorin wrote: >>> These tests: >>> >>> Tests/Lib/ldap/schema/test_tokenizer.py >>> Tests/Lib/ldap/test_modlist.py >>> Tests/Lib/test_ldapurl.py >>> >>> only signal failure by printing to stdout. This makes them cumbersome to run >>> automatically. >>> Is there a reason for this behavior? >> >> Ancient historic artefacts before automated testing got normal. ;-) >> >>> Would you accept a patch that converts them to unittest and adds them to >>> runtests.sh? > > It's low priority, other projects are taking up a lot of my time, but I'm slowly > working on this. Take your time. > I noticed that the last two test cases in Tests/Lib/ldap/schema/test_tokenizer.py > (marked "for Oracle") fail. > Unfortunately LDAP tokenization is beyond my experience, I can't tell what the behavior > should be here. Can you shed some light on it? Well, I tried to work around some interop issues with non-LDAPv3-compliant schema definitions in Oracle Internet Directory (OID) and therefore added these test-cases. This was ages ago and not successful anyway. I also don't have access to OID myself and therefore I can't tell whether that's still relevant today. So I'd recommend to simply ignore (comment) this test-cases. Ciao, Michael. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3829 bytes Desc: S/MIME Cryptographic Signature URL: From dave at krondo.com Wed Feb 1 23:22:26 2017 From: dave at krondo.com (Dave Peticolas) Date: Thu, 02 Feb 2017 04:22:26 +0000 Subject: [python-ldap] result_type of connection.result() Message-ID: Hi, I've noticed (using versions 2.4.19 and 2.4.29) that the result_type tuple returned by result() is sometimes RES_SEARCH_ENTRY when all=1 using the default from not passing in "all" despite the docstring which seems to say it should always be RES_SEARCH_RESULT. The scenario where this is happening is that multiple asynchronous searches are started and then the results are requested for each one by explicitly passing specific msgid values for each one, but with all=1. Note this doesn't always happen. It seems to be related to the order the results are returned from the server. Am I just reading the docstring incorrectly? thanks, dave -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael at stroeder.com Thu Feb 2 02:24:21 2017 From: michael at stroeder.com (=?UTF-8?Q?Michael_Str=c3=b6der?=) Date: Thu, 2 Feb 2017 08:24:21 +0100 Subject: [python-ldap] result_type of connection.result() In-Reply-To: References: Message-ID: Dave Peticolas wrote: > Hi, I've noticed (using versions 2.4.19 and 2.4.29) that the result_type tuple > returned by result() is sometimes RES_SEARCH_ENTRY when all=1 using the default from > not passing in "all" despite the docstring which seems to say it should always be > RES_SEARCH_RESULT. Hmm, I'm not 100 % sure about the SearchResultDone: https://tools.ietf.org/html/rfc4511#section-4.5.2 > The scenario where this is happening is that multiple asynchronous searches are started > and then the results are requested for each one by explicitly passing specific msgid > values for each one, but with all=1. > > Note this doesn't always happen. It seems to be related to the order the results are > returned from the server. > > Am I just reading the docstring incorrectly? Could you please provide some example code for testing this? It might also be relevant which LDAP server you're using. Ciao, Michael. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3829 bytes Desc: S/MIME Cryptographic Signature URL: From dave at krondo.com Thu Feb 2 09:52:46 2017 From: dave at krondo.com (Dave Peticolas) Date: Thu, 02 Feb 2017 14:52:46 +0000 Subject: [python-ldap] result_type of connection.result() In-Reply-To: References: Message-ID: On Wed, Feb 1, 2017 at 11:31 PM Michael Str?der wrote: > Dave Peticolas wrote: > > Hi, I've noticed (using versions 2.4.19 and 2.4.29) that the result_type > tuple > > returned by result() is sometimes RES_SEARCH_ENTRY when all=1 using the > default from > > not passing in "all" despite the docstring which seems to say it should > always be > > RES_SEARCH_RESULT. > > Hmm, I'm not 100 % sure about the SearchResultDone: > > https://tools.ietf.org/html/rfc4511#section-4.5.2 > > > The scenario where this is happening is that multiple asynchronous > searches are started > > and then the results are requested for each one by explicitly passing > specific msgid > > values for each one, but with all=1. > > > > Note this doesn't always happen. It seems to be related to the order the > results are > > returned from the server. > > > > Am I just reading the docstring incorrectly? > > Could you please provide some example code for testing this? > It might also be relevant which LDAP server you're using. > > Oh, right, it's OpenLDAP 2.4.31. I'll try to work up a more detailed example code but here is the gist: conn = msg_ids = [conn.search(*query_args) for query_args in queries] all_results = [] for msg_id in msg_ids: kind, results = conn.result(msg_id) if kind != RES_SEARCH_RESULT: results = [] all_results.extend(results) Sometimes "kind" will be RES_SEARCH_ENTRY and the results will be ignored. This is extracted from the django-auth-ldap library which is using code like the above to search for user records across multiple queries. https://bitbucket.org/psagers/django-auth-ldap/issues/65/ldapsearchunion-bug -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael at stroeder.com Thu Feb 2 10:26:24 2017 From: michael at stroeder.com (=?UTF-8?Q?Michael_Str=c3=b6der?=) Date: Thu, 2 Feb 2017 16:26:24 +0100 Subject: [python-ldap] result_type of connection.result() In-Reply-To: References: Message-ID: <02844d90-a7e2-6df9-492c-d5c62c3714db@stroeder.com> Dave Peticolas wrote: > Sometimes "kind" will be RES_SEARCH_ENTRY and the results will be ignored. What makes you think you can ignore results with type RES_SEARCH_ENTRY? Please see: https://tools.ietf.org/html/rfc4511#section-4.5.2 Ciao, Michael. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3829 bytes Desc: S/MIME Cryptographic Signature URL: From dave at krondo.com Thu Feb 2 11:01:05 2017 From: dave at krondo.com (Dave Peticolas) Date: Thu, 02 Feb 2017 16:01:05 +0000 Subject: [python-ldap] result_type of connection.result() In-Reply-To: <02844d90-a7e2-6df9-492c-d5c62c3714db@stroeder.com> References: <02844d90-a7e2-6df9-492c-d5c62c3714db@stroeder.com> Message-ID: On Thu, Feb 2, 2017 at 7:26 AM Michael Str?der wrote: > Dave Peticolas wrote: > > Sometimes "kind" will be RES_SEARCH_ENTRY and the results will be > ignored. > > What makes you think you can ignore results with type RES_SEARCH_ENTRY? > > Please see: https://tools.ietf.org/html/rfc4511#section-4.5.2 I see that, yes. And just to be clear, it's not me that thinks you can ignore those results, it's the django-auth-ldap library :) I'm just trying to figure out why things are going wrong. But there's also this bit of documentation in python-ldap for the result() method: *For all set to 0, result tuples trickle in (with the same message id), and with the result types RES_SEARCH_ENTRY and RES_SEARCH_REFERENCE, until the final result which has a result type of RES_SEARCH_RESULT and a (usually) empty data field. When all is set to 1, only one result is returned, with a result type of RES_SEARCH_RESULT, and all the result tuples listed in the data field.* Please see: https://www.python-ldap.org/doc/html/ldap.html#ldap.LDAPObject.result To me, that documentation suggests that when all=1 you will only ever see a result type of RES_SEARCH_RESULT, but that's not what I am observing. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave at krondo.com Fri Feb 3 11:48:34 2017 From: dave at krondo.com (Dave Peticolas) Date: Fri, 03 Feb 2017 16:48:34 +0000 Subject: [python-ldap] result_type of connection.result() In-Reply-To: References: <02844d90-a7e2-6df9-492c-d5c62c3714db@stroeder.com> Message-ID: On Thu, Feb 2, 2017 at 8:01 AM Dave Peticolas wrote: > On Thu, Feb 2, 2017 at 7:26 AM Michael Str?der > wrote: > > Dave Peticolas wrote: > > Sometimes "kind" will be RES_SEARCH_ENTRY and the results will be > ignored. > > What makes you think you can ignore results with type RES_SEARCH_ENTRY? > > Please see: https://tools.ietf.org/html/rfc4511#section-4.5.2 > > > I see that, yes. And just to be clear, it's not me that thinks you can > ignore those results, it's the django-auth-ldap library :) I'm just trying > to figure out why things are going wrong. But there's also this bit of > documentation in python-ldap for the result() method: > > *For all set to 0, result tuples trickle in (with the same message id), > and with the result types RES_SEARCH_ENTRY and RES_SEARCH_REFERENCE, until > the final result which has a result type of RES_SEARCH_RESULT and a > (usually) empty data field. When all is set to 1, only one result is > returned, with a result type of RES_SEARCH_RESULT, and all the result > tuples listed in the data field.* > > Please see: > https://www.python-ldap.org/doc/html/ldap.html#ldap.LDAPObject.result > > To me, that documentation suggests that when all=1 you will only ever see > a result type of RES_SEARCH_RESULT, but that's not what I am observing. > Note the OpenLDAP docs for the underlying C function are a lot less specific about the result type you can expect when all=1. https://www.openldap.org/software/man.cgi?query=ldap_result&sektion=3&apropos=0&manpath=OpenLDAP+2.4-Release Is it possible that the python-ldap docs are simply incorrect here? -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael at stroeder.com Fri Feb 3 11:54:05 2017 From: michael at stroeder.com (=?UTF-8?Q?Michael_Str=c3=b6der?=) Date: Fri, 3 Feb 2017 17:54:05 +0100 Subject: [python-ldap] result_type of connection.result() In-Reply-To: References: <02844d90-a7e2-6df9-492c-d5c62c3714db@stroeder.com> Message-ID: Dave Peticolas wrote: > Note the OpenLDAP docs for the underlying C function are a lot less specific about the > result type you can expect when all=1. > > https://www.openldap.org/software/man.cgi?query=ldap_result&sektion=3&apropos=0& > manpath=OpenLDAP+2.4-Release > > Is it possible that the python-ldap docs are simply incorrect here? Yes, I'm inclined to simply hunk out this sentence for all=1 (IIRC not written by me). Ciao, Michael. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3829 bytes Desc: S/MIME Cryptographic Signature URL: From dave at krondo.com Fri Feb 3 12:10:23 2017 From: dave at krondo.com (Dave Peticolas) Date: Fri, 03 Feb 2017 17:10:23 +0000 Subject: [python-ldap] result_type of connection.result() In-Reply-To: References: <02844d90-a7e2-6df9-492c-d5c62c3714db@stroeder.com> Message-ID: On Fri, Feb 3, 2017 at 8:54 AM Michael Str?der wrote: > Dave Peticolas wrote: > > Note the OpenLDAP docs for the underlying C function are a lot less > specific about the > > result type you can expect when all=1. > > > > > https://www.openldap.org/software/man.cgi?query=ldap_result&sektion=3&apropos=0& > > manpath=OpenLDAP+2.4-Release > > > > Is it possible that the python-ldap docs are simply incorrect here? > > Yes, I'm inclined to simply hunk out this sentence for all=1 (IIRC not > written by me). > > I think that might be the right course. My C is pretty rusty, but looking at the result.c code it looks like when all=1 the "result type" is the result type of the first message in a linked list and the order of those results is pretty sensitive to the order they are received in and the mix of message IDs seen up till then. https://github.com/openldap/openldap/blob/master/libraries/libldap/result.c -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael at stroeder.com Wed Feb 8 04:56:27 2017 From: michael at stroeder.com (=?UTF-8?Q?Michael_Str=c3=b6der?=) Date: Wed, 8 Feb 2017 10:56:27 +0100 Subject: [python-ldap] ANN: python-ldap 2.4.30 Message-ID: Find a new release of python-ldap: https://pypi.python.org/pypi/python-ldap/2.4.30 python-ldap provides an object-oriented API to access LDAP directory servers from Python programs. It mainly wraps the OpenLDAP 2.x libs for that purpose. Additionally it contains modules for other LDAP-related stuff (e.g. processing LDIF, LDAP URLs and LDAPv3 schema). Project's web site: http://www.python-ldap.org/ Ciao, Michael. ---------------------------------------------------------------- Released 2.4.30 2017-02-08 Changes since 2.4.29: Lib/ * compability fix in ldap.controls.deref to be compatible to recent pyasn1 0.2.x (thanks to Ilya Etingof) -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3829 bytes Desc: S/MIME Cryptographic Signature URL: From pviktori at redhat.com Tue Feb 14 05:48:34 2017 From: pviktori at redhat.com (Petr Viktorin) Date: Tue, 14 Feb 2017 11:48:34 +0100 Subject: [python-ldap] Tests run by hand In-Reply-To: <33e398a8-d0ac-e18a-41d2-82e428ea475e@stroeder.com> References: <2613548c-6300-3205-77cd-aa5f2115741d@redhat.com> <422c7ca2-f4b9-a459-bebb-b541173068f4@stroeder.com> <33e398a8-d0ac-e18a-41d2-82e428ea475e@stroeder.com> Message-ID: <636e1e25-afb1-e1dc-d566-3e5b4c63bc08@redhat.com> On 02/01/2017 07:43 PM, Michael Str?der wrote: > Petr Viktorin wrote: >> On 01/09/2017 05:32 PM, Michael Str?der wrote: >>> Petr Viktorin wrote: >>>> These tests: >>>> >>>> Tests/Lib/ldap/schema/test_tokenizer.py >>>> Tests/Lib/ldap/test_modlist.py >>>> Tests/Lib/test_ldapurl.py >>>> >>>> only signal failure by printing to stdout. This makes them cumbersome to run >>>> automatically. >>>> Is there a reason for this behavior? >>> >>> Ancient historic artefacts before automated testing got normal. ;-) >>> >>>> Would you accept a patch that converts them to unittest and adds them to >>>> runtests.sh? >> >> It's low priority, other projects are taking up a lot of my time, but I'm slowly >> working on this. > > Take your time. Here is a patch. If you have some other idea on organization (like moving the files to t_*.py with the other unittest-based ones) , let me know. >> I noticed that the last two test cases in Tests/Lib/ldap/schema/test_tokenizer.py >> (marked "for Oracle") fail. >> Unfortunately LDAP tokenization is beyond my experience, I can't tell what the behavior >> should be here. Can you shed some light on it? > > Well, I tried to work around some interop issues with non-LDAPv3-compliant schema > definitions in Oracle Internet Directory (OID) and therefore added these test-cases. > > This was ages ago and not successful anyway. I also don't have access to OID myself and > therefore I can't tell whether that's still relevant today. So I'd recommend to simply > ignore (comment) this test-cases. Thanks; commented. -- Petr Viktorin -------------- next part -------------- A non-text attachment was scrubbed... Name: use-unittest.patch Type: text/x-patch Size: 22262 bytes Desc: not available URL: From michael at stroeder.com Tue Feb 14 08:32:42 2017 From: michael at stroeder.com (=?UTF-8?Q?Michael_Str=c3=b6der?=) Date: Tue, 14 Feb 2017 14:32:42 +0100 Subject: [python-ldap] Tests run by hand In-Reply-To: <636e1e25-afb1-e1dc-d566-3e5b4c63bc08@redhat.com> References: <2613548c-6300-3205-77cd-aa5f2115741d@redhat.com> <422c7ca2-f4b9-a459-bebb-b541173068f4@stroeder.com> <33e398a8-d0ac-e18a-41d2-82e428ea475e@stroeder.com> <636e1e25-afb1-e1dc-d566-3e5b4c63bc08@redhat.com> Message-ID: <78e51237-dfc0-b7d8-ee73-760b7a054541@stroeder.com> Petr Viktorin wrote: > Here is a patch. Thanks! > If you have some other idea on organization (like moving the files to t_*.py with the > other unittest-based ones) , let me know. Tests/ is a historic huge mess, especially my local working copy. ;-] I've reorganized everything so all test scripts which should be automatically run match Tests/t_*.py Please ignore everything else when invoking tests during package builds. Ouch! We already had Tests/t_ldapurl.py. => I've merged the test cases from Tests/Lib/test_ldapurl.py script. This works now: for I in Tests/t_*.py ; do python $I ; done Please provide a patch for whatever Ciao, Michael. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3829 bytes Desc: S/MIME Cryptographic Signature URL: From michael at stroeder.com Tue Feb 14 17:05:55 2017 From: michael at stroeder.com (=?UTF-8?Q?Michael_Str=c3=b6der?=) Date: Tue, 14 Feb 2017 23:05:55 +0100 Subject: [python-ldap] ANN: python-ldap 2.4.32 Message-ID: Find a new release of python-ldap: https://pypi.python.org/pypi/python-ldap/2.4.32 python-ldap provides an object-oriented API to access LDAP directory servers from Python programs. It mainly wraps the OpenLDAP 2.x libs for that purpose. Additionally it contains modules for other LDAP-related stuff (e.g. processing LDIF, LDAP URLs and LDAPv3 schema). Project's web site: https://www.python-ldap.org/ Ciao, Michael. ---------------------------------------------------------------- Released 2.4.32 2017-02-14 Running tests made easier: - python setup.py test - added tox.ini ---------------------------------------------------------------- Released 2.4.31 2017-02-14 Changes since 2.4.30: Tests/ * new test scripts t_ldap_schema_tokenizer.py and t_ldap_modlist.py on former raw scripts (thanks to Petr Viktorin) * new test-cases in t_ldapurl.py based on former raw scripts (thanks to Petr Viktorin) * new test-cases in t_ldap_dn.py * moved a script to Demo/ -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3829 bytes Desc: S/MIME Cryptographic Signature URL: From joeyhendricks20 at gmail.com Wed Feb 15 06:19:04 2017 From: joeyhendricks20 at gmail.com (Joey Hendricks) Date: Wed, 15 Feb 2017 12:19:04 +0100 Subject: [python-ldap] Password history / password policy In-Reply-To: References: Message-ID: Hi guys, I,m busy with the Python-Ldap module and i,m running into a bit of trouble with my company's password policy. We dont want a user to be able to reset his password to a password he has used before we have set our password policy the following way: Enforce password history : 24 passwords remembered Maximum password age : 42 days Minimum password age : 1 days Minimum password length : 8 characters Password must meet complexity requirements : Disabled i,m using the following Pyhton code to change the password server = LDAP_IP ldap.set_option(ldap.OPT_X_TLS_REQUIRE_CERT, ldap.OPT_X_TLS_NEVER) conn = ldap.initialize(server) conn.set_option(ldap.OPT_REFERRALS, 0) conn.set_option(ldap.OPT_PROTOCOL_VERSION, 3) conn.set_option(ldap.OPT_X_TLS, ldap.OPT_X_TLS_DEMAND) conn.set_option(ldap.OPT_X_TLS_DEMAND, True) conn.set_option(ldap.OPT_DEBUG_LEVEL, 4095) conn.simple_bind_s(base64.b64decode(BIND_DN), base64.b64decode(BIND_PASS)) password_value1 = '"{0}\"'.format(pwd).encode("utf-16-le") add_pass = [(ldap.MOD_REPLACE, "UnicodePwd", password_value1)] conn.modify_s(CN_NAME, add_pass) conn.unbind_s()" Or is there a way that the ad wont change the password if the password has been used before. So that I completely obey my password policy. I hope someone can help me Kind regards Joey Hendricks -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael at stroeder.com Wed Feb 15 11:04:56 2017 From: michael at stroeder.com (=?UTF-8?Q?Michael_Str=c3=b6der?=) Date: Wed, 15 Feb 2017 17:04:56 +0100 Subject: [python-ldap] Password history / password policy In-Reply-To: References: Message-ID: <6b763e1c-8a0d-e3e6-f2e9-52c88c6dd533@stroeder.com> Joey, basically your LDAP server (MS AD?) is responsible for enforcing all password policy rules including password history check. Hence it's not really a python-ldap question. So I'd recommend to simply test it. Ciao, Michael. P.S.: Please subscribe to the low-traffic mailing list so I don't have to accept your postings manually and to avoid missing a response. Joey Hendricks wrote: > Hi guys, > > I,m busy with the Python-Ldap module and i,m running into a bit of trouble > with my company's password policy. > We dont want a user to be able to reset his password to a password he has > used before we have set our password policy the following way: > > Enforce password history : 24 passwords > remembered > Maximum password age : 42 days > Minimum password age : 1 days > Minimum password length : 8 characters > Password must meet complexity requirements : Disabled > > > i,m using the following Pyhton code to change the password > > server = LDAP_IP > ldap.set_option(ldap.OPT_X_TLS_REQUIRE_CERT, > ldap.OPT_X_TLS_NEVER) > conn = ldap.initialize(server) > conn.set_option(ldap.OPT_REFERRALS, 0) > conn.set_option(ldap.OPT_PROTOCOL_VERSION, 3) > conn.set_option(ldap.OPT_X_TLS, ldap.OPT_X_TLS_DEMAND) > conn.set_option(ldap.OPT_X_TLS_DEMAND, True) > conn.set_option(ldap.OPT_DEBUG_LEVEL, 4095) > conn.simple_bind_s(base64.b64decode(BIND_DN), > base64.b64decode(BIND_PASS)) > password_value1 = '"{0}\"'.format(pwd).encode("utf-16-le") > add_pass = [(ldap.MOD_REPLACE, "UnicodePwd", password_value1)] > conn.modify_s(CN_NAME, add_pass) > conn.unbind_s()" > > Or is there a way that the ad wont change the password if the password has > been used before. > > So that I completely obey my password policy. > > I hope someone can help me > > Kind regards > > Joey Hendricks -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3829 bytes Desc: S/MIME Cryptographic Signature URL: From christian at python.org Sat Feb 18 04:33:02 2017 From: christian at python.org (Christian Heimes) Date: Sat, 18 Feb 2017 10:33:02 +0100 Subject: [python-ldap] Performance improvement for schema.tokenizer.split_tokens Message-ID: Hi, I have been running into performance issues with split_tokens from the schema parser. The first request to a new WSGI process spends about 25 to 30% in split_tokens() while parsing LDAP schema. Consecutive requests benefit from a schema cache. I was able to come up with a new implementation of split_tokens() which is about 8 times faster on Python 2. The new implementation uses a regular expression to split the schema string into tokens. It is successfully able to parse over 3,000 schema lines from 389-DS and FreeIPA with same result as the curent split_tokens() function. Personally I find it easier to read and understand, too. Please review my implementation and consider it for python-ldap. Implementation with tests: https://github.com/tiran/fast_split_tokens Background information: https://github.com/pyldap/pyldap/issues/85 https://fedorahosted.org/freeipa/ticket/6679 Regards, Christian From michael at stroeder.com Sat Feb 18 05:08:16 2017 From: michael at stroeder.com (=?UTF-8?Q?Michael_Str=c3=b6der?=) Date: Sat, 18 Feb 2017 11:08:16 +0100 Subject: [python-ldap] Performance improvement for schema.tokenizer.split_tokens In-Reply-To: References: Message-ID: <4c5c8f4a-72bf-f08e-499c-b0aaa19fd532@stroeder.com> Could you please also test with Tests/t_ldap_schema_tokenizer.py in recent python-ldap 2.4.32? Maybe you already did. Ciao, Michael. Christian Heimes wrote: > I have been running into performance issues with split_tokens from the > schema parser. The first request to a new WSGI process spends about 25 > to 30% in split_tokens() while parsing LDAP schema. Consecutive requests > benefit from a schema cache. > > I was able to come up with a new implementation of split_tokens() which > is about 8 times faster on Python 2. The new implementation uses a > regular expression to split the schema string into tokens. It is > successfully able to parse over 3,000 schema lines from 389-DS and > FreeIPA with same result as the curent split_tokens() function. > Personally I find it easier to read and understand, too. > > Please review my implementation and consider it for python-ldap. > > Implementation with tests: > https://github.com/tiran/fast_split_tokens > > Background information: > https://github.com/pyldap/pyldap/issues/85 > https://fedorahosted.org/freeipa/ticket/6679 > > Regards, > Christian -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3829 bytes Desc: S/MIME Cryptographic Signature URL: From christian at python.org Sat Feb 18 05:43:20 2017 From: christian at python.org (Christian Heimes) Date: Sat, 18 Feb 2017 11:43:20 +0100 Subject: [python-ldap] Performance improvement for schema.tokenizer.split_tokens In-Reply-To: <4c5c8f4a-72bf-f08e-499c-b0aaa19fd532@stroeder.com> References: <4c5c8f4a-72bf-f08e-499c-b0aaa19fd532@stroeder.com> Message-ID: <79e208b3-170b-35b6-37b9-8546fd75eae2@python.org> On 2017-02-18 11:08, Michael Str?der wrote: > Could you please also test with Tests/t_ldap_schema_tokenizer.py in recent python-ldap > 2.4.32? Maybe you already did. No, I didn't. I haven't noticed that you added tests for the tokenizer 3 days ago. Thanks for the hint. Your test cases revealed two issue in my implementation: empty quoted string and () without surrounding white space. I addressed both edge cases and fixed my regular expression. Christian From michael at stroeder.com Sat Feb 18 09:51:25 2017 From: michael at stroeder.com (=?UTF-8?Q?Michael_Str=c3=b6der?=) Date: Sat, 18 Feb 2017 15:51:25 +0100 Subject: [python-ldap] Performance improvement for schema.tokenizer.split_tokens In-Reply-To: References: Message-ID: Christian, Christian Heimes wrote: > I was able to come up with a new implementation of split_tokens() which > is about 8 times faster on Python 2. thanks for your code submission. [1] http://www.openldap.org/devel/contributing.html#notice > The new implementation uses a > regular expression to split the schema string into tokens. It is > successfully able to parse over 3,000 schema lines from 389-DS and > FreeIPA with same result as the curent split_tokens() function. > Personally I find it easier to read and understand, too. Note that 389-DS schema is somewhat broken (re-use of same OIDs) but for testing this particular function it works. I've also successfully tested your implementation with my own monster schema collection for OpenLDAP. > Please review my implementation and consider it for python-ldap. I've prepared all to commit your improvement (slightly modified) but I have to insist on an IPR notice for your submission (I've been to careless in the past). Please have a look at [1] for some sample text and replace "OpenLDAP Software" with "python-ldap module". Ciao, Michael. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3829 bytes Desc: S/MIME Cryptographic Signature URL: From christian at python.org Sat Feb 18 10:08:12 2017 From: christian at python.org (Christian Heimes) Date: Sat, 18 Feb 2017 16:08:12 +0100 Subject: [python-ldap] Performance improvement for schema.tokenizer.split_tokens In-Reply-To: References: Message-ID: <4221c5db-1bc0-09cc-dc9a-3e0666faa49c@python.org> On 2017-02-18 15:51, Michael Str?der wrote: > Christian, > > Christian Heimes wrote: >> I was able to come up with a new implementation of split_tokens() which >> is about 8 times faster on Python 2. > > thanks for your code submission. > > [1] http://www.openldap.org/devel/contributing.html#notice > >> The new implementation uses a >> regular expression to split the schema string into tokens. It is >> successfully able to parse over 3,000 schema lines from 389-DS and >> FreeIPA with same result as the curent split_tokens() function. >> Personally I find it easier to read and understand, too. > > Note that 389-DS schema is somewhat broken (re-use of same OIDs) but for testing this > particular function it works. I've also successfully tested your implementation with my > own monster schema collection for OpenLDAP. > >> Please review my implementation and consider it for python-ldap. > > I've prepared all to commit your improvement (slightly modified) but I have to insist on > an IPR notice for your submission (I've been to careless in the past). Please have a look > at [1] for some sample text and replace "OpenLDAP Software" with "python-ldap module". Hi Michael, I see no problem with the IPR notice. I added a note to the doc string of fast_split_tokens.py. In the mean time a user complained that my tokenizer silently ignores invalid schema lines. The last version does some rudimentary checks and refuses additional residue. It also handles one of the two special Oracle cases from you test file. Regards, Christian From michael at stroeder.com Sat Feb 18 10:26:19 2017 From: michael at stroeder.com (=?UTF-8?Q?Michael_Str=c3=b6der?=) Date: Sat, 18 Feb 2017 16:26:19 +0100 Subject: [python-ldap] Performance improvement for schema.tokenizer.split_tokens In-Reply-To: <4221c5db-1bc0-09cc-dc9a-3e0666faa49c@python.org> References: <4221c5db-1bc0-09cc-dc9a-3e0666faa49c@python.org> Message-ID: Christian Heimes wrote: > I see no problem with the IPR notice. I added a note to the doc string > of fast_split_tokens.py. Please repeat it here for the mailing list archive because I don't know what will happen to your github repo in the future (python-ldap is much older than github). Even if the mailing list archive on python.org is abandoned I have a local copy. > In the mean time a user complained that my tokenizer silently ignores > invalid schema lines. Please provide additional test-cases here to be added to Tests/t_ldap_schema_tokenizer.py before. > The last version does some rudimentary checks and > refuses additional residue. It also handles one of the two special > Oracle cases from you test file. I see you're coming close to my own (non-published) attempts of working around some schema bugs. But is it still that much faster then? ;-) For the record the ABNF is here: https://tools.ietf.org/html/rfc4512#section-4.1 I'm inclined to trade speed over these ancient and seriously broken Oracle test-cases (see also https://tools.ietf.org/html/draft-thomson-postel-was-wrong). And while you're at it: TESTCASES_ESCAPED_QUOTES = ( ("BLUBBER DI 'BLU\'BB ER' DA 'BLAH' ", ["BLUBBER", "DI", "BLU'BB ER", "DA", "BLAH"]), ) Ciao, Michael. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3829 bytes Desc: S/MIME Cryptographic Signature URL: From michael at stroeder.com Sat Feb 18 12:39:32 2017 From: michael at stroeder.com (=?UTF-8?Q?Michael_Str=c3=b6der?=) Date: Sat, 18 Feb 2017 18:39:32 +0100 Subject: [python-ldap] Performance improvement for schema.tokenizer.split_tokens In-Reply-To: References: <4221c5db-1bc0-09cc-dc9a-3e0666faa49c@python.org> Message-ID: Michael Str?der wrote: > I see you're coming close to my own (non-published) attempts of working around some > schema bugs. But is it still that much faster then? ;-) Overall schema parsing done in ldap.schema.SubSchema.__init__() is twice as fast with your recent implementation. That's great! Ciao, Michael. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3829 bytes Desc: S/MIME Cryptographic Signature URL: From christian at python.org Mon Feb 20 04:59:38 2017 From: christian at python.org (Christian Heimes) Date: Mon, 20 Feb 2017 10:59:38 +0100 Subject: [python-ldap] Performance improvement for schema.tokenizer.split_tokens In-Reply-To: References: <4221c5db-1bc0-09cc-dc9a-3e0666faa49c@python.org> Message-ID: On 2017-02-18 18:39, Michael Str?der wrote: > Michael Str?der wrote: >> I see you're coming close to my own (non-published) attempts of working around some >> schema bugs. But is it still that much faster then? ;-) > > Overall schema parsing done in ldap.schema.SubSchema.__init__() is twice as fast with > your recent implementation. That's great! > > Ciao, Michael. Hi, here is a patch against latest CVS head including some additional test cases. I changed the regular expression to handle backslash quoting of single quotes within single quotes. A co-worker also noticed that my implementation did not check for closing parenthesis before opening parenthesis. I added a check for the special case, too (thx Standa). ----- The attached patch file is derived from python-ldap module. All of the modifications to python-ldap module represented in the following patch(es) were developed by Christian Heimes . I have not assigned rights and/or interest in this work to any party. ----- Regards, Christian -------------- next part -------------- A non-text attachment was scrubbed... Name: fast_split_tokens.patch Type: text/x-patch Size: 6558 bytes Desc: not available URL: From pviktori at redhat.com Mon Feb 20 08:39:05 2017 From: pviktori at redhat.com (Petr Viktorin) Date: Mon, 20 Feb 2017 14:39:05 +0100 Subject: [python-ldap] Test for ldap.schema.subentry.urlfetch using file:// Message-ID: Hello, Martin recently wrote a test for ldap.schema.subentry.urlfetch (downloading & parsing a schema file). I've changed it to use unittest and I'm sending it here for consideration. -- Petr Viktorin -------------- next part -------------- A non-text attachment was scrubbed... Name: t_ldap_schema_urlfetch.py Type: text/x-python Size: 4765 bytes Desc: not available URL: From michael at stroeder.com Wed Feb 22 16:24:19 2017 From: michael at stroeder.com (=?UTF-8?Q?Michael_Str=c3=b6der?=) Date: Wed, 22 Feb 2017 22:24:19 +0100 Subject: [python-ldap] Performance improvement for schema.tokenizer.split_tokens In-Reply-To: References: <4221c5db-1bc0-09cc-dc9a-3e0666faa49c@python.org> Message-ID: <41d14a00-1b80-dd5f-bc34-6fb946d3d41a@stroeder.com> Christian Heimes wrote: > On 2017-02-18 18:39, Michael Str?der wrote: >> Michael Str?der wrote: >>> I see you're coming close to my own (non-published) attempts of working around some >>> schema bugs. But is it still that much faster then? ;-) >> >> Overall schema parsing done in ldap.schema.SubSchema.__init__() is twice as fast with >> your recent implementation. That's great! > > here is a patch against latest CVS head including some additional test > cases. I've slightly modified your patch for backward compability to older Python versions. > I changed the regular expression to handle backslash quoting of > single quotes within single quotes. A co-worker also noticed that my > implementation did not check for closing parenthesis before opening > parenthesis. I added a check for the special case, too (thx Standa). Thanks. This revealed an error in my ?-DIR suppl. schema for web2ldap. Maybe I did something wrong but one bad case does not raise ValueError as expected: $ PYTHONPATH=Lib python Tests/t_ldap_schema_tokenizer.py .Fxx. ====================================================================== FAIL: test_broken (__main__.TestSplitTokens) ---------------------------------------------------------------------- Traceback (most recent call last): File "Tests/t_ldap_schema_tokenizer.py", line 115, in test_broken self._run_failure_tests(TESTCASES_BROKEN) File "Tests/t_ldap_schema_tokenizer.py", line 84, in _run_failure_tests should_have_failed, AssertionError: 1 value(s) should have raised ValueError: ['( BLUB )) DA ('] ---------------------------------------------------------------------- Ran 5 tests in 0.001s FAILED (failures=1, expected failures=2) Ciao, Michael. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3829 bytes Desc: S/MIME Cryptographic Signature URL: From pviktori at redhat.com Tue Feb 28 09:58:45 2017 From: pviktori at redhat.com (Petr Viktorin) Date: Tue, 28 Feb 2017 15:58:45 +0100 Subject: [python-ldap] Performance improvement for schema.tokenizer.split_tokens In-Reply-To: <41d14a00-1b80-dd5f-bc34-6fb946d3d41a@stroeder.com> References: <4221c5db-1bc0-09cc-dc9a-3e0666faa49c@python.org> <41d14a00-1b80-dd5f-bc34-6fb946d3d41a@stroeder.com> Message-ID: <65671e69-9a16-b11e-d243-4ff17ece3763@redhat.com> On 02/22/2017 10:24 PM, Michael Str?der wrote: > Christian Heimes wrote: >> On 2017-02-18 18:39, Michael Str?der wrote: >>> Michael Str?der wrote: >>>> I see you're coming close to my own (non-published) attempts of working around some >>>> schema bugs. But is it still that much faster then? ;-) >>> >>> Overall schema parsing done in ldap.schema.SubSchema.__init__() is twice as fast with >>> your recent implementation. That's great! >> >> here is a patch against latest CVS head including some additional test >> cases. > > I've slightly modified your patch for backward compability to older Python versions. > >> I changed the regular expression to handle backslash quoting of >> single quotes within single quotes. A co-worker also noticed that my >> implementation did not check for closing parenthesis before opening >> parenthesis. I added a check for the special case, too (thx Standa). > > Thanks. This revealed an error in my ?-DIR suppl. schema for web2ldap. > > Maybe I did something wrong but one bad case does not raise ValueError as expected: > > $ PYTHONPATH=Lib python Tests/t_ldap_schema_tokenizer.py > .Fxx. > ====================================================================== > FAIL: test_broken (__main__.TestSplitTokens) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "Tests/t_ldap_schema_tokenizer.py", line 115, in test_broken > self._run_failure_tests(TESTCASES_BROKEN) > File "Tests/t_ldap_schema_tokenizer.py", line 84, in _run_failure_tests > should_have_failed, > AssertionError: 1 value(s) should have raised ValueError: ['( BLUB )) DA ('] > > ---------------------------------------------------------------------- > Ran 5 tests in 0.001s > > FAILED (failures=1, expected failures=2) > > Ciao, Michael. It's actually two cases, apply the attached patch to see the other one. -- Petr Viktorin -------------- next part -------------- A non-text attachment was scrubbed... Name: testfix.patch Type: text/x-patch Size: 570 bytes Desc: not available URL: From michael at stroeder.com Tue Feb 28 15:39:41 2017 From: michael at stroeder.com (=?UTF-8?Q?Michael_Str=c3=b6der?=) Date: Tue, 28 Feb 2017 21:39:41 +0100 Subject: [python-ldap] Performance improvement for schema.tokenizer.split_tokens In-Reply-To: <65671e69-9a16-b11e-d243-4ff17ece3763@redhat.com> References: <4221c5db-1bc0-09cc-dc9a-3e0666faa49c@python.org> <41d14a00-1b80-dd5f-bc34-6fb946d3d41a@stroeder.com> <65671e69-9a16-b11e-d243-4ff17ece3763@redhat.com> Message-ID: Petr Viktorin wrote: > On 02/22/2017 10:24 PM, Michael Str?der wrote: >> Maybe I did something wrong but one bad case does not raise ValueError as expected: > > It's actually two cases, apply the attached patch to see the other one. Argh! Yes, Committed. Ciao, Michael. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3829 bytes Desc: S/MIME Cryptographic Signature URL: From moritz.sichert at googlemail.com Thu Mar 23 12:27:45 2017 From: moritz.sichert at googlemail.com (Moritz Sichert) Date: Thu, 23 Mar 2017 17:27:45 +0100 Subject: [python-ldap] Patch: include errnum in LDAPException Message-ID: <48d26745-37be-540d-6c1f-26d5bc8a867a@googlemail.com> Hi, the attached patch adds the errnum value to the LDAPException. This is useful when you get an error that python-ldap doesn't know about, because right now you will only get the error message string. Best regards, Moritz -------------- next part -------------- A non-text attachment was scrubbed... Name: errnum-in-exception.patch Type: text/x-patch Size: 667 bytes Desc: not available URL: