From pspacek at redhat.com Mon Jan 5 10:09:06 2015 From: pspacek at redhat.com (Petr Spacek) Date: Mon, 05 Jan 2015 10:09:06 +0100 Subject: [python-ldap] Is it possible to bind using a kerberos keytab In-Reply-To: <387DA264871BE742972A79C3684DB29B35C40D84@LONS00110044.mercury.intra> References: <387DA264871BE742972A79C3684DB29B35C4055B@LONS00110044.mercury.intra> <549BF64D.6040500@stroeder.com> <387DA264871BE742972A79C3684DB29B35C40D84@LONS00110044.mercury.intra> Message-ID: <54AA54B2.9010800@redhat.com> On 30.12.2014 13:43, Kev SMITH wrote: > ldap.LOCAL_ERROR: {'info': 'SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server not found in Kerberos database)', 'desc': 'Local error'} "Server not found in Kerberos database" usually indicates that: - client is for some reason requesting wrong server name - DNS configuration is wrong and DNS name canonization in Kerberos libs leads to wrong server name even though client originally requested correct name. I would recommend you to run tcpdump/wireshark and compare 'Server name' in requests sent from ldapsearch and Python LDAP client. -- Petr Spacek @ Red Hat From michael at stroeder.com Sat Jan 10 18:21:02 2015 From: michael at stroeder.com (=?UTF-8?Q?Michael_Str=c3=b6der?=) Date: Sat, 10 Jan 2015 18:21:02 +0100 Subject: [python-ldap] ANN: python-ldap 2.4.19 Message-ID: <54B15F7E.5010205@stroeder.com> Find a new release of python-ldap: http://pypi.python.org/pypi/python-ldap/2.4.19 python-ldap provides an object-oriented API to access LDAP directory servers from Python programs. It mainly wraps the OpenLDAP 2.x libs for that purpose. Additionally it contains modules for other LDAP-related stuff (e.g. processing LDIF, LDAP URLs and LDAPv3 schema). Project's web site: http://www.python-ldap.org/ Ciao, Michael. ---------------------------------------------------------------- Released 2.4.19 2015-01-10 Changes since 2.4.18: Lib/ * Fixed missing ReconnectLDAPObject._reconnect_lock when pickling (see SF#64, thanks to Dan O'Reilly) * Added ldap.controls.pagedresults which is pure Python implementation of Simple Paged Results Control (see RFC 2696) and delivers the correct result size From mark at proseconsulting.co.uk Wed Jan 21 00:16:24 2015 From: mark at proseconsulting.co.uk (Mark R Bannister) Date: Tue, 20 Jan 2015 23:16:24 +0000 Subject: [python-ldap] ldap.OPT_DESC, async ops and paged search controls Message-ID: <54BEE1C8.70806@proseconsulting.co.uk> Hi, I've been using the new ldap.OPT_DESC feature introduced in python-ldap 2.4.17 and have a question concerning the use of it with asynchronous search operations and paged search controls. I have a daemon (http://dbis.sf.net) that launches async searches with paged search controls (page size 1000), and then uses the ldap.OPT_DESC feature to get the file descriptor back. It then adds the fd to a select() call, with the intention to be woken up when there is something to do. When it wakes up it calls ldap.result(). So far this makes sense, right? Now, the behaviour I'm seeing is not necessarily what I would have expected, so I want to check with others to see if this is correct or not. I have an example search operation that returns about 80,000 entries. With a page size of 1000, would you not expect to go round the loop and call select() 80 times? Well that's not what I'm getting. It's looping round and calling select() 80,000 times, once per entry. Each time select() wakes me up, I call ldap.result() which for the first 999 iterations returns (None, None). Then, the 1,000th time, it returns 1,000 entries. I then switch to the next page and go round the loop again, and the same thing happens again. Thinking that there was something wrong with my code, I ran strace to confirm that when I was calling ldap.result() and got (None, None) returned, there was some data transferred over the file descriptor. Does this seem right to you and is there anyway to optimise this? All 80,000 entries are taking about 15 seconds to read into Python using the python-ldap module compared with 5 seconds for native C. Thanks, Mark. From rich.megginson at gmail.com Wed Jan 21 00:36:04 2015 From: rich.megginson at gmail.com (Rich Megginson) Date: Tue, 20 Jan 2015 16:36:04 -0700 Subject: [python-ldap] ldap.OPT_DESC, async ops and paged search controls In-Reply-To: <54BEE1C8.70806@proseconsulting.co.uk> References: <54BEE1C8.70806@proseconsulting.co.uk> Message-ID: <54BEE664.3090708@gmail.com> On 01/20/2015 04:16 PM, Mark R Bannister wrote: > Hi, > > I've been using the new ldap.OPT_DESC feature introduced in > python-ldap 2.4.17 and have a question concerning the use of it with > asynchronous search operations and paged search controls. > > I have a daemon (http://dbis.sf.net) that launches async searches with > paged search controls (page size 1000), and then uses the > ldap.OPT_DESC feature to get the file descriptor back. It then adds > the fd to a select() call, with the intention to be woken up when > there is something to do. When it wakes up it calls ldap.result(). > > So far this makes sense, right? > > Now, the behaviour I'm seeing is not necessarily what I would have > expected, so I want to check with others to see if this is correct or > not. I have an example search operation that returns about 80,000 > entries. With a page size of 1000, would you not expect to go round > the loop and call select() 80 times? Well that's not what I'm > getting. It's looping round and calling select() 80,000 times, once > per entry. Each time select() wakes me up, I call ldap.result() which > for the first 999 iterations returns (None, None). Then, the 1,000th > time, it returns 1,000 entries. I then switch to the next page and go > round the loop again, and the same thing happens again. > > Thinking that there was something wrong with my code, I ran strace to > confirm that when I was calling ldap.result() and got (None, None) > returned, there was some data transferred over the file descriptor. > > Does this seem right to you and is there anyway to optimise this? All > 80,000 entries are taking about 15 seconds to read into Python using > the python-ldap module compared with 5 seconds for native C. What LDAP server are you using? It may be something specific to your server. > > Thanks, > Mark. > > _______________________________________________ > python-ldap mailing list > python-ldap at python.org > https://mail.python.org/mailman/listinfo/python-ldap From michael at stroeder.com Wed Jan 21 10:16:54 2015 From: michael at stroeder.com (=?UTF-8?Q?Michael_Str=c3=b6der?=) Date: Wed, 21 Jan 2015 10:16:54 +0100 Subject: [python-ldap] ldap.OPT_DESC, async ops and paged search controls In-Reply-To: <54BEE1C8.70806@proseconsulting.co.uk> References: <54BEE1C8.70806@proseconsulting.co.uk> Message-ID: <54BF6E86.1020104@stroeder.com> Mark R Bannister wrote: > I've been using the new ldap.OPT_DESC feature introduced in python-ldap 2.4.17 > and have a question concerning the use of it with asynchronous search > operations and paged search controls. Never used that myself. Why are you using paging? This only makes sense if you want to retrieve more than 1000 entries from MS AD. > Does this seem right to you and is there anyway to optimise this? All 80,000 > entries are taking about 15 seconds to read into Python using the python-ldap > module compared with 5 seconds for native C. For better comparison of the numbers could you please also test Python code without using the ldap.OPT_DESC feature: 1. using LDAPObject.search_ext_s() 2. using ldap.resiter 3. 1. and 2. with and without paging I'd also try to see what wakes up the select() by using wireshark. Ciao, Michael. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4252 bytes Desc: S/MIME Cryptographic Signature URL: From mark at proseconsulting.co.uk Thu Jan 22 14:33:16 2015 From: mark at proseconsulting.co.uk (Mark R Bannister) Date: Thu, 22 Jan 2015 13:33:16 +0000 Subject: [python-ldap] ldap.OPT_DESC, async ops and paged search controls In-Reply-To: <54BF6E86.1020104@stroeder.com> References: <54BEE1C8.70806@proseconsulting.co.uk> <54BF6E86.1020104@stroeder.com> Message-ID: <54C0FC1C.201@proseconsulting.co.uk> On 21/01/2015 09:16, Michael Str?der wrote: > Mark R Bannister wrote: >> I've been using the new ldap.OPT_DESC feature introduced in python-ldap 2.4.17 >> and have a question concerning the use of it with asynchronous search >> operations and paged search controls. > Never used that myself. > > Why are you using paging? This only makes sense if you want to retrieve more > than 1000 entries from MS AD. I've tested this now without paged search controls and get the same problem. The directory server is OpenLDAP 2.4.30 on Solaris 11.2 (it's listening on localhost). I've attached two test scripts that examine this problem. The output of testldap1.py run on a test DN that has 7 child entries: ldap.bind complete, fd 4 ldap.search started calling select with ([4], [], []) , returned: ([4], [], []) ldap.result returned: (97, []) calling select with ([4], [], []) , returned: ([4], [], []) ldap.result returned: (None, None) calling select with ([4], [], []) , returned: ([4], [], []) ldap.result returned: (None, None) calling select with ([4], [], []) , returned: ([4], [], []) ldap.result returned: (None, None) calling select with ([4], [], []) , returned: ([4], [], []) ldap.result returned: (None, None) calling select with ([4], [], []) , returned: ([4], [], []) ldap.result returned: (None, None) calling select with ([4], [], []) , returned: ([4], [], []) ldap.result returned: (None, None) calling select with ([4], [], []) , returned: ([4], [], []) ldap.result returned: (None, None) calling select with ([4], [], []) , returned: ([4], [], []) ldap.result returned: (101, [ ... results here ... ]) So why is select being woken up once per entry? Why isn't select being woken up once with the entire result set? This is a simple case, and for 7 entries one could argue it doesn't matter, but when I'm dealing with 80,000 entries, that's a lot of unnecessary wake-ups. >> Does this seem right to you and is there anyway to optimise this? All 80,000 >> entries are taking about 15 seconds to read into Python using the python-ldap >> module compared with 5 seconds for native C. > For better comparison of the numbers could you please also test Python code > without using the ldap.OPT_DESC feature: > > 1. using LDAPObject.search_ext_s() > > 2. using ldap.resiter > > 3. 1. and 2. with and without paging > > I'd also try to see what wakes up the select() by using wireshark. > > Ciao, Michael. > > I can't use wireshark for this one because it's localhost. However, truss works fine for me and demonstrates that each time ldap.result() is called it reads one entry from the fd, then returns [None, None]. I have to call ldap.result() as many times as there are entries before I actually get any meaningful results. This doesn't seem right to me. If I run the second attached test script (testldap2.py) and run truss on that, I see the same behaviour under the covers - a poll() and a separate read() for each individual entry. I haven't tried ldap.resiter, but I'm sure it's the same as it uses LDAPObject.result3(). Perhaps this is some tuning I've missed from OpenLDAP, which seems to want to drip-feed one result at a time ... or is this normal? It seems quite inefficient to me. Thanks, Mark. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- #!/usr/bin/python # LDAP async search for all entries directly underneath given base import sys import ldap from select import select base = sys.argv[1] filter = "objectclass=*" scope = ldap.SCOPE_ONELEVEL l = ldap.initialize("ldap://localhost") l.bind("", "") fd = l.get_option(ldap.OPT_DESC) print "ldap.bind complete, fd %d" % fd l.search(base, scope, filter) print "ldap.search started" d = None while d is None or len(d) == 0: print "calling select with ([%d], [], [])" % fd, sys.stdout.flush() r, w, e = select([fd], [], []) print ", returned: %s" % str((r, w, e)) (t, d) = l.result(timeout=0) print "ldap.result returned: %s" % str((t, d)) l.unbind() -------------- next part -------------- #!/usr/bin/python # LDAP async search for all entries directly underneath given base import sys import ldap base = sys.argv[1] filter = "objectclass=*" scope = ldap.SCOPE_ONELEVEL l = ldap.initialize("ldap://localhost") l.bind_s("", "") print "ldap.bind complete" result = l.search_s(base, scope, filter) print "ldap.search returned: %s" % str(result) l.unbind() From michael at stroeder.com Thu Jan 22 15:15:04 2015 From: michael at stroeder.com (=?UTF-8?Q?Michael_Str=c3=b6der?=) Date: Thu, 22 Jan 2015 15:15:04 +0100 Subject: [python-ldap] ldap.OPT_DESC, async ops and paged search controls In-Reply-To: <54C0FC1C.201@proseconsulting.co.uk> References: <54BEE1C8.70806@proseconsulting.co.uk> <54BF6E86.1020104@stroeder.com> <54C0FC1C.201@proseconsulting.co.uk> Message-ID: <54C105E8.2030504@stroeder.com> Mark R Bannister wrote: > I can't use wireshark for this one because it's localhost. Usually I grab the trace with tcpdump in a raw pcap file and view it with wireshark on my local workstation. Could you do that? I'd be interested to see whether there's something like keepalive etc. which wakes up select(). > However, truss > works fine for me and demonstrates that each time ldap.result() is called it > reads one entry from the fd, then returns [None, None]. But can truss also output what's transmitted on the network interface? Could you also try to access slapd over ldapi:// ? > If I run the second attached test script (testldap2.py) and run truss on that, > I see the same behaviour under the covers - a poll() and a separate read() for > each individual entry. I haven't tried ldap.resiter, but I'm sure it's the > same as it uses LDAPObject.result3(). But ldap.resiter internally blocks. > Perhaps this is some tuning I've missed from OpenLDAP, which seems to want to > drip-feed one result at a time ... or is this normal? It seems quite > inefficient to me. Let me do some tests on my local system and then we see whether we should take this over to openldap-technical mailing list. Ciao, Michael. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4252 bytes Desc: S/MIME Cryptographic Signature URL: From michael at stroeder.com Thu Jan 22 16:16:35 2015 From: michael at stroeder.com (=?UTF-8?Q?Michael_Str=c3=b6der?=) Date: Thu, 22 Jan 2015 16:16:35 +0100 Subject: [python-ldap] ldap.OPT_DESC, async ops and paged search controls In-Reply-To: <54C0FC1C.201@proseconsulting.co.uk> References: <54BEE1C8.70806@proseconsulting.co.uk> <54BF6E86.1020104@stroeder.com> <54C0FC1C.201@proseconsulting.co.uk> Message-ID: <54C11453.10000@stroeder.com> Mark R Bannister wrote: > On 21/01/2015 09:16, Michael Str?der wrote: >> Mark R Bannister wrote: >>> I've been using the new ldap.OPT_DESC feature introduced in python-ldap 2.4.17 >>> and have a question concerning the use of it with asynchronous search >>> operations and paged search controls. >> Never used that myself. >> >> Why are you using paging? This only makes sense if you want to retrieve more >> than 1000 entries from MS AD. > > I've tested this now without paged search controls and get the same problem. Ok, my local tests show the same results (openSUSE Linux 13.2, OpenLDAP RE24 from git, python-ldap 2.4.19). And ldap.resiter is not really usable for msgid=ldap.RES_ANY. I vaguely remember having seen this many years ago. Because at that time I had rewritten LDAPObject.result() to internally poll for results which made the process a CPU hog. :-/ So yes, you should take this to openldap-technical mailing list. For there you should ideally present C code depmonstrating the issue. ;-) Ciao, Michael. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4252 bytes Desc: S/MIME Cryptographic Signature URL: From mark at proseconsulting.co.uk Tue Jan 27 13:38:04 2015 From: mark at proseconsulting.co.uk (Mark R Bannister) Date: Tue, 27 Jan 2015 12:38:04 +0000 Subject: [python-ldap] ldap.OPT_DESC, async ops and paged search controls In-Reply-To: <54C11453.10000@stroeder.com> References: <54BEE1C8.70806@proseconsulting.co.uk> <54BF6E86.1020104@stroeder.com> <54C0FC1C.201@proseconsulting.co.uk> <54C11453.10000@stroeder.com> Message-ID: <54C786AC.1030407@proseconsulting.co.uk> On 22/01/2015 15:16, Michael Str?der wrote: > Mark R Bannister wrote: >> On 21/01/2015 09:16, Michael Str?der wrote: >>> Mark R Bannister wrote: >>>> I've been using the new ldap.OPT_DESC feature introduced in python-ldap 2.4.17 >>>> and have a question concerning the use of it with asynchronous search >>>> operations and paged search controls. >>> Never used that myself. >>> >>> Why are you using paging? This only makes sense if you want to retrieve more >>> than 1000 entries from MS AD. >> I've tested this now without paged search controls and get the same problem. > Ok, my local tests show the same results (openSUSE Linux 13.2, OpenLDAP RE24 > from git, python-ldap 2.4.19). And ldap.resiter is not really usable for > msgid=ldap.RES_ANY. > > I vaguely remember having seen this many years ago. Because at that time I had > rewritten LDAPObject.result() to internally poll for results which made the > process a CPU hog. :-/ > > So yes, you should take this to openldap-technical mailing list. > For there you should ideally present C code depmonstrating the issue. ;-) > > Ciao, Michael. > I had time to play with this some more, and it emerged that if I did this: result = ldap.result(timeout=0.0001) (i.e. instead out timeout=0), it behaves much more the way I expected. The number of select() calls drops from over 80,000 to about 3,000 and testing different values this seemed to be the most optimal setting (on the hardware I was testing on anyway). So this is definitely one for the openldap-technical mailing list. It seems that timeout=0 implies that I want one result at a time, when in fact I need a way of requesting all the data currently available in kernel buffers but without waiting around for new network packets to arrive. Best regards, Mark. -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael at stroeder.com Tue Jan 27 16:39:33 2015 From: michael at stroeder.com (=?UTF-8?Q?Michael_Str=c3=b6der?=) Date: Tue, 27 Jan 2015 16:39:33 +0100 Subject: [python-ldap] ldap.OPT_DESC, async ops and paged search controls In-Reply-To: <54C786AC.1030407@proseconsulting.co.uk> References: <54BEE1C8.70806@proseconsulting.co.uk> <54BF6E86.1020104@stroeder.com> <54C0FC1C.201@proseconsulting.co.uk> <54C11453.10000@stroeder.com> <54C786AC.1030407@proseconsulting.co.uk> Message-ID: <54C7B135.8060403@stroeder.com> Mark R Bannister wrote: > I had time to play with this some more, Me too. > and it emerged that if I did this: > > result = ldap.result(timeout=0.0001) > > (i.e. instead out timeout=0), it behaves much more the way I expected. The > number of select() calls drops from over 80,000 to about 3,000 and testing > different values this seemed to be the most optimal setting (on the hardware I > was testing on anyway). How many results? On my system timeout=0.00001 is only 10% worse either with ~1200 LDAP results always returned in one chunk and calling result() ~800 times. I also tried to use select.epoll(). But I'm not sure about the exact flags. But it this seems worse and is not available on all platforms. Overall both solutions are not ideal but do not perform so much worse than synchronous methods. > So this is definitely one for the openldap-technical mailing list. Yes, please post there. Note that it would be probably a solution to have one global blocking result() collecting thread which dispatches the results to the outstanding request queues. It would be nice to have this encapsulated in a variant of the LDAPObject class. Ciao, Michael. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4252 bytes Desc: S/MIME Cryptographic Signature URL: From michael at stroeder.com Tue Jan 27 21:52:28 2015 From: michael at stroeder.com (=?UTF-8?Q?Michael_Str=c3=b6der?=) Date: Tue, 27 Jan 2015 21:52:28 +0100 Subject: [python-ldap] ldap.OPT_DESC, async ops and paged search controls In-Reply-To: <54C7F3FD.3000902@proseconsulting.co.uk> References: <54BEE1C8.70806@proseconsulting.co.uk> <54BF6E86.1020104@stroeder.com> <54C0FC1C.201@proseconsulting.co.uk> <54C11453.10000@stroeder.com> <54C786AC.1030407@proseconsulting.co.uk> <54C7B135.8060403@stroeder.com> <54C7F3FD.3000902@proseconsulting.co.uk> Message-ID: <54C7FA8C.1090101@stroeder.com> Mark R Bannister wrote: > The original design limits the number of threads in an attempt to be more > scalable. There is a fixed number of workers that can each be responsible for > a larger number of LDAP connections. This is by design. If I launched a new > blocking thread for each LDAP connection, it would be easy to overload the > system with too many threads by sending in many different requests > simultaneously. I don't know what you're after. I thought you have many incoming requests and you want to handle them completely asynchronously placing incoming requests into a queue. You could send many LDAP requests to the server without blocking and dispatch the incoming responses to the original requests in the queue. Ciao, Michael. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4252 bytes Desc: S/MIME Cryptographic Signature URL: From mark at proseconsulting.co.uk Tue Jan 27 21:24:29 2015 From: mark at proseconsulting.co.uk (Mark R Bannister) Date: Tue, 27 Jan 2015 20:24:29 +0000 Subject: [python-ldap] ldap.OPT_DESC, async ops and paged search controls In-Reply-To: <54C7B135.8060403@stroeder.com> References: <54BEE1C8.70806@proseconsulting.co.uk> <54BF6E86.1020104@stroeder.com> <54C0FC1C.201@proseconsulting.co.uk> <54C11453.10000@stroeder.com> <54C786AC.1030407@proseconsulting.co.uk> <54C7B135.8060403@stroeder.com> Message-ID: <54C7F3FD.3000902@proseconsulting.co.uk> On 27/01/2015 15:39, Michael Str?der wrote: > Mark R Bannister wrote: >> I had time to play with this some more, > Me too. > >> and it emerged that if I did this: >> >> result = ldap.result(timeout=0.0001) >> >> (i.e. instead out timeout=0), it behaves much more the way I expected. The >> number of select() calls drops from over 80,000 to about 3,000 and testing >> different values this seemed to be the most optimal setting (on the hardware I >> was testing on anyway). > How many results? 80,000. > Yes, please post there. Note that it would be probably a solution to > have one global blocking result() collecting thread which dispatches > the results to the outstanding request queues. It would be nice to > have this encapsulated in a variant of the LDAPObject class. Ciao, > Michael. The original design limits the number of threads in an attempt to be more scalable. There is a fixed number of workers that can each be responsible for a larger number of LDAP connections. This is by design. If I launched a new blocking thread for each LDAP connection, it would be easy to overload the system with too many threads by sending in many different requests simultaneously. Best regards, Mark. From mark at proseconsulting.co.uk Tue Jan 27 22:13:24 2015 From: mark at proseconsulting.co.uk (Mark R Bannister) Date: Tue, 27 Jan 2015 21:13:24 +0000 Subject: [python-ldap] ldap.OPT_DESC, async ops and paged search controls In-Reply-To: <54C7FA8C.1090101@stroeder.com> References: <54BEE1C8.70806@proseconsulting.co.uk> <54BF6E86.1020104@stroeder.com> <54C0FC1C.201@proseconsulting.co.uk> <54C11453.10000@stroeder.com> <54C786AC.1030407@proseconsulting.co.uk> <54C7B135.8060403@stroeder.com> <54C7F3FD.3000902@proseconsulting.co.uk> <54C7FA8C.1090101@stroeder.com> Message-ID: <54C7FF74.10008@proseconsulting.co.uk> On 27/01/2015 20:52, Michael Str?der wrote: > Mark R Bannister wrote: >> The original design limits the number of threads in an attempt to be more >> scalable. There is a fixed number of workers that can each be responsible for >> a larger number of LDAP connections. This is by design. If I launched a new >> blocking thread for each LDAP connection, it would be easy to overload the >> system with too many threads by sending in many different requests >> simultaneously. > I don't know what you're after. > > I thought you have many incoming requests and you want to handle them > completely asynchronously placing incoming requests into a queue. You could > send many LDAP requests to the server without blocking and dispatch the > incoming responses to the original requests in the queue. > > Ciao, Michael. > Indeed, which is more-or-less how's it's done. But you spoke of having a blocking thread reading results back from the directory server, and I don't see how that could be an improvement. Here is a rough diagram of how it currently works.One half of the jigsaw puzzle is implemented by my Pyloom library (https://sourceforge.net/p/dbis/code/ci/default/tree/src/pyloom/__init__.py.in). One listen socket, some marshal threads that pick up incoming requests and dispatch to a separate group of worker threads: [listen socket] ____________|____________ | | | [marshal] [marshal] [marshal] ____|____ ____|____ ____|____ | | | | | | | | | [w] [w] [w] [w] [w] [w] [w] [w] [w] The worker thread itself is implemented in the DBIS Server class (https://sourceforge.net/p/dbis/code/ci/default/tree/src/dbis/server.py) and has a queue of work, each which may have its own LDAP object (or a reference to another session that is currently looking up the information it needs): [w] +--> [session 1] ---> [LDAP object] | +--> [session 2] ---> [LDAP object] | +--> [session 3] ---> [waiting on another session] | ... etc. As you can see, I may have many LDAP objects in different states, and it's all asynchronous, but importantly I have a fixed number of threads. The only thing that is dynamic is the size of a worker's queue, i.e. the number of sessions a worker thread is actively managing may go up or down, to an uppper limit, but the total number of running threads stays the same. Each worker relies on a select() call to wake it up, and there may be a number of things that wakes a worker up (new incoming request from marshal thread, new incoming data from running LDAP search operation, data now available in watched session, server shutdown message). Now what I don't follow is your suggestion of using a blocking thread to read results back from an LDAP search operation. For that to work, I would need a much greater number of threads running, one per LDAP object. I don't see how that would improve anything, I can only see it making the server perform more poorly when under heavy load. But maybe I'm missing something here ... Best regards, Mark. From michael at stroeder.com Tue Jan 27 22:26:18 2015 From: michael at stroeder.com (=?UTF-8?Q?Michael_Str=c3=b6der?=) Date: Tue, 27 Jan 2015 22:26:18 +0100 Subject: [python-ldap] ldap.OPT_DESC, async ops and paged search controls In-Reply-To: <54C7FF74.10008@proseconsulting.co.uk> References: <54BEE1C8.70806@proseconsulting.co.uk> <54BF6E86.1020104@stroeder.com> <54C0FC1C.201@proseconsulting.co.uk> <54C11453.10000@stroeder.com> <54C786AC.1030407@proseconsulting.co.uk> <54C7B135.8060403@stroeder.com> <54C7F3FD.3000902@proseconsulting.co.uk> <54C7FA8C.1090101@stroeder.com> <54C7FF74.10008@proseconsulting.co.uk> Message-ID: <54C8027A.9090505@stroeder.com> Mark R Bannister wrote: > As you can see, I may have many LDAP objects in different states, There is no internal state of a LDAPObject instance. And I wonder why you open a separate LDAP connection for each worker thread. I don't think you need that since you can send several LDAP requests on one connection and dispatch the incoming results my msg_id to the queued request (see result type of LDAPObject.result2() and higher). 1. Just queue the incoming requests, 2. send the accompanying LDAP request on the single connection, 3. attach msg_id tp queued request and 4. have *one* result thread repeatedly calling LDAPObject.result2() and 5. finally dispatch the results by msg_id to the queued incoming request. With this approach you wouldn't even need so many worker threads and you're not overloading the server with so many connections. It's not trivial of course because you also have to take care of the result type to determine wheter all results for a single msg_id were received. But that's what you have to take care of anyway when calling LDAPObject.result(msgid=RES_ANY) like in your examples. Also decent caching is mandatory. YMMV though. Ciao, Michael. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4252 bytes Desc: S/MIME Cryptographic Signature URL: From mark at proseconsulting.co.uk Tue Jan 27 22:43:09 2015 From: mark at proseconsulting.co.uk (Mark R Bannister) Date: Tue, 27 Jan 2015 21:43:09 +0000 Subject: [python-ldap] ldap.OPT_DESC, async ops and paged search controls In-Reply-To: <54C8027A.9090505@stroeder.com> References: <54BEE1C8.70806@proseconsulting.co.uk> <54BF6E86.1020104@stroeder.com> <54C0FC1C.201@proseconsulting.co.uk> <54C11453.10000@stroeder.com> <54C786AC.1030407@proseconsulting.co.uk> <54C7B135.8060403@stroeder.com> <54C7F3FD.3000902@proseconsulting.co.uk> <54C7FA8C.1090101@stroeder.com> <54C7FF74.10008@proseconsulting.co.uk> <54C8027A.9090505@stroeder.com> Message-ID: <54C8066D.70507@proseconsulting.co.uk> On 27/01/2015 21:26, Michael Str?der wrote: > Mark R Bannister wrote: >> As you can see, I may have many LDAP objects in different states, > There is no internal state of a LDAPObject instance. And I wonder why you open > a separate LDAP connection for each worker thread. I don't think you need that > since you can send several LDAP requests on one connection and dispatch the > incoming results my msg_id to the queued request (see result type of > LDAPObject.result2() and higher). > > 1. Just queue the incoming requests, > 2. send the accompanying LDAP request on the single connection, > 3. attach msg_id tp queued request and > 4. have *one* result thread repeatedly calling LDAPObject.result2() and > 5. finally dispatch the results by msg_id to the queued incoming request. > > With this approach you wouldn't even need so many worker threads and you're > not overloading the server with so many connections. > > It's not trivial of course because you also have to take care of the result > type to determine wheter all results for a single msg_id were received. But > that's what you have to take care of anyway when calling > LDAPObject.result(msgid=RES_ANY) like in your examples. > > Also decent caching is mandatory. > > YMMV though. > > Ciao, Michael. > So if I have just one thread reading results and dispatching results, I have a bottleneck don't I? If that thread is busy reading a large result set for one request, and there is also a large amount of data waiting for another ten separate requests, there will be a delay in processing the others. Whereas if I have ten separate workers reading their own data, I am spreading the load and improving the performance. It is a more scalable solution with performance that is directly proportional to the number of requests, not exponential. That said, there may be some mileage in considering having a single LDAP object per worker, which behaves much as you describe, but the scope is limited to that worker's queue. Best regards, Mark. From michael at stroeder.com Tue Jan 27 22:56:48 2015 From: michael at stroeder.com (=?UTF-8?Q?Michael_Str=c3=b6der?=) Date: Tue, 27 Jan 2015 22:56:48 +0100 Subject: [python-ldap] ldap.OPT_DESC, async ops and paged search controls In-Reply-To: <54C8066D.70507@proseconsulting.co.uk> References: <54BEE1C8.70806@proseconsulting.co.uk> <54BF6E86.1020104@stroeder.com> <54C0FC1C.201@proseconsulting.co.uk> <54C11453.10000@stroeder.com> <54C786AC.1030407@proseconsulting.co.uk> <54C7B135.8060403@stroeder.com> <54C7F3FD.3000902@proseconsulting.co.uk> <54C7FA8C.1090101@stroeder.com> <54C7FF74.10008@proseconsulting.co.uk> <54C8027A.9090505@stroeder.com> <54C8066D.70507@proseconsulting.co.uk> Message-ID: <54C809A0.4010401@stroeder.com> Mark R Bannister wrote: > So if I have just one thread reading results and dispatching results, I have a > bottleneck don't I? If that thread is busy reading a large result set for one > request, and there is also a large amount of data waiting for another ten > separate requests, there will be a delay in processing the others. Whereas if > I have ten separate workers reading their own data, I am spreading the load > and improving the performance. It is a more scalable solution with > performance that is directly proportional to the number of requests, not > exponential. After all your LDAP server will be also limited, especially when you really want to enumerate all objects in your maps. If enumeration is really crucial in your environment, then consider maintaining a local maps cache database. With this you can query the LDAP server with syncrepl or (modifyTimestamp=>) to keep LDAP result sets small during normal update operation. sssd is doing a lot like this. But it's complex and error-prone. Personally I don't think that enumeration is really needed. BTDT, but YMMV since the OS platforms out there are quite different. Ciao, Michael. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4252 bytes Desc: S/MIME Cryptographic Signature URL: From mark at proseconsulting.co.uk Tue Jan 27 23:18:52 2015 From: mark at proseconsulting.co.uk (Mark R Bannister) Date: Tue, 27 Jan 2015 22:18:52 +0000 Subject: [python-ldap] ldap.OPT_DESC, async ops and paged search controls In-Reply-To: <54C809A0.4010401@stroeder.com> References: <54BEE1C8.70806@proseconsulting.co.uk> <54BF6E86.1020104@stroeder.com> <54C0FC1C.201@proseconsulting.co.uk> <54C11453.10000@stroeder.com> <54C786AC.1030407@proseconsulting.co.uk> <54C7B135.8060403@stroeder.com> <54C7F3FD.3000902@proseconsulting.co.uk> <54C7FA8C.1090101@stroeder.com> <54C7FF74.10008@proseconsulting.co.uk> <54C8027A.9090505@stroeder.com> <54C8066D.70507@proseconsulting.co.uk> <54C809A0.4010401@stroeder.com> Message-ID: <54C80ECC.1000006@proseconsulting.co.uk> On 27/01/2015 21:56, Michael Str?der wrote: > Mark R Bannister wrote: >> So if I have just one thread reading results and dispatching results, I have a >> bottleneck don't I? If that thread is busy reading a large result set for one >> request, and there is also a large amount of data waiting for another ten >> separate requests, there will be a delay in processing the others. Whereas if >> I have ten separate workers reading their own data, I am spreading the load >> and improving the performance. It is a more scalable solution with >> performance that is directly proportional to the number of requests, not >> exponential. > After all your LDAP server will be also limited, especially when you really > want to enumerate all objects in your maps. > > If enumeration is really crucial in your environment, then consider > maintaining a local maps cache database. With this you can query the LDAP > server with syncrepl or (modifyTimestamp=>) to keep LDAP result sets small > during normal update operation. sssd is doing a lot like this. But it's > complex and error-prone. > > Personally I don't think that enumeration is really needed. BTDT, but YMMV > since the OS platforms out there are quite different. > > Ciao, Michael. > Enumeration is not that important here, but it may be somewhere else, which is why I did want to limit the design with unnecessary bottlenecks. Best regards, Mark. From michael at stroeder.com Thu Jan 29 14:59:23 2015 From: michael at stroeder.com (=?UTF-8?Q?Michael_Str=c3=b6der?=) Date: Thu, 29 Jan 2015 14:59:23 +0100 Subject: [python-ldap] Fwd: LDAPCon 2015 Call for Papers In-Reply-To: <20150129135012.GA26897@slab.skills-1st.co.uk> References: <20150129135012.GA26897@slab.skills-1st.co.uk> Message-ID: <54CA3CBB.8040103@stroeder.com> HI! Please excuse me if this CfP reaches you through several mailing lists. But it might be interesting for some mailing list members. The PC is looking forward to your submissions. Ciao, Michael. -------- Forwarded Message -------- Subject: LDAPCon 2015 Call for Papers Date: Thu, 29 Jan 2015 13:50:12 +0000 From: Andrew Findlay Reply-To: enquiries at lists.ldapcon.org To: openldap-technical at openldap.org LDAPCon 2015 ============ The fifth International Conference on LDAP and Directory Services will be held in the UK at the University of Edinburgh School of Informatics Forum. Tutorials: 11th November 2015 Conference: 12th and 13th November 2015 Call for papers and tutorials ============================= Topics You are using LDAP in interesting projects? You do LDAP client or server development? You have used LDAP in a new way? You do identity and access management on top of LDAP? Why not share your ideas and experiences with others? We are looking for speakers who are willing to talk about any topic related to LDAP and identity management, including: LDAP technology implementation (Servers, API, User interfaces etc.) LDAP Usage (Schema, Security, Operations, Scaling, big data, etc.) LDAP related technologies (PKI, XACML, SAML, etc.) LDAP and Beyond (IAM, Identity Federation, Authentication on the web, etc.) Best Practices for directory services. Accepted talks will be grouped into tracks such as a standards/development and deployment/administration. Deadlines & Important Dates Submission Deadline: 28th June Author Notification: 10th July Final Papers due: 10th October 2015 Tutorials: 11th November 2015 Conference: 12th-13th November 2015 Talk Submissions Main presentations should last about 45 minutes including discussion; we will also provide smaller slots of 15 minutes and 5 minutes for poster presentations or lightning talks. Please tell us which duration you prefer when proposing your talk. The talk must be in English. The one and only way to submit your abstract (approximately 200-800 words, accompanied by your biography of about 100-300 words) is via email to submissions at lists.ldapcon.org. Abstracts must reach the Program Committee by 28th June 2015. Early submission is encouraged. All abstracts will be reviewed by the program committee. For accepted talks we expect you to submit slides and/or a paper of approximately 2-10 pages (A4 or US Letter format, 25mm borders, preferably LaTeX source or OpenOffice). For 5-minute talks, a brief abstract is required. A short paper, slides or a poster should be provided for accepted talks. We will provide display boards for posters throughout the conference. By submitting a paper you grant the conference organizers the non-exclusive right to publish your paper in the conference proceedings and on the website; you maintain the right to publish it elsewhere at your discretion. Tutorial Submissions We are looking for high-quality tutorials on LDAP and related subjects, at any level from introductory to advanced. Tutorial length can range from an hour to a full day. Wireless Internet access will be available if required. The purpose of the tutorials is focussed education, so they should cover established topics and best practice rather than presenting new work. Tutorials will be on Wednesday 11th November 2015. The Programme Committee has an open mind about the format of the tutorial day, but has a limited number of rooms available. Make your proposal early and we will aim to build an attractive programme for the day. Expenses Speakers get free access to the conference, including the social event. If requested in advance we will provide accommodation for speakers. Travel expenses might also be covered in special cases. If you need this, please contact us early so we can try to arrange it. Website http://ldapcon.org/2015/ Contacts General enquiries: enquiries at lists.ldapcon.org Paper/Tutorial submissions: submissions at lists.ldapcon.org -- ----------------------------------------------------------------------- | From Andrew Findlay, Skills 1st Ltd | | Consultant in large-scale systems, networks, and directory services | | http://www.skills-1st.co.uk/ +44 1628 782565 | ----------------------------------------------------------------------- -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4252 bytes Desc: S/MIME Cryptographic Signature URL: From anurag.chourasia at gmail.com Wed Feb 11 09:18:25 2015 From: anurag.chourasia at gmail.com (Anurag Chourasia) Date: Wed, 11 Feb 2015 05:18:25 -0300 Subject: [python-ldap] Error in Installing python-ldap - RHEL 6.5 Server - Python 2.7.9 Message-ID: I am installing python-ldap on a RHEL 6.5 Server. I am on Python 2.7.9 I am using the following command to installl pip2.7 install python-ldap The compilation process fails with lots of errors. Could someone please guide me? The session transcript is at http://dpaste.com/2BQFWF8 Regards, Guddu -------------- next part -------------- An HTML attachment was scrubbed... URL: From pspacek at redhat.com Wed Feb 11 09:52:10 2015 From: pspacek at redhat.com (Petr Spacek) Date: Wed, 11 Feb 2015 09:52:10 +0100 Subject: [python-ldap] Error in Installing python-ldap - RHEL 6.5 Server - Python 2.7.9 In-Reply-To: References: Message-ID: <54DB183A.5010303@redhat.com> On 11.2.2015 09:18, Anurag Chourasia wrote: > I am installing python-ldap on a RHEL 6.5 Server. I am on Python 2.7.9 > > I am using the following command to installl > > pip2.7 install python-ldap > > The compilation process fails with lots of errors. Could someone please > guide me? The session transcript is at http://dpaste.com/2BQFWF8 Hello, why you do not want to use $ yum install python-ldap ? If you really want to use pip - you are most probably missing openldap-devel package. I would recommend you to download source package and look into .spec file in the src.rpm to see how it is compiled in RHEL, what packages are in build requirement set etc. Have a nice day! -- Petr Spacek @ Red Hat From michael at stroeder.com Wed Feb 11 09:58:16 2015 From: michael at stroeder.com (=?UTF-8?Q?Michael_Str=c3=b6der?=) Date: Wed, 11 Feb 2015 09:58:16 +0100 Subject: [python-ldap] Error in Installing python-ldap - RHEL 6.5 Server - Python 2.7.9 In-Reply-To: <54DB183A.5010303@redhat.com> References: <54DB183A.5010303@redhat.com> Message-ID: <54DB19A8.4070408@stroeder.com> Petr Spacek wrote: > On 11.2.2015 09:18, Anurag Chourasia wrote: >> I am installing python-ldap on a RHEL 6.5 Server. I am on Python 2.7.9 >> >> I am using the following command to installl >> >> pip2.7 install python-ldap >> >> The compilation process fails with lots of errors. Could someone please >> guide me? The session transcript is at http://dpaste.com/2BQFWF8 > > Hello, > > why you do not want to use > $ yum install python-ldap As you can see above the original poster wants to use Python 2.7.9. Is that available with recent python-ldap on RHEL 6.5? Ciao, Michael. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4252 bytes Desc: S/MIME Cryptographic Signature URL: From michael at stroeder.com Wed Feb 11 09:56:28 2015 From: michael at stroeder.com (=?UTF-8?Q?Michael_Str=c3=b6der?=) Date: Wed, 11 Feb 2015 09:56:28 +0100 Subject: [python-ldap] Error in Installing python-ldap - RHEL 6.5 Server - Python 2.7.9 In-Reply-To: References: Message-ID: <54DB193C.8010305@stroeder.com> Anurag Chourasia wrote: > I am installing python-ldap on a RHEL 6.5 Server. I am on Python 2.7.9 > > I am using the following command to installl > > pip2.7 install python-ldap > > The compilation process fails with lots of errors. Could someone please > guide me? The session transcript is at http://dpaste.com/2BQFWF8 You don't have the necessary include files (library headers) installed as indicated by these lines in your build log: -------------------------- snip -------------------------- In file included from Modules/LDAPObject.c:9: Modules/errors.h:8:18: error: lber.h: No such file or directory Modules/errors.h:9:18: error: ldap.h: No such file or directory -------------------------- snip -------------------------- Please look at the build prerequisites section in the docs: http://www.python-ldap.org/doc/html/installing.html#prerequisites I guess you have to install the *-devel packages of OpenLDAP, OpenSSL and Cyrus-SASL. I don't know the exact RPM package names on your platform though. Installing the OpenLDAP -devel package should pull in all others based on the RPM dependencies. Ciao, Michael. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4252 bytes Desc: S/MIME Cryptographic Signature URL: From anurag.chourasia at gmail.com Wed Feb 11 10:28:38 2015 From: anurag.chourasia at gmail.com (Anurag Chourasia) Date: Wed, 11 Feb 2015 06:28:38 -0300 Subject: [python-ldap] Error in Installing python-ldap - RHEL 6.5 Server - Python 2.7.9 In-Reply-To: <54DB19A8.4070408@stroeder.com> References: <54DB183A.5010303@redhat.com> <54DB19A8.4070408@stroeder.com> Message-ID: Thanks Petr/Michael, RHEL 6.5 comes with Python 2.6.6 but i installed Python 2.7.9 using the IUS repo. After installing the development package (openldap-devel) indicated by you my problem got resolved. Regards, Guddu On Wed, Feb 11, 2015 at 5:58 AM, Michael Str?der wrote: > Petr Spacek wrote: > > On 11.2.2015 09:18, Anurag Chourasia wrote: > >> I am installing python-ldap on a RHEL 6.5 Server. I am on Python 2.7.9 > >> > >> I am using the following command to installl > >> > >> pip2.7 install python-ldap > >> > >> The compilation process fails with lots of errors. Could someone please > >> guide me? The session transcript is at http://dpaste.com/2BQFWF8 > > > > Hello, > > > > why you do not want to use > > $ yum install python-ldap > > As you can see above the original poster wants to use Python 2.7.9. Is that > available with recent python-ldap on RHEL 6.5? > > Ciao, Michael. > > > _______________________________________________ > python-ldap mailing list > python-ldap at python.org > https://mail.python.org/mailman/listinfo/python-ldap > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ziadbad at gmail.com Tue Feb 24 11:29:18 2015 From: ziadbad at gmail.com (Ziad Badawi) Date: Tue, 24 Feb 2015 12:29:18 +0200 Subject: [python-ldap] Getting all active directory attributes Message-ID: Greetings, So I'm trying (according to my research) res = l.search_s('CN=Schema,CN=Configuration,DC=domain,DC=local', ldap.SCOPE_BASE, '(objectclass=*)', ['*','+'] ) To get all attributes of my domain (server 2012).But i'm only getting those: ['distinguishedName', 'repsFrom', 'replUpToDateVector', 'cn', 'objectCategory', 'objectClass', 'masteredBy', 'objectGUID', 'repsTo', 'whenChanged', 'whenCreated', 'uSNCreated', 'msDs-masteredBy', 'uSNChanged', 'instanceType', 'dSCorePropagationData', 'name'] which are for sure not all of them, the attribute editor on the server has a lot more. Am I missing something? Any help is appreciated. -- Z -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael at stroeder.com Tue Feb 24 12:05:41 2015 From: michael at stroeder.com (=?UTF-8?Q?Michael_Str=c3=b6der?=) Date: Tue, 24 Feb 2015 12:05:41 +0100 Subject: [python-ldap] Getting all active directory attributes In-Reply-To: References: Message-ID: <54EC5B05.8090108@stroeder.com> Ziad Badawi wrote: > Greetings, > > So I'm trying (according to my research) > > res = l.search_s('CN=Schema,CN=Configuration,DC=domain,DC=local', > ldap.SCOPE_BASE, '(objectclass=*)', ['*','+'] ) > > To get all attributes of my domain (server 2012).But i'm only getting those: > > ['distinguishedName', 'repsFrom', 'replUpToDateVector', 'cn', > 'objectCategory', 'objectClass', 'masteredBy', 'objectGUID', 'repsTo', > 'whenChanged', 'whenCreated', 'uSNCreated', 'msDs-masteredBy', > 'uSNChanged', 'instanceType', 'dSCorePropagationData', 'name'] > > which are for sure not all of them, the attribute editor on the server has > a lot more. > Am I missing something? I assume you are talking about MS Active Directory W2K12. 1. AFAIK MS AD does *not* support getting all operational attributes by requesting '+' like defined in RFC 3673. 2. There are some operational attributes in AD you have to explicitly request. You have to find out which ones. One example is e.g. attribute 'tokenGroups' because it computes all nested group membership relation and therefore consumes a lot of resources. All in all this is not strictly related to python-ldap. It's rather a general AD question. You should ask in appropriate MS forums to get more information. Ciao, Michael. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4252 bytes Desc: S/MIME Cryptographic Signature URL: From ziadbad at gmail.com Tue Feb 24 16:03:11 2015 From: ziadbad at gmail.com (Ziad Badawi) Date: Tue, 24 Feb 2015 17:03:11 +0200 Subject: [python-ldap] Modifying user attributes Message-ID: Hello, I am trying to modify the telephoneNumber user attribute as follows #!/usr/bin/python import ldap import ldap.modlist as modlist ldap.set_option(ldap.OPT_X_TLS_REQUIRE_CERT, ldap.OPT_X_TLS_NEVER) l = ldap.initialize('ldaps://domain.local:636') l.simple_bind_s('user', 'passwd') dn="CN=John Smith,OU=Domain Users,DC=domain,DC=local" old = {'telephoneNumber': '1234'} new = {'telephoneNumber':'123456'} ldif = modlist.modifyModlist(old,new) l.modify_s(dn,ldif) l.unbind_s() When I execute this, I get ldap.SERVER_DOWN: {'info': 'TLS error -5939:No more entries in the directory', 'desc': "Can't contact LDAP server"} and I am not sure what it means, but the strange thing is when I execute this another time, it works fine and the attribute is modified Any idea what is going on here? and what is the solution? -- Z -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael at stroeder.com Tue Feb 24 17:32:18 2015 From: michael at stroeder.com (=?UTF-8?Q?Michael_Str=c3=b6der?=) Date: Tue, 24 Feb 2015 17:32:18 +0100 Subject: [python-ldap] Modifying user attributes In-Reply-To: References: Message-ID: <54ECA792.10707@stroeder.com> Ziad Badawi wrote: > l = ldap.initialize('ldaps://domain.local:636') > > l.simple_bind_s('user', 'passwd') > [..] > When I execute this, I get > ldap.SERVER_DOWN: {'info': 'TLS error -5939:No more entries in the > directory', 'desc': "Can't contact LDAP server"} This is a TLS related error probably occuring at l.simple_bind_s(). Use ldap.initialize('ldaps://domain.local:636',trace_level=1) (or higher trace level) to see details. Beware that password is sent to trace output. I don't see where you set the CA cert(s) used to validate the server cert. > and I am not sure what it means, but the strange thing is when I execute > this another time, it works fine and the attribute is modified > > Any idea what is going on here? and what is the solution? Is there a load balancer in front of ldaps://domain.local:636 and your requests are passed to different replicas/instances? Ciao, Michael. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4252 bytes Desc: S/MIME Cryptographic Signature URL: From ziadbad at gmail.com Wed Feb 25 08:21:52 2015 From: ziadbad at gmail.com (Ziad Badawi) Date: Wed, 25 Feb 2015 09:21:52 +0200 Subject: [python-ldap] Modifying user attributes In-Reply-To: <54ECA792.10707@stroeder.com> References: <54ECA792.10707@stroeder.com> Message-ID: On Tue, Feb 24, 2015 at 6:32 PM, Michael Str?der wrote: > Ziad Badawi wrote: > > l = ldap.initialize('ldaps://domain.local:636') > > > > l.simple_bind_s('user', 'passwd') > > [..] > > When I execute this, I get > > ldap.SERVER_DOWN: {'info': 'TLS error -5939:No more entries in the > > directory', 'desc': "Can't contact LDAP server"} > > This is a TLS related error probably occuring at l.simple_bind_s(). Use > ldap.initialize('ldaps://domain.local:636',trace_level=1) (or higher trace > level) to see details. Beware that password is sent to trace output. > > After checking the trace, it does fail on simple_bind_s. > I don't see where you set the CA cert(s) used to validate the server cert. I do not know how to use that and I used ldap.set_option(ldap.OPT_X_TLS_REQUIRE_CERT, ldap.OPT_X_TLS_NEVER) to avoid using TLS as I learned. > > and I am not sure what it means, but the strange thing is when I execute > > this another time, it works fine and the attribute is modified > > > > Any idea what is going on here? and what is the solution? > > Is there a load balancer in front of ldaps://domain.local:636 and your > requests are passed to different replicas/instances? > > No load balancers > Ciao, Michael. > > > -- Z -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael at stroeder.com Wed Feb 25 08:48:33 2015 From: michael at stroeder.com (=?UTF-8?Q?Michael_Str=c3=b6der?=) Date: Wed, 25 Feb 2015 08:48:33 +0100 Subject: [python-ldap] Modifying user attributes In-Reply-To: References: <54ECA792.10707@stroeder.com> Message-ID: <54ED7E51.9010202@stroeder.com> Ziad Badawi wrote: > On Tue, Feb 24, 2015 at 6:32 PM, Michael Str?der > wrote: > >> Ziad Badawi wrote: >>> l = ldap.initialize('ldaps://domain.local:636') >>> >>> l.simple_bind_s('user', 'passwd') >>> [..] >>> When I execute this, I get >>> ldap.SERVER_DOWN: {'info': 'TLS error -5939:No more entries in the >>> directory', 'desc': "Can't contact LDAP server"} >> >> This is a TLS related error probably occuring at l.simple_bind_s(). Use >> ldap.initialize('ldaps://domain.local:636',trace_level=1) (or higher trace >> level) to see details. Beware that password is sent to trace output. >> >> After checking the trace, it does fail on simple_bind_s. > >> I don't see where you set the CA cert(s) used to validate the server cert. > > I do not know how to use that and I used > ldap.set_option(ldap.OPT_X_TLS_REQUIRE_CERT, ldap.OPT_X_TLS_NEVER) to avoid > using TLS as I learned. See Demo/initialize.py in the source distribution. Ciao, Michael. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4252 bytes Desc: S/MIME Cryptographic Signature URL: From ziadbad at gmail.com Wed Feb 25 12:06:15 2015 From: ziadbad at gmail.com (Ziad Badawi) Date: Wed, 25 Feb 2015 13:06:15 +0200 Subject: [python-ldap] Modifying user attributes In-Reply-To: <54ED7E51.9010202@stroeder.com> References: <54ECA792.10707@stroeder.com> <54ED7E51.9010202@stroeder.com> Message-ID: I will give that shot after getting the certificate. But I am wondering if I could just bypass the certificate check and connect successfully without it. I am also still wondering how does it work correctly after executing the script several more times On Wed, Feb 25, 2015 at 9:48 AM, Michael Str?der wrote: > Ziad Badawi wrote: > > On Tue, Feb 24, 2015 at 6:32 PM, Michael Str?der > > wrote: > > > >> Ziad Badawi wrote: > >>> l = ldap.initialize('ldaps://domain.local:636') > >>> > >>> l.simple_bind_s('user', 'passwd') > >>> [..] > >>> When I execute this, I get > >>> ldap.SERVER_DOWN: {'info': 'TLS error -5939:No more entries in the > >>> directory', 'desc': "Can't contact LDAP server"} > >> > >> This is a TLS related error probably occuring at l.simple_bind_s(). Use > >> ldap.initialize('ldaps://domain.local:636',trace_level=1) (or higher > trace > >> level) to see details. Beware that password is sent to trace output. > >> > >> After checking the trace, it does fail on simple_bind_s. > > > >> I don't see where you set the CA cert(s) used to validate the server > cert. > > > > I do not know how to use that and I used > > ldap.set_option(ldap.OPT_X_TLS_REQUIRE_CERT, ldap.OPT_X_TLS_NEVER) to > avoid > > using TLS as I learned. > > See Demo/initialize.py in the source distribution. > > Ciao, Michael. > > -- Z -------------- next part -------------- An HTML attachment was scrubbed... URL: From neerajthakur11 at gmail.com Mon Mar 2 07:18:29 2015 From: neerajthakur11 at gmail.com (Neeraj Thakur) Date: Mon, 2 Mar 2015 11:48:29 +0530 Subject: [python-ldap] installing python-ldap on windows 64bit Message-ID: I am trying to use the python-ldap library with Python 2.7.6 (64 bit) on Windows 7 64 bit. My Python 2.7.6 is built with MSVC 2010. I was able to get python_ldap?2.4.19?cp27?none?win_amd64.whl , but the modules from Mr. Gohlke are built with MSVC 2008. *experimental*: If I install the python_ldap?2.4.19?cp27?none?win_amd64.whl on my Python 2.7 import ldap fails with File "c:\Python27_64\lib\site-packages\ldap\__init__.py", line 22, in import _ldap ImportError: DLL load failed: %1 is not a valid Win32 application. now the question is, I want to build python-ldap with MSVC 2010, for this I need 1. OpenLDAP 2.4.11+ client libs 2. Cyrus SASL 2.1.x or newer 3. MIT Kerberos or heimdal libs but none of them have build environment for windows, how can I hack/build the libs and modules with MSVC 2010? -- With Regards Neeraj -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron at heyaaron.com Fri Mar 6 03:03:50 2015 From: aaron at heyaaron.com (Aaron C. de Bruyn) Date: Thu, 5 Mar 2015 18:03:50 -0800 Subject: [python-ldap] 'Missing' OPT_X_TLS_NEVER Message-ID: I've been beating my head against the wall for about 24 hours. The short problem: Deploying an app I wrote to one particular machine has problems because the ldap module does not have OPT_X_TLS_NEVER (AttributeError: 'module' object has no attribute 'OPT_X_TLS_NEVER'). My guess is that python-ldap doesn't compile in TLS support if there's a particular TLS library unavailable. The long problem: I have three identical machines. My development laptop running Ubuntu 14.04, a stanging/test machine running Ubuntu 14.04, and a production server running Ubuntu 14.04. The production server is of course the one that's failing. All three machines have Dokku installed. (If you've never heard about it, it's basically an open source version of Heroku. You 'git push' your code, it creates a container, and it runs your code.) Every box I push to should have the same build process. A docker container is built that resembles the heroku cedar stack, python-ldap (and other libraries) are installed, and the application is deployed. The python-ldap build/install from the production machine is here: http://dpaste.com/2SBT39J Once the application container is deployed, I can connect in to it, fire up a python shell, and run: >>> import ldap >>> ldap.__version__ '2.4.18' >>> ldap.OPT_X_TLS_REQUIRE_CERT 24582 >>> ...just not on the production box. I get: ldap.OPT_X_TLS_REQUIRE_CERT: ldap.OPT_X_TLS_NEVER, AttributeError: 'module' object has no attribute 'OPT_X_TLS_NEVER' I know this isn't really an issue with python-ldap itself, I'm just hoping there might be someone here who might be familiar with the technologies involved and can give me a shove in the right direction. Thanks, -A From michael at stroeder.com Fri Mar 6 08:42:53 2015 From: michael at stroeder.com (=?UTF-8?Q?Michael_Str=c3=b6der?=) Date: Fri, 06 Mar 2015 08:42:53 +0100 Subject: [python-ldap] 'Missing' OPT_X_TLS_NEVER In-Reply-To: References: Message-ID: <54F95A7D.20809@stroeder.com> Aaron C. de Bruyn wrote: > The short problem: Deploying an app I wrote to one particular machine > has problems because the ldap module does not have OPT_X_TLS_NEVER > (AttributeError: 'module' object has no attribute 'OPT_X_TLS_NEVER'). > > My guess is that python-ldap doesn't compile in TLS support if there's > a particular TLS library unavailable. > [..] Ubuntu 14.04. How are you building python-ldap? Note that libldap on Debian and probably Ubuntu is linked against GnuTLS, not OpenSSL. This might make difference in some options. I'd check whether the -devel packages are all the same on the different machines. There has to be some minimal difference somewhere. > ldap.OPT_X_TLS_REQUIRE_CERT: ldap.OPT_X_TLS_NEVER, > AttributeError: 'module' object has no attribute 'OPT_X_TLS_NEVER' It's not clear to me whether ldap.OPT_X_TLS_REQUIRE_CERT is present but ldap.OPT_X_TLS_NEVER is not. Or both do not exist? Ciao, Michael. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4252 bytes Desc: S/MIME Cryptographic Signature URL: From dwmaillist at gmail.com Thu Mar 12 19:43:15 2015 From: dwmaillist at gmail.com (Daniel Watrous) Date: Thu, 12 Mar 2015 13:43:15 -0500 Subject: [python-ldap] SERVER_DOWN: {'desc': "Can't contact LDAP server"} (no ssl) Message-ID: I have the following class that creates an instance variable from ldap.initialize. There is a long running instance that is part of a web application. When first deployed, the app works as expected and connects to the LDAP directory and can run queries. class dao: host = 'ldap://ldap.company.com:123' scope = ldap.SCOPE_SUBTREE def __init__(self, base='o=hp.com'): self.base = base self.ldapconn = ldap.initialize(self.host) def execute_ldap_search(self, filter, attrs=None): """Execute an LDAP search based on the provided filter and attributes""" return self.ldapconn.search_s(self.base, self.scope, filter, attrs) After running for some time however, self.ldapconn appears to go bad and I get this error: Traceback (most recent call last): File "/app/.python/lib/python2.7/site-packages/bottle.py", line 862, in _handle return route.call(**args) File "/app/.python/lib/python2.7/site-packages/bottle.py", line 1732, in wrapper rv = callback(*a, **ka) File "wsgi.py", line 22, in search_people matching_people = persondao.find_by_last_first_comma(search, city, country) File "/home/stackato/app/dao.py", line 83, in find_by_last_first_comma return self.ldap_response_to_objects(self.execute_ldap_search(filter), self.model_class) File "/home/stackato/app/dao.py", line 27, in execute_ldap_search return self.ldapconn.search_s(self.base, self.scope, filter, attrs) File "/app/.python/lib/python2.7/site-packages/ldap/ldapobject.py", line 559, in search_s return self.search_ext_s(base,scope,filterstr,attrlist,attrsonly,None,None,timeout=self.timeout) File "/app/.python/lib/python2.7/site-packages/ldap/ldapobject.py", line 552, in search_ext_s msgid = self.search_ext(base,scope,filterstr,attrlist,attrsonly,serverctrls,clientctrls,timeout,sizelimit) File "/app/.python/lib/python2.7/site-packages/ldap/ldapobject.py", line 548, in search_ext timeout,sizelimit, File "/app/.python/lib/python2.7/site-packages/ldap/ldapobject.py", line 106, in _ldap_call result = func(*args,**kwargs) SERVER_DOWN: {'desc': "Can't contact LDAP server"} Google provides lots of links for this error, but they all have to do with SSL. I can connect just fine, but the connection goes bad. How can I reconnect? Is there a better way to handle this? -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael at stroeder.com Thu Mar 12 21:46:28 2015 From: michael at stroeder.com (=?UTF-8?Q?Michael_Str=c3=b6der?=) Date: Thu, 12 Mar 2015 21:46:28 +0100 Subject: [python-ldap] SERVER_DOWN: {'desc': "Can't contact LDAP server"} (no ssl) In-Reply-To: References: Message-ID: <5501FB24.7090405@stroeder.com> Daniel Watrous wrote: > I have the following class that creates an instance variable from > ldap.initialize. There is a long running instance that is part of a web > application. When first deployed, the app works as expected and connects to > the LDAP directory and can run queries. > [..] > After running for some time however, self.ldapconn appears to go bad and I > get this error: > [..] > SERVER_DOWN: {'desc': "Can't contact LDAP server"} When using persistent connection you have to be prepared to handle this because servers might close connections after a while. Basically that's what ReconnectLDAPObject is for: http://www.python-ldap.org/doc/html/ldap.html#ldap.ReconnectLDAPObject Make sure to use recent python-ldap release 2.4.19 to avoid running into bugs recently fixed. Ciao, Michael. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4272 bytes Desc: S/MIME Cryptographic Signature URL: From rene.kijewski at fu-berlin.de Tue Mar 17 10:04:18 2015 From: rene.kijewski at fu-berlin.de (rene.kijewski at fu-berlin.de) Date: Tue, 17 Mar 2015 10:04:18 +0100 Subject: [python-ldap] ldap_sasl_bind_s() Message-ID: <20150317100418.0b4b2acb@inf.fu-berlin.de> Hello, in [1] I added a method to invoke ldap_sasl_bind_s. This function is especially useful I your application is only a "man-in-the-middle" and you don't want/need to know the password of the client: 1) Invoke ldap_sasl_bind_s(self->ldap, NULL, "DIGEST-MD5", NULL, NULL, NULL, &challenge) 2) Send the challenge to the client 3) Let the client calculate the response, e.g. using [2] 4) Invoke ldap_sasl_bind_s(self->ldap, NULL, "DIGEST-MD5", &response, NULL, NULL, NULL) With the added ldap_sasl_bind_s and [3] it was very easy to implement an authenticator that does not send the plaintext password over the wire at all. I cannot tell if my change fits the coding standards of your fine project, neither do I know if you it is bug free. I do not know if returning either a string, an integer or raising an exception really is a proper interface. Maybe returning a tuple (ldaperror, challenge) would be better? Either way, please provide ldap_sasl_bind_s in the master branch, since it might be advisable to transfer as little (exploitable) information as possible. Best Ren? 1: https://github.com/Kijewski/python-ldap/commit/3b666b5a4205a30b7ad04695846409b3c89feb58 2: https://github.com/jaredhanson/js-sasl-digest-md5 3: https://github.com/kennethreitz/flask-sockets -- ?I can give you everything you desire,? said the fairy. ?Wealth, a crown and scepter, fame, a long happy life. Choose. But what little you ask for you will have to pay dearly.? From michael at stroeder.com Thu Mar 19 22:07:17 2015 From: michael at stroeder.com (=?UTF-8?Q?Michael_Str=c3=b6der?=) Date: Thu, 19 Mar 2015 22:07:17 +0100 Subject: [python-ldap] Reminder: OpenLDAP, web2ldap and python-ldap at CLT 2015 In-Reply-To: <54D8B940.6020500@stroeder.com> References: <54D8B940.6020500@stroeder.com> Message-ID: <550B3A85.4020803@stroeder.com> HI! A short reminder that you can meet me this weekend in Chemnitz, Germany. Peter Gietz, Dieter Kl?nter and meinereiner will be present OpenLDAP at Chemnitzer Linuxtage 2015 and will answer questions about OpenLDAP server, clients, and integration. I've prepared some nice demos of ?-DIR, RADIUS/LDAP for WLAN-AP, dhcpd and powerdns with LDAP backend and all the entries managed with web2ldap. Of course I will answer questions related to python-ldap as well. Date: Saturday/Sunday, 21./22. March 2015 Booth: https://chemnitzer.linux-tage.de/2015/en/programm/beitrag/134 My ?-DIR presentation 22. March 2015, 11:00: https://chemnitzer.linux-tage.de/2015/en/programm/beitrag/135 Hope to see some of you there. Ciao, Michael. From michael at stroeder.com Sun Mar 22 23:18:39 2015 From: michael at stroeder.com (=?UTF-8?Q?Michael_Str=c3=b6der?=) Date: Sun, 22 Mar 2015 23:18:39 +0100 Subject: [python-ldap] ldap_sasl_bind_s() In-Reply-To: <20150317100418.0b4b2acb@inf.fu-berlin.de> References: <20150317100418.0b4b2acb@inf.fu-berlin.de> Message-ID: <550F3FBF.2060301@stroeder.com> rene.kijewski at fu-berlin.de wrote: > in [1] I added a method to invoke ldap_sasl_bind_s. > > This function is especially useful I your application is only a > "man-in-the-middle" and you don't want/need to know the password of the > client: Thanks for your very interesting contribution. I will look into it as time constraints permits. Do you have a small test application with which I could test it? Ciao, Michael. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4272 bytes Desc: S/MIME Cryptographic Signature URL: From rene.kijewski at fu-berlin.de Mon Mar 23 15:18:49 2015 From: rene.kijewski at fu-berlin.de (=?UTF-8?B?UmVuw6k=?= Kijewski) Date: Mon, 23 Mar 2015 15:18:49 +0100 Subject: [python-ldap] ldap_sasl_bind_s() In-Reply-To: <550F3FBF.2060301@stroeder.com> References: <20150317100418.0b4b2acb@inf.fu-berlin.de> <550F3FBF.2060301@stroeder.com> Message-ID: <20150323151849.68b56017@inf.fu-berlin.de> Am Sun, 22 Mar 2015 23:18:39 +0100 schrieb Michael Str?der : > rene.kijewski at fu-berlin.de wrote: > > in [1] I added a method to invoke ldap_sasl_bind_s. > > > > This function is especially useful I your application is only a > > "man-in-the-middle" and you don't want/need to know the password of the > > client: > > Do you have a small test application with which I could test it? Hello Michael, in [1] I wrote a small example application that uses my patch i.e. ldap_sasl_bind_s. The login uses a websocket so you retain the connection that gave you the "challenge". The mechanism works like that: LDAP server web application web browser ---------------------------------------------------------------------- asks user for credentials <--- opens a web socket What's your name? ---> <--- I'm $name test if the user name is valid according the local naming standards <--- asks for DIGEST-MD5 challenge gives challenge ---> forward challenge ---> <--- calculate response <--- forward response confirms response Afterwards the connection is bound, but the web application never saw the password. The challenge contains a nonce, so even if there is a malicious 3rd party between the application and the user, it can't use the response. Of course you should still use LDAPS and HTTPS. Please ask if something is unclear. Best regards, Ren? 1: https://github.com/Kijewski/ldap_sasl_bind_s-example -- ?Sometimes there is no point in giving up.? ?Louis Wu