[python-ldap] LDAP pagination cookie not working across AD server restart

[Stephen J. Butler]

VLV lets you sort and specify offsets.

I wouldn't expect the paged results cookie to survive reconnects or server
restarts. The MS docs don't say what causes it to be invalidated, but they
do specify that clients treat it as a blob and should restart queries where
it fails:

https://technet.microsoft.com/en-us/windows-server-docs/identity/ad-ds/manage/how-ldap-server-cookies-are-handled

I don't see any guarantees mentioned for when the cookie will succeed.
Which to me means your strategy should be a rare case (where retry is
acceptable) and not a standard path.


On Thu, Jan 5, 2017 at 1:07 PM, Ritesh Nadhani <riteshn at gmail.com> wrote:

> On Thu, Jan 5, 2017 at 10:56 AM, Michael Ströder <michael at stroeder.com>
> wrote:
> > Ritesh Nadhani wrote:
> >> I am writing a script that would poll an AD with 250K user
> >> information. It looks something like (from which I took code
> >> inspiration):
> >>
> >> https://gist.github.com/mattfahrner/c228ead9c516fc322d3a
> >
> > So you're using the Simple Paged Control (see RFC 2696):
> >
> > https://tools.ietf.org/html/rfc2696
> >
>
> Yes.
>
> >> To be resilient of server restarts and my app crashing, I am writing
> >> the cookie into a file and reading it from there in the next
> >> iteration.
> >
> > Hmm...
> >
> >> Everything works and even my script crashes, on restart it picks up
> >> the right cookie file from storage and starts from the correct offset
> >> (ignoring users that have already been fetched).
> >
> > I wonder why this works in your case.
> > In general I doubt that it's guaranteed to work.
>
> Well, I will be honest and this is not something that I found over the
> docs or anything. I just wrote the implementation and during my tests,
> it seemed to work so I just assumed. My bad.
>
> >
> >> The problem seems to be that, while the operation is going on if I
> >> restart the server, I get SERVER_DOWN exception, on which case I just
> >> re-setup the connection again after the server is back and continue on
> >> the loop. Unfortunately, in this case, using the older cookie gives
> >> me:
> >>
> >> UNAVAILABLE_CRITICAL_EXTENSION: {'info': '00000057: LdapErr:
> >> DSID-0C0907B8, comment: Error processing control, data 0, v2580',
> >> 'desc': 'Critical extension is unavailable'}
> >>
> >> ..
> >>
> >> and I cannot figure out the issue. The cookie way works brilliant if
> >> my app and the network IO behaves correctly. If my app crashes, I can
> >> restart and reuse the cookie with a new connection object.
> >>
> >> But if I restart the AD server itself, reconnecting and using the
> >> cookie does not work.
> >
> > In general I would never expect this (both cases) to work since the
> server can
> > throw away any context of your former stale LDAP connection. The first
> case may
> > work with AD but likely does not work with other LDAP servers. AFAICS
> there is
> > no text in RFC 2696 clarifying this.
> >
> > I wonder why AD returns UNAVAILABLE_CRITICAL_EXTENSION though. But hey,
> RFC 2696
> > uses pretty blurry text like "it SHOULD return the appropriate error".
> >
>
> So I guess my question becomes:
>
> If i have to efficiently pull all user information from a directory
> service where the network IO can be a bit sketchy, whats the best to
> query when I want to ignore already fetched users information?
>
> As far as I can see, there is no way to specifiy ordering, offseting
> during the query option.
>
> Or I am just out of luck?
>
>
> > Ciao, Michael.
> >
> >
>
>
>
> --
> Ritesh
> _______________________________________________
> python-ldap mailing list
> python-ldap at python.org
> https://mail.python.org/mailman/listinfo/python-ldap
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ldap/attachments/20170105/65c982d1/attachment-0001.html>