Real-world use of concurrent.futures

[Andrew McLean]

On 08/05/2014 20:06, Chris Angelico wrote:
> On Fri, May 9, 2014 at 4:55 AM, Andrew McLean <lists at andros.org.uk> wrote:
>> Because of latency in the DNS lookup this could
>> benefit from multithreading.
> Before you go too far down roads that are starting to look
> problematic: A DNS lookup is a UDP packet out and a UDP packet in
> (ignoring the possibility of TCP queries, which you probably won't be
> doing here). Maybe it would be easier to implement it as asynchronous
> networking? I don't know that Python makes it easy for you to
> construct DNS requests and parse DNS responses; that's something more
> in Pike's line of work. But it may be more possible to outright do the
> DNS query asynchronously. TBH I haven't looked into it; but it's
> another option to consider.
>
> Separately from your programming model, though, how are you handling
> timeouts? Any form of DNS error (NXDOMAIN being the most likely), and
> the sort-of-error-but-not-error state of getting a response with no
> answer, indicates that the address is invalid; but what if you just
> don't hear back from the server? Will that mark addresses off as dead?
>
> ChrisA

I've done this on a very small scale in the past. I used

http://www.dnspython.org/

to do the heavy lifting. The relevant bits of code looks like:

> # Set up the default dns resolver and add a cache
> dns.resolver.default_resolver = dns.resolver.Resolver()
> dns.resolver.default_resolver.cache = dns.resolver.Cache()
and
>     try:
>         result = dns.resolver.query(domain, 'MX')
>         return True
>     except dns.resolver.NXDOMAIN:
>         return False
>     except dns.resolver.NoAnswer:
>         return False
>     except dns.resolver.Timeout:
>         print "*** timeout looking for the MX record for the domain:
> %s" % domain
>         return False

You are right, I'll need to do something more sophisticated when I
encounter a timeout, but I think that's a matter of detail.

Andrew