CPython thread starvation

[Paul Rubin]

Roy Smith <roy at panix.com> writes:
> I agree that application-level name cacheing is "wrong", but sometimes 
> doing it the wrong way just makes sense.  I could whip up a simple 
> cacheing wrapper around getaddrinfo() in 5 minutes.  Depending on the 
> environment (both technology and bureaucracy), getting a cacheing 
> nameserver installed might take anywhere from 5 minutes to a few days to ...

IMHO this really isn't one of those times.  The in-app wrapper would
only be usable to just that process, and we already know that the OP has
multiple processes running the same app on the same machine.  They would
benefit from being able to share the cache, so now your wrapper gets
more complicated.  If it's not a nameserver then it's something that
fills in for one.  And then, since the application appears to be a large
scale web spider, it probably wants to run on a cluster, and the cache
should be shared across all the machines.  So you really probably want
an industrial strength nameserver with a big persistent cache, and maybe
a smaller local cache because of high locality when crawling specific
sites, etc.

Also, since this is a production application, doing something in 5
minutes is less important than making it solid and configurable.