url question - extracting (2 types of) domains

lazy arunmail at gmail.com
Tue May 15 22:04:02 EDT 2007


Hi,
Im trying to extract the domain name from an url. lets say I call
it full_domain and significant_domain(which is the homepage domain)

Eg: url=http://en.wikipedia.org/wiki/IPod ,
full_domain=en.wikipedia.org ,significant_domain=wikipedia.org

Using urlsplit (of urlparse module), I will be able to get the
full_domain, but Im wondering how to get significant_domain. I will
not be able to use like counting the number of dots. etc

Some domains maybe like foo.bar.co.in (where significant_domain=
bar.co.in)
I have around 40M url list. Its ok, if I fallout in few(< 1%) cases.
Although I agree that measuring this error rate itself is not clear,
maybe just based on ituition.

Anybody have clues about existing url parsers in python to do this.
Searching online couldnt help me much other than
the urlparse/urllib module.

Worst case is to try to build a table of domain
categories(like .com, .co.il etc and look for it in the suffix rather
than counting dots and just extract the part till the preceding dot),
but Im afraid if I do this, I might miss some domain category.




More information about the Python-list mailing list