[Tutor] python and Beautiful soup question

[Joshua Valdez]

I'm having trouble making this script work to scrape information from a
series of Wikipedia articles.

What I'm trying to do is iterate over a series of wiki URLs and pull out
the page links on a wiki portal category (e.g.
https://en.wikipedia.org/wiki/Category:Electronic_design).

I know that all the wiki pages I'm going through have a page links section.
However when I try to iterate through them I get this error message:

Traceback (most recent call last):
  File "./wiki_parent.py", line 37, in <module>
    cleaned = pages.get_text()AttributeError: 'NoneType' object has no
attribute 'get_text'

Why do I get this error?

The file I'm reading in the first part looks like this:

1 Category:Abrahamic_mythology2 Category:Abstraction3
Category:Academic_disciplines4 Category:Activism5 Category:Activists6
Category:Actors7 Category:Aerobics8 Category:Aerospace_engineering9
Category:Aesthetics

and it is stored in a the port_ID dict like so:

{1: 'Category:Abrahamic_mythology', 2: 'Category:Abstraction', 3:
'Category:Academic_disciplines', 4: 'Category:Activism', 5:
'Category:Activists', 6: 'Category:Actors', 7: 'Category:Aerobics', 8:
'Category:Aerospace_engineering', 9: 'Category:Aesthetics', 10:
'Category:Agnosticism', 11: 'Category:Agriculture'...}

The desired output is:

parent_num, page_ID, page_num

I realize the code is a little hackish but I'm just trying to get this
working:

#!/usr/bin/env pythonimport os,re,nltkfrom bs4 import
BeautifulSoupfrom urllib import urlopen
url = "https://en.wikipedia.org/wiki/"+'Category:Furniture'

rootdir = '/Users/joshuavaldez/Desktop/L1/en.wikipedia.org/wiki'

reg = re.compile('[\w]+:[\w]+')
number=1
port_ID = {}for root,dirs,files in os.walk(rootdir):
    for file in files:
        if reg.match(file):
            port_ID[number]=file
            number+=1


test_file = open('test_file.csv', 'w')
for key, value in port_ID.iteritems():

    url = "https://en.wikipedia.org/wiki/"+str(value)
    raw = urlopen(url).read()
    soup=BeautifulSoup(raw)
    pages = soup.find("div" , { "id" : "mw-pages" })
    cleaned = pages.get_text()
    cleaned = cleaned.encode('utf-8')
    pages = cleaned.split('\n')
    pages = pages[4:-2]
    test = test = port_ID.items()[0]

    page_ID = 1
    for item in pages:
        test_file.write('%s %s %s\n' % (test[0],item,page_ID))
        page_ID+=1
    page_ID = 1


Hi I posted this on stackoverflow and didn't really get any help so I
though I would ask here to see if someone could help!  Thanks!




*Joshua Valdez*
*Computational Linguist : Cognitive Scientist
     *

(440)-231-0479
jdv12 at case.edu <jdv2 at uw.edu> | jdv2 at uw.edu | joshv at armsandanchors.com
<http://www.linkedin.com/in/valdezjoshua/>