[Madison] Python to accept terms and condition form a website

[Eric Gierach]

You probably have this figured out by now, but mechanize's Browser object
has a method, select_form, which allows you to set the browser's focus on a
particular form and submit it with the "click" method.  Use the select_form
method's predicate argument to pass a pointer to a function you define to
find the right form based on its content.  It's easier than it sounds.

Example code:

from mechanize import Browser
from urllib2 import URLError

# initialize browser, set user agent
browser = Browser()
browser.addheaders = [('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT
5.1; it; rv:1.8.1.11)')]

# open the URL containing your TOS form
try:
    browser.open('http://www.example.com/TOS.html')
except URLError:
    print "couldn't open the page"

# if your bot got a valid response
if browser.viewing_html():
    # if your bot found the TOS form and gave it focus
    if browser.select_form(predicate=find_form):
        # optionally set other form fields
        browser.form["YOUR_NAME"] = "Mr. Spider"
        browser.form["NUM_RECORDS"] = "35"
        # browser.click generates a Request object which you can pass to
browser.open to submit the form.
        browser.open(browser.click())
        print "mission complete"

def find_form(form):
    """
    The browser calls this function with each form on the page.  You need to
find something unique
    about the form you're interested in and return true if the passed-in
form has it.  So, in this example,
    your TOS form has a <input type="submit" name="TOS_BTN".../> field.
 Let's search for that.
    """
    return "TOS_BTN" in form

Here's the reference for mechanize forms:
http://wwwsearch.sourceforge.net/mechanize/forms.html
(it's a mess)

Eric

On Tue, May 3, 2011 at 1:04 PM, Nicola Branzoli <nbranzol at ssc.wisc.edu>wrote:

> Hey, I am writing a code in python to access public data online (using
> BeautifulSoup).
> The task is relatively easy but the code does not get to the page I want
> because I need to accept the terms and condition of the website first
> (by a standard 'Click  the Accept').
> I need to tell python how to automatically accept the terms and
> condition and proceed to the web address specified. I am new in
> pyhton, my guess is that I have to use mechanize because cookielib is
> not good for this job. Am I right? What other resources can I use? Any
> link with an example similar to my problem would be great...
>
> Thanks a lot!
>
>
>
> --
> Nicola Branzoli
> Ph.D. Candidate - University of Wisconsin Madison
> William H. Sewell Social Science Building
> 1180 Observatory Drive
> Madison, WI 53706-1393
>
>
> _______________________________________________
> Madison mailing list
> Madison at python.org
> http://mail.python.org/mailman/listinfo/madison
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/madison/attachments/20110505/12f11a41/attachment.html>