[Tutor] Making http posts

[Rich Lovely]

2009/11/5 Kent Johnson <kent37 at tds.net>:
> On Thu, Nov 5, 2009 at 5:06 AM, Stephen Nelson-Smith <sanelson at gmail.com> wrote:
>> Hello,
>>
>> I want to make an HTTP POST which will require authentication.
>
> What kind of authentication? Basic auth is easy to use from Python;
> form-based auth is a bit tricker.
>
>> This is because we are using a web tool without an API, and we want a
>> programatic way of getting content into the tool.  Tech support of the
>> web tool have said we can just make a POST to the http endpoint.
>>
>> >From the below source, it looks like I just post some text to
>> /project/4254/stories, with the ID of addStoryForm.
>
> IIRC the form ID is not part of the post. The text has to be formatted
> as name=value pairs.
>
>> Is the best way to do this just a simple urllib script?
>
> urllib2. See my writeup here:
> http://personalpages.tds.net/~kent37/kk/00010.html
>
>> What's the best way to find out from the browser exactly what is being
>> posted when I use the web interface?  Tcpdump?  Or is the a better
>> tool?
>
> There are Firefox addons for this. It's been a while so I'm not sure
> which ones but I think either Firebug or Tamper Data will do it.
>
> Kent
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> To unsubscribe or change subscription options:
> http://mail.python.org/mailman/listinfo/tutor
>

If you're really stuck, look into mechanize:  it links together
urllib, beautiful soup and a form handler.  To log in to a form, all
you need to do is:

br = mechanize.Browser()
br.open("http://path.to.form.com/form")
#br.select_form(...) if it's not the first form on the page - see docs
br['name'] = "username"
br['password'] = "password"
response = br.submit()

then response is a beautifulsoup object of the returned page.  The
browser object also updates to act on the new page as well, and has
methods to click on links and so on, exactly as you would in a
browser.  Basically, if you can do it with a keyboard and your
prefered webbrowser, you can do it with mechanize, and a few things
beside.  (I've bypassed a (copypasta) javascript md5 function on a
forum login before now...)
-- 
Rich "Roadie Rich" Lovely

There are 10 types of people in the world: those who know binary,
those who do not, and those who are off by one.