SUBMIT "ACCEPT" button - Python - beautifulsoap
peter.neumaier at gmail.com
peter.neumaier at gmail.com
Wed Jan 28 11:36:45 EST 2015
I am totally new to Python and please accept my apologies upfront for potential newbie errors. I am trying to parse a 'simple' web page: http://flow.gassco.no/
When opening the page first time in my browser I need to confirm T&C with an accept button. After accepting T&C I would like to scrape some data from that follow up page. It appears that when opening in a browser directly http://flow.gassco.no/acceptDisclaimer I would get around that T&C.
But not when I open the URL via beautifulsoap
My parsing/scraping tool is implemented in bs, but I fail to parse the content as I am not getting around T&C. When printing "response.text" from BS, I get below code. How do I get around this form for accepting terms & conditions so that I can parse/scrape data from that page?
Here is what I am doing:
#!/usr/bin/env python
import requests
import bs4
index_url='http://flow.gassco.no/acceptDisclaimer'
def get_video_page_urls():
response = requests.get(index_url)
soup = bs4.BeautifulSoup(response.text)
return soup
print(get_video_page_urls())
++++
PRINTOUT from response.text:
<form action="acceptDisclaimer" method="get">
<input class="accept" type="submit" value="Accept"/>
<input class="decline" name="decline" onclick="window.location ='http://www.gassco.no'" type="button" value="Decline"/>
</form></div></div></div></div></div>
<script type="text/javascript">
var _gaq = _gaq || [];
_gaq.push(['_setAccount', 'UA-30727768-1']);
_gaq.push(['_trackPageview']);
(function() {
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
})();
</script>
More information about the Python-list
mailing list