Urllib's urlopen and urlretrieve

[qoresucks at gmail.com]

Initially I was just trying the html, but later when I attempted more complicated sites that weren't my own I noticed that large bulks of the site were lost in the process. The urllib code essentially looks like what I was trying but it didn't work as I had expected.

To be more specific, after I got it working for my own little page, I attempted to take it further and get all the lessons from Learn Python The Hard Way. When I tried the same method on the first intro page to see if I was even getting it right, the html code was all there but upon opening it I noticed the format was all wrong, colors were off for the background, images, etc... were all missing. So clearly I ended up misunderstanding something and its something critical I need to understand. 

As for the OS, I primarily use Mac OS, however well versed in linux and windows if there is anything specific out there that might help. 

As for which version if Python, I have been using Python 2 to learn on as I heard that Python 3 was still largely unadopted due to a lack of library support etc... by comparison. Are people adopting it fast enough now that I should consider learning on 3 instead of 2?

Also, it isn't so much to do it for technical reasons but rather I thought it would be something interesting and fun to learn some form of internet/network programming. Granted, its not the best approach, but I'm not really aware of too many others, and I it does seem interesting to me. 

Python programming probably isn't the best way to initially approach this I agree, but I wasn't sure what to research on or to get a better grasp of network/internet/web programming so I figured I would just dive head first and figure things out, and reinforce more programming while learning internet/network programming was my initial goal. 

Thank you all for your responses though. :)


On Thursday, February 21, 2013 7:59:26 AM UTC-5, Michael Herman wrote:
> Are you just trying to get the html? If so, you can use this code-
> 
> 
> 
> import urllib
> 
> 
> # fetch the and download a webpage, nameing it test.html
> urllib.urlretrieve("http://www.web2py.com/", filename="test.html")
> 
> 
> 
> 
> 
> 
> I recommend using the requests library, as it's easier to use and more powerful:
> 
> 
> 
> 
> 
> 
> 
> 
> 
> import requests
> 
> # retrive the webpage
> r = requests.get("http://www.web2py.com/")
> 
> # write the content to test_request.html
> with open("test_requests.html", "wb") as code:   
> 
> 
> 
> 
> 
> code.write(r.content)
> 
> If you want to get up to speed quickly on internet programming, I have a course I am developing. It's on kickstarter - http://kck.st/VQj8hq. The first section of the book dives into web fundamentals and internet programming. 
> 
> 
> 
> 
> 
> 
> 
> On Thu, Feb 21, 2013 at 4:12 AM,  <qore... at gmail.com> wrote:
> 
> 
> I only just started Python and given that I know nothing about network programming or internet programming of any kind really, I thought it would be interesting to try write something that could create an archive of a website for myself. With this I started trying to use the urllib library, however I am having a problem understanding why certain things wont work with the urllib.urlretrieve and urllib.urlopen then reading.
> 
> 
> 
> 
> 
> Why is it that when using urllib.urlopen then reading or urllib.urlretrieve, does it only give me parts of the sites, loosing the formatting, images, etc...? How can I get around this?
> 
> 
> 
> Lastly, while its a bit off topic, I lack a good understanding of network programming as a whole. From making programs communicate or to simply extract data from URL's, I don't know where to even begin, which has lead me to learning python to better understand it hopefully then carry it over to other languages I know. Can anyone give me some advice on where to begin learning this information? Even if its in another language.
> 
> 
> 
> --
> 
> http://mail.python.org/mailman/listinfo/python-list