Help regarding urllib

[malhar vora]

On Saturday, August 24, 2013 4:15:01 PM UTC+5:30, malhar vora wrote:
> Hello All,
> 
> 
> 
> 
> 
> I am simply fetching data from robots.txt of a url. Below is my code.
> 
> 
> 
> siteurl = siteurl.rstrip("/")

Sorry for last complete. It was sent by mistake.

Here is my code.

siteurl = siteurl.rstrip("/")
roboturl = siteurl + r'/robots.txt'
robotdata = urllib.urlopen(roboturl).read() # Reading robots.txt of given url
print robotdata

In above code siteurl is fetched simply from local text file.


Whenever I run above code. In place of "/" before robots.txt, it writes "\\" in url as I found in error. The error is given below.

This is main function
Main URL : www.bestrecipes.com.au
$$$$$$$$$$:www.bestrecipes.com.au
###########-->www.bestrecipes.com.au/robots.txt
Traceback (most recent call last):
  File "dataintegrator.py", line 104, in <module>
    main()
  File "dataintegrator.py", line 81, in main
    print "Sitemap Url : " + getSiteMapUrl(i)
  File "D:\Malhar Data\Projects\Data Parsing\My Code\Final Part\libs\datareader.
py", line 50, in getSiteMapUrl
    robotdata = urllib.urlopen(roboturl).read() # Reading robots.txt of given ur
l
  File "C:\Python26\lib\urllib.py", line 87, in urlopen
    return opener.open(url)
  File "C:\Python26\lib\urllib.py", line 203, in open
    return getattr(self, name)(url)
  File "C:\Python26\lib\urllib.py", line 461, in open_file
    return self.open_local_file(url)
  File "C:\Python26\lib\urllib.py", line 475, in open_local_file
    raise IOError(e.errno, e.strerror, e.filename)
IOError: [Errno 2] The system cannot find the path specified: 'www.bestrecipes.c
om.au\\robots.txt'

I am new to Python and not able to figure out this problem. Please help me.

Thank you,

Malhar Vora