[Tutor] Python Scraping problem help

CPECC张涛 sccdzt at foxmail.com
Thu May 27 01:38:39 EDT 2021


Dear sir :
Could you please help solve the below problem that can run but no result ,the program is runing,but can noly retrieve one picture ,Could you help tell me how to solve this problem .
import requests
from bs4 import BeautifulSoup
import urllib
import time,os


headers={'User-Agent': 'Mozilla/5.0 (Windows NT 6.1;'
                       'Win64; x64) AppleWebKit/537.36 (KHTML, like'
                       'Gecko) Chrome/69.0.3497.100'
                       'Safari/537.36','Referer': "https://www.douban.com/group/topic/206388806/?type=rec"}
url=r'https://www.douban.com/group/topic/206388806/?type=rec'
wbdata=requests.get(url,headers=headers)
# print(response)
soup=BeautifulSoup(wbdata.content,'lxml')
imgs=soup.select('div.image-wrapper>img')
# print(imgs)
number=1
for item in imgs:
    img=item.get('src')
    print(img)
    path=r'F:\Python crawler'

    if not os.path.isdir(path):
        os.makedirs(path)
    paths=path+str(number)+'.jpg'
    time.sleep(1)
    try:
        urllib.request.urlretrieve(img,paths)
        number+=1
        # print('%sDownloaded'%img)
    except:
        continue


More information about the Tutor mailing list