[Tutor] Web Page Scraping

Crusier crusier at gmail.com
Mon May 23 23:17:54 EDT 2016


Dear All,

I am trying to scrape a web site using Beautiful Soup. However, BS
doesn't show any of the data. I am just wondering if it is Javascript
or some other feature which hides all the data.

I have the following questions:

1) Please advise how to scrape the following data from the website:

'http://www.dbpower.com.hk/en/quote/quote-warrant/code/10348'

Type, Listing Date (Y-M-D), Call / Put, Last Trading Day (Y-M-D),
Strike Price, Maturity Date (Y-M-D),  Effective Gearing (X),Time to
Maturity (D),
Delta (%), Daily Theta (%),  Board Lot.......

2) I am able to scrape most of the data from the same site

'http://www.dbpower.com.hk/en/quote/quote-cbbc/code/63852'

 Please advise what is the difference between these two sites.
Attached is my code

Thank you

Regards,
Hank

from bs4 import BeautifulSoup
import requests
import json
import re

warrants = ['10348']

def web_scraper(warrants):

    url = "http://www.dbpower.com.hk/en/quote/quote-warrant/code/"

    # Scrape from the Web
    for code in warrants:
        new_url = url + code
        response = requests.get(new_url)
        html = response.content
        soup = BeautifulSoup(html,"html.parser")
        print(soup)


        name = soup.findAll('div', attrs={'class': 'article_content'})
        #print(name)

        for n in name:
            name1 = str(n.text)
            s_code = name1[:4]
            print(name1)


web_scraper(warrants)


More information about the Tutor mailing list