亮点:

1、系统分析目标网页

2、html标签数据解析方法

3、海量数据一键保存

环境介绍:

python 3.8
pycharm 2021专业版
selenium >>> pip install selenium==3.141.0

代码

导入模块

from selenium import webdriver
import time
import csv
1
2
3

1. 打开谷歌浏览器

driver = webdriver.Chrome()

for page in range(0, 38):
    print(f'------------------------正在爬取第{page+1}页------------------------')
1
2
3
4

2. 打开网页

    driver.get(f'https://search.suning.com/%E4%B8%9D%E8%A2%9C%E6%83%85%E8%B6%A3/&iy=0&isNoResult=0&cp={page}')
    # 灵活等待
    driver.implicitly_wait(1)
    drop_down()
    get_next()
1
2
3
4
5

3. 获取商品数据

    # 网页 html: <div class="">茅台1 <div>茅台2 <div>茅台3</div></div></div>   <div>茅台2</div>   <img src="图片"/>
    # <a href="网页链接"></a>:
    divs = driver.find_elements_by_css_selector('.item-bg')
1
2
3

所有的商品都拿出来

    for div in divs:
1

标题

        title = div.find_element_by_css_selector('.title-selling-point a').text
1

价格

        price = div.find_element_by_css_selector('.def-price').text
1

评论数

        try:
            comment_num = div.find_element_by_css_selector('.info-evaluate a').text
        except:
            comment_num = 0
1
2
3
4

商铺名字

        store_name = div.find_element_by_css_selector('.store-stock a').text
1

详情页链接

get_attribute: 获取标签属性名

       detail_url = div.find_element_by_css_selector('.title-selling-point a').get_attribute('href')
        print(title, price, comment_num, store_name, detail_url)
1
2

4. 保存数据

        # a: 追加写入
        # wb: 以二进制方式覆盖写入
        with open('苏宁.csv', mode='a', encoding='utf-8', newline='') as f:
            csv_writer = csv.writer(f)
            csv_writer.writerow([title, price, comment_num, store_name, detail_url])

with open('苏宁.csv', mode='a', encoding='utf-8', newline='') as f:
    csv_writer = csv.writer(f)
    csv_writer.writerow(['title', 'price', 'comment_num', 'store_name', 'detail_url'])
1
2
3
4
5
6
7
8
9

5. 翻页保存

def drop_down():
    """执行页面滚动的操作"""
    for x in range(1, 12, 2):
        time.sleep(1)
        j = x / 9
        js = 'document.documentElement.scrollTop = document.documentElement.scrollHeight * %f' % j
        driver.execute_script(js)

def get_next():
1
2
3
4
5
6
7
8
9