API例子：用Python驱动Firefox采集网页数据 | 一起学习Python网络爬虫

内容导读

互联网集市收集整理的这篇技术教程文章主要介绍了API例子：用Python驱动Firefox采集网页数据 | 一起学习Python网络爬虫，小编现在分享给大家，供广大互联网技能从业者学习和参考。文章包含3524字，纯文字阅读大概需要6分钟。

内容图文

1，引言

• 本次测试的环境是：Windows10， Python3.4.3 32bit
• 安装Scrapy ： $ pip install Scrapy #实际安装时，由于服务器状态的不稳定，出现好几次中途退出的情况

3，编写运行第一个Scrapy爬虫

3.1. 生成一个新项目：tutorial

$ scrapy startproject tutorial

项目目录结构如下：

3.2. 定义要抓取的item

# -*- coding: utf-8 -*-

# Define here the models for your scraped items
#
# See documentation in:
# http://doc.scrapy.org/en/latest/topics/items.html

import scrapy

class DmozItem(scrapy.Item):
    title = scrapy.Field()
    link = scrapy.Field()
    desc = scrapy.Field()

3.3. 定义Spider

import scrapy
from tutorial.items import DmozItem

class DmozSpider(scrapy.Spider):
    name = "dmoz"
    allowed_domains = ["dmoz.org"]
    start_urls = [
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"
    ]

    def parse(self, response):
        for sel in response.xpath('//ul/li'):
            item = DmozItem()
            item['title'] = sel.xpath('a/text()').extract()
            item['link'] = sel.xpath('a/@href').extract()
            item['desc'] = sel.xpath('text()').extract()
            yield item

3.4. 运行

$ scrapy crawl dmoz -o item.json

1) 结果报错：
A) ImportError: cannot import name '_win32stdio'
B) ImportError: No module named 'win32api'

3) 解决过程：

A) 需要手工去下载

twisted/internet

下的 _win32stdio 和 _pollingfile，存放到python目录的lib\sitepackages\twisted\internet下

B) 下载并安装

pywin32

再次运行，成功！在控制台上可以看到scrapy的输出信息，待运行完成退出后，到项目目录打开结果文件items.json， 可以看到里面以json格式存储的爬取结果。

[
{"title": ["        About       "], "desc": [" ", " "], "link": ["/docs/en/about.html"]},
{"title": ["   Become an Editor "], "desc": [" ", " "], "link": ["/docs/en/help/become.html"]},
{"title": ["            Suggest a Site          "], "desc": [" ", " "], "link": ["/docs/en/add.html"]},
{"title": [" Help             "], "desc": [" ", " "], "link": ["/docs/en/help/helpmain.html"]},
{"title": [" Login                       "], "desc": [" ", " "], "link": ["/editors/"]},
{"title": [], "desc": [" ", " Share via Facebook "], "link": []},
{"title": [], "desc": [" ", "  Share via Twitter  "], "link": []},
{"title": [], "desc": [" ", " Share via LinkedIn "], "link": []},
{"title": [], "desc": [" ", " Share via e-Mail   "], "link": []},
{"title": [], "desc": [" ", " "], "link": []},
{"title": [], "desc": [" ", "  "], "link": []},
{"title": ["        About       "], "desc": [" ", " "], "link": ["/docs/en/about.html"]},
{"title": ["   Become an Editor "], "desc": [" ", " "], "link": ["/docs/en/help/become.html"]},
{"title": ["            Suggest a Site          "], "desc": [" ", " "], "link": ["/docs/en/add.html"]},
{"title": [" Help             "], "desc": [" ", " "], "link": ["/docs/en/help/helpmain.html"]},
{"title": [" Login                       "], "desc": [" ", " "], "link": ["/editors/"]},
{"title": [], "desc": [" ", " Share via Facebook "], "link": []},
{"title": [], "desc": [" ", "  Share via Twitter  "], "link": []},
{"title": [], "desc": [" ", " Share via LinkedIn "], "link": []},
{"title": [], "desc": [" ", " Share via e-Mail   "], "link": []},
{"title": [], "desc": [" ", " "], "link": []},
{"title": [], "desc": [" ", "  "], "link": []}
]

第一次运行scrapy的测试成功。
4，接下来的工作
接下来，我将使用GooSeeker API来实现网络爬虫，省掉对每个item人工去生成和测试xpath的工作量。目前有2个计划：

1. 在gsExtractor中封装一个方法：从xslt内容中自动提取每个item的xpath
2. 从gsExtractor的提取结果中自动提取每个item的结果

具体选择哪个方案，将在接下来的实验中确定，并发布到gsExtractor新版本中

5，文档修改历史

2016-11-04：V1.0，首次发布

内容总结

以上是互联网集市为您收集整理的API例子：用Python驱动Firefox采集网页数据 | 一起学习Python网络爬虫全部内容，希望文章能够帮你解决API例子：用Python驱动Firefox采集网页数据 | 一起学习Python网络爬虫所遇到的程序开发问题。 如果觉得互联网集市技术教程内容还不错，欢迎将互联网集市网站推荐给程序员好友。

内容备注

版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 gblab@vip.qq.com 举报，一经查实，本站将立刻删除。

内容手机端

---