【菜鸟学Python】使用Scrapy框架爬取糗事百科
内容导读
互联网集市收集整理的这篇技术教程文章主要介绍了【菜鸟学Python】使用Scrapy框架爬取糗事百科，小编现在分享给大家，供广大互联网技能从业者学习和参考。文章包含3138字，纯文字阅读大概需要5分钟。
内容图文
第一步：
创建项目
scrapy stratproject [name]
如 scrapy startproject choushibaike
第二步：
进入到项目的文件夹目录创建APP
scrapy gensider baike lovehhy.net
第三步：
配置baike.py文件
# -*- coding: utf-8 -*-
import scrapy
from ..items import ChoushibaikeItem
class BaikeSpider(scrapy.Spider):
name = ‘baike‘
allowed_domains = [‘lovehhy.net‘]
start_urls = [‘http://www.lovehhy.net/joke/Detail/QSBK‘]
def parse(self, response):
titles = response.xpath(‘//div[@class="cat_llb"]/h3/a/text()‘).extract()
contents = response.xpath(‘//div[@class="cat_llb"]/div[@id="endtext"]/text()‘).extract()
times = response.xpath(‘//div[@class="cat_llb"]/text()‘).extract()
title_list = []
for title in titles:
title_list.append(title)
content_list = []
for content in contents:
content_list.append(content)
time_list = []
for time in times:
time_list.append(time)
item_lists = zip(title_list, content_list, time_list)
for item_list in item_lists:
item = ChoushibaikeItem()
item[‘title‘] = item_list[0]
item[‘content‘] = item_list[1]
item[‘time‘] = item_list[2][0:22]
item[‘click‘] = item_list[2][22:-1]
yield item
next_url = response.xpath(‘//a[text()="下一页>>"]/@href‘).extract_first()
url = response.urljoin(next_url)
yield scrapy.Request(url=url, callback=self.parse)
第四步：
配置items.py文件
import scrapy
class ChoushibaikeItem(scrapy.Item):
# define the fields for your item here like:
# name = scrapy.Field()
title = scrapy.Field()
content = scrapy.Field()
time = scrapy.Field()
click = scrapy.Field()
第五步：
配置pipelines.py文件
import pymongo
class MongoPipeline(object):
def __init__(self, mongo_uri, mongo_db):
self.mongo_uri = mongo_uri
self.mongo_db = mongo_db
@classmethod
def from_crawler(cls, crawler):
return cls(
mongo_uri=crawler.settings.get(‘MONGO_URI‘),
mongo_db=crawler.settings.get(‘MONGO_DB‘)
)
def open_spider(self, spider):
self.client = pymongo.MongoClient(self.mongo_uri)
self.db = self.client[self.mongo_db]
def process_item(self, item, spider):
name = item.__class__.__name__
self.db[name].insert(dict(item))
return item
def close_spider(self, spider):
self.client.close()
第六步：
配置settings.py文件
#
-*- coding: utf-8 -*-
#
Scrapy settings for choushibaike project
#
#
For simplicity, this file contains only settings considered important or
#
commonly used. You can find more settings consulting the documentation:
#
#
https://docs.scrapy.org/en/latest/topics/settings.html
#
https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
#
https://docs.scrapy.org/en/latest/topics/spider-middleware.html
BOT_NAME = ‘choushibaike‘
SPIDER_MODULES = [‘choushibaike.spiders‘]
NEWSPIDER_MODULE = ‘choushibaike.spiders‘# Crawl responsibly by identifying yourself (and your website) on the user-agent
USER_AGENT = ‘Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36‘# Obey robots.txt rules
ROBOTSTXT_OBEY = False
ITEM_PIPELINES = {
# ‘choushibaike.pipelines.ChoushibaikePipeline‘: 300,‘choushibaike.pipelines.MongoPipeline‘: 400,
}
MONGO_URI = ‘mongodb://admin:123456@localhost/‘
MONGO_DB = ‘choushibaike‘
第七步：
运行项目
scrapy crawl baike
原文：https://www.cnblogs.com/liduo0413/p/11687319.html
内容总结
以上是互联网集市为您收集整理的【菜鸟学Python】使用Scrapy框架爬取糗事百科全部内容，希望文章能够帮你解决【菜鸟学Python】使用Scrapy框架爬取糗事百科所遇到的程序开发问题。 如果觉得互联网集市技术教程内容还不错，欢迎将互联网集市网站推荐给程序员好友。
内容备注
版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 gblab@vip.qq.com 举报，一经查实，本站将立刻删除。
内容手机端
扫描二维码推送至手机访问。