利用scrapy將爬到的數(shù)據(jù)保存到mysql（防止重復(fù)）

2020-01-04 15:29:07

字體：大中小

來源：轉(zhuǎn)載

供稿：網(wǎng)友

前言

本文主要給大家介紹了關(guān)于 scrapy爬到的數(shù)據(jù)保存到mysql（防止重復(fù)）的相關(guān)內(nèi)容，分享出來供大家參考學(xué)習(xí)，下面話不多說了，來一起看看詳細的介紹吧。

1.環(huán)境建立

1.使用xmapp安裝php, mysql ,phpmyadmin

2.安裝python3，pip

3.安裝pymysql

3.(windows 略)我這邊是mac，安裝brew，用brew 安裝scrapy

2.整個流程

1. 創(chuàng)建數(shù)據(jù)庫和數(shù)據(jù)庫表，準備保存

2.寫入爬蟲目標URL，進行網(wǎng)絡(luò)請求

3.對爬返回數(shù)據(jù)進行處理，得到具體數(shù)據(jù)

4.對于具體數(shù)據(jù)保存到數(shù)據(jù)庫中

2.1.創(chuàng)建數(shù)據(jù)庫

首先創(chuàng)建一個數(shù)據(jù)庫叫scrapy，然后創(chuàng)建一個表article，我們這里給body加了唯一索引，防止重復(fù)插入數(shù)據(jù)

---- Database: `scrapy`-- -- -------------------------------------------------------- ---- 表的結(jié)構(gòu) `article`-- CREATE TABLE `article` ( `id` int(11) NOT NULL, `body` varchar(200) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL, `author` varchar(50) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL, `createDate` datetime NOT NULL) ENGINE=InnoDB DEFAULT CHARSET=latin1;  ---- Indexes for table `article`--ALTER TABLE `article` ADD PRIMARY KEY (`id`), ADD UNIQUE KEY `uk_body` (`body`);

scrapy,保存mysql,保存到mysql,避免重復(fù)爬取

弄好以后是這樣的。

2.2 先看下整個爬蟲項目的結(jié)構(gòu)

scrapy,保存mysql,保存到mysql,避免重復(fù)爬取

quotes_spider.py是核心，負責(zé)對網(wǎng)絡(luò)請求和對內(nèi)容進行處理，然后對整理好的內(nèi)容拋給pipelines進行具體處理，保存到數(shù)據(jù)庫中，這樣不會影響速度。

其他的看圖說明

2.2 寫入爬蟲目標URL，進行網(wǎng)絡(luò)請求

import scrapyfrom tutorial.items import TutorialItemclass QuotesSpider(scrapy.Spider): name = "quotes" def start_requests(self):  url = 'http://quotes.toscrape.com/tag/humor/'  yield scrapy.Request(url) def parse(self, response):  item = TutorialItem()  for quote in response.css('div.quote'):   item['body'] = quote.css('span.text::text').extract_first()   item['author'] = quote.css('small.author::text').extract_first()   yield item  next_page = response.css('li.next a::attr("href")').extract_first()  if next_page is not None:   yield response.follow(next_page, self.parse)

start_requests 就是要寫入具體要爬的URL

parse就是核心的對返回的數(shù)據(jù)進行處理的地方，然后以item的形式拋出，接下來定義好下一個要爬的內(nèi)容

2.3 items

# -*- coding: utf-8 -*-# Define here the models for your scraped items## See documentation in:# https://doc.scrapy.org/en/latest/topics/items.htmlimport scrapyclass TutorialItem(scrapy.Item): body = scrapy.Field() author = scrapy.Field() pass

2.4 pipelines

# -*- coding: utf-8 -*-# Define your item pipelines here## Don't forget to add your pipeline to the ITEM_PIPELINES setting# See: https://doc.scrapy.org/en/latest/topics/item-pipeline.htmlimport pymysqlimport datetimefrom tutorial import settingsimport loggingclass TutorialPipeline(object): def __init__(self):  self.connect = pymysql.connect(   host = settings.MYSQL_HOST,   db = settings.MYSQL_DBNAME,   user = settings.MYSQL_USER,   passwd = settings.MYSQL_PASSWD,   charset = 'utf8',   use_unicode = True  )  self.cursor = self.connect.cursor();  def process_item(self, item, spider):  try:   self.cursor.execute(    "insert into article (body, author, createDate) value(%s, %s, %s) on duplicate key update author=(author)",    (item['body'],     item['author'],     datetime.datetime.now()     ))   self.connect.commit()  except Exception as error:   logging.log(error)  return item def close_spider(self, spider):  self.connect.close();

2.5 配置

ITEM_PIPELINES = { 'tutorial.pipelines.TutorialPipeline':300}MYSQL_HOST = 'localhost'MYSQL_DBNAME = 'scrapy'MYSQL_USER = 'root'MYSQL_PASSWD = '123456'MYSQL_PORT = 3306

3.啟動爬蟲