(8)分布式下的爬蟲Scrapy應該如何做-圖片下載(源碼放送)

2019-11-14 16:54:20

字體：大中小

來源：轉載

供稿：網友

轉載主注明出處：http://m.survivalescaperooms.com/codefish/p/4968260.html

在爬蟲中，我們遇到比較多需求就是文件下載以及圖片下載，在其它的語言或者框架中，我們可能在經過數據篩選，然后異步的使用文件下載類來達到目的，Scrapy框架中本身已經實現了文件及圖片下載的文件，相當的方便，只要幾行代碼，就可以輕松的搞定下載。下面我將演示如何使用scrapy下載豆瓣的相冊首頁內容。

優點介紹：

1)自動去重

2)異步操作，不會阻塞

3)可以生成指定尺寸的縮略圖

4)計算過期時間

5)格式轉化

編碼過程：

一，定義Item

# -*- coding: utf-8 -*-# Define here the models for your scraped items## See documentation in:# http://doc.scrapy.org/en/latest/topics/items.htmlimport scrapyfrom scrapy import Item,Fieldclass DoubanImgsItem(scrapy.Item):    # define the fields for your item here like:    # name = scrapy.Field()    image_urls = Field()    images = Field()    image_paths = Field()    pass

二，定義spider

#coding=utf-8from scrapy.spiders import Spiderimport refrom douban_imgs.items import DoubanImgsItemfrom scrapy.http.request import Request# please pay attention to the encoding of info,otherwise raise errorimport sysreload(sys)sys.setdefaultencoding('utf8')class download_douban(Spider):    name = 'download_douban'    def __init__(self, url='152686895', *args, **kwargs):        self.allowed_domains = ['douban.com']        self.start_urls = [                'http://www.douban.com/photos/album/%s/' %(url) ]        #call the father base function        self.url = url        super(download_douban, self).__init__(*args, **kwargs)    def parse(self, response):        """        :type response: response infomation        """        list_imgs = response.xpath('//div[@class="photolst clearfix"]//img/@src').extract()        if list_imgs:            item = DoubanImgsItem()            item['image_urls'] = list_imgs            yield item

三，定義piepline

# -*- coding: utf-8 -*-# Define your item pipelines here## Don't forget to add your pipeline to the ITEM_PIPELINES setting# See: http://doc.scrapy.org/en/latest/topics/item-pipeline.htmlfrom scrapy.pipelines.images import ImagesPipelinefrom scrapy.exceptions import DropItemfrom scrapy import Requestfrom scrapy import logclass DoubanImgsPipeline(object):    def PRocess_item(self, item, spider):        return itemclass DoubanImgDownloadPieline(ImagesPipeline):    def get_media_requests(self,item,info):        for image_url in item['image_urls']:            yield Request(image_url)    def item_completed(self, results, item, info):        image_paths = [x['path'] for ok, x in results if ok]        if not image_paths:            raise DropItem("Item contains no images")        item['image_paths'] = image_paths        return item

四，定義setting.py，啟用item處理器

# Configure item pipelines# See http://scrapy.readthedocs.org/en/latest/topics/item-pipeline.htmlITEM_PIPELINES = {   'douban_imgs.pipelines.DoubanImgDownloadPieline': 300,}IMAGES_STORE='C://doubanimgs'IMAGES_EXPIRES = 90

運行效果：