1.為什么寫這個(gè)？
一些簡單的頁面，無需用比較大的框架來進(jìn)行爬取，自己純手寫又比較麻煩
因此針對這個(gè)需求寫了talonspider:
•1.針對單頁面的item提取 - 具體介紹點(diǎn)這里
•2.spider模塊 - 具體介紹點(diǎn)這里
2.介紹&&使用
2.1.item
這個(gè)模塊是可以獨(dú)立使用的，對于一些請求比較簡單的網(wǎng)站（比如只需要get請求），單單只用這個(gè)模塊就可以快速地編寫出你想要的爬蟲，比如(以下使用python3，python2見examples目錄)：
2.1.1.單頁面單目標(biāo)
比如要獲取這個(gè)網(wǎng)址http://book.qidian.com/info/1004608738 的書籍信息，封面等信息，可直接這樣寫：
import timefrom talonspider import Item, TextField, AttrFieldfrom pprint import pprintclass TestSpider(Item): author = TextField(css_select='a.writer') cover = AttrField(css_select='a#bookImg>img', attr='src') def tal_title(self, title): return title def tal_cover(self, cover): return 'http:' + coverif __name__ == '__main__': item_data = TestSpider.get_item(url='http://book.qidian.com/info/1004608738') pprint(item_data)
2.1.1.單頁面多目標(biāo)
比如獲取豆瓣250電影首頁展示的25部電影，這一個(gè)頁面有25個(gè)目標(biāo)，可直接這樣寫：
from talonspider import Item, TextField, AttrFieldfrom pprint import pprint# 定義繼承自item的爬蟲類class DoubanSpider(Item): target_item = TextField(css_select='div.item') cover = AttrField(css_select='div.pic>a>img', attr='src') abstract = TextField(css_select='span.inq') def tal_title(self, title): if isinstance(title, str): return title else: return ''.join([i.text.strip().replace('/xa0', '') for i in title])if __name__ == '__main__': items_data = DoubanSpider.get_items(url='https://movie.douban.com/top250') result = [] for item in items_data: result.append({ 'title': item.title, 'cover': item.cover, 'abstract': item.abstract, }) pprint(result)2.2.spider
當(dāng)需要爬取有層次的頁面時(shí)，比如爬取豆瓣250全部電影，這時(shí)候spider部分就派上了用場：
# !/usr/bin/env pythonfrom talonspider import Spider, Item, TextField, AttrField, Requestfrom talonspider.utils import get_random_user_agent# 定義繼承自item的爬蟲類class DoubanItem(Item): target_item = TextField(css_select='div.item') cover = AttrField(css_select='div.pic>a>img', attr='src') abstract = TextField(css_select='span.inq') def tal_title(self, title): if isinstance(title, str): return title else: return ''.join([i.text.strip().replace('/xa0', '') for i in title])class DoubanSpider(Spider): # 定義起始url，必須 start_urls = ['https://movie.douban.com/top250'] # requests配置 request_config = { 'RETRIES': 3, 'DELAY': 0, 'TIMEOUT': 20 } # 解析函數(shù) 必須有 def parse(self, html): # 將html轉(zhuǎn)化為etree etree = self.e_html(html) # 提取目標(biāo)值生成新的url pages = [i.get('href') for i in etree.cssselect('.paginator>a')] pages.insert(0, '?start=0&filter=') headers = { "User-Agent": get_random_user_agent() } for page in pages: url = self.start_urls[0] + page yield Request(url, request_config=self.request_config, headers=headers, callback=self.parse_item) def parse_item(self, html): items_data = DoubanItem.get_items(html=html) # result = [] for item in items_data: # result.append({ # 'title': item.title, # 'cover': item.cover, # 'abstract': item.abstract, # }) # 保存 with open('douban250.txt', 'a+') as f: f.writelines(item.title + '/n')if __name__ == '__main__': DoubanSpider.start()控制臺(tái)：
/Users/howie/anaconda3/envs/work3/bin/python /Users/howie/Documents/programming/python/git/talonspider/examples/douban_page_by_spider.py2017-06-07 23:17:30,346 - talonspider - INFO: talonspider started2017-06-07 23:17:30,693 - talonspider_requests - INFO: GET a url: https://movie.douban.com/top2502017-06-07 23:17:31,074 - talonspider_requests - INFO: GET a url: https://movie.douban.com/top250?start=25&filter=2017-06-07 23:17:31,416 - talonspider_requests - INFO: GET a url: https://movie.douban.com/top250?start=50&filter=2017-06-07 23:17:31,853 - talonspider_requests - INFO: GET a url: https://movie.douban.com/top250?start=75&filter=2017-06-07 23:17:32,523 - talonspider_requests - INFO: GET a url: https://movie.douban.com/top250?start=100&filter=2017-06-07 23:17:33,032 - talonspider_requests - INFO: GET a url: https://movie.douban.com/top250?start=125&filter=2017-06-07 23:17:33,537 - talonspider_requests - INFO: GET a url: https://movie.douban.com/top250?start=150&filter=2017-06-07 23:17:33,990 - talonspider_requests - INFO: GET a url: https://movie.douban.com/top250?start=175&filter=2017-06-07 23:17:34,406 - talonspider_requests - INFO: GET a url: https://movie.douban.com/top250?start=200&filter=2017-06-07 23:17:34,787 - talonspider_requests - INFO: GET a url: https://movie.douban.com/top250?start=225&filter=2017-06-07 23:17:34,809 - talonspider - INFO: Time usage：0:00:04.462108Process finished with exit code 0
此時(shí)當(dāng)前目錄會(huì)生成douban250.txt，具體見douban_page_by_spider.py。
3.說明
學(xué)習(xí)之作，待完善的地方還有很多，歡迎提意見，項(xiàng)目地址talonspider。
新聞熱點(diǎn)
疑難解答
圖片精選
