python scrapy版 極客學(xué)院爬蟲V2

1 基本技術(shù) 使用scrapy

2 這個(gè)爬蟲的難點(diǎn)是 Request中的headers和cookies 嘗試過好多次才成功（模擬登錄），否則只能抓免費(fèi)課程

3文件樹如下

│ jike.json 存儲(chǔ)json文件
│ run.py 運(yùn)行爬蟲
└─jkscrapy
│ items.py 設(shè)置字段
│ pipelines.py 處理字段
│ settings.py 基本設(shè)置
│ init.py
└─spiders
jkspiders.py 爬蟲
jkspiders.pyc
init.py

coding:utf-8

from jkscrapy.items import JkscrapyItem
from scrapy.http import Request
import re
from scrapy.spiders import BaseSpider
from scrapy.selector import Selector
from jkscrapy.settings import *
import sys
reload(sys)
sys.setdefaultencoding("utf-8")

爬蟲

class jikespider(BaseSpider):
name = "jike"
baseurl = " starturls = [" allowed_domains = ["www.jikexueyuan.com", "search.jikexueyuan.com", "jikexueyuan.com"]
def __init(self):
self.headers = headers
self.cookies = cookies
pass

爬蟲 重寫 BaseSpider parse

-1、在首頁中獲取標(biāo)簽及課程對應(yīng)的地址
-2、eghttp://www.jikexueyuan.com/course/python/
def parse(self, response):
s_total = Selector(text=response.body).xpath(
'//*[@id="pager"]/div[1]/div[1]/ul/li/div/div/div/dl/dd/a/@href').extract()
if len(s_total) > 0:
for page in s_total:
yield Request(page, callback=self.get_course_pages,headers=self.headers,cookies=self.cookies)
else:
pass

爬蟲 get_course_pages 獲取課程連接

-1、scrapy Selector xpath 獲取課程地址
-2、eg http://www.jikexueyuan.com/course/1860.html
def get_course_pages(self, response):
x_couses = Selector(text=response.body).xpath('//*[@id="changeid"]/ul/li/div[1]/a')
for x in x_couses:
try:
href = x.select('@href').extract()[0]
title = x.select('img/@title').extract()[0]
yield Request(href, callback=self.get_course_detail,headers=self.headers,cookies=self.cookies)
except:
pass

爬蟲 get_course_detail獲取課程

-1、scrapy Selector xpath 獲取課程地址
-2、eg http://www.jikexueyuan.com/course/271_3.html?ss=1
def get_course_detail(self, response):
d_couses = Selector(text=response.body).xpath('//*[@id="pager"]/div[3]/div[2]/div[2]/ul/li/div/h2/a')
for d in d_couses:
try:
href = d.select('@href').extract()[0]
PRint(href)
title = d.select('text()').extract()[0]

1. # print(" %s %s" % (href, title))
2. meta ={}
3. meta["href"]= href
4. meta["title"]= title
5. yieldRequest(href, callback=self.get_down_urls, meta={"meta": meta},headers=self.headers,cookies=self.cookies)
6. except:
7. pass

爬蟲 get_down_urls獲取課程下地址

-1、正則 獲取課程下載地址，這個(gè)是調(diào)用Flash播放地址 嘗試過很多方法 最后發(fā)現(xiàn)正則可以
-2、eg http://cv3.jikexueyuan.com/201509071527/df51514a02286dac0b30245eaa4dd166/html5/course_mob/01/video/c271b_03_h264_sd_960_540.mp4
def get_down_urls(self, response):
meta = response.meta["meta"]
course_down = re.findall(r'source src="(.*?)"', response.body, re.S)
item = JkscrapyItem()
if course_down:
item["course_id"] = meta["href"]
item["course_name"] = meta["title"]
item["course_path"] = course_down[0]
yield item

爬蟲為了測試簡要的存儲(chǔ)為json 結(jié)果只存儲(chǔ)了幾個(gè)字段，可以根據(jù)需要進(jìn)行調(diào)整共爬取了4500多條