以前講過利用phantomjs做爬蟲抓網頁 //www.jb51.net/article/55789.htm 是配合選擇器做的
利用 beautifulSoup(文檔 ：http://www.crummy.com/software/BeautifulSoup/bs4/doc/)這個python模塊，可以很輕松的抓取網頁內容
# coding=utf-8import urllibfrom bs4 import BeautifulSoupurl ='http://www.baidu.com/s'values ={'wd':'網球'}encoded_param = urllib.urlencode(values)full_url = url +'?'+ encoded_paramresponse = urllib.urlopen(full_url)soup =BeautifulSoup(response)alinks = soup.find_all('a')上面可以抓取百度搜出來結果是網球的記錄。
beautifulSoup內置了很多非常有用的方法。
幾個比較好用的特性：
構造一個node元素
代碼如下:
soup = BeautifulSoup('<b class="boldest">Extremely bold</b>')
tag = soup.b
type(tag)
# <class 'bs4.element.Tag'>
屬性可以使用attr拿到，結果是字典
代碼如下:
tag.attrs
# {u'class': u'boldest'}
或者直接tag.class取屬性也可。
也可以自由操作屬性
tag['class'] = 'verybold'tag['id'] = 1tag# <blockquote class="verybold" id="1">Extremely bold</blockquote>del tag['class']del tag['id']tag# <blockquote>Extremely bold</blockquote>tag['class']# KeyError: 'class'print(tag.get('class'))# None還可以隨便操作，查找dom元素，比如下面的例子
1.構建一份文檔
html_doc = """<html><head><title>The Dormouse's story</title></head><p><b>The Dormouse's story</b></p><p>Once upon a time there were three little sisters; and their names were<a id="link1">Elsie</a>,<a id="link2">Lacie</a> and<a id="link3">Tillie</a>;and they lived at the bottom of a well.</p><p>...</p>"""from bs4 import BeautifulSoupsoup = BeautifulSoup(html_doc)
2.各種搞
soup.head# <head><title>The Dormouse's story</title></head>soup.title# <title>The Dormouse's story</title>soup.body.b# <b>The Dormouse's story</b>soup.a# <a class="sister" id="link1">Elsie</a>soup.find_all('a')# [<a class="sister" id="link1">Elsie</a>,# <a class="sister" id="link2">Lacie</a>,# <a class="sister" id="link3">Tillie</a>]head_tag = soup.headhead_tag# <head><title>The Dormouse's story</title></head>head_tag.contents[<title>The Dormouse's story</title>]title_tag = head_tag.contents[0]title_tag# <title>The Dormouse's story</title>title_tag.contents# [u'The Dormouse's story']len(soup.contents)# 1soup.contents[0].name# u'html'text = title_tag.contents[0]text.contentsfor child in title_tag.children: print(child)head_tag.contents# [<title>The Dormouse's story</title>]for child in head_tag.descendants: print(child)# <title>The Dormouse's story</title># The Dormouse's storylen(list(soup.children))# 1len(list(soup.descendants))# 25title_tag.string# u'The Dormouse's story'
新聞熱點
疑難解答
