使用urllib2和python正确阅读超文本标记语言

Question

我打开这个链接"http://www.amazon.com/s?rh=n： 1"与urllib2和我试图获取下一页链接（href="/s/ref=lp_1_pg_2？rh=n： 283155， n：！1000， n： 1

Answer 1

尝试

import urllib2
from HTMLParser import HTMLParser

    class MyHTMLParser(HTMLParser):
        href = []
        def handle_starttag(self, tag, attrs):
            if tag == "a":
                for attr in attrs:
                    if attr[0] == "href" and 'page' in attr[1] and 'rh' in attr[1]:
                        self.href.append(attr[1])


    def _get(url):
        response = urllib2.urlopen(url)
        html = response.read()
        parser = MyHTMLParser()
        parser.feed(html.decode('utf-8'))
        href = parser.href
        print href

    _get('http://www.amazon.com/s?rh=n%3A1')

Answer 2

它这样做是因为你没有标题。我试过：

from mechanize import Browser
from bs4 import BeautifulSoup

browser = Browser()

html_page = browser.open("http://www.amazon.com/s?rh=n%3A1")
soup = BeautifulSoup(html_page)
link = soup.find("a", {"title" : "Next Page"})
print link

输出：

<a title="Next Page" id="pagnNextLink" class="pagnNext" href="/s?rh=n%3A1&amp;page=2">
<span id="pagnNextString">Next Page</span>
<span class="srSprite pagnNextArrow"></span>
</a>

然后我添加了标题：

from mechanize import Browser
from bs4 import BeautifulSoup
browser = Browser()

browser.addheaders = [('User-agent', 'Mozilla/5.0\
    (Windows NT 6.2; WOW64) AppleWebKit/537.11 (KHTML, like Gecko)\
    Chrome/23.0.1271.97 Safari/537.11')]

html_page = browser.open("http://www.amazon.com/s?rh=n%3A1")
soup = BeautifulSoup(html_page)
link = soup.find("a", {"title" : "Next Page"})
print link

输出：

<a title="Next Page" id="pagnNextLink" class="pagnNext" href="/s/ref=lp_1_pg_2/177-4872792-4084836?rh=n%3A283155%2Cn%3A%211000%2Cn%3A1&amp;page=2&amp;ie=UTF8&amp;qid=1376771097">
<span id="pagnNextString">Next Page</span>
<span class="srSprite pagnNextArrow"></span>
</a>

所以只需像这样添加标题信息

例子：

from bs4 import BeautifulSoup
import urllib2

req = urllib2.Request("http://www.amazon.com/s?rh=n%3A1")
req.add_header('User-agent', 'Mozilla/5.0\
            (Windows NT 6.2; WOW64) AppleWebKit/537.11 (KHTML, like Gecko)\
            Chrome/23.0.1271.97 Safari/537.11')

html_page = urllib2.urlopen(req)

if html_page.getcode() == 200:
    soup = BeautifulSoup(html_page)
    link = soup.find("a", {"title" : "Next Page"})
    print link['href']

else:
    print "Error loading page"

输出：

/s/ref=lp_1_pg_2/176-2670743-2970243?rh=n%3A283155%2Cn%3A%211000%2Cn%3A1&page=2&ie=UTF8&qid=1376771750