使用lxml的Python脚本，xpath返回空列表

Question

我尝试使用xpath和lxml从html标记中提取href链接。但是xpath返回空列表，而它是单独测试的，并且似乎可以工作。

代码返回空值，而xpath似乎工作正常。

page = self.opener.open(link).read()
doc=html.fromstring(str(page))
ref = doc.xpath('//ul[@class="s-result-list s-col-1 s-col-ws-1 s-result-list-hgrid s-height-equalized s-list-view s-text-condensed s-item-container-height-auto"]/li/div/div[@class="a-fixed-left-grid"]/div/div[@class="a-fixed-left-grid-col a-col-left"]/div/div/a')
for post in ref:
    print(post.get("href"))

我使用一个代理服务器来访问链接，它似乎可以工作，因为“doc”变量正在填充html内容。我已经检查了这些链接，现在正处于获取此xpath的正确页面上。

这是我试图从中获取数据的链接：https://www.amazon.com/s/ref=lp_266162_nr_n_0?fst=as:off

Answer 1

您的xpath选择器无效。尝试css选择器，如下所示

import requests
import lxml, lxml.html

url = 'https://www.amazon.com/s/ref=lp_266162_nr_n_0?fst=as%3Aoff&rh=n%3A283155%2Cn%3A%211000%2Cn%3A1%2Cn%3A173508%2Cn%3A266162%2Cn%3A3564986011&bbn=266162&ie=UTF8&qid=1550120216&rnid=266162'
r = requests.get(url)
html = lxml.html.fromstring(r.content)
links = html.cssselect('.a-fixed-left-grid-col .a-col-left a')
for link in links:
    print(link.attrib['href'])

https://www.amazon.com/Top-500-Instant-Pot-Recipes/dp/1730885209
https://www.amazon.com/Monthly-Budget-Planner-Organizer-Notebook/dp/1978202865
https://www.amazon.com/Edge-Order-Daniel-Libeskind/dp/045149735X
https://www.amazon.com/Man-Glass-House-Johnson-Architect/dp/0316126438
https://www.amazon.com/Versailles-Private-Invitation-Guillaume-Picon/dp/2080203371
https://www.amazon.com/Palm-Springs-Modernist-Tim-Street-Porter/dp/0847861872
https://www.amazon.com/Building-Chicago-Architectural-John-Zukowsky/dp/0847848701
https://www.amazon.com/Taverns-American-Revolution-Adrian-Covert/dp/160887785X
https://www.amazon.com/TRAVEL-MOSAIC-Color-Number-Relaxation/dp/1717562221
https://www.amazon.com/Understanding-Cemetery-Symbols-Historic-Graveyards/dp/1547047216
https://www.amazon.com/Soviet-Bus-Stops-Christopher-Herwig/dp/099319110X
https://www.amazon.com/Famous-Movie-Scenes-Dot-Dot/dp/1977747043

certifi==2018.11.29
chardet==3.0.4
cssselect==1.0.3
idna==2.8
lxml==4.3.1
requests==2.21.0
urllib3==1.24.1

Answer 2

我想你在寻找书籍：艺术中的链接

import requests
from lxml.html import fromstring

link = 'https://www.amazon.com/s/ref=lp_266162_nr_n_0?fst=as%3Aoff&rh=n%3A283155%2Cn%3A%211000%2Cn%3A1%2Cn%3A173508%2Cn%3A266162%2Cn%3A3564986011&bbn=266162&ie=UTF8&qid=1550120216&rnid=266162'
r = requests.get(link,headers={"User-Agent":"Mozilla/5.0"})
htmlcontent = fromstring(r.text)
itemlinks = htmlcontent.xpath('//*[@id="mainResults"]//*[contains(@class,"s-access-detail-page")]')
for link in itemlinks:
    print(link.get('href'))

如果您想使用css选择器，那么以下操作应该可以：

'#mainResults .s-access-detail-page'