刮擦不产生结果（已爬网0页）

Question

试图找出scrapy的工作原理，并使用它在论坛上查找信息。

items.py

import scrapy


class BodybuildingItem(scrapy.Item):
    # define the fields for your item here like:
    title = scrapy.Field()
    pass

spider.py

from scrapy.spider import BaseSpider
from scrapy.selector import Selector
from bodybuilding.items import BodybuildingItem

class BodyBuildingSpider(BaseSpider):
    name = "bodybuilding"
    allowed_domains = ["forum.bodybuilding.nl"]
    start_urls = [
        "https://forum.bodybuilding.nl/fora/supplementen.22/"
    ]

    def parse(self, response):
        responseSelector = Selector(response)
        for sel in responseSelector.css('li.past.line.event-item'):
            item = BodybuildingItem()
            item['title'] = sel.css('a.data-previewUrl::text').extract()
            yield item

在这个例子中，我试图获取帖子标题的论坛是：https://forum.bodybuilding.nl/fora/supplementen.22/

然而，我一直没有得到任何结果：

类BodyBuildingSpider（BaseSpider）：2017-10-07 00:42:28[scrapy.utils.log]信息：scrapy 1.4.0启动（机器人：健美）2017-10-07 00:42:28[scrapy.utils.log]信息：覆盖设置：{'NEWSPIDER_模块'：'bodybuilding.SPIDER'，'SPIDER_模块'：['bodybuilding.SPIDER'，'ROBOTSTXT_服从]：True，'bot_NAME'：'bodybuilding']2017-10-07 00:42:28[scrapy.middleware]信息：启用的扩展：['scrapy.extensions.memusage.MemorySage'，'scrapy.extensions.logstats.logstats'，'scrapy.extensions.corestats.corestats']2017-10-07 00:42:28[scrapy.middleware]信息：启用的下载程序中间件：['scrapy.downloadermiddleware.robotstxt.RobotsTxtMiddleware'，'scrapy.downloadermiddleware.httpauth.HttpAuthMiddleware'，'scrapy.downloadermiddleware.downloaderTimeout.downloadermiddleware'，'scrapy.downloadermiddleware.DefaultHeadersMiddleware'，'scrapy.downloadermiddleware.useragent.UserAgentMiddleware'，'scrapy.downloader中间件.retry.RetryMiddleware'，scrapy.DownloaderMiddleware.redirect.MetaRefreshMiddleware'，scrapy.DownloaderMiddleware.httpcompression中间件'，scrapy.DownloaderMiddleware.redirect.RedirectMiddleware'，scrapy.DownloaderMiddleware.cookies中间件'，scrapy.DownloaderMiddleware.HttpProxyMiddlewar电子“，”scrapy.downloadermiddleware.stats.DownloaderStats']2017-10-07 00:42:28[scrapy.middleware]信息：启用的蜘蛛中间件：['scrapy.SpiderMiddleware.httperror.HttpErrorMiddleware'，'scrapy.SpiderMiddleware.offsite.OffsiteMiddleware'，'scrapy.SpiderMiddleware.referer.RefererMiddleware'，'scrapy.SpiderMiddleware.urllength.UrlLengthMiddleware'，'scrapy.SpiderMiddleware.depth.DepthMiddleware']2017-10-07 00:42:28[scrapy.middleware]信息：启用的项目管道：[]2017-10-07 00:42:28[scrapy.core.engine]信息：蜘蛛打开2017-10-07 00:42:28[scrapy.extensions.logstats]信息：爬网0页（以0页/分钟），爬网0项（以0项/分钟）2017-10-07 00:42:28[scrapy.core.engine]调试：爬网（404）https://forum.bodybuilding.nl/robots.txt

我在这里一直遵循指南：http://blog.florian-hopf.de/2014/07/scrapy-and-elasticsearch.html

更新1：

正如有人告诉我的，我需要将代码更新到新标准，我做到了，但结果没有改变：

from scrapy.spider import BaseSpider
from scrapy.selector import Selector
from bodybuilding.items import BodybuildingItem

class BodyBuildingSpider(BaseSpider):
    name = "bodybuilding"
    allowed_domains = ["forum.bodybuilding.nl"]
    start_urls = [
        "https://forum.bodybuilding.nl/fora/supplementen.22/"
    ]

    def parse(self, response):
        for sel in response.css('li.past.line.event-item'):
            item = BodybuildingItem()
            yield {'title': title.css('a.data-previewUrl::text').extract_first()}
            yield item

最后一次更新修复

经过一些很好的帮助，我终于可以使用这个蜘蛛了：

import scrapy

class BlogSpider(scrapy.Spider):
    name = 'bodybuilding'
    start_urls = ['https://forum.bodybuilding.nl/fora/supplementen.22/']

    def parse(self, response):
        for title in response.css('h3.title'):
            yield {'title': title.css('a::text').extract_first()}
            next_page_url = response.xpath("//a[text()='Volgende >']/@href").extract_first()
            if next_page_url:
                 yield response.follow(next_page_url, callback=self.parse)

Answer 1

您应该使用response.css（'li.pass.line.event item'），并且不需要responseSelector=Selector（response）。

此外，您正在使用的CSSli.past.line.event项目，不再有效，因此您需要首先根据最新的网页更新这些内容

要获取下一页URL，可以使用

>>> response.css("a.text::attr(href)").extract_first()
'fora/supplementen.22/page-2'

然后使用response.follow跟随这个相对的url

编辑-2：下一页处理更正

上一次编辑不起作用，因为在下一页上它与上一页url匹配，所以您需要使用下面的

next_page_url = response.xpath("//a[text()='Volgende >']/@href").extract_first()
if next_page_url:
   yield response.follow(next_page_url, callback=self.parse)

Edit-1：下一页处理

next_page_url = response.css("a.text::attr(href)").extract_first()
if next_page_url:
   yield response.follow(next_page_url, callback=self.parse)