Scrapy Splash为任何站点返回403

Question

由于某些原因，在使用Splash时，我有403用于任何请求。我做错了什么？

以下https://github.com/scrapy-plugins/scrapy-splash我设置了所有设置：

SPLASH_URL = 'http://localhost:8050'
DOWNLOADER_MIDDLEWARES = {
    'scrapy_splash.SplashCookiesMiddleware': 723,
    'scrapy_splash.SplashMiddleware': 725,
    'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
}
SPIDER_MIDDLEWARES = {
    'scrapy_splash.SplashDeduplicateArgsMiddleware': 100,
}
DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'
HTTPCACHE_STORAGE = 'scrapy_splash.SplashAwareFSCacheStorage'

开始和docker吵架

sudo docker run-p 8050:8050刮水/飞溅

蜘蛛代码：

import scrapy

from scrapy import Selector
from scrapy_splash import SplashRequest


class VestiaireSpider(scrapy.Spider):
    name = "vestiaire"
    base_url = "https://www.vestiairecollective.com"
    rotate_user_agent = True

    def start_requests(self):
        urls = ["https://www.vestiairecollective.com/men-clothing/jeans/"]
        for url in urls:
            yield SplashRequest(url=url, callback=self.parse, meta={'args': {"wait": 0.5}})

    def parse(self, response):
        data = Selector(response)
        category_name = data.xpath('//h1[@class="campaign campaign-title clearfix"]/text()').extract_first().strip()
        self.log(category_name)

然后我运行蜘蛛：

搔痒爬行试验

并返回403请求url：

2017-12-19 22:55:17[scrapy.utils.log]信息：scrapy 1.4.0启动（机器人程序：爬虫）2017-12-19 22:55:17[scrapy.utils.log]信息：覆盖的设置：{'DUPEFILTER_类'：'scrapy_splash.SplashAwareDupeFilter'，'CONCURRENT_请求：'NEWSPIDER_模块'：'crawler.SPIDER'，'SPIDER_模块'：['rs.SPIDER']，'ROBOTSTXT_-obe'：True，'COOKIES_-ENABLED'：False，'BOT_-NAME'：'crawler'，'HTTPCACHE_-STORAGE'：'scrapy_-splash.SplashAwareFSCacheStorage'}2017-12-19 22:55:17[scrapy.middleware]信息：启用的扩展：['scrapy.extensions.telnet.TelnetConsole'，'scrapy.extensions.logstats.logstats'，'scrapy.extensions.memusage.MemoryUsage'，'scrapy.extensions.corestats.corestats']2017-12-19 22:55:17[scrapy.middleware]信息：启用的下载中间件：['scrapy.downloadermiddleware.robotstxt.RobotsTxtMiddleware'，'scrapy.downloadermiddleware.httpauth.HttpAuthMiddleware'，'scrapy.downloadermiddleware.downloaderTimeout.downloadermiddleware'，'scrapy.downloadermiddleware.DefaultHeadersMiddleware'，'scrapy.downloadermiddleware.useragent.UserAgentMiddleware'，'scrapy.downloader中间件.retry.RetryMiddleware'，scrapy.Downloadermiddleware.redirect.MetaRefreshMiddleware'，scrapy.Downloadermiddleware.redirect.RedirectMiddleware'，scrapy_splash.SplashCookiesMiddleware'，scrapy_splash.SplashMiddleware'，scrapy.Downloadermiddleware.httpproxy中间件'，scrapy.Downloadermiddleware'，scrapypressionMiddleware“，”scrapy.downloaderMiddleware.stats.DownloaderStats“]2017-12-19 22:55:17[scrapy.middleware]信息：启用的蜘蛛中间件：[scrapy.SpiderMiddleware.httperror.HttpErrorMiddleware'，scrapy_splash.splash重复数据消除中间件'，scrapy.SpiderMiddleware.offsite.OffsiteIddleware'，scrapy.SpiderMiddleware.referer.RefererMiddleware'，scrapy.SpiderMiddleware.UrlLengthMiddleware'，scrapy.SpiderMiddleware.DepthMiddleware']2017-12-19 22:55:17[scrapy.pipelines.images.ImagesPipeline']2017-12-19 22:55:17[scrapy.core.engine]信息：Spider打开2017-12-19 22:55:17[scrapy.extensions.logstats]信息：爬网0页（以0页/分钟的速度），爬网0项（以0项/分钟的速度）2017-12-19 22:55:17[scrapy.extensions.telnet]调试：Telnet控制台监听127.0.0.1:6023 2017-12-19 22:55:20[scrapy.core.engine]调试：爬网（200）https://www.vestiairecollective.com/robots.txt

Answer 1

问题出在用户代理中。许多网站需要它来访问。访问该网站并避免禁令的最简单方法是使用此库随机化用户代理。https://github.com/cnu/scrapy-random-useragent