如何从收到的响应中提取html代码？

Question

我可以通过Inspect工具找到HTML。我的代码返回的是html，我可以通过“查看页面源代码”工具看到它。因此，这意味着taht Javascript在嵌入代码之前会修改代码。但是，splash的角色是运行javascript并返回HTML，不是吗？？回应。body返回页面的源代码，而不返回我上面提到的响应所需的html代码。

import scrapy
from scrapy_splash import SplashRequest
from bs4 import BeautifulSoup

class NetherSplashSpider(scrapy.Spider):
    name = 'nether_splash'
    download_delay = 10

    custom_settings = {
        'SPLASH_URL': 'http://localhost:8050',
        'DOWNLOADER_MIDDLEWARES': {
            'scrapy_splash.SplashCookiesMiddleware': 723,
            'scrapy_splash.SplashMiddleware': 725,
            'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
        },
        'SPIDER_MIDDLEWARES': {
            'scrapy_splash.SplashDeduplicateArgsMiddleware': 100,
        },
        'DUPEFILTER_CLASS': 'scrapy_splash.SplashAwareDupeFilter',
    }

    def start_requests(self):
        yield SplashRequest(
            url='https://www.gaslicht.com/stroom-vergelijken?partial=true&aanbieders=eneco&skip=0&take=10&_=1559207102962',
            callback=self.parse,
        )


    def parse(self, response):



        filename = 'splash.html'
        with open(filename, 'wb') as f:
            f.write(response.body)

Answer 1

为了加载整个页面，您需要添加“wait”参数。尝试将“args={'wait'：1.0}”添加到您的请求中。

yield SplashRequest(
            url='https://www.gaslicht.com/stroom-vergelijken?partial=true&aanbieders=eneco&skip=0&take=10&_=1559207102962',
            callback=self.parse, args={'wait': 1.0}
        )