Scrapy输出JSON或CSV

Question

FEED_EXPORT_ENCODING = 'utf-8'

import datetime
now = datetime.datetime.now ()
formatted = now.strftime ("%Y%m%d_%H%M")
FEED_URI = f'\\C:\\Users\\Acer\\Desktop\\{formatted}.csv'
FEED_TYPE = 'csv'

# -*- coding: utf-8 -*-
import scrapy
import datetime


class SpecialOffersSpider(scrapy.Spider):
    name = 'special_offers'
    allowed_domains = ['www.tinydeal.com']

    def start_requests(self):
        yield scrapy.Request(url='https://www.tinydeal.com/specials.html', callback=self.parse, headers={
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36'
        })

    def parse(self, response):
        for product in response.xpath("//ul[@class='productlisting-ul']/div/li"):
            yield {
                'title': product.xpath(".//a[@class='p_box_title']/text()").get(),
                'url': response.urljoin(product.xpath(".//a[@class='p_box_title']/@href").get()),
                'discounted_price': product.xpath(".//div[@class='p_box_price']/span[1]/text()").get(),
                'original_price': product.xpath(".//div[@class='p_box_price']/span[2]/text()").get(),
                'User-Agent': response.request.headers['User-Agent'].decode('utf-8'),
                'datetime': datetime.datetime.now().strftime("%Y%m%d %H%M")

            }

        next_page = response.xpath("//a[@class='nextPage']/@href").get()

        if next_page:
            yield scrapy.Request(url=next_page, callback=self.parse, headers={
                'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36'
            })

scrapy crawl special_offers

当我导出CSV时，数据与我使用EXCEL打开时预期的不同

CSV数据示例{“标题”：“树莓皮3型号B和树莓皮2 E-524988的ABS塑料外壳”，“URL”：“https://www.tinydeal.com/abs-plastic-case-for-raspberry-pi-3-model-b-raspberry-pi-2-p163950.html”，“discounted_price”：“R$12.74”，“original_price”：“R$13.66”，“user-agent”：“mozilla/5.0(Windows NT 10.0；Win64；x64)AppleWebKit/537.36（KHTML,like Gecko）Chrome/76.0.3809.100 Safari/537.36“，”日期“：”20200420 2330“}{”标题“：”3M 9001 KN90防尘口罩呼吸器防尘PM2.5工业建筑polle ；RTH-562440“，”URL“：”https://www.tinydeal.com/3M-9001-KN90-Dust-Masks-Rephilator-Anti-Dust-PM25-Industrial-Construction-Polle-P-179487.html“，”Discounted_Price“：”R$10.29“”用户代理“：”Mozilla/5.0（Windows NT 10.0；Win64；x64)AppleWebKit/537.36（KHTML,like Gecko）Chrome/76.0.3809.100 Safari/537.36“，”日期“：”20200420 2330“}{”标题“：”2合1复古蓝莱茵石项链+耳环珠宝套装 ；DJA-562974“，”URL“：”https://www.tinydeal.com/2-in-1-Vintage-Blue-Rhinestone-Necklace-Earring-Jewelry-Set-P-180097.html“，”Discounted_Price“：”R$11.77“，”Original_Price“：”R$30.77“，”用户代理“：”Mozilla/5.0（Windows NT 10.0；Win64；x64)AppleWebKit/537.36（KHTML,like Gecko）Chrome/76.0.3809.100 Safari/537.36“，”DateTime“：”20200420 2330“}{”title“：”64GB USB 2.0闪存驱动器USB笔驱动器U盘 ；EFM-561923“，”URL“：”https://www.tinydeal.com/64GB-USB-20-Flash-Drive-USB-Pen-Drive-U-Disk-P-178875.html“，”Discounted_Price“：”R$34.83“，”Original_Price“：”R$99.43“，”User-Agent“.0（Windows NT 10.0；Win64；x64)AppleWebKit/537.36（KHTML,like Gecko）Chrome/76.0.3809.100 Safari/537.36“,”DateTime“：”20200420 2330“}

JSON数据示例

{“标题”：“树莓皮3型号B和树莓皮2 E-524988的ABS塑料外壳”，“URL”：“https://www.tinydeal.com/abs-plastic-case-for-raspberry-pi-3-model-b-raspberry-pi-2-p163950.html”，“discounted_price”：“R$12.74”，“原始价格”：“R$13.66”，“用户代理”：“mozilla/5.0(Windows NT 10.0；Win64；x64)AppleWebKit/537.36（KHTML,like Gecko）Chrome/76.0.3809.100 Safari/537.36“，”日期“：”20200420 2329“}{”标题“：”3M 9001 KN90防尘口罩呼吸器防尘PM2.5工业建筑polle ；RTH-562440“，”URL“：”https://www.tinydeal.com/3M-9001-KN90-Dust-Masks-Rephilator-Anti-Dust-PM25-Industrial-Construction-Polle-P-179487.html“，”Discounted_Price“：”R$10.29“”用户代理“：”Mozilla/5.0（Windows NT 10.0；Win64；x64)AppleWebKit/537.36（KHTML,like Gecko）chrome/76.0.3809.100 safari/537.36“，”datetime“：”20200420 2329“}{”title“：”2合1复古蓝色莱茵石项链+耳环珠宝套装 ；DJA-562974“，”URL“：”https://www.tinydeal.com/2-in-1-vintage-blue-rhinestone-necklace-earring-jewelry-set-p-180097.html“，”discounted_price“：”r$11.77“，”original_price代理“：”Mozilla/5.0（Windows NT 10.0；Win64；x64)AppleWebKit/537.36（KHTML,like Gecko）Chrome/76.0.3809.100 Safari/537.36“,”DateTime“：”20200420 2329“}

谁能告诉我，在这些输出中，哪里出了问题？

Answer 1

你是怎么得到刮下来的数据的？从你展示的来看，我怀疑你是从终端机上复制的。是这样吗？如果是，可以使用以下命令将其直接保存到文件中：

Scrapy crawl special_offers-o<保存文件的位置>/special_offers.json

希望这能解决你的问题。拜托，让我知道。