请求url： h中缺少方案

Question

我是一个初学者在刮，python。我试图在scrapinghub中部署spider代码，但遇到了以下错误。下面是代码。

import scrapy
from bs4 import BeautifulSoup,SoupStrainer
import urllib2
from scrapy.selector import Selector
from scrapy.http import HtmlResponse
import re
import pkgutil
from pkg_resources import resource_string
from tues1402.items import Tues1402Item

data = pkgutil.get_data("tues1402","resources/urllist.txt")
class SpiderTuesday (scrapy.Spider):     
    name = 'tuesday'
    self.start_urls = [url.strip() for url in data]
    def parse(self, response):
       story = Tues1402Item()
       story['url'] = response.url
       story['title'] = response.xpath("//title/text()").extract()
       return story

是我spider.py代码

import scrapy
class Tues1402Item(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    title = scrapy.Field()
    url = scrapy.Field() 

就是这些项目。py代码和

from setuptools import setup, find_packages
setup(
    name         = 'tues1402',
    version      = '1.0',
    packages     = find_packages(),
    entry_points = {'scrapy': ['settings = tues1402.settings']},
    package_data = {'tues1402':['resources/urllist.txt']},
    zip_safe = False,
)

这是设置。py代码。

以下是错误。

回溯（最近一次调用）：文件“/usr/local/lib/python2.7/site packages/scrapy/core/engine.py”，第126行，在启动请求的第70行，文件“/usr/local/lib/python2.7/site packages/scrapy/spider/init.py”。从url（url）文件“/usr/local/lib/python2.7/site packages/scrapy/spider/init.py”中发出请求，第73行，从url返回请求（url，dont\u filter=True）文件“/usr/local/lib/python2.7/site packages/scrapy/http/Request/init.py”中发出请求，第25行，在init self中_设置url（url）文件“/usr/local/lib/python2.7/site packages/scrapy/http/request/init.py”，第57行，在“设置url提升值错误”（“请求url中缺少方案：%s”%self.\u url）值错误：请求url中缺少方案：h

先谢谢你

Answer 1

您的错误意味着urlh不是有效的url。你应该打印出你的self.start_urls，看看你有什么网址，你最有可能有一个字符串h作为你的第一个网址。

您的爬行器似乎在文本中而不是在URL列表中进行迭代：

data = pkgutil.get_data("tues1402","resources/urllist.txt")
class SpiderTuesday (scrapy.Spider):     
    name = 'tuesday'
    self.start_urls = [url.strip() for url in data]

假设您在urlist中使用一些分隔符存储URL。txt文件您应该拆分该文件：

# assuming file has url in every line
self.start_urls = [url.strip() for url in data.splitlines()]