python请求对url进行编码

Question

我试图通过参数在谷歌搜索，它的工作，当我搜索一个词，但一个我做空间它坏了我知道有一种方法来编码网址。

import urllib.request
from urllib.parse import urlencode, quote_plus
from fake_useragent import UserAgent
import time
import requests
from bs4 import BeautifulSoup

keyword = "host free"
url = "https://www.google.co.il/search?q=%s" % (keyword)
print(url)

thepage = urllib.request.Request(url, headers=request_headers)
page = urllib.request.urlopen(thepage)

//Continue...

回溯：

https://www.google.co.il/search?q=host free
Traceback (most recent call last):
  File "C:\Users\Maor Ben Lulu\Desktop\Maor\Python\google\Google_Bot_new.py", line 42, in <module>
    page = urllib.request.urlopen(thepage)
  File "C:\Program Files (x86)\Python37-32\lib\urllib\request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "C:\Program Files (x86)\Python37-32\lib\urllib\request.py", line 531, in open
    response = meth(req, response)
  File "C:\Program Files (x86)\Python37-32\lib\urllib\request.py", line 641, in http_response
    'http', request, response, code, msg, hdrs)
  File "C:\Program Files (x86)\Python37-32\lib\urllib\request.py", line 569, in error
    return self._call_chain(*args)
  File "C:\Program Files (x86)\Python37-32\lib\urllib\request.py", line 503, in _call_chain
    result = func(*args)
  File "C:\Program Files (x86)\Python37-32\lib\urllib\request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: Bad Request
[Finished in 0.7s with exit code 1]
[shell_cmd: python -u "C:\Users\Maor Ben Lulu\Desktop\Maor\Python\google\Google_Bot_new.py"]
[dir: C:\Users\Maor Ben Lulu\Desktop\Maor\Python\google]
[path: C:\Program Files (x86)\Python37-32\Scripts\;C:\Program Files (x86)\Python37-32\;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;D:\Program Files\Git\cmd;C:\Users\Maor Ben Lulu\AppData\Local\Microsoft\WindowsApps;]

还有一次，我用希伯来文写道：

UnicodeEncodeError:“ascii”编解码器无法对位置14-18中的字符进行编码：序号不在范围内（128）

Answer 1

请求库可以为您做Gahan提到的。通过字典将查询参数和标头传递到request.get（）：

headers = {
    'User-agent':
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582'
     # other headers (if needed)
}

params = {
  'q': 'how to create minecraft server',   # query 
  'gl': 'us',                              # country to search from (United States in this case)
  'hl': 'en'                               # language
   # other params (if needed)
}

html = requests.get('https://www.google.com/search', headers=headers, params=params)
soup = BeautifulSoup(html.text, 'lxml')

在线IDE中的代码和示例：

from bs4 import BeautifulSoup
import requests, lxml

headers = {
    'User-agent':
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582'
}

params = {
  'q': 'how to create minecraft server',
  'gl': 'us',
  'hl': 'en',
}

html = requests.get('https://www.google.com/search', headers=headers, params=params).text
soup = BeautifulSoup(html, 'lxml')

for result in soup.select('.tF2Cxc'):
  title = result.select_one('.yuRUbf').text
  link = result.select_one('.yuRUbf a')['href']
  print(title, link, sep='\n')

---------
'''
How to Setup a Minecraft: Java Edition Server – Home
https://help.minecraft.net/hc/en-us/articles/360058525452-How-to-Setup-a-Minecraft-Java-Edition-Server
Minecraft Server Download
https://www.minecraft.net/en-us/download/server
Setting Up Your Own Minecraft Server - iD Tech
https://www.idtech.com/blog/creating-minecraft-server
Tutorials/Setting up a server - Minecraft Wiki
https://minecraft.fandom.com/wiki/Tutorials/Setting_up_a_server
# other results
'''

或者，您也可以使用SerpApi的Google Organic Results API实现同样的功能。这是一个免费的付费API。

你的情况的不同之处在于，如果问题不仅仅是在请求标题中传递用户代理，你不必花时间弄清楚这些事情或者如何绕过谷歌的块。

相反，您需要使用所需的参数（params）在结构化JSON上迭代，并获取所需的数据。

要集成的示例代码：

import os
from serpapi import GoogleSearch

params = {
  "engine": "google",
  "q": "tesla",
  "hl": "en",
  "gl": "us",
  "api_key": os.getenv("API_KEY"),
}

search = GoogleSearch(params)
results = search.get_dict()

# scrapes first page of Google results
for result in results["organic_results"]:
  print(result['title'])
  print(result['link'])


---------
'''
How to Setup a Minecraft: Java Edition Server – Home
https://help.minecraft.net/hc/en-us/articles/360058525452-How-to-Setup-a-Minecraft-Java-Edition-Server
Minecraft Server Download
https://www.minecraft.net/en-us/download/server
Setting Up Your Own Minecraft Server - iD Tech
https://www.idtech.com/blog/creating-minecraft-server
Tutorials/Setting up a server - Minecraft Wiki
https://minecraft.fandom.com/wiki/Tutorials/Setting_up_a_server
# other results
'''

免责声明，我为SerpApi工作。

Answer 2

有一种方法可以使用urllib对url进行编码。作语法分析报价，但有一个请求模块在所有此类情况下都非常有用，您可以按如下方式使用它：

import requests
base_url = 'https://www.google.co.il/search'
res = requests.get(base_url, params={'q': 'host free'})  # query parameter and value in dict format to be passed as params kwarg

如上所示，您可以将查询参数作为关键字参数传递