使用Python/C#进行动态Google搜索

Question

我想检索谷歌搜索结果计数（106,000,000个结果（0.58秒））。 我用Python写了这个脚本:

import requests, webbrowser
from bs4 import BeautifulSoup

user_input = input("Type in query: ")
print("Googling..")
link = "http://www.google.com/search?q=" + user_input
google_search = requests.get(link)
print(google_search.headers)

#print it out as file

with open("Output.html", "w") as text_file:
    print("{}".format(google_search.text), file=text_file)

但当我查看文件时，结果统计数据就是不见了。 除了Google Search API，还有其他方法可以做到这一点吗？Google Search API很糟糕，因为它很有限，甚至无法得到正确的结果。 我写Python和C#是因为我两个都懂。

Answer 1

要从Google获得正确的结果，您必须设置正确的user-agenthttp头:

import requests
from bs4 import BeautifulSoup


user_input = input("Type in query: ")
print("Googling for keyword={}..".format(user_input))

params = {
    'q': user_input,
    'hl': 'en'   # <-- set hl=en to obtain english only results.
}
headers = {
    'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0'
}

google_search = requests.get("https://www.google.com/search", params=params, headers=headers)
soup = BeautifulSoup(google_search.content, 'html.parser')
print(soup.select_one('#result-stats').text)

打印（例如）:

Type in query: moon
Googling for keyword=moon..
About 1,720,000,000 results (0.99 seconds)