在抓取Web时定义用户代理字符串Selenium WebDriver

Question

初始化类时，我的实现代码如下。执行以下操作时发生错误。

FirefoxOptions options = new FirefoxOptions();
                         ^
SyntaxError: invalid syntax

请帮助初始化具有用户代理的 Web 驱动程序。我希望我能避免机器人自然刮擦。使用： “Mozilla/5.0 （视窗 NT 6.1;赢64;x64;rv：47.0） 壁虎/20100101火狐/47.0“ 作为代理

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.action_chains import ActionChains
from time import sleep
from bs4 import BeautifulSoup
import pandas as pd
class DataExtract:
    def __init__(self):
        FirefoxOptions options = new FirefoxOptions();
        String userAgent = ""Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0";
        options.addPreference("general.useragent.override",userAgent);
        WebDriver webDriver = new FirefoxDriver(options);
        options.add_argument('--allow-running-insecure-content')
        options.add_argument('--ignore-certificate-errors')
        self.driver = webdriver.PhantomJS(executable_path=r"C:/Pathtoexec/phantomjs/bin/phantomjs.exe")
        self.accept_untrusted_certs = True

Answer 1

经过一番冲浪，我发现了适合我的东西。请建议我如何检查是否相应地设置了Web代理。

from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
from time import sleep
from bs4 import BeautifulSoup
import pandas as pd

dcap = dict(DesiredCapabilities.PHANTOMJS)
dcap["phantomjs.page.settings.userAgent"] = (
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/53 "
"(KHTML, like Gecko) Chrome/15.0.87")

driver = webdriver.PhantomJS(desired_capabilities=dcap,executable_path=r"C:/PathtoExec/phantomjs.exe")
driver.get("https://www.webpagecontainingtables.com")
soup=BeautifulSoup(driver.page_source,'lxml')
table = soup.find_all('table')[4]
df = pd.read_html(str(table),header=0)
print(df)