BS4和刮取更新表

Question

我正试图从whoscored.com（下面的变量根链接）中删除到EPL所有球员的链接，这里是代码：

from bs4 import BeautifulSoup
from selenium import webdriver
root = "https://www.whoscored.com/Regions/252/Tournaments/2/Seasons/6335/Stages/13796/PlayerStatistics/England-Premier-League-2016-2017"
driver = webdriver.PhantomJS()
driver.get(root)
page = driver.page_source
soup = BeautifulSoup(page, "html.parser")
players = soup.find("div", {'id':'statistics-table-summary'})

print(players)

如果你进入这个页面，你会看到一个玩家列表和一个下一个按钮来显示下一个10个玩家（其中有284个在29页）我想要的输出：保存链接到每个10个玩家的配置文件，然后移动到下一个页面与下一个10个玩家直到完成

Answer 1

在获取.page_source:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# ...

driver.get(root)

# wait for at least one player to be present in the statistics table
wait = WebDriverWait(driver, 10)
wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "#statistics-table-summary .player-link")))

page = driver.page_source
driver.close()

# ...