Python Selenium-调整pause_time以在无限页面中向下滚动

Question

我试图刮所有的链接可用在一个无限的页面，向下滚动，并获得新的链接可用。但是，time.sleep（）不允许在一次又一次向下滚动之前暂停驱动程序一段合理的时间。

for i in range(1,20):
    time.sleep(i)

下面是我在“如何在Python中使用selenium webdriver滚动web页面”中使用的一个建议中使用的代码：

from selenium import webdriver

scroll_pause_time = 5
scraped_links = []

driver = webdriver.Chrome(executable_path=driver_path)
driver.get(url)
links = driver.find_elements_by_xpath(links_filepath)
for link in links:
    if link not in scraped_links:
        scraped_links.append(link)
        print(link)
last_height = driver.execute_script("return document.body.scrollHeight")
while True:
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    time.sleep(scroll_pause_time)
    new_height = driver.execute_script("return document.body.scrollHeight")
    if new_height == last_height:
        break
    last_height = new_height
    links = driver.find_elements_by_xpath(links_filepath)
    for link in links:
        if link not in scraped_links:
            scraped_links.append(link)
            print(link)

在20-30次迭代之后，由于time.sleep（）与网页的刷新速度相比太低，代码会中断。

Answer 1

如果您不想每次猜测加载页面需要多长时间，并随机设置几秒钟的睡眠时间，则可以使用显式等待。示例：

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Firefox()
driver.get("http://somedomain/url_that_delays_loading")
try:
    element = WebDriverWait(browser, 10).until(
                                    EC.presence_of_element_located((By.ID, "myDynamicElement"))
                                )
except common.exceptions.TimeoutException:
    print('TimeoutException')
finally:
    driver.quit()

# do what you want after necessary elements are loaded

当time.sleep（）与网页刷新速度相比变得太低时，这将解决问题。