StaleElementReferenceException即使在使用Web Scraping从维基百科收集数据时添加了等待

Question

我是上网的新手。请原谅我的愚蠢错误，如果有的话。

我一直在做一个项目，我需要一个电影列表作为我的数据。我正试图通过网络搜索从维基百科中收集数据。

下面是我的代码：

def MoviesList(years, driver):
    for year in years:
        driver.implicitly_wait(150)
        year.click()
        table = driver.find_element_by_xpath('/html/body/div[3]/div[3]/div[5]/div[1]/table[2]/tbody')
        movies = table.find_elements_by_xpath('tr/td[1]/i/a')
        for movie in movies:
            print(movie.text)
        driver.back()
years = driver.find_elements_by_partial_link_text('List of Bollywood films of')
del years[:2]
MoviesList(years, driver)

试图从此页面获取年份列表并将其存储在years变量中。然后，我循环了所有这些年，试图提取出年度最佳10部电影。见此参考

产出：

Tanhaji
Baaghi 3
...
...
Panga
# Top movies of the year 2020
selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document (from line year.click())

Tanhaji  
...
...
War  # First movie of the year 2019
Saaho
...
...
Vikram Urvashi  # Last movie of the year 1920
# Top movies of the year from 2020 to 1920

Answer 1

为了使用Selenium和python从wikipedia的宝莱坞电影列表中收集数据，您必须为visibility_of_all_elements_locited()导入WebDriverWait，您可以使用以下定位器策略：

注：作为示范，此计划仅限于收集前三（3）年全球总票房最高的电影

>

代码块：

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

options = webdriver.ChromeOptions() 
options.add_argument("start-maximized")
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get("https://en.wikipedia.org/wiki/Lists_of_Bollywood_films")
parent_window  = driver.current_window_handle
years = [my_elem.get_attribute("href") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.PARTIAL_LINK_TEXT, "List of Bollywood films of")))[2:5]]
print(years)
for year in years:
    driver.execute_script("window.open('" + year +"')")
    WebDriverWait(driver, 10).until(EC.number_of_windows_to_be(2))
    windows_after = driver.window_handles
    new_window = [x for x in windows_after if x != parent_window][0]
    driver.switch_to_window(new_window)
    print([my_elem.get_attribute("innerHTML") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//table/caption//following::tbody[1]//td/i/a")))])
    driver.close()
    driver.switch_to_window(parent_window)
driver.quit()

['Tanhaji', 'Baaghi 3', 'Street Dancer 3D', 'Shubh Mangal Zyada Saavdhan', 'Malang', 'Chhapaak', 'Love Aaj Kal', 'Jawaani Jaaneman', 'Thappad', 'Panga']
['War', 'Saaho', 'Kabir Singh', 'Uri: The Surgical Strike', 'Bharat', 'Good Newwz', 'Mission Mangal', 'Housefull 4', 'Gully Boy', 'Dabangg 3']
['Sanju', 'Padmaavat', 'Andhadhun', 'Simmba', 'Thugs of Hindostan', 'Race 3', 'Baaghi 2', 'Hichki', 'Badhaai Ho', 'Pad Man']

• 如何在webtable中打开多个hrefs
• 在Python中使用Selenium WebScraping JavaScript呈现的内容
• 在访问第一个元素--Webscraping Selenium Python之后，无法通过循环中的xpaths访问其余元素
• 如何通过Python使用Selenium在新选项卡中打开网站中的每个产品