在访问第一个元素--Webscraping Selenium Python之后，无法在循环中通过xpaths访问其余元素

Question

我试图从sciencedirect网站上搜索数据。我试图通过创建一个xpaths列表并循环这些xpaths来一个接一个地访问journal issues，从而自动化刮痧过程。当im运行该循环时，im在访问第一个日记之后无法访问其余的元素。这个过程对我在另一个网站工作，但不是在这个。

我还想知道除了这个过程之外，还有什么更好的方法来访问这些元素。

#Importing libraries
 import requests
 import os
 import json
 from selenium import webdriver
 import pandas as pd
 from bs4 import BeautifulSoup  
 import time
 import requests
 from time import sleep

 from selenium.webdriver.common.by import By
 from selenium.webdriver.support.ui import WebDriverWait
 from selenium.webdriver.support import expected_conditions as EC

 #initializing the chromewebdriver|
 driver=webdriver.Chrome(executable_path=r"C:/selenium/chromedriver.exe")

 #website to be accessed
 driver.get("https://www.sciencedirect.com/journal/journal-of-corporate-finance/issues")

 #generating the list of xpaths to be accessed one after the other
 issues=[]
 for i in range(0,20):
     docs=(str(i))
     for j in range(1,7):
         sets=(str(j))
         con=("//*[@id=")+('"')+("0-accordion-panel-")+(docs)+('"')+("]/section/div[")+(sets)+("]/a")
         issues.append(con)

 #looping to access one issue after the other
 for i in issues:
     try:
         hat=driver.find_element_by_xpath(i)
         hat.click()
         sleep(4)
         driver.back()
     except:
         print("no more issues",i)

Answer 1

要从sciencedirect网站https://www.sciencedirect.com/journal/journal-of-composition-finance/issures中获取数据，可以执行以下步骤：

>

先打开所有的手风琴。

然后使用Ctrl+click()在adjustant选项卡中打开每个问题。

  from selenium import webdriver
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.support import expected_conditions as EC
  from selenium.webdriver.common.action_chains import ActionChains
  from selenium.webdriver.common.keys import Keys

  options = webdriver.ChromeOptions() 
  options.add_argument("start-maximized")
  options.add_experimental_option("excludeSwitches", ["enable-automation"])
  options.add_experimental_option('useAutomationExtension', False)
  driver = webdriver.Chrome(options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
  driver.get('https://www.sciencedirect.com/journal/journal-of-corporate-finance/issues')
  accordions = WebDriverWait(driver, 30).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "li.accordion-panel.js-accordion-panel>button.accordion-panel-title>span")))
  for accordion in accordions:
      ActionChains(driver).move_to_element(accordion).click(accordion).perform()
  issues = WebDriverWait(driver, 30).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "a.anchor.js-issue-item-link.text-m span.anchor-text")))
  windows_before  = driver.current_window_handle
  for issue in issues:
      ActionChains(driver).key_down(Keys.CONTROL).click(issue).key_up(Keys.CONTROL).perform()
      WebDriverWait(driver, 10).until(EC.number_of_windows_to_be(2))
      windows_after = driver.window_handles
      new_window = [x for x in windows_after if x != windows_before][0]
      driver.switch_to_window(new_window)
      WebDriverWait(driver, 30).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "a#journal-title>span")))
      print(WebDriverWait(driver, 30).until(EC.visibility_of_element_located((By.XPATH, "//h2"))).get_attribute("innerHTML"))
      driver.close()
      driver.switch_to_window(windows_before)
  driver.quit()

  Institutions, Governance and Finance in a Globally Connected Environment
  Volume 58
  Corporate Governance in Multinational Enterprises
  .
  .
  .

您可以在以下内容中找到一些相关的详细讨论：

• 如何使用Selenium WebDriver的控件+单击，在主选项卡中，在同一窗口的新选项卡中打开嵌入在webelement中的链接
• 如何在webtable中打开多个hrefs
• 在Python中使用Selenium WebScraping JavaScript呈现的内容
• StaleElementReferenceException，即使在使用Web搜索从维基百科收集数据时添加了等待
• 如何通过Python使用Selenium在新选项卡中打开网站中的每个产品