使用selenium和python在抓取数据时迭代单击

Question

我正试图从这个网页上抓取数据

http://stats.espncricinfo.com/ci/engine/stats/index.html?class=1;团队=5人；模板=结果；类型=击球

我需要从表中复制内容并将其放入csv文件中，然后转到下一页并将这些页面的内容附加到同一文件中。我可以刮表，但是当我尝试使用SeleniumWebDriver的click循环单击next按钮时，它会转到下一页并停止。这是我的密码。

    driver = webdriver.Chrome(executable_path = 'path')
    url = 'http://stats.espncricinfo.com/ci/engine/stats/index.html?class=1;team=5;template=results;type=batting'
def data_from_cricinfo(url):
    driver.get(url)
    pgsource = str(driver.page_source)
    soup = BeautifulSoup(pgsource, 'html5lib')
    data = soup.find_all('div', class_ = 'engineTable')
    for tr in data:
        info = tr.find_all('tr')
             # grab data

    next_link = driver.find_element_by_class_name('PaginationLink')
    next_link.click()
data_from_cricinfo(url)

有没有无论如何点击下一个使用循环的所有页面，并将所有页面的内容复制到同一个文件？提前感谢。

Answer 1

您可以执行如下操作来遍历所有页面（通过Next按钮）并解析表中的数据：

from selenium import webdriver
from bs4 import BeautifulSoup

URL = 'http://stats.espncricinfo.com/ci/engine/stats/index.html?class=1;team=5;template=results;type=batting'

driver = webdriver.Chrome()
driver.get(URL)

while True:
    soup = BeautifulSoup(driver.page_source, 'html5lib')
    table = soup.find_all(class_='engineTable')[2]
    for info in table.find_all('tr'):
        data = [item.text for item in info.find_all("td")]
        print(data)

    try:
        driver.find_element_by_partial_link_text('Next').click()
    except:
        break

driver.quit()