Python Selenium Web抓取隐藏Div

Question

正如标题所示，我正在尝试使用Selenium从网站（示例）中获取一些数据，但是我在从Pro结果表中获取隐藏在每一行中的数据时遇到了问题，即单击Show Details按钮（）时显示的数据。

这是我的代码：

from bs4 import BeautifulSoup

from selenium import webdriver

# Set some Selenium Options
options = webdriver.ChromeOptions()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')

# Webdriver
wd = webdriver.Chrome('chromedriver',options=options)

# URL
url = 'https://www.tapology.com/fightcenter/fighters/30449-sultan-aliev'

# Load URL
wd.get(url)

# Get HTML
soup = BeautifulSoup(wd.page_source, 'html.parser')

# All rows of the Pro Record table 
rows = soup.findAll('div', {'class': 'result'})

print(len(rows)) 

# [Out] 18

# Try to find all hidden data
hidden = soup.findAll('div', {'class': 'detail tall'})

print(hidden)

# [Out] []

正如您所看到的，我可以很容易地获取表中的行，但是当我试图获取隐藏数据时，我就是找不到获取它的方法。

我对Selenium也不是很熟悉，所以欢迎提供任何指导。

Answer 1

如果你想使用selenium只尝试以下代码。您需要单击每个展开按钮以获取下一个表信息。然后使用element.get_attribute（"text Content"）

代码：

from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait

driver=webdriver.Chrome()
driver.get("https://www.tapology.com/fightcenter/fighters/30449-sultan-aliev")
WebDriverWait(driver,10).until(EC.element_to_be_clickable((By.CSS_SELECTOR,"span.closebutton_closeButton--3abym"))).click()
tablerecords=WebDriverWait(driver,20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR,"div.result")))
print(len(tablerecords))
for row in range(len(tablerecords)):
    tablerecords = WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div.result")))
    try:
        expand_btn=tablerecords[row].find_element_by_xpath(".//div[@class='more']/i")
        driver.execute_script("arguments[0].click();", expand_btn)
        time.sleep(2)
        hiddenelements=tablerecords[row].find_element_by_xpath("./following-sibling::div[1]").get_attribute('textContent')
        print(hiddenelements)
    except:
        continue

输出：

25
Billing:Preliminary CardDuration:3 x 5 Minute RoundsWeight:Welterweight · 170 lbs (77.1 kg) · Weigh-In 170.0 lbs (77.1 kgs)Odds:-120 · Near EvenReferee:Leon Roberts
UFC on ESPN+ 7
UFC 230: Cormier vs. Lewis· Aliev Injury
Billing:Preliminary CardDuration:3 x 5 Minute RoundsWeight:Welterweight · 170 lbs (77.1 kg) · Weigh-In 171.0 lbs (77.6 kgs)Odds:+250 · Moderate UnderdogReferee:Osiris Maia
UFC on FOX 26· Aliev Injury
Billing:Preliminary CardDuration:3 x 5 Minute RoundsWeight:Welterweight · 170 lbs (77.1 kg) · Weigh-In 171.0 lbs (77.6 kgs)Odds:+135 · Slight UnderdogReferee:Ed CollantesDisclosed Pay:$20,000 ($10K Base, $10K Bonus)
UFC 202: Diaz vs. McGregor 2· Aliev Injury
Billing:Preliminary CardDuration:3 x 5 Minute RoundsWeight:Welterweight · 170 lbs (77.1 kg) · Weigh-In 170.0 lbs (77.1 kgs)Odds:-180 · Slight FavoriteReferee:Bobby RehmanUFC on FOX 14 Performance of the Night
Billing:Main CardDuration:3 x 5 Minute RoundsWeight:Light Heavyweight · 205 lbs (93.0 kg)
Billing:Main CardDuration:3 x 5 Minute RoundsWeight:Middleweight · 185 lbs (84.0 kg) · Weigh-In 185.4 lbs (84.1 kgs)Referee:Valentin Tarasov
Billing:Main CardDuration:3 x 5 Minute RoundsWeight:Middleweight · 185 lbs (83.9 kg) · Weigh-In 185.8 lbs (84.3 kgs)Odds:-350 · Moderate FavoriteReferee:Herb Dean
Billing:Preliminary CardDuration:3 x 5 Minute RoundsWeight:Middleweight · 185 lbs (83.9 kg) · Weigh-In 185.5 lbs (84.1 kgs)Odds:+145 · Slight UnderdogReferee:Joseph Hawes
Billing:Main CardDuration:3 x 5 Minute RoundsWeight:Light Heavyweight · 205 lbs (93.0 kg)
Billing:Main CardDuration:2 x 5 Minute Rounds
Billing:Main CardDuration:3 x 5 Minute Rounds
Title Bout:Tournament ChampionshipBilling:Main CardDuration:3 x 5 Minute RoundsWeight:Light Heavyweight · 205 lbs (93.0 kg)
ProFC 39: Global Grand Prix (Stage 6)· Omari Akhmedov injury
Title Bout:Tournament ChampionshipBilling:Main CardDuration:2 x 5 Minute RoundsWeight:Light Heavyweight · 205 lbs (93.0 kg)
Billing:Preliminary CardDuration:3 x 5 Minute RoundsWeight:Light Heavyweight · 205 lbs (93.0 kg)
Billing:Main CardDuration:2 x 5 Minute RoundsWeight:Light Heavyweight · 205 lbs (93.0 kg)
Title Bout:Tournament ChampionshipBilling:Main EventDuration:2 x 5 Minute RoundsWeight:Light Heavyweight · 205 lbs (93.0 kg)

Answer 2

也许您不需要从HTML中提取数据。Chrome开发者工具中的一个快速检查告诉我，这个站点有API来查询数据，但是你需要使用完全相同的请求头。

JSON格式的internal_fighters

JSON格式的内部\u排名\u项目

该问题的另一个替代方法是模拟按钮上的“单击”操作。

隐藏div的问题是当用户单击（）按钮时，div标记是动态添加的。

# click submit button
submit_button = wd.find_elements_by_xpath('//*[@id="fighterRecord"]/section[1]/ul/li[1]/div/div[4]/i')[0]
submit_button.click()

Answer 3

包含所需信息的json正在使用js请求从tapology api获取
要检索此信息，请安装seleniumwire并使用：

from seleniumwire import webdriver
import requests
# ...
driver = webdriver.Firefox()
driver.scopes = [ 'api.tapology.com'] # filter api.tapology.com requests only 
driver.get('https://www.tapology.com/fightcenter/fighters/30449-sultan-aliev')

for request in driver.requests:
    print(request.path)
    r = requests.get(request.path, headers=request.headers)
    print(r.json())  # the info you need is here

https://api.tapology.com/v1/internal_ranking_items/47211352261#排名数据https://api.tapology.com/v1/internal_fighters/472130449#战斗机数据