无法使用BeautifulSoup find_all或pandas.read_html函数从表中刮取数据

Question

import mechanicalsoup
from bs4 import BeautifulSoup
import pandas

root_url="https://www.pro-football-reference.com"

#Opens the main pro-football page with list of 2018 games
browser=mechanicalsoup.StatefulBrowser()
browser.open("https://www.pro-football-reference.com/years/2018/games.htm")
main_page = browser.get_current_page()
browser.close()
data=main_page.find_all("tr")

#Finds the link to box-score information for the first game.  
#Will iterate over all games in a for loop later on.
box_score_tag = data[1].find("td",{"data-stat":"boxscore_word"})
box_score_link = root_url+box_score_tag.a.get("href")

#Opens the box-score page for the first game
browser2=mechanicalsoup.StatefulBrowser()
browser2.open(box_score_link)
boxscorepage=browser2.get_current_page()
browser2.close()

#attempt to scrape all the tables using Pandas
tables = pandas.read_html(box_score_link)
print(len(tables))

BeautifulSoup尝试（替换最后3行）

#attempt to scrape the specific table in question using BeautifulSoup
game_info = boxscorepage.find_all("table",{"id":"game_info"})
print(game_info)

这不会输出任何东西--在这个页面上，找到一些标签（divs、spans等）可以工作，但另一些则不行。在本例中，它没有按照预期找到带有game_info的表。

Answer 1

不需要用硒。这些表可以在HTML的注释中找到。只要把那些拔出来，你就能抓到所有的桌牌。该特定表是第二个表（在索引位置1）。

代码：

import requests
from bs4 import BeautifulSoup
from bs4 import Comment
import pandas as pd


url = 'https://www.pro-football-reference.com/boxscores/201809060phi.htm'
response = requests.get(url)

soup = BeautifulSoup(response.text, 'html.parser')
comments = soup.find_all(string=lambda text: isinstance(text, Comment))

tables = []
for each in comments:
    if 'table' in each:
        try:
            tables.append(pd.read_html(each)[0])
        except:
            continue
        
print (tables[1].loc[1:])

输出：

print (tables[1].loc[1:])
            0                         1
1    Won Toss         Eagles (deferred)
2        Roof                  outdoors
3     Surface                     grass
4    Duration                      3:19
5  Attendance                     69696
6     Weather    81 degrees, wind 8 mph
7  Vegas Line  Philadelphia Eagles -1.0
8  Over/Under              44.5 (under)