把一张有漂亮汤的html表格刮成熊猫

Question

我试图使用beautiful soup刮一个html表，并将其导入熊猫--http://www.baseball-reference.com/teams/nym/2017.shtml--“Team Batting”表。

找表没问题：

table = soup.find('div', attrs={'class': 'overthrow table_container'})
table_body = table.find('tbody')

for i in table.findAll('tr')[2]: #increase to 3 to get next row in table...
    print(i.get_text())

table_head = table.find('thead')

for i in table_head.findAll('th'):
    print(i.get_text())

现在我很难把所有的东西放在一个数据帧中。以下是我目前掌握的信息：

header = []    
for th in table_head.findAll('th'):
        key = th.get_text()
        header.append(key)

row= []
for tr in table.findAll('tr')[2]:
    value = tr.get_text()
    row.append(value)

od = OrderedDict(zip(head, row))
df = pd.DataFrame(d1, index=[0])

这一次只适用于一行。我的问题是如何同时对表中的每一行都这样做？

Answer 1

我已经测试了以下将为您的目的工作。基本上，您需要创建一个列表，循环播放器，使用该列表填充一个数据帧。建议不要逐行创建DataFrame，因为这样可能会慢得多。

import collections as co
import pandas as pd

from bs4 import BeautifulSoup

with open('team_batting.html','r') as fin:
    soup = BeautifulSoup(fin.read(),'lxml')

table = soup.find('div', attrs={'class': 'overthrow table_container'})
table_body = table.find('tbody')

table_head = table.find('thead')
header = []    
for th in table_head.findAll('th'):
    key = th.get_text()
    header.append(key)

# loop over table to find number of rows with '' in first column
endrows = 0
for tr in table.findAll('tr'):
    if tr.findAll('th')[0].get_text() in (''):
        endrows += 1

rows = len(table.findAll('tr'))
rows -= endrows + 1 # there is a pernicious final row that begins with 'Rk' 

list_of_dicts = []
for row in range(rows):
    the_row = []
    try:
        table_row = table.findAll('tr')[row]
        for tr in table_row:
            value = tr.get_text()
            the_row.append(value)
        od = co.OrderedDict(zip(header,the_row))
        list_of_dicts.append(od)
    except AttributeError:
        continue 

df = pd.DataFrame(list_of_dicts)