用bs4 Beautiful Soup刮多页-只刮第一页

Question

***我的代码只用于练习！

我试图从FPL的网站上删除每个玩家的名字和团队https://www.premierleague.com/我的代码有一些问题。

问题是它只得到的页面与'-1'在网址的末尾，whch我甚至没有灌输在我的页面列表！

页面没有任何逻辑-基本url是https://www.premierleague.com/players?se=363

我的代码：

import requests
from bs4 import BeautifulSoup
import pandas

plplayers = []

pl_url = 'https://www.premierleague.com/players?se=363&cl='
pages_list = ['1', '2', '131', '34']
for page in pages_list:
    r = requests.get(pl_url + page)
    c = r.content
    soup = BeautifulSoup(c, 'html.parser')
    player_names = soup.find_all('a', {'class': 'playerName'})



    for x in player_names:
        player_d = {}
        player_teams = []
        player_href = x.get('href')
        player_info_url = 'https://www.premierleague.com/' + player_href
        player_r = requests.get(player_info_url, headers=headers)
        player_c = player_r.content
        player_soup = BeautifulSoup(player_c, 'html.parser')
        team_tag = player_soup.find_all('td', {'class': 'team'})
        for team in team_tag:
            try:
                team_name = team.find('span', {'class': 'long'}).text
                if '(Loan)' in team_name:
                    team_name.replace('  (Loan) ', '')
                if team_name not in player_teams:
                    player_teams.append(team_name)
                player_d['NAME'] = x.text
                player_d['TEAMS'] = player_teams
            except:
                pass
        plplayers.append(player_d)


df = pandas.DataFrame(plplayers)
df.to_csv('plplayers.txt')

Answer 1

我会对此发表评论，但我是新来的，没有足够的声誉，所以我必须在回答中保留它。

看起来，当您请求存储在player\r中时，您指定了headers参数，但实际上没有创建headers变量。

如果替换player\u r=请求。使用player\u r=requests获取（player\u info\u url，headers=headers）。获取（player\u info\u url）相反，您的代码应该可以完美运行。至少在我的机器上是这样的。