美丽的汤，即使有硒也不能找到所有的东西

Question

我看了stackoverflow上关于漂亮汤的大部分问题，只从网站上抓取了一半的数据，但是到目前为止没有一个有效。我尝试过将该功能更改为lxml或html5lib等。我也尝试使用硒，现在我尝试用硒向下滚动网站，加载网站上的所有内容，并使用漂亮的汤来抓取数据，但是当需要超过100个项目时，它只保留抓取16个项目。我在下面附上了我的代码。

我试图刮掉的网站链接：https://www.ranker.com/list/kpop-disbanded-groups/ranker-music?ref=listed_on

from selenium import webdriver
from selenium.webdriver.common import timeouts
from selenium.webdriver.common.keys import Keys
import os
from bs4 import BeautifulSoup
import requests
import time

url = 'https://www.ranker.com/list/kpop-disbanded-groups/ranker-music?ref=listed_on&pos=2'
driver = webdriver.Safari()
driver.get(url)

SCROLL_PAUSE_TIME = 3
# Get scroll height
last_height = driver.execute_script("return document.body.scrollHeight")

while True:
    # Scroll down to bottom
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

    # Wait to load page
    time.sleep(SCROLL_PAUSE_TIME)
    # Calculate new scroll height and compare with last scroll height
    new_height = driver.execute_script("return document.body.scrollHeight")
    if new_height == last_height:
        html_content = driver.execute_script('return document.body.innerHTML')
        soup = BeautifulSoup(html_content, 'html.parser')
        for years in soup.findAll('div', class_= 'gridItem_itemDescription__2Etxm gridItem_blather__2Mozw'):
            print(years.p.text)
        break
    last_height = new_height

Answer 1

import requests
from bs4 import BeautifulSoup


def main(url):
    params = {
        "limit": 200,
        "offset": 0,
        "useDefaultNodeLinks": "false",
        "liCacheKey": "decacb20-5d77-4b04-a871-7b2c54e3db15",
        "include": "votes,wikiText,rankings,serviceProviders,openListItemContributors,taggedLists",
        "propertyFetchType": "ALL"
    }
    r = requests.get(url, params=params).json()['listItems']
    for x in r:
        yr = x.get('blather', 'N/A')
        soup = BeautifulSoup(yr, 'lxml')
        print("Name: {:<30}, Year: {}".format(
            x['name'], soup.get_text(strip=True)))


main('https://api.ranker.com/lists/2713714/items')

输出：

Name: 2NE1                          , Year: 2009–2016
Name: Wanna One                     , Year: 2017-2019
Name: SISTAR                        , Year: 2010–2017
Name: 4Minute                       , Year: 2009–2016
Name: I.O.I                         , Year: 2016–2017
Name: Wonder Girls                  , Year: 2007–2017
Name: X1                            , Year: 2019–2020
Name: T-ara                         , Year: N/A
Name: Miss A                        , Year: 2010–2017
Name: PRISTIN                       , Year: N/A
Name: IZ*ONE                        , Year: 2018–2021
Name: KARA                          , Year: 2007-2016
Name: Triple H                      , Year: N/A
Name: Orange Caramel                , Year: N/A
Name: After School                  , Year: N/A
Name: GFriend                       , Year: 2015–2021
Name: Pristin V                     , Year: N/A
Name: Nine Muses                    , Year: 2010-2019
Name: 2AM                           , Year: 2008–2017
Name: JBJ                           , Year: 2017–2018
Name: Boyfriend                     , Year: N/A
Name: HELLOVENUS                    , Year: N/A
Name: Sistar19                      , Year: 2011–2017
Name: UNB                           , Year: N/A
Name: Rainbow                       , Year: 2009–2016
Name: The Ark                       , Year: N/A
Name: History                       , Year: 2013–2017
Name: UNI.T                         , Year: N/A
Name: MADTOWN                       , Year: 2014–2017
Name: MYTEEN                        , Year: N/A
Name: Speed                         , Year: 2012–2016
Name: SPICA                         , Year: 2012–2017
Name: Stellar                       , Year: 2011–2018
Name: FIESTAR                       , Year: 2012-2018
Name: Secret                        , Year: 2009–2018
Name: B.I.G                         , Year: 2014- 2020
Name: Melody Day                    , Year: 2012-2018
Name: RaNia                         , Year: N/A
Name: C-Clown                       , Year: 2012–2015
Name: Hi Suhyun                     , Year: N/A
Name: BESTie                        , Year: 2013-2018
Name: Playback                      , Year: N/A
Name: HIGH4                         , Year: N/A
Name: Seo Taiji and Boys            , Year: N/A
Name: 15&                           , Year: N/A
Name: Glam                          , Year: 2012–2015
Name: CHI CHI                       , Year: N/A
Name: BIGSTAR                       , Year: N/A
Name: Coed School                   , Year: 2010-2013
Name: Bambino                       , Year: N/A
Name: 1Punch                        , Year: 2015
Name: EvoL                          , Year: 2012–2015
Name: A-Jax                         , Year: N/A
Name: Rainz                         , Year: N/A
Name: Leessang                      , Year: 2002–2017
Name: Tiny-G                        , Year: 2012–2015
Name: M&D                           , Year: N/A
Name: LIPBUBBLE                     , Year: N/A
Name: Homme                         , Year: 2010–2018
Name: Drug Restaurant               , Year: N/A
Name: 2YOON                         , Year: 2013–2016
Name: HONEYST                       , Year: N/A
Name: Jewelry                       , Year: 2001–2015
Name: D.Holic                       , Year: 2014–2017
Name: Lucky J                       , Year: 2014-2016
Name: M.I.B                         , Year: 2011–2017
Name: Tahiti                        , Year: N/A
Name: The Legend                    , Year: 2014–2017
Name: Wassup                        , Year: N/A
Name: Rainbow Pixie                 , Year: 2009–2016
Name: 14U                           , Year: N/A
Name: The East Light                , Year: N/A
Name: F-ve Dolls                    , Year: 2011–2015
Name: 8Eight                        , Year: 2007–2014
Name: 2EYES                         , Year: N/A
Name: Untouchable                   , Year: N/A
Name: DMTN                          , Year: 2010–2014
Name: Baby V.O.X                    , Year: 2013-2015
Name: LC9                           , Year: 2013–2015
Name: ChoColat                      , Year: 2011–2017
Name: CoCoSoRi                      , Year: N/A
Name: MyB                           , Year: 2015–2016
Name: I.B.I                         , Year: 2016–2017
Name: A-Prince                      , Year: 2012–2015
Name: M.Pire                        , Year: 2013-2015
Name: Bob Girls                     , Year: 2014–2015
Name: Gangkiz                       , Year: 2012–2014
Name: TraxX                         , Year: N/A
Name: GI                            , Year: 2013–2016
Name: BTL                           , Year: 2014–2016
Name: SKarf                         , Year: 2012–2014
Name: T-max                         , Year: 2007–2012
Name: Sunny Days                    , Year: 2012–2016
Name: SeeYa                         , Year: 2006–2011
Name: 4L                            , Year: 2014–2016
Name: Blady                         , Year: 2011-2017
Name: Phantom                       , Year: 2011–2017
Name: Puretty                       , Year: 2012–2014
Name: Double-A                      , Year: 2011–2015
Name: The SeeYa                     , Year: 2012–2015
Name: D-Unit                        , Year: 2012–2013
Name: Unicorn                       , Year: 2015–2017
Name: N-Sonic                       , Year: 2011–2016
Name: Supreme Team                  , Year: 2009–2013
Name: GP Basic                      , Year: 2010–2015
Name: Shu-I                         , Year: 2009 -2015
Name: Big Mama                      , Year: 2003—2012
Name: N-Train                       , Year: 2011–2013
Name: NOM                           , Year: 2013–2016
Name: Ledt                          , Year: 2010–2016
Name: PARAN                         , Year: 2005-2011
Name: N.EX.T                        , Year: 1992-1997, 2003-2014
Name: Kiha & The Faces              , Year: N/A
Name: Rumble Fish                   , Year: 2003-2010