用漂亮的汤刮除超过渲染的数据

Question

我从谷歌应用商店抓取应用名称，每个网址作为输入，我只得到60个应用（因为如果用户不向下滚动，网站会呈现60个应用）。它是如何工作的，我如何才能从一个页面刮所有的应用程序使用美丽的汤和/或硒？

非常感谢。

这是我的密码：

urls = []

urls.extend(["https://play.google.com/store/apps/category/NEWS_AND_MAGAZINES/collection/topselling_paid"])

for i in urls:
    response = get(i)
    html_soup = BeautifulSoup(response.text, 'html.parser')
    app_container = html_soup.find_all('div', class_="card no-rationale square-cover apps small")
    file = open("./InputFiles/applications.txt","w+")
    for i in range(0, len(app_container)):
        #print(app_container[i].div['data-docid'])
        file.write(app_container[i].div['data-docid'] + "\n")

    file.close()
num_lines = sum(1 for line in open('./InputFiles/applications.txt'))
print("Applications : " + str(num_lines) )

Answer 1

在这种情况下，您需要使用Selenium。我为你尝试一下，并获得所有应用程序。我会尽力解释，希望你能理解。

使用Selenium比其他Python函数更强大。我使用了ChromeDriver，所以如果你还没有安装，你可以在中安装它

http://chromedriver.chromium.org/

from time import sleep
from selenium import webdriver


options = webdriver.ChromeOptions()
driver=webdriver.Chrome(chrome_options=options, 
executable_path=r'This part is your Driver path')
driver.get('https://play.google.com/store/apps/category/NEWS_AND_MAGAZINES/collection/topselling_paid')

driver.execute_script("window.scrollTo(0, document.body.scrollHeight);") ## Scroll to bottom of page with using driver
sleep(5) ## Give a delay for allow to page scroll . If we dont program will already take 60 element without letting scroll
x = driver.find_elements_by_css_selector("div[class='card-content id-track-click id-track-impression']") ## Declare which class

for a in x:
  print a.text
driver.close()

输出：

1. Pocket Casts
Podcast Media LLC
₺24,99
2. Broadcastify Police Scanner Pro
RadioReference.com LLC
₺18,99
3. Relay for reddit (Pro)
DBrady
₺8,00
4. Sync for reddit (Pro)
Red Apps LTD
₺15,00
5. reddit is fun golden platinum (unofficial)
TalkLittle
₺9,99
... **UP TO 75**

注:

不要介意钱。这是我的国家货币，所以它会换成你的。

根据您的评论更新：

span标记中也有相同的数据docid。您可以使用get\u属性获取它。只需将以下代码添加到您的项目中。

y = driver.find_elements_by_css_selector("span[class=preview-overlay-container]")

for b in y :
   print b.get_attribute('data-docid')

输出

au.com.shiftyjelly.pocketcasts
com.radioreference.broadcastifyPro
reddit.news
com.laurencedawson.reddit_sync.pro
com.andrewshu.android.redditdonation
com.finazzi.distquakenoads
com.twitpane.premium
org.fivefilters.kindleit
.... UP TO 75