捕获多个相同标签的更好方法？

Question

我正在尝试创建一个刮取下载链接的刮取器，我想使用regex但这对我来说将是一个噩梦，我找到了这个名为BeautifulSoup的库，我正在尝试捕获div class=“article-content”的子级中的URL，它是

标记，这个

是URL的名称，我不想将所有URL组合在一个列表中，而是使用了dictionary，它是key是名称(

)和value是URL的列表，这里的讨论已经足够了。

import requests
from bs4 import BeautifulSoup

def scrape():
    resp = requests.get('https://www.animeout.xyz/love-live-nijigasaki-gakuen-school-idol-doukoukai-1080p-300mb720p-150mbepisode-1/')
    soup = BeautifulSoup(resp.text,'html.parser')
    contents = soup.find('div',class_='article-content')
    output = {}
    for tag in contents.children:
        if tag.name == 'h3':
            name = tag.text
            links = []
            for sibling in tag.next_siblings:
                if sibling.name == 'p':
                    for link in sibling.find_all('a',text='Direct Download'):
                        links.append(link.get('href'))
                if sibling.name == 'h3':
                    output.update({name:links})
                    break

到目前为止，我只捕获了1个密钥，有没有一个pythonic的方法来实现这一点？

Answer 1

您可能要尝试以下操作：

import json

import requests
from bs4 import BeautifulSoup


def scrape(source_url):
    soup = BeautifulSoup(
        requests.get(source_url).text,
        'html.parser',
    ).find_all(lambda t: t.name == "a" and "Direct" in t.text)

    return {idx: a["href"] for idx, a in enumerate(soup, start=1)}


data = scrape("https://www.animeout.xyz/love-live-nijigasaki-gakuen-school-idol-doukoukai-1080p-300mb720p-150mbepisode-1/")

print(json.dumps(data, indent=2))

您只有一个键的原因是键必须是唯一的，但链接的名称不是唯一的。用一些独特的东西来改变它，例如，索引号。

示例输出：

  "1": "http://nimbus.animeout.com/series/00RAPIDBOT/Love Live Nijigasaki Gakuen School Idol Doukoukai/[AnimeOut] Love Live Nijigasaki Gakuen School Idol Doukoukai - 01 [1080pp][1080pp][Erai-raws][RapidBot].mkv",
  "2": "http://nimbus.animeout.com/series/00RAPIDBOT/Love Live Nijigasaki Gakuen School Idol Doukoukai/[AnimeOut] Love Live Nijigasaki Gakuen School Idol Doukoukai - 01 [v2][1080pp][1080pp][Erai-raws][RapidBot].mkv",
  "3": "http://nimbus.animeout.com/series/00RAPIDBOT/Love Live Nijigasaki Gakuen School Idol Doukoukai/[AnimeOut] Love Live Nijigasaki Gakuen School Idol Doukoukai - 01 [1080pp][1080pp][Erai-raws][RapidBot].mkv",
  "4": "http://nimbus.animeout.com/series/00RAPIDBOT/Love Live Nijigasaki Gakuen School Idol Doukoukai/[AnimeOut] Love Live Nijigasaki Gakuen School Idol Doukoukai - 02 [1080pp][1080pp][Erai-raws][RapidBot].mkv",
  "5": "http://nimbus.animeout.com/series/00RAPIDBOT/Love Live Nijigasaki Gakuen School Idol Doukoukai/[AnimeOut] Love Live Nijigasaki Gakuen School Idol Doukoukai - 03 [v0][1080pp][1080pp][Erai-raws][RapidBot].mkv",

and so on ...