使用Python抓取多个网页

Question

我想刮从多个网站与类似的网址的，如https://woollahra.ljhooker.com.au/our-team, https://chinatown.ljhooker.com.au/our-team和https://bondibeach.ljhooker.com.au/our-team.

我已经写了一个脚本，第一个网站的工作，但我不知道如何告诉它从其他两个网站刮。

我的代码：

from urllib.request import urlopen as uReq
from bs4 import BeautifulSoup as soup

my_url = "https://woollahra.ljhooker.com.au/our-team"

page_soup = soup(page_html, "html.parser")  
containers = page_soup.findAll("div", {"class":"team-details"})

for container in containers:
    agent_name = container.findAll("div", {"class":"team-name"})
    name = agent_name[0].text

    phone = container.findAll("span", {"class":"phone"})
    mobile = phone[0].text

    print("name: " + name)
    print("mobile: " + mobile)

有没有一种方法可以让我简单地列出url的不同部分（woollahra、chinatown、bondibeach），这样脚本就可以使用我已经编写的代码在每个网页中循环？

Answer 1

你只需要一个循环

for team in ["woollahra", "chinatown", "bondibeach"]:
    my_url = "https://{}.ljhooker.com.au/our-team".format(team)
    page_soup = soup(page_html, "html.parser")  

    # make sure you indent the rest of the code 

Answer 2

locations = ['woollahra', 'chinatown', 'bondibeach']
for location in locations:
    my_url = 'https://' + location + '.ljhooker.com.au/our-team'

后面是代码的其余部分，它们将查看列表中的每个元素，您可以稍后添加更多位置