使用Python-beautifulsou创建Web抓取表和数据

Question

我已经使用Python Beautifulsoup从该网站的所有页面中，将此表中的数据刮取到字典中，如下面的代码所示。

然而，我也在努力为每一家有自己独立页面的公司拼凑，并将其编入词典。

import requests 
from bs4 import BeautifulSoup
from pprint import pprint 

company_data = []

for i in range(1, 3):
    page = requests.get(f'https://web.archive.org/web/20121007172955/http://www.nga.gov/collection/anZ1.htm{i}?')
    soup = BeautifulSoup(page.text, "lxml")

    row_info = soup.select('div.accordion_heading.panel-group.s_list_table')

    for row_info in row_info:
        comapny_info = {}
        comapny_info['Name'] = row_info.select_one('div.col_1 a').text.strip()

pprint(company_data)

Answer 1

我刚刚和一家200万人的公司合作过，我相信这会有所帮助。

import requests
from bs4 import BeautifulSoup
res=requests.get("https://web.archive.org/web/20121007172955/http://www.nga.gov/collection/anZ1.htm").text
soup=BeautifulSoup(res,'html.parser')
comapny_info={}
comapny_info['Profile'] = soup.select('div.text-desc-members')
if len(soup.select('div.text-desc-members'))==0:
  comapny_info['Profile']  = soup.select('div.list-sub')[0].text.strip()

comapny_info['ACOP']=[item['href'] for item in soup.select(".table.table-striped a.files")]
comapny_info['QuestionAnswer']=["Question:" + q.text.strip() + " Answer:" +a.text.strip() for q ,a in zip(soup.select("div.list-reports .m_question"),soup.select("div.list-reports .m_answer")) ]

print(comapny_info)