BeautifulSoup4-不选择span类的所有实例

Question

我正在尝试从使用非特定span类来格式化/显示内容的网站中刮取数据。这些页面提供了有关化学产品的信息，并且每个产品都在一个div类中进行描述。我首先用那个div类进行解析，然后从那里提取我需要的数据。我已经能够得到许多东西，但我似乎无法提取的部分是在span类“ppisreportspan”中。如果您查看代码，您会注意到它在每个化学描述中出现多次。

<tr>
    <td><h4 id='stateprod'>MAINE STATE PRODUCT REPORT</h4><hr class='report'><span style="color:Maroon;" Class="subtitle">Company Number: </span><span style='color:black;' Class="subtitle">38</span><br /><span Class="subtitle">MONSANTO COMPANY                                                                                    <br/>800 N. LINDBERGH BOULEVARD                                  <br/>MAIL STOP FF4B                                              <br/>ST LOUIS&nbsp;MO&nbsp;63167-0001<br/></span><br/><span style="color:Maroon;" Class="subtitle">Number of Currently Registered Products: </span><span style='color:black; font-size:14px' class="subtitle">80</span><br /><br/><p class='noprint'><img alt='' src='images/epalogo.png' />&nbsp;&nbsp;View the label in the US EPA Pesticide Product Label System (PPLS).<br /><img alt='' src='images/alstar.png' />&nbsp;&nbsp;View the label in the Accepted Labels State Tracking and Repository (ALSTAR).<br /></p>
        <hr class='report'>
            <div class='nopgbrk'>
                <span class='ppisreportspanprodname'>PRECEPT INSECTICIDE </span>
                <br/>EPA Registration Number: <a href = "http://iaspub.epa.gov/apex/pesticides/f?p=PPLS:102:::NO::P102_REG_NUM:100-1075" target='blank'>100-1075-524 <img alt='EPA PPLS Link' src='images/pplslink.png'/></a>
                <span class='line-break'></span>
                <span class=ppisProd>ME Product Number: </span>
                <**span class="ppisreportspan"**>2014000996</span>
                <br />Registration Year: &nbsp;&nbsp;<**span class="ppisreportspan"**>2019</span>
                &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Type: &nbsp;&nbsp;<span class="ppisreportspan">RESTRICTED</span><br/><br/>
                <table width='100%'>
                    <tr>
                        <td width='13%'>Percent</td>
                        <td style='width:87%;align:left'>Active Ingredient</td>
                    </tr>
                    <tr>
                        <td><span class="ppisreportspan">3.0000</span></td>
                        <td><span class="ppisreportspan">Tefluthrin (128912)</span></td>
                    </tr>
                </table><hr />
        </div>

        <div class='nopgbrk'>
            <span class='ppisreportspanprodname' >ACCELERON IC-609 INSECTICIDE SEED TREATMENT FOR CORN</span>
            <br/>EPA Registration Number: <a href = "http://iaspub.epa.gov/apex/pesticides/f?p=PPLS:102:::NO::P102_REG_NUM:264-789" target='blank'>264-789-524 <img alt='EPA PPLS Link' src='images/pplslink.png'/>
            </a><span class='line-break'></span>
            <span class=ppisProd>ME Product Number: <a href = "alstar_label.aspx?LabelId=116671" target = 'blank'>2009005053</span>
                <img alt='ALSTAR Link' src='images/alstar.png'/></a>
                <br />Registration Year: &nbsp;&nbsp;<span class="ppisreportspan">2019</span>
                <br/>
                <table width='100%'>
                    <tr>
                        <td width='13%'>Percent</td>
                        <td style='width:87%;align:left'>Active Ingredient</td>
                    </tr>
                    <tr>
                        <td><span class="ppisreportspan">48.0000</span></td>
                        <td><span class="ppisreportspan">Clothianidin (44309)</span></td>
                    </tr>
                </table><hr />
            </div>

这个样品包括两种化学物质。一个有“Alstar”ID和链接，一个没有。两者都有注册年限。这些数据点让我心烦意乱。

您可能还注意到，在第一个示例中，“ppisreportspan”中存储了一个10位数的代码。我能够将它提取为没有Alstar链接的nay记录的“ppisprod”span的一部分。我不明白为什么，但它强调了一点，似乎我的解析过程忽略了那个span类。

在过去的两天里，我尝试了各种方法，基于SO上的各种不同答案，所以我不可能一一列举。我似乎可以得到从第一个“跨度”到最后一个跨度的任何东西，或者我得到“非类型”错误或空列表。

这个最接近：

它为许多div块返回正确的跨距，但对于任何具有alstar链接的块，它仍然跳过（返回空白tuple[])，如示例中的第二个块。一张显示数据的图片，然后是一系列的三组空括号，这些括号应该放在数据的位置

import urllib.request, urllib.parse, urllib.error
from bs4 import BeautifulSoup
import ssl
import re

url = input('Enter URL:')
hand = open(url)
soup = BeautifulSoup(hand, 'html.parser')
#create a list of chunks by product (div)
products = soup.find_all('div' , class_ ='nopgbrk')
print(type(products))
print(len(products))

tempalstars =[]
rptspanclasses = []
regyears = []
alstarIDs = []
asltrlinks = []
# read the span tags
for product in products:
    tempalstar = product.find_all('span', class_= "ppisreportspan")
    tempalstars.append(tempalstar)

print(tempalstar)

最终，我希望能够为每一个div块从这些span语句中选择一年的文本以及Alstar链接，但当我能够得到查找该类所有实例的代码时，我将跨过这座桥。

或者-是否有更简单的方法可以获取注册年份和Alstar链接（例如2009005053)而不是我正在尝试执行的操作？

我使用的是Python3.7.2，谢谢！

Answer 1

我设法从这个网站上得到了一些数据。您只需知道公司编号，如果是Monsanto，则该编号为38（在选择Maine并在搜索框中键入Monsanto后，该编号显示在：

import re
import requests
from bs4 import BeautifulSoup

url_1 = 'http://npirspublic.ceris.purdue.edu/state/state_menu.aspx?state=ME'
url_2 = 'http://npirspublic.ceris.purdue.edu/state/company.aspx'

company_name = 'monsanto'
company_number = '38'

with requests.session() as s:
    r = s.get(url_1)

    soup = BeautifulSoup(r.text, 'lxml')

    data = {i['name']: '' for i in soup.select('input[name]')}
    for i in soup.select('input[value]'):
        data[i['name']] = i['value']

    data['ctl00$ContentPlaceHolder1$search'] = 'company'
    data['ctl00$ContentPlaceHolder1$TextBoxInput1'] = company_name

    r = s.post(url_1, data=data)

    soup = BeautifulSoup(r.text, 'lxml')

    data = {i['name']: '' for i in soup.select('input[name]')}
    for i in soup.select('input[value]'):
        data[i['name']] = i['value']

    data = {k: v for k, v in data.items() if not k.startswith('ctl00$ContentPlaceHolder1$')}
    data['ctl00$ContentPlaceHolder1${}'.format(company_number)] = 'Display+Products'

    r = s.post(url_2, data=data)
    soup = BeautifulSoup(r.text, 'lxml')

    for div in soup.select('.nopgbrk'):
        #extract name
        print(div.select_one('.ppisreportspanprodname').text)

        #extract ME product number:
        s = ''.join(re.findall(r'\d{10}', div.text))
        print(s)

        #extract alstar link
        s = div.select_one('a[href*="alstar_label.aspx"]')
        if s:
            print(s['href'])
        else:
            print('No ALSTAR link')

        #extract Registration year:
        s = div.find(text=lambda t: 'Registration Year:' in t)
        if s:
            print(s.next.text)
        else:
            print('No registration year.')

        print('-' * 80)

打印：

PRECEPT INSECTICIDE                                                                                                                                                                                     
2014000996
No ALSTAR link
2019
--------------------------------------------------------------------------------
ACCELERON IC-609 INSECTICIDE SEED TREATMENT FOR CORN                                                                                                                                                    
2009005053
alstar_label.aspx?LabelId=117531
2019
--------------------------------------------------------------------------------
ACCELERON D-342 FUNGICIDE SEED TREATMENT                                                                                                                                                                
2015000498
alstar_label.aspx?LabelId=117538
2019
--------------------------------------------------------------------------------
ACCELERON DX-309                                                                                                                                                                                        
2009005026
alstar_label.aspx?LabelId=117559
2019
--------------------------------------------------------------------------------

... and so on.