用BeautifulSoup或Pandas刮表数据

Question

我对使用python有点陌生，我接到了一个任务，需要从表中抓取数据。我也不太懂html。我以前从来没有这样做过，花了几天时间研究各种刮桌子的方法。不幸的是，所有的例子都是一个看起来比我所处理的更简单的网页布局。我尝试了很多不同的方法，但没有一种方法允许我选择所需的表数据。

下面网页底部的“每日水位”选项卡下的表怎么刮？

url=https://apps.wrd.state.or.us/apps/gw/gw_info/gw_hydrograph/hydrograph.aspx？gw_logid=harn0052657

我已经尝试使用了以下链接中的方法，其他未在此显示的方法：

漂亮的刮汤台

用美味可口的汤刮桌子

from bs4 import BeautifulSoup
import requests

html = requests.get(url).text
soup = BeautifulSoup(html, "html.parser")
data = soup.find_all("table")  # {"class": "xxxx"})  

import pandas as pd
df_list = pd.read_html(url)
df_list

Answer 1

下面尝试使用Python-Requests的方法，当涉及请求时，需要简单、直接、可靠、快速和更少的代码。我已经从网站本身获取API URL后，检查了网络部分的谷歌chrome浏览器。

下面的脚本到底在做什么：

>

首先，它将使用API URL并执行一个带有动态参数（以CAPS为单位）的GET请求，您可以更改Well No、Start和end date的值以获得所需的结果。

 import json
 import requests
 from urllib3.exceptions import InsecureRequestWarning
 requests.packages.urllib3.disable_warnings(InsecureRequestWarning)
 import csv

 def scrap_daily_water_level():

 file_path = '' #Input File path here
 file_name = 'daily_water_level_data.csv' #File name

 #CSV headers
 csv_headers = ['Line #','GW Log Id','GW Site Id', 'Land Surface Elevation', 'Record Date','Restrict to OWRD only', 'Reviewed Status', 'Reviewed Status Description', 'Water level ft above mean sea level', 'Water level ft below land surface'] 
 list_of_water_readings = []

 #Dynamic Params
 WELL_NO = 'HARN0052657'
 START_DATE = '1/1/1905'
 END_DATE = '12/30/2050'

 #API URL
 URL = 'https://apps.wrd.state.or.us/apps/gw/gw_data_rws/api/' + WELL_NO + '/gw_recorder_water_level_daily_mean_public/?start_date=' + START_DATE + '&end_date=' + END_DATE + '&reviewed_status=&restrict_to_owrd_only=n'

 response = requests.get(URL,verify=False) #GET API call
 json_result = json.loads(response.text) #JSON loads to parse JSON data

 print('Daily water level data count ',json_result['feature_count']) # Prints no. of data counts
 extracted_data = json_result['feature_list'] #Extracted data in JSON form

 for idx, item in enumerate(extracted_data): #Iterate over the list of extracted data
     list_of_water_readings.append({ #append and create list of data with headers for further usage
                                 'Line #': idx + 1, 
                                 'GW Log Id' : item['gw_logid'],
                                 'GW Site Id': item['gw_site_id'],
                                 'Land Surface Elevation': item['land_surface_elevation'], 
                                 'Record Date': item['record_date'],
                                 'Restrict to OWRD only': item['restrict_to_owrd_only'],
                                 'Reviewed Status':item['reviewed_status'],
                                 'Reviewed Status Description': item['reviewed_status_description'],
                                 'Water level ft above mean sea level': item['waterlevel_ft_above_mean_sea_level'],
                                 'Water level ft below land surface': item['waterlevel_ft_below_land_surface']
                                 })
 #Create CSV and write data in to it.
 with open(file_path + file_name ,'a+') as daily_water_level_data_CSV: #Open file in a+ mode 
     csvwriter = csv.DictWriter(daily_water_level_data_CSV, delimiter=',', lineterminator='\n',fieldnames=csv_headers)
     print('Writing CSV header now...')
     csvwriter.writeheader() #Write headers in CSV file
     for item in list_of_water_readings: #iterate over the appended data and save them in to the CSV file.
         print('Writing data rows now..')
         print(item)            
         csvwriter.writerow(item)

 scrap_daily_water_level()