CSV到字典列表-更好的方法？

Question

我正在使用一个函数，该函数接收CSV文件名，将每一行转换为字典，然后返回创建的字典列表（以便能够在以后的函数中进行迭代和组织。我已经通过执行以下操作实现了我想要的功能，但觉得必须有更好的方法。有改进建议吗？

import re

def import_incidents(filename):
    """Imports CSV and returns list of dictionaries for each incident"""
    with open(filename, 'r') as file:
        data = file.read()
        data = data.split('\n')
        list_of_data = []
        headers = True
        for line in data:
            line = line.split('","')
            if headers == True:
                #Skip header and set to false
                headers = False
            elif len(line) == 1 or line[3] == '':
                #File always has a 1 lenth final line, skip it.
                #Events can leave blank policies, skip those too.
                pass
            else:
                temp_dict = {}
                temp_dict['id'] = re.sub('"', '', line[0])
                temp_dict['time'] = re.sub('GMT-0600','',line[1])
                temp_dict['source'] = line[2]
                temp_dict['policy'] = line[3]
                temp_dict['destination'] = line[5]
                temp_dict['status'] = line[10]
                list_of_data.append(temp_dict)

return list_of_data

print(import_incidents('Incidents (Yesterday Only).csv'))

CSV内容示例：

"ID","Incident Time","Source","Policies","Channel","Destination","Severity","Action","Maximum Matches","Transaction Size","Status",
"9511564","29 Dec. 2015, 08:33:59 AM GMT-0600","Doe, John","Encrypted files","HTTPS","blah.blah.com","Medium","Permitted","0","47.7 KB","Closed - Authorized",
"1848446","29 Dec. 2015, 08:23:36 AM GMT-0600","Smith, Joe","","HTTP","google.com","Low","Permitted","0","775 B","Closed"

Answer 1

你可以用熊猫。它速度快，可以在几行内完成；

import pandas as pd
df = pd.read_csv('incidents.csv')
df['Incident Time'] = df['Incident Time'].str.replace('GMT-0600', '')
list_of_data = df.dropna(subset=['Policies']).to_dict(orient='records')

现在list_of_data包含：

[{'Action': 'Permitted',
  'Channel': 'HTTPS',
  'Destination': 'blah.blah.com',
  'ID': 9511564,
  'Incident Time': '29 Dec. 2015, 08:33:59 AM ',
  'Maximum Matches': 0,
  'Policies': 'Encrypted files',
  'Severity': 'Medium',
  'Source': 'Doe, John',
  'Status': 'Closed - Authorized',
  'Transaction Size': '47.7 KB',
  'Unnamed: 11': nan}]

.dropna（subset='Policies'）删除列Policies中包含NaNs的行，即缺少值。

如果您不想要DICT列表，请保留数据框：

df = pd.read_csv('incidents.csv', parse_dates=[1]).dropna(subset=['Policies'])

这将Incient Time读取为非常方便的datetime64[ns]对象。数据框如下所示：

Answer 2

您重新发明了csv。DictReader（）类，恐怕：

import csv

def import_incidents(filename):
    with open(filename, 'r', newline='') as file:
        reader = csv.DictReader(file)
        for row in reader:
            if not row or not row['Policies']:
                continue
            row['Incident Time'] = re.sub('GMT-0600', '', row['Incident Time'])
            yield row

这依赖于字典键的标题行。您可以使用DictReader（）的fieldnames参数定义自己的字典键（fieldnames字段按顺序与文件中的列相匹配），但是文件中的第一行仍然像其他行一样读取。您可以使用next（）函数跳过行（请参阅使用Python编辑csv文件时跳过标题）。