Python：如何读取和处理一个18GB的csv文件？[副本]

Question

下面是我所做的代码：

df=pd.read_csv('/Users/gaoyingqiang/Desktop/D989_Leistung.csv',usecols=[1,2],sep=';',encoding='gbk',iterator=True,chunksize=1000000)
df=pd.concat(df,ignore_index=True)

U1=df['Kanal 1-1 [V]']
I1=df['Kanal 1-2 [V]']

c=[]
for num in range(0,16333660,333340):
    lu=sum(U1[num:num+333340]*U1[num:num+333340])/333340
    li=sum(I1[num:num+333340]*I1[num:num+333340])/333340
    lui=sum(I1[num:num+333340]*U1[num:num+333340])/333340
    c.append(180*mt.acos(2*lui/mt.sqrt(4*lu*li))/np.pi)
    lu=0
    li=0
    lui=0

phase=pd.DataFrame(c)
phase.to_excel('/Users/gaoyingqiang/Desktop/Phaseverschiebung_1.xlsx',sheet_name='Sheet1')

有没有办法加速这个过程？

Answer 1

你读了一百万块，然后把它分成一个巨大的df，然后处理它。一整块地阅读，处理它（写它？）会更快然后再读下一段？

作为对您的评论的答复，当您

df_chunks = pd.read_csv(..... chunksize=1000000)

您将得到一个pandas.io对象（或类似的对象）

for chunk in df_chunks:
    # do something, eg..
    U1=df['Kanal 1-1 [V]']
    I1=df['Kanal 1-2 [V]']

    c=[]
    for num in range(0,16333660,333340):
        lu=sum(U1[num:num+333340]*U1[num:num+333340])/333340
        li=sum(I1[num:num+333340]*I1[num:num+333340])/333340
        lui=sum(I1[num:num+333340]*U1[num:num+333340])/333340
        c.append(180*mt.acos(2*lui/mt.sqrt(4*lu*li))/np.pi)
        lu=0
        li=0
        lui=0

    phase=pd.DataFrame(c)
    # append phase to a csv file (i'd have to google how to do that but I'm sure you can)