Backtrader量化平台教程-Portfolio级别的回测（九）

AD：（本人录制的backtrader视频课程，大家多多支持哦~ https://edu.csdn.net/course/detail/9040）   

 无意中发现了一个巨牛的人工智能教程，忍不住分享一下给大家。教程不仅是零基础，通俗易懂，而且非常风趣幽默，像看小说一样！觉得太牛了，所以分享给大家。教程链接：https://www.cbedai.net/qtlyx       

 

    我们知道，有时候我们要测试的可不仅仅是一个标的，但是之前backtrader的教程中，我们都是针对的是一个标的的回测。如果接触过优矿、ricequant这样的平台的同学，可能觉得backtrader不适合做这样的portfolio层面的回测。确实，似乎backtrader整个官方教程里面，没有任何讲到这种全市场、组合的回测demo，但是backtrader其实也是可以胜任这样的任务的。

       前段时间，笔者就做了这样的一个事情，让backtrader能够完成我们想要的组合层面的回测。

1.最终的效果

       和一般的portfolio层面的回测平台一样，我们希望，最后我们实现一个策略只要进行一些设置就可以了。笔者利用backtrader封装了一个函数，实现了几乎和优矿一样的功能。

        使用的时候，笔者的函数只需要如下的设置：

 

start_date = "2017-04-01"
end_date = "2017-06-20"
trading_csv_name = 'trading_data_two_year.csv'
portfolio_csv_name = 'port_two_year.csv'
benchmark_csv_name = None

         然后我们就可以回测了。笔者把回测的类封装了起来，只要调用笔者的回测类就可以了。

begin = datetime.datetime.now()
result_dict = bt_backtest.backtrader_backtest(start_date=start_date, end_date=end_date, trading_csv_name=trading_csv_name, \
                                              portfolio_csv_name=portfolio_csv_name, bechmark_csv_name=benchmark_csv_name)

        然后，回测结束后输出评价指标。

end = datetime.datetime.now()
print "time elapse:", (end - begin)
print 'Start Portfolio Value: %.2f' % result_dict['start_cash']
print 'Final Portfolio Value: %.2f' % result_dict['final_value']
print 'Total Return:', result_dict['total_return']
print 'Sharpe Ratio :', result_dict['sharpe_ratio'] #* 2 # todo there should be consider!
print 'Max Drowdown:', result_dict['max_drowdown'] * 2
print 'Max Drowdown Money:', result_dict['max_drowdown_money']
print "Trade Information", result_dict['trade_info']

result = pd.read_csv('result.csv', index_col=0)
result.plot()
plt.show()
position_info = pd.read_csv('position_info.csv', index_col=0)

        接下来说一下我们要输入的这些csv文件吧。

trading_csv_name = 'trading_data_two_year.csv'
portfolio_csv_name = 'port_two_year.csv'
benchmark_csv_name = None

        第一个是交易行情的数据，数据格式如下：

 

tradingdate,ticker,_open,_high,_low,_close,_volume

20070104,000001,14.65,15.32,13.83,14.11,69207082.0

20070104,000002,15.7,16.56,15.28,15.48,75119519.0

20070104,000004,4.12,4.12,3.99,4.06,1262915.0

20070104,000005,2.51,2.53,2.46,2.47,14123749.0

20070104,000006,13.5,14.07,13.39,13.7,15026054.0

20070104,000007,2.44,2.44,2.35,2.36,3014956.0

20070104,000008,4.16,4.24,4.15,4.16,818282.0

20070104,000009,0.0,0.0,0.0,4.23,0.0

20070104,000010,0.0,0.0,0.0,5.37,0.0

20070104,000011,5.78,5.85,5.42,5.45,4292318.0

20070104,000012,11.15,11.3,10.66,10.95,6934903.0

        所以，这一部分的数据会相当的大，一天就会有3000条左右的记录，毕竟，A股的股票数目就是这样。如果是用的更小级别的数据，那么数据量必然更大了。

 

portfolio_csv_name = 'port_two_year.csv'

 

        这是我们调仓日的目标仓位，只需要sigdate,secucode,weight三个字段就行。

        benchmark_csv_name自然就是benchmark的daily return数据文件了。这里可以不设置。

2.回测函数

         接下来就是核心的回测的函数了。

# -*- coding: utf-8 -*-

from __future__ import (absolute_import, division, print_function,
                        unicode_literals)

import datetime
import backtrader as bt
from backtrader import Order
import pandas as pd
import matplotlib.pyplot as plt


#  Create a Stratey
class AlphaPortfolioStrategy(bt.Strategy):

    def log(self, txt, dt=None):
        dt = dt or self.datas[0].datetime.date(0)
        print('%s, %s' % (dt.isoformat(), txt))

    def __init__(self, dataId_to_secId_dict, secId_to_dataId_dict, adj_df, end_date, backtesting_length=None, benchmark=None, result_csv_name='result'):
        # 1.get target portfolio weight
        self.adj_df = adj_df
        self.backtesting_length = backtesting_length
        self.end_date = end_date
        # 2.backtrader data_id and secId transfer diction
        self.dataId_to_secId_dict = dataId_to_secId_dict
        self.secId_to_dataId_dict = secId_to_dataId_dict
        self.benchmark = benchmark
        # 3.store the untradable day due to the up and down floor, empty it in a new adjustment day
        self.order_line_dict = {}
        self.pre_position_data_id = list()
        self.value_for_plot = {}
        # 4.keep track of pending orders and buy price/commission
        self.order = None
        self.buyprice = None
        self.buycomm = None
        self.value = 10000000.0
        self.cash = 10000000.0

        self.positions_info = open('position_info.csv', 'wb')
        self.positions_info.write('date, bt_id,sec_code, size, last price\n')

    def start(self):
        print("the world call me!")

    def notify_fund(self, cash, value, fundvalue, shares):
        # update the market value every day
        # print("actual value:", value)
        self.value = value - 10000000.0  # we give the broker more 10 million for the purpose of illiquidity
        # self.value = value  # we give the broker more 10 million for the purpose of illiquidity

        self.value_for_plot[self.datetime.datetime()] = self.value / 10000000.0
        self.cash = cash
        # print("cash:", cash)

    def notify_order(self, order):
        if order.status in [order.Submitted, order.Accepted]:
            # Buy/Sell order submitted/accepted to/by broker - Nothing to do
            return
        # Check if an order has been completed
        # Attention: broker could reject order if not enougth cash
        if order.status in [order.Completed]:
            if order.isbuy():
                # order.
                self.log(
                    'BUY EXECUTED, Price: %.2f, Cost: %.2f, Comm %.2f' %
                    (order.executed.price,
                     order.executed.value,
                     order.executed.comm))
                self.buyprice = order.executed.price
                self.buycomm = order.executed.comm
            else:  # Sell
                self.log('SELL EXECUTED, Price: %.2f, Cost: %.2f, Comm %.2f' %
                         (order.executed.price,
                          order.executed.value,
                          order.executed.comm))
            self.bar_executed = len(self)
        elif order.status in [order.Canceled, order.Rejected]:
            self.log('Order Canceled/Rejected')
        elif order.status == order.Margin:
            self.log('Order Margin')
        self.order = None

    def notify_trade(self, trade):
        if not trade.isclosed:
            return
        self.log('OPERATION PROFIT, GROSS %.2f, NET %.2f' %
                 (trade.pnl, trade.pnlcomm))

    def next(self):
        # 0.Check if an order is pending ... if yes, we cannot send a 2nd one
        # if self.order:
        #     return
        bar_time = self.datetime.datetime()
        bar_time_str = bar_time.strftime('%Y-%m-%d')
        trading_date = bar_time + datetime.timedelta(days=1)
        # bar_time_str = (self.datetime.datetime() + datetime.timedelta(1)).strftime('%Y-%m-%d')
        print(bar_time_str, self.value)
        print("bar day===:", bar_time_str, "===============")
        for (k, v) in self.dataId_to_secId_dict.items():
            size = self.positions[self.datas[k]].size
            price = self.datas[k].close[-1]
            self.positions_info.write('%s ,%s ,%s, %s, %s' % (bar_time_str, k, v, size, price))
            self.positions_info.write('\n')

        # 1. no matter the adjustment day, up/down floor blocked order should deal with each bar
        for (k, v) in self.order_line_dict.items():
            bar = self.datas[k]
            buyable = False if bar.low[1] / bar.close[0] > 1.0950 else True
            sellable = False if bar.high[1] / bar.close[0] < 0.910 else True

            # buyable = False if (bar.open[1] / bar.close[0] > 1.0950) and (bar.close[1] / bar.close[0] > 1.0950) else True
            # sellable = False if (bar.open[1] / bar.close[0] < 0.910 and (bar.close[1] / bar.close[0] < 0.91)) else True

            if v > 0:
                if buyable:
                    del self.order_line_dict[k]
                    self.log('%s BUY CREATE, %.2f, vlo is %s' % (self.dataId_to_secId_dict[k], bar.open[1], v))
                    self.order = self.buy(data=bar, size=v, exectype=Order.Market)
                else:
                    print("############:")
                    print("unbuyable:", self.dataId_to_secId_dict[k])
            elif v < 0:
                if sellable:
                    del self.order_line_dict[k]
                    self.log('%s SELL CREATE, %.2f, vol is %s' % (self.dataId_to_secId_dict[k], bar.open[1], v))
                    self.order = self.sell(data=bar, size=abs(v), exectype=Order.Market)
                else:
                    print("############:")
                    print("unsellable:", self.dataId_to_secId_dict[k])

        # 2. ensure the adjustment day
        # 2.1 get the current bar time
        adj_sig = self.adj_df[self.adj_df['sigdate'] == trading_date.strftime('%Y-%m-%d')][['secucode', 'hl_weight']]
        # 2.2 check the adjustment day
        if len(adj_sig) == 0 or bar_time_str == self.end_date:
            return
        # 3. adjust the portfolio
        # 3.1 set two dicts to store the buy order and sell order spearately
        buy_dict = {}
        sell_dict = {}
        self.order_line_dict = {}
        current_position_data_id = list()
        # 3.2 iterate the portfolio instruments and divide into buy group and sell group
        for index in adj_sig.index:
            # get current instrument code and transfer to the data_id
            sec_id = adj_sig.loc[index]['secucode']
            data_id = self.secId_to_dataId_dict[sec_id]
            if self.backtesting_length and data_id >= self.backtesting_length:
                continue
            bar = self.datas[data_id]
            current_position_data_id.append(data_id)
            # get the  target weight value
            target_weight = adj_sig.loc[index]['hl_weight']
            # calculate the current weight value
            current_position = self.positions[self.datas[data_id]]
            current_mv = current_position.size * bar.close[0]
            current_weight = current_mv / float(self.value)
            diff_weight = (target_weight - current_weight)
            if bar.open[1] == 0:
                continue
            diff_volume = int(diff_weight * self.value / bar.open[1] / 100) * 100
            print("the weight difference", diff_weight)
            if diff_volume > 0:
                buy_dict[data_id] = diff_volume
            elif diff_volume < 0:
                sell_dict[data_id] = diff_volume

        # 3.3 make order work
        for (k, v) in sell_dict.items():
            bar = self.datas[k]
            # sellable = False if (bar.high[1] == bar.low[1]) and (bar.open[1] / bar.close[0] < 0.920) else True
            sellable = False if (bar.open[1] / bar.close[0] < 0.920) else True
            # sellable = False if (bar.open[1] / bar.close[0] < 0.910 and (bar.close[1] / bar.close[0] < 0.91)) else True

            if sellable:
                self.log('%s SELL CREATE, %.2f, vol is %s' % (self.dataId_to_secId_dict[k], bar.open[1], v))
                self.order = self.sell(data=bar, size=abs(v), exectype=Order.Market)
            else:
                print("############:")
                print("unsellable:", self.dataId_to_secId_dict[k])
                self.order_line_dict[k] = v

        for (k, v) in buy_dict.items():
            bar = self.datas[k]
            # buyable = False if (bar.low[1] == bar.high[1]) and (bar.open[1] / bar.close[0]) > 1.0950 else True
            buyable = False if (bar.open[1] / bar.close[0]) > 1.0950 else True
            # buyable = False if (bar.open[1] / bar.close[0] > 1.0950) and (bar.close[1] / bar.close[0] > 1.0950) else True

            if buyable:
                self.log('%s BUY CREATE, %.2f, vlo is %s' % (self.dataId_to_secId_dict[k], bar.open[1], v))
                self.order = self.buy(data=bar, size=v, exectype=Order.Market)
            else:
                print("############:")
                print("unbuyable:", self.dataId_to_secId_dict[k])
                self.order_line_dict[k] = v

        # 3.4 close position, when the data_id is not in current portfolio and in last portfolio, we close the position
        if self.pre_position_data_id:
            clost_data_id = [di for di in self.pre_position_data_id if di not in current_position_data_id]
            for di in clost_data_id:
                bar = self.datas[di]
                print('CLOSE POSITION:',  self.dataId_to_secId_dict[di])
                self.order = self.close(data=bar)
        # 3.5 update the position data id for next checking
        self.pre_position_data_id = current_position_data_id

    def stop(self):
        # plot the net value and the benchmark curves
        plot_df = pd.concat([pd.Series(self.value_for_plot, name="net value").to_frame(), self.benchmark], axis=1, join='inner')
        plot_df.to_csv('result.csv')
        self.positions_info.close()
        print("death")

def backtrader_backtest(start_date, end_date, trading_csv_name, portfolio_csv_name, bechmark_csv_name):
    # 1.backtest parameters setting:start time, end time, the assets number(backtest_length) and the benckmark data and new a cerebro
    start_date, end_date = datetime.datetime.strptime(start_date, "%Y-%m-%d"), datetime.datetime.strptime(end_date, "%Y-%m-%d")
    backtest_length = None
    # benchmark series
    if bechmark_csv_name:
        benchmark = pd.read_csv(bechmark_csv_name, date_parser=True, dtype={'date': str})
        benchmark['date'] = benchmark['date'].apply(lambda x: datetime.datetime.strptime(x, "%Y-%m-%d"))
        benchmark = benchmark.set_index('date')
    else:
        benchmark = None

    # result_df = pd.DataFrame()
    cerebro = bt.Cerebro()

    # 2.get required trading data
    # 2.1 get trading data(total trading data)
    trading_data_df = pd.read_csv(trading_csv_name, dtype={'sigdate': str, 'secucode': str})
    trading_data_df.rename(columns={'sigdate': 'tradingdate', 'secucode': 'ticker'}, inplace=True)
    transer = lambda x: datetime.datetime.strptime(x, "%Y-%m-%d")
    trading_data_df['tradingdate'] = trading_data_df['tradingdate'].apply(transer)
    trading_data_df = trading_data_df[(trading_data_df['tradingdate'] > start_date) & (trading_data_df['tradingdate'] < end_date)]
    trading_data_df['openinterest'] = 0
    trading_data_df = trading_data_df.set_index('tradingdate')
    # 2.2 get target portfolio data for the target assets filter
    adj_df = pd.read_csv(portfolio_csv_name, dtype={'secucode': str})
    adj_df = adj_df[['sigdate', 'secucode', 'hl_weight']]

    parser1 = lambda x: datetime.datetime.strptime(x, "%Y/%m/%d")
    parser2 = lambda x: x.strftime("%Y-%m-%d")
    adj_df['sigdate'] = adj_df['sigdate'].apply(parser1)
    adj_df = adj_df[(adj_df['sigdate'] > start_date) & (adj_df['sigdate'] < end_date)]

    adj_df['sigdate'] = adj_df['sigdate'].apply(parser2)
    # 2.3 generate two diction to trans between secId and backtrader id
    sec_id_list = adj_df['secucode'].drop_duplicates().tolist()
    data_id_list = [i for i in range(len(sec_id_list))]

    dataId_to_secId_dict = dict(zip(data_id_list, sec_id_list))
    secId_to_dataId_dict = dict(zip(sec_id_list, data_id_list))

    print("total stocks' number", len(sec_id_list), '.', 'data feeding......')
    # 2.4 feed required datafeed and add them to the cerebro
    for index, sec_id in enumerate(sec_id_list[0:backtest_length]):
        sec_df = trading_data_df[trading_data_df['ticker'] == sec_id]
        # sec_df = sec_df.set_index('tradingdate')
        sec_raw_start = sec_df.index[0]
        if sec_raw_start != start_date.strftime("%Y-%m-%d"):
            na_fill_value = sec_df.head(1)['open'].values[0]
            df_temp = pd.DataFrame(index=pd.date_range(start=start_date, end=sec_raw_start, freq='D')[:-1],
                                    columns=['open', 'high', 'low', 'close', 'volume', 'openinterest']
                                   ).fillna(na_fill_value)
            frames = [df_temp, sec_df]
            sec_df = pd.concat(frames)
        data_feed = bt.feeds.PandasData(dataname=sec_df,
                                        fromdate=start_date,
                                        todate=end_date
                                        )
        cerebro.adddata(data_feed, name=sec_id)
    print('data feed finish!')
    # 3.cerebero config
    cerebro.addstrategy(AlphaPortfolioStrategy,
                        dataId_to_secId_dict, secId_to_dataId_dict, adj_df, end_date.strftime("%Y-%m-%d"), backtest_length, benchmark)
    cerebro.broker.setcash(20000000.0)
    # cerebro.broker.setcash(10000000.0)

    cerebro.broker.setcommission(commission=0.0008)
    cerebro.broker.set_slippage_fixed(0.02)
    cerebro.addanalyzer(bt.analyzers.Returns, _name="Returns")
    cerebro.addanalyzer(bt.analyzers.SharpeRatio, _name='SharpeRatio', riskfreerate=0.00, stddev_sample=True, annualize=True)
    cerebro.addanalyzer(bt.analyzers.AnnualReturn, _name='AnnualReturn')
    cerebro.addanalyzer(bt.analyzers.DrawDown, _name='DW')
    cerebro.addanalyzer(bt.analyzers.TradeAnalyzer, _name='TradeAnalyzer')
    ### information print
    start_cash = (cerebro.broker.getvalue() - 10000000.0)
    # start_cash = (cerebro.broker.getvalue())

    # print('Starting Portfolio Value: %.2f' % (cerebro.broker.getvalue() - 10000000.0))
    print("start cerebro.run()")
    results = cerebro.run()  # run the cerebro
    # 4. show the result
    strat = results[0]
    final_value = (cerebro.broker.getvalue() - 10000000.0)
    total_return = ((cerebro.broker.getvalue() - 10000000.0) / 10000000.0 - 1)
    # final_value = (cerebro.broker.getvalue())
    # total_return = ((cerebro.broker.getvalue()) / 10000000.0 - 1)

    sharpe_ratio = strat.analyzers.SharpeRatio.get_analysis()['sharperatio']
    max_drowdown = strat.analyzers.DW.get_analysis()['max']['drawdown']
    max_drowdown_money = strat.analyzers.DW.get_analysis()['max']['moneydown']
    trade_info = strat.analyzers.TradeAnalyzer.get_analysis()
    return_dict = {'start_cash': start_cash, 'final_value': final_value, 'total_return': total_return,\
                   'sharpe_ratio': sharpe_ratio, 'max_drowdown': max_drowdown, 'max_drowdown_money': max_drowdown_money,\
                   'trade_info': trade_info}
    return return_dict

        基本上，代码的注释很全面了，基本实现了涨跌停不能买入，进入排队等待，当日开盘价买入，滑点设置等等这些功能。Backtrader有一点，可能是为了加快速度，特别不方便，就是datafeed不能按照名字来实现查找，而是用index来寻找，所以我们需要建立一个全局的dict，能够实现data id到股票代码以及反过来的一一对应的dict。别的，其实实现起来还是比较方便的，但是性能有待提高，等后续需要继续启动这个项目的时候，笔者继续努力吧。譬如如何修改一个查找的逻辑和数据类型的使用。
 

PS.文章中提到的port_two_year.csv文件其实就是目标组合文件，格式大概是：

sigdate,secucode,weight

2019-01-07,000001,0.5

2019-01-07,000003,0.2

.......