梯度下降simpleNet分类问题的Numpy实现

本文从最基础的梯度下降原理出发，使用numpy库搭建一个简单的神经网络。
主要函数：
softmax：对于分类问题，使用恒等函数作为显然并不合适，因此设计softmax函数，其基本原理就是将用二进制矩阵表示的输出经过softmax将其最大预测概率输出，比如某一类别真实值为[0,0,1],经过softmax函数后输出为[0.1,0.2,0.7]，那么就可认为最大的概率0.7为“1",其余各位为0，从而输出[0,0,1]

def softmax(x):          #概率表示
    if x.ndim == 2:
        x = x.T
        x = x - np.max(x, axis=0)
        y = np.exp(x) / np.sum(np.exp(x), axis=0)
        return y.T 

    x = x - np.max(x) # 溢出对策
    return np.exp(x) / np.sum(np.exp(x))

交叉熵误差：计算误差的一种常见方法。

def cross_entropy_error(y, t):   #交叉熵误差
    if y.ndim == 1:
        t = t.reshape(1, t.size)
        y = y.reshape(1, y.size)
        
    # 监督数据是one-hot-vector的情况下，转换为正确解标签的索引
    if t.size == y.size:
        t = t.argmax(axis=1)
             
    batch_size = y.shape[0]
    return -np.sum(np.log(y[np.arange(batch_size), t] + 1e-7)) / batch_size

梯度计算：

def numerical_gradient(f, x):  #梯度
    h = 1e-4 # 0.0001
    grad = np.zeros_like(x)
    
    it = np.nditer(x, flags=['multi_index'], op_flags=['readwrite'])
    while not it.finished:
        idx = it.multi_index
        tmp_val = x[idx]
        x[idx] = float(tmp_val) + h
        fxh1 = f(x) # f(x+h)
        
        x[idx] = tmp_val - h 
        fxh2 = f(x) # f(x-h)
        grad[idx] = (fxh1 - fxh2) / (2*h)
        
        x[idx] = tmp_val # 还原值
        it.iternext()   
        
    return grad

简单神经网络搭建：

class simpleNet:
    def __init__(self):
        self.W = np.random.randn(4,3)  #高斯分布随机设定初始权重(输入特证数，输出类别数)

    def predict(self, x):
        return np.dot(x, self.W)      #初始点与权重的乘积

    def loss(self, x, t):             #计算损失值
        z = self.predict(x)
        y = softmax(z)
        loss = cross_entropy_error(y, t)
        
        return loss

导入数据：

from sklearn.datasets import load_iris
from sklearn.preprocessing import label_binarize
x, y = load_iris(return_X_y=True)  #输入输出
t = label_binarize(y, np.arange(3)) #二进制化

开始训练：

lr=0.1 #步长
step_num=100  #训练次数
# x = np.array([[6, 4],[5,2]])   #初始点
# t = np.array([[1, 0, 0, 0],[0,0,0,1]])     #真实值

net = simpleNet()
for i in range(step_num):
    f = lambda w: net.loss(x, t)    #最优化函数
    dW = numerical_gradient(f, net.W)  #梯度
    net.W =net.W-lr*dW  #梯度下降
    print("Epoch:{},   Loss：{}".format(i+1 ,net.loss(x, t)))
print("权重：\n{}".format(dW) ) #最终权重

结果：

Epoch:1, Loss：4.469604981030863
Epoch:2, Loss：3.396881506373018
Epoch:3, Loss：2.554322729988447
Epoch:4, Loss：1.3206329220647133
Epoch:5, Loss：1.079182319284441
Epoch:6, Loss：0.9707371004901388
Epoch:7, Loss：0.9457430927217405
Epoch:8, Loss：0.9290112643942359
Epoch:9, Loss：0.9104523042440371
Epoch:10, Loss：0.899485864036335
Epoch:11, Loss：0.886088099670014
Epoch:12, Loss：0.8792119507227469
Epoch:13, Loss：0.869767254986812
Epoch:14, Loss：0.864876995213135
Epoch:15, Loss：0.8582763218188074
Epoch:16, Loss：0.853072728726504
Epoch:17, Loss：0.8486170811870465
Epoch:18, Loss：0.8414159272145784
Epoch:19, Loss：0.8390064274439943
Epoch:20, Loss：0.8291222072378025
Epoch:21, Loss：0.8290062753492333
Epoch:22, Loss：0.8164407424621777
Epoch:23, Loss：0.8188614527630587
Epoch:24, Loss：0.8038424574551353
Epoch:25, Loss：0.808895504970726
Epoch:26, Loss：0.7916403646513253
Epoch:27, Loss：0.7992920168501539
Epoch:28, Loss：0.7799620529820273
Epoch:29, Loss：0.7901046058788422
Epoch:30, Loss：0.768825508248329
Epoch:31, Loss：0.7813177923231995
Epoch:32, Loss：0.7582021730232279
Epoch:33, Loss：0.7728909664271106
Epoch:34, Loss：0.7480495917761614
Epoch:35, Loss：0.7647795225379398
Epoch:36, Loss：0.7383244589877087
Epoch:37, Loss：0.7569424572411679
Epoch:38, Loss：0.7289867180118752
Epoch:39, Loss：0.7493441158365557
Epoch:40, Loss：0.7200003678221459
Epoch:41, Loss：0.7419539616925939
Epoch:42, Loss：0.7113332755071491
Epoch:43, Loss：0.73474587259771
Epoch:44, Loss：0.7029567656904927
Epoch:45, Loss：0.7276974308572848
Epoch:46, Loss：0.6948452043438799
Epoch:47, Loss：0.7207893136991993
Epoch:48, Loss：0.6869756260964092
Epoch:49, Loss：0.7140047898200519
Epoch:50, Loss：0.6793274104337468
Epoch:51, Loss：0.7073293070695685
Epoch:52, Loss：0.6718820024792488
Epoch:53, Loss：0.7007501550195521
Epoch:54, Loss：0.6646226724145446
Epoch:55, Loss：0.6942561884370216
Epoch:56, Loss：0.6575343077858617
Epoch:57, Loss：0.6878376000863937
Epoch:58, Loss：0.6506032335661945
Epoch:59, Loss：0.6814857333722724
Epoch:60, Loss：0.6438170556022663
Epoch:61, Loss：0.6751929271343579
Epoch:62, Loss：0.6371645237823125
Epoch:63, Loss：0.6689523863846322
Epoch:64, Loss：0.6306354119294747
Epoch:65, Loss：0.6627580740342025
Epoch:66, Loss：0.6242204119580923
Epoch:67, Loss：0.6566046196435363
Epoch:68, Loss：0.6179110402795248
Epoch:69, Loss：0.6504872420561403
Epoch:70, Loss：0.6116995547997274
Epoch:71, Loss：0.6444016833948858
Epoch:72, Loss：0.6055788811356811
Epoch:73, Loss：0.638344152430882
Epoch:74, Loss：0.5995425468998744
Epoch:75, Loss：0.6323112757260998
Epoch:76, Loss：0.5935846230863947
Epoch:77, Loss：0.6263000552601905
Epoch:78, Loss：0.5876996717308693
Epoch:79, Loss：0.6203078315217482
Epoch:80, Loss：0.5818826991357091
Epoch:81, Loss：0.6143322512180702
Epoch:82, Loss：0.5761291140489179
Epoch:83, Loss：0.6083712389329796
Epoch:84, Loss：0.5704346902642862
Epoch:85, Loss：0.602422972173765
Epoch:86, Loss：0.5647955331752041
Epoch:87, Loss：0.59648585934622
Epoch:88, Loss：0.5592080498712695
Epoch:89, Loss：0.5905585202912426
Epoch:90, Loss：0.5536689224236769
Epoch:91, Loss：0.5846397690588289
Epoch:92, Loss：0.5481750840312667
Epoch:93, Loss：0.5787285986604382
Epoch:94, Loss：0.5427236977553389
Epoch:95, Loss：0.5728241675810071
Epoch:96, Loss：0.5373121375858654
Epoch:97, Loss：0.566925787862167
Epoch:98, Loss：0.5319379716259516
Epoch:99, Loss：0.5610329146037137
Epoch:100, Loss：0.5265989471941406
权重：
[[ 0.04295729 -1.19884135 1.15588407]
[ 0.0042395 -0.53714437 0.53290487]
[ 0.06584665 -0.86142451 0.79557786]
[ 0.02509257 -0.2589142 0.23382163]]