分配张量时的OOM

Question

分配张量时如何解决ResourceExhaustedError： OOM的问题？

ResourceExhaustedError（回溯见上文）：使用形状分配张量时OOM[10000,32,28,28]

我包含了几乎所有的代码

learning_rate = 0.0001
epochs = 10
batch_size = 50

# declare the training data placeholders
# input x - for 28 x 28 pixels = 784 - this is the flattened image data that is drawn from
# mnist.train.nextbatch()
x = tf.placeholder(tf.float32, [None, 784])
# dynamically reshape the input
x_shaped = tf.reshape(x, [-1, 28, 28, 1])
# now declare the output data placeholder - 10 digits
y = tf.placeholder(tf.float32, [None, 10])
def create_new_conv_layer(input_data, num_input_channels, num_filters, filter_shape, pool_shape, name):
    # setup the filter input shape for tf.nn.conv_2d
    conv_filt_shape = [filter_shape[0], filter_shape[1], num_input_channels,
                      num_filters]

    # initialise weights and bias for the filter
    weights = tf.Variable(tf.truncated_normal(conv_filt_shape, stddev=0.03),
                                      name=name+'_W')
    bias = tf.Variable(tf.truncated_normal([num_filters]), name=name+'_b')

    # setup the convolutional layer operation
    out_layer = tf.nn.conv2d(input_data, weights, [1, 1, 1, 1], padding='SAME')

    # add the bias
    out_layer += bias

    # apply a ReLU non-linear activation
    out_layer = tf.nn.relu(out_layer)

    # now perform max pooling
    ksize = [1, 2, 2, 1]
    strides = [1, 2, 2, 1]
    out_layer = tf.nn.max_pool(out_layer, ksize=ksize, strides=strides,
                               padding='SAME')

    return out_layer
# create some convolutional layers
layer1 = create_new_conv_layer(x_shaped, 1, 32, [5, 5], [2, 2], name='layer1')
layer2 = create_new_conv_layer(layer1, 32, 64, [5, 5], [2, 2], name='layer2')

flattened = tf.reshape(layer2, [-1, 7 * 7 * 64])

# setup some weights and bias values for this layer, then activate with ReLU
wd1 = tf.Variable(tf.truncated_normal([7 * 7 * 64, 1000], stddev=0.03), name='wd1')
bd1 = tf.Variable(tf.truncated_normal([1000], stddev=0.01), name='bd1')
dense_layer1 = tf.matmul(flattened, wd1) + bd1
dense_layer1 = tf.nn.relu(dense_layer1)

# another layer with softmax activations
wd2 = tf.Variable(tf.truncated_normal([1000, 10], stddev=0.03), name='wd2')
bd2 = tf.Variable(tf.truncated_normal([10], stddev=0.01), name='bd2')
dense_layer2 = tf.matmul(dense_layer1, wd2) + bd2
y_ = tf.nn.softmax(dense_layer2)
cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=dense_layer2, labels=y))


# add an optimiser
optimiser = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cross_entropy)

# define an accuracy assessment operation
correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

# setup the initialisation operator
init_op = tf.global_variables_initializer() 



 with tf.Session() as sess:
            # initialise the variables
            sess.run(init_op)
            total_batch = int(len(mnist.train.labels) / batch_size)
            for epoch in range(epochs):
                avg_cost = 0
                for i in range(total_batch):
                    batch_x, batch_y = mnist.train.next_batch(batch_size=batch_size)
                    _, c = sess.run([optimiser, cross_entropy], feed_dict={x: 
         batch_x, 
            y: batch_y})
                    avg_cost += c / total_batch
                test_acc = sess.run(accuracy,feed_dict={x: mnist.test.images, y: 
            mnist.test.labels})
                print("Epoch:", (epoch + 1), "cost =", "{:.3f}".format(avg_cost), "  
            test accuracy: {:.3f}".format(test_acc))

            print("\nTraining complete!")
            print(sess.run(accuracy, feed_dict={x: mnist.test.images, y: 
            mnist.test.labels}))

错误中引用的行是：create\u new\u conv\u layer-function

快跑。。在训练循环中

下面列出了我从调试器输出复制的更多错误（有更多行，但我认为这些是主要错误，其他错误是由此引起的…）

tensorflow.python.framework.errors_impl。ResourceExhaustedError： OOM当分配张量与形状[10000,32,28,28][[节点： Conv2D=Conv2D[T=DT_FLOAT，data_format="NHWC"，填充="SAME"，步幅=[1,1,1]，use_cudnn_on_gpu=true，_device="/作业：localhost/replica： 0/任务： 0/gpu： 0"]（重塑，layer1_W/读取）]]

我第二次运行它时出现了以下错误我同时拥有cpu和GPU，如下面的输出所示，我可以理解与cpu问题相关的一些错误可能是因为我的tensorflow没有编译以使用这些功能，我在windows 10上安装了cuda 8和cudnn 6、python 3.5、tensorflow 1.3.0。

2017-10-03 03:53:58.944371： W C：\tf_jenkins\home\workspace\rel-win\M\Windows-gpu\PY\35\tenorflow\core\Platform\cpu_feature_guard.cc:45]TensorFlow库没有编译成使用AVX指令，但这些指令在您的机器上可用，可以加快CPU计算速度。2017-10-03 03:53:58.945563： W C：\tf_jenkins\home\workspace\rel-win\M\Windows-gpu\PY\35\tenorflow\core\Platform\cpu_feature_guard.cc:45]TensorFlow库没有编译成使用AVX2指令，但这些指令在您的机器上可用，可以加快CPU计算速度。2017-10-03 03:53:59.230761： I C：\tf_jenkins\home\workspace\rere-win\M\windows-gpu\PY\35\tenorflow\core\common_runtime\gpu\gpu_device.cc:955]找到设备0与属性：名称：Quadro K620主要：5次要：0 MemyClock速率（GHz）1.124 pciBusID0000:01:00.0总内存： 2.00GiB免费内存： 1.66GiB2017-10-03 03:53:59.231109： I C：\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tenorflow\core\common_runtime\gpu\gpu_device.cc:976]DMA：0 2017-10-03 03:53:59.231229： I C：\tf_jenkins\home\workspace\rel-win\M\Windows-gpu\PY\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:986] 0： Y2017-10-03 03:53:59.231363： I C：\tf_jenkins\home\workspace\rel-win\M\Windows-gpu\PY\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:1045]正在创建TensorFlow设备（/gpu： 0）-

Answer 1

进程因内存不足（OOM）而失败，因为您一次推送了整个测试集进行评估（请参见此问题）。很容易看出，10000*32*28*28*4几乎是1Gb，而您的GPU总共只有1.66Gb可用，而且大部分已经被网络本身占用。

解决方案是不仅为训练，而且为测试提供神经网络批次。结果精度将是所有批次的平均值。而且，你不需要在每个时代之后都这样做：你真的对所有中间网络的测试结果感兴趣吗？

您的第二条错误消息很可能是由于以前的故障造成的，因为CUDNN驱动程序似乎不再工作。我建议重新启动你的机器。