无法测试和部署用于推理的深度实验室v3-mobilenetv2张量流-精简分段模型

Question

我们正试图使用深度实验室v3和移动网络2在Android上运行语义分割模型。我们遵循了官方张量流精简版转换程序，使用TOCO并在bazel的帮助下tflite_convert。源冻结图是从官方的张量流DeepLab模型动物园获得的。

我们能够使用以下命令成功转换模型：-

CUDA_VISIBLE_DEVICES=“0”toco--output_file=toco256.tflite--graph_def_file=路径/to/deeplab/deeplab v3_mnv2_pascal_trainval/frozen_reference_graph。pb--输入数组=图像传感器--输出数组=语义预测--输入形状=1256256,3--推断输入类型=QUANTIZED_UINT8--推断类型=FLOAT--平均值=128--标准偏差值=127--allow_custom_ops--训练后量化

tflite文件的大小约为2.25 Mb.但是当我们尝试使用官方基准测试工具测试模型时，它失败了，并显示以下错误报告：-

bazel run -c opt tensorflow/contrib/lite/tools/benchmark:benchmark_model -- --graph=`realpath toco256.tflite`
INFO: Analysed target //tensorflow/contrib/lite/tools/benchmark:benchmark_model (0 packages loaded).
INFO: Found 1 target...
Target //tensorflow/contrib/lite/tools/benchmark:benchmark_model up-to-date:
  bazel-bin/tensorflow/contrib/lite/tools/benchmark/benchmark_model
INFO: Elapsed time: 0.154s, Critical Path: 0.00s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/tensorflow/contrib/lite/tools/benchmark/benchmark_model '--graph=path/to/deeplab/venINFO: Build completed successfully, 1 total action
STARTING!
Num runs: [50]
Inter-run delay (seconds): [-1]
Num threads: [1]
Benchmark name: []
Output prefix: []
Warmup runs: [1]
Graph: path/to/venv/tensorflow/toco256.tflite]
Input layers: []
Input shapes: []
Use nnapi : [0]
Loaded model path/to/venv/tensorflow/toco256.tflite
resolved reporter
Initialized session in 45.556ms
Running benchmark for 1 iterations 
tensorflow/contrib/lite/kernels/pad.cc:96 op_context.dims != 4 (3 != 4)
Node number 24 (PAD) failed to prepare.

Failed to invoke!
Aborted (core dumped)

我们还尝试了相同的命令，但没有包括“allow_custom_ops”和“post_training_quantize”选项，甚至使用了与1513513,3相同的输入大小；但结果是一样的。

此问题似乎与以下github问题类似：(https://github.com/tensorflow/tensorflow/issues/21266). 然而，在TensorFlow的最新版本中，这个问题应该已经解决。

型号：http://download.tensorflow.org/models/deeplabv3_mnv2_pascal_trainval_2018_01_29.tar.gzTensorflow版本：1.11 Bazel版本：0.17.2 OS:Ubuntu 18.04

此外，android应用程序无法正确加载模型（tflite解释器）

那么，我们如何将分割模型正确转换为可用于Android设备推断的tflite格式呢？

更新：-

使用tenorflow 1.12，我们得到了一个新的错误：

$ bazel run -c opt tensorflow/lite/tools/benchmark:benchmark_model -- --graph=`realpath /path/to/research/deeplab/venv/tensorflow/toco256.tflite`

    tensorflow/lite/kernels/depthwise_conv.cc:99 params->depth_multiplier * SizeOfDimension(input, 3) != SizeOfDimension(filter, 3) (0 != 32)
    Node number 30 (DEPTHWISE_CONV_2D) failed to prepare.

此外，在使用同一型号的较新版本时(3 Mb。pb文件)的深度乘数=0.5，我们得到了一个不同的错误:-

F tensorflow/lite/toco/graph_transformations/propagate_fixed_sizes.cc:116] Check failed: dim_x == dim_y (3 vs. 32)Dimensions must match

在这种情况下，我们对tflite转换使用了上述相同的命令；但是我们甚至不能生成一个“tflite”文件作为输出。这似乎是深度倍增值的问题。(甚至我们尝试在转换时将depth_multiplier参数作为参数)。

Answer 1

我也有同样的问题。从https://github.com/tantara/JejuNet，我看到他成功地将模型转换为tflite。我发短信向他求助，但不幸的是现在没有回应。

Answer 2

我也遇到了这个问题。转换过程中似乎有两个问题：

• 输入张量具有动态形状，即 [?,?,?,3]
• pad_to_bounding_box节点部分未自动转换为静态形状

对于下面的解决方案，这是在以下条件下测试的:

< li>Tensorflow 1.15 < li>Ubuntu 16.0.4

我假设您已经使用deeplab文件夹中的export_model.py文件创建了一个. pb文件，并将该文件命名为deeplab_mobilenet_v2.pb。

• 下载optimize_for_inference.py https://raw.githubusercontent.com/benoitsteiner/tensorflow-opencl/master/tensorflow/python/tools/optimize_for_inference.py
• 运行优化（根据您的配置更改参数）

python3 optimize_for_inference.py \
        --input "path/to/your/deeplab_mobilenet_v2.pb" \
        --output "path/to/deeplab_mobilenet_v2_opt.pb" \
        --frozen_graph True \
        --input_names ImageTensor \
        --output_names SemanticPredictions \
        --placeholder_type_enum=4

placeholder_type_enum=4是uint8数据类型(dtypes.uint8.as_datatype_enum)

确保您已经安装了bazel，并且已经从github下载了tensorflow r1.15分支。然后从tensorflow repo制作transform_graph工具:

巴泽尔构建张量流/工具/graph_transforms：transform_graph

然后运行transform_graph工具（确保将形状设置为用作输入的任何形状）：

bazel-bin/tensorflow/tools/graph_transforms/transform_graph \
--in_graph="/path/to/deeplab_mobilenet_v2_opt.pb" \
--out_graph="/path/to/deeplab_mobilenet_v2_opt_flatten.pb" \
--inputs='ImageTensor' \
--outputs='SemanticPredictions' \
--transforms='
    strip_unused_nodes(type=quint8, shape="1,400,225,3")
    flatten_atrous_conv
    fold_constants(ignore_errors=true, clear_output_shapes=false)
    fold_batch_norms
    fold_old_batch_norms
    remove_device
    sort_by_execution_order'

运行下面的python文件，确保将model_filepath、save_folder和save_name更改为适合您需要的任何内容。

import tensorflow as tf
import numpy as np
from tensorflow.contrib import graph_editor as ge

def freeze_session(session, keep_var_names=None, output_names=None, clear_devices=True):
    """
    Freezes the state of a session into a pruned computation graph.

    Creates a new computation graph where variable nodes are replaced by
    constants taking their current value in the session. The new graph will be
    pruned so subgraphs that are not necessary to compute the requested
    outputs are removed.
    @param session The TensorFlow session to be frozen.
    @param keep_var_names A list of variable names that should not be frozen,
                          or None to freeze all the variables in the graph.
    @param output_names Names of the relevant graph outputs.
    @param clear_devices Remove the device directives from the graph for better portability.
    @return The frozen graph definition.
    """
    graph = session.graph
    with graph.as_default():
        freeze_var_names = list(set(v.op.name for v in tf.global_variables()).difference(keep_var_names or []))
        output_names = output_names or []
        output_names += [v.op.name for v in tf.global_variables()]
        input_graph_def = graph.as_graph_def()
        if clear_devices:
            for node in input_graph_def.node:
                node.device = ""
        frozen_graph = tf.graph_util.convert_variables_to_constants(
            session, input_graph_def, output_names, freeze_var_names)
        return frozen_graph

def load_convert_save_graph(model_filepath, save_folder, save_name):
    '''
    Lode trained model.
    '''
    print('Loading model...')
    graph = tf.Graph()
    sess = tf.InteractiveSession(graph = graph)

    with tf.gfile.GFile(model_filepath, 'rb') as f:
        graph_def = tf.GraphDef()
        graph_def.ParseFromString(f.read())

    print('Check out the input placeholders:')
    nodes = [n.name + ' => ' +  n.op for n in graph_def.node if n.op in ('Placeholder')]
    for node in nodes:
        print(node)

    # Define input tensor
    input = tf.placeholder(np.uint8, shape = [1,400,225,3], name='ImageTensor')

    tf.import_graph_def(graph_def, {'ImageTensor': input}, name='')

    print('Model loading complete!')

    # remove the pad to bounding box node
    name = "pad_to_bounding_box"
    print(name)
    sgv = ge.make_view_from_scope(name, tf.get_default_graph())
    print("\t" + sgv.inputs[0].name)
    for node in sgv.inputs:
        print("name in = " + node.name)
    for node in sgv.outputs:
        print("name out = " + node.name)
    print("\t" + sgv.outputs[len(sgv.outputs)-1].name)
    sgv = sgv.remap_inputs([0])
    sgv = sgv.remap_outputs([len(sgv.outputs)-1])
    (sgv2, det_inputs) = ge.bypass(sgv)


    frozen_graph = freeze_session(sess,
                              output_names=['SemanticPredictions'])
    tf.train.write_graph(frozen_graph, save_folder, save_name, as_text=False)


load_convert_save_graph("path/to/deeplab_mobilenet_v2_opt_flatten.pb", "/path/to", "deeplab_mobilenet_v2_opt_flatten_static.pb")

tflite_convert \
  --graph_def_file="/path/to/deeplab_mobilenet_v2_opt_flatten_static.pb" \
  --output_file="/path/to/deeplab_mobilenet_v2_opt_flatten_static.tflite" \
  --output_format=TFLITE \
  --input_shape=1,400,225,3 \
  --input_arrays="ImageTensor" \
  --inference_type=FLOAT \
  --inference_input_type=QUANTIZED_UINT8 \
  --std_dev_values=128 \
  --mean_values=128 \
  --change_concat_input_ranges=true \
  --output_arrays="SemanticPredictions" \
  --allow_custom_ops

您现在可以运行您的tflite模型了