Bazel入门
1 Bazel简介：
bazel是Google开源的一套编译构建工具，广泛应用于Google内部，包括TensorFlow项目。修改TensorFlow内部源码，需要使用bazel来编译，
故有必要了解下bazel。bazel优点很多，主要有
1 构建快。支持增量编译。对依赖关系进行了优化，从而支持并发执行。
2 可构建多种语言。bazel可用来构建Java C++ Android ios等很多语言和框架，并支持mac windows linux等不同平台
3 可伸缩。可处理任意大小的代码库，可处理多个库，也可以处理单个库
4 可扩展。使用bazel扩展语言可支持新语言和新平台。
2 Bazel项目结构：
和Makefile一样，使用bazel编译也必须满足它的项目结构要求。bazel顶层，也就是根目录下为工作区workspace，workspace下包含多个package，
每个package又包含多个编译目标target。
2.1 工作区workspace:
要进行构建的文件系统，根目录下必须包含一个文件名为WORKSPACE的文件，即使它内容为空。它指明了构建的根目录。文件系统中包括源文件，
头文件，输出目录的符号链接等。下面是TensorFlow源码根目录下的WORKSPACE
workspace(name = "org_tensorflow")
http_archive(
name = "io_bazel_rules_closure",
sha256 = "110fe68753413777944b473c25eed6368c4a0487cee23a7bac1b13cc49d3e257",
strip_prefix = "rules_closure-4af89ef1db659eb41f110df189b67d4cf14073e1",
urls = [
"https://mirror.bazel.build/github.com/bazelbuild/rules_closure/archive/4af89ef1db659eb41f110df189b67d4cf14073e1.tar.gz",
"https://github.com/bazelbuild/rules_closure/archive/4af89ef1db659eb41f110df189b67d4cf14073e1.tar.gz", # 2017-08-28
],
)
load("@io_bazel_rules_closure//closure:defs.bzl", "closure_repositories")
closure_repositories()
load("//tensorflow:workspace.bzl", "tf_workspace")
# Uncomment and update the paths in these entries to build the Android demo.
#android_sdk_repository(
# name = "androidsdk",
# api_level = 23,
# # Ensure that you have the build_tools_version below installed in the
# # SDK manager as it updates periodically.
# build_tools_version = "26.0.1",
# # Replace with path to Android SDK on your system
# path = "<PATH_TO_SDK>",
#)
#
#android_ndk_repository(
# name="androidndk",
# path="<PATH_TO_NDK>",
# # This needs to be 14 or higher to compile TensorFlow.
# # Please specify API level to >= 21 to build for 64-bit
# # archtectures or the Android NDK will automatically select biggest
# # API level that it supports without notice.
# # Note that the NDK version is not the API level.
# api_level=14)
# Please add all new TensorFlow dependencies in workspace.bzl.
tf_workspace()
new_http_archive(
name = "inception5h",
build_file = "models.BUILD",
sha256 = "d13569f6a98159de37e92e9c8ec4dae8f674fbf475f69fe6199b514f756d4364",
urls = [
"http://storage.googleapis.com/download.tensorflow.org/models/inception5h.zip",
"http://download.tensorflow.org/models/inception5h.zip",
],
)
new_http_archive(
name = "mobile_ssd",
build_file = "models.BUILD",
sha256 = "bddd81ea5c80a97adfac1c9f770e6f55cbafd7cce4d3bbe15fbeb041e6b8f3e8",
urls = [
"http://storage.googleapis.com/download.tensorflow.org/models/object_detection/ssd_mobilenet_v1_android_export.zip",
"http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v1_android_export.zip",
],
)
new_http_archive(
name = "mobile_multibox",
build_file = "models.BUILD",
sha256 = "859edcddf84dddb974c36c36cfc1f74555148e9c9213dedacf1d6b613ad52b96",
urls = [
"http://storage.googleapis.com/download.tensorflow.org/models/mobile_multibox_v1a.zip",
"http://download.tensorflow.org/models/mobile_multibox_v1a.zip",
],
)
new_http_archive(
name = "stylize",
build_file = "models.BUILD",
sha256 = "3d374a730aef330424a356a8d4f04d8a54277c425e274ecb7d9c83aa912c6bfa",
urls = [
"http://storage.googleapis.com/download.tensorflow.org/models/stylize_v1.zip",
"http://download.tensorflow.org/models/stylize_v1.zip",
],
)
new_http_archive(
name = "speech_commands",
build_file = "models.BUILD",
sha256 = "c3ec4fea3158eb111f1d932336351edfe8bd515bb6e87aad4f25dbad0a600d0c",
urls = [
"http://storage.googleapis.com/download.tensorflow.org/models/speech_commands_v0.01.zip",
"http://download.tensorflow.org/models/speech_commands_v0.01.zip",
],
)
new_local_repository(
name = "aarch64_compiler",
path = "/",
build_file = "aarch64_compiler.BUILD",
)
http_archive：下载bazel文件，然后解压它，这个bazel 目录文件中必须包含BUILD文件。上面的http_archive中指明了要
下载io_bazel_rules_closure文件，以及它的下载地址。
new_http_archive: 下载文件，然后解压它，然后和其中包含的build_file一起创建bazel目录。
load：从.bzl文件中加载一些内容，如上面从defs.bzl文件中加载内容。
android_sdk_repository：构建Android app时使用，指定Android sdk目录。
android_ndk_repository：构建Android app时使用，指定Android ndk目录。
2.2 包package
一个WORKSPACE工作区下可以包括多个包package，每个package可以实现一个子模块，从而让各个模块进行解耦。
每个package下必须包含一个BUILD文件，它指定了package的编译构建规则。
# Description:
# TensorFlow is a computational framework, primarily for use in machine
# learning applications.
package(
default_visibility = ["//visibility:public"],
)
licenses(["notice"]) # Apache 2.0
load(
"//tensorflow:tensorflow.bzl",
"tf_cc_test",
"tf_cc_binary",
"tf_copts",
"tf_gen_op_wrappers_cc",
"cc_library_with_android_deps",
)
cc_library(
name = "gradients",
srcs = [
"framework/gradients.cc",
"framework/while_gradients.cc",
"framework/while_gradients.h",
],
hdrs = ["framework/gradients.h"],
deps = [
":cc_ops",
":cc_ops_internal",
":grad_op_registry",
":ops",
":scope",
":scope_internal",
":while_loop",
"//tensorflow/core:core_cpu",
"//tensorflow/core:framework",
"//tensorflow/core:lib",
"//tensorflow/core:lib_internal",
],
)
tf_cc_test(
name = "framework_gradients_test",
srcs = ["framework/gradients_test.cc"],
deps = [
":cc_ops",
":client_session",
":grad_op_registry",
":grad_ops",
":gradients",
":testutil",
"//tensorflow/core:all_kernels",
"//tensorflow/core:framework",
"//tensorflow/core:framework_internal",
"//tensorflow/core:protos_all_cc",
"//tensorflow/core:test",
"//tensorflow/core:test_main",
"//tensorflow/core:testlib",
],
)
tf_cc_test(
name = "framework_while_gradients_test",
size = "small",
srcs = ["framework/while_gradients_test.cc"],
deps = [
":cc_ops",
":client_session",
":grad_op_registry",
":grad_ops",
":gradients",
":testutil",
":while_loop",
"//tensorflow/core:all_kernels",
"//tensorflow/core:framework",
"//tensorflow/core:framework_internal",
"//tensorflow/core:protos_all_cc",
"//tensorflow/core:test",
"//tensorflow/core:test_main",
"//tensorflow/core:testlib",
],
)
cc_library(
name = "gradient_checker",
srcs = ["framework/gradient_checker.cc"],
hdrs = ["framework/gradient_checker.h"],
deps = [
":cc_ops",
":client_session",
":gradients",
":ops",
":scope",
"//tensorflow/core:framework",
"//tensorflow/core:lib",
"//tensorflow/core:lib_internal",
],
)
tf_cc_test(
name = "framework_gradient_checker_test",
srcs = ["framework/gradient_checker_test.cc"],
deps = [
":cc_ops",
":grad_op_registry",
":grad_ops",
":gradient_checker",
":testutil",
"//tensorflow/core:all_kernels",
"//tensorflow/core:framework",
"//tensorflow/core:framework_internal",
"//tensorflow/core:test",
"//tensorflow/core:test_main",
"//tensorflow/core:testlib",
],
)
cc_library(
name = "grad_ops",
deps = [
":array_grad",
":data_flow_grad",
":math_grad",
":nn_grad",
],
)
cc_library(
name = "grad_testutil",
testonly = 1,
srcs = ["gradients/grad_testutil.cc"],
hdrs = ["gradients/grad_testutil.h"],
deps = [
":grad_op_registry",
":ops",
":scope",
],
)
cc_library_with_android_deps(
name = "ops",
srcs = ["framework/ops.cc"],
hdrs = ["framework/ops.h"],
android_deps = ["//tensorflow/core:android_tensorflow_lib"],
deps = [
"//tensorflow/core:core_cpu",
"//tensorflow/core:framework",
"//tensorflow/core:lib",
"//tensorflow/core:lib_internal",
"//tensorflow/core:protos_all_cc",
],
)
cc_library_with_android_deps(
name = "scope",
srcs = [
"framework/scope.cc",
"framework/scope_internal.h",
],
hdrs = ["framework/scope.h"],
android_deps = ["//tensorflow/core:android_tensorflow_lib"],
common_deps = [
":ops",
],
deps = [
"//tensorflow/core:core_cpu",
"//tensorflow/core:framework",
"//tensorflow/core:lib",
"//tensorflow/core:protos_all_cc",
],
)
cc_library_with_android_deps(
name = "scope_internal",
hdrs = ["framework/scope_internal.h"],
common_deps = [
":scope",
],
deps = [],
)
tf_cc_test(
name = "framework_scope_test",
srcs = ["framework/scope_test.cc"],
deps = [
":ops",
":scope",
"//tensorflow/core:framework",
"//tensorflow/core:test",
"//tensorflow/core:test_main",
"//tensorflow/core:testlib",
],
)
cc_library_with_android_deps(
name = "client_session",
srcs = ["client/client_session.cc"],
hdrs = ["client/client_session.h"],
android_deps = ["//tensorflow/core:android_tensorflow_lib"],
common_deps = [
":ops",
":scope",
],
deps = [
"//tensorflow/core:core_cpu",
"//tensorflow/core:lib",
"//tensorflow/core:protos_all_cc",
],
)
tf_cc_test(
name = "client_client_session_test",
srcs = ["client/client_session_test.cc"],
deps = [
":cc_ops",
":client_session",
"//tensorflow/core:all_kernels",
"//tensorflow/core:core_cpu_internal",
"//tensorflow/core:framework",
"//tensorflow/core:lib",
"//tensorflow/core:tensorflow",
"//tensorflow/core:test",
"//tensorflow/core:test_main",
"//tensorflow/core:testlib",
],
)
cc_library_with_android_deps(
name = "const_op",
srcs = ["ops/const_op.cc"],
hdrs = ["ops/const_op.h"],
android_deps = [
"//tensorflow/core:android_tensorflow_lib",
],
common_deps = [
":ops",
":scope",
],
deps = [
"//tensorflow/core:core_cpu",
"//tensorflow/core:framework",
],
)
tf_cc_test(
name = "ops_const_op_test",
srcs = ["ops/const_op_test.cc"],
deps = [
":const_op",
"//tensorflow/core:framework",
"//tensorflow/core:test",
"//tensorflow/core:test_main",
"//tensorflow/core:testlib",
],
)
cc_library_with_android_deps(
name = "while_loop",
srcs = ["ops/while_loop.cc"],
hdrs = ["ops/while_loop.h"],
android_deps = [
"//tensorflow/core:android_tensorflow_lib",
],
common_deps = [
":cc_ops",
":cc_ops_internal",
":ops",
":scope",
":scope_internal",
],
deps = [
"//tensorflow/core:core_cpu",
"//tensorflow/core:framework",
],
)
tf_cc_test(
name = "ops_while_loop_test",
size = "small",
srcs = ["ops/while_loop_test.cc"],
deps = [
":cc_ops",
":client_session",
":testutil",
":while_loop",
"//tensorflow/core:core_cpu",
"//tensorflow/core:test",
"//tensorflow/core:test_main",
"//tensorflow/core:testlib",
],
)
cc_library(
name = "grad_op_registry",
srcs = ["framework/grad_op_registry.cc"],
hdrs = ["framework/grad_op_registry.h"],
deps = [
":ops",
":scope",
],
)
cc_library(
name = "array_grad",
srcs = ["gradients/array_grad.cc"],
deps = [
":cc_ops",
":cc_ops_internal",
":grad_op_registry",
":gradients",
"//tensorflow/core:lib_proto_parsing",
],
alwayslink = 1,
)
tf_cc_test(
name = "gradients_array_grad_test",
srcs = ["gradients/array_grad_test.cc"],
deps = [
":array_grad",
":cc_ops",
":cc_ops_internal",
":grad_op_registry",
":grad_testutil",
":gradient_checker",
":testutil",
"//tensorflow/core:test",
"//tensorflow/core:test_main",
"//tensorflow/core:testlib",
],
)
cc_library(
name = "math_grad",
srcs = ["gradients/math_grad.cc"],
deps = [
":cc_ops",
":cc_ops_internal",
":grad_op_registry",
":gradients",
],
alwayslink = 1,
)
tf_cc_test(
name = "gradients_math_grad_test",
srcs = ["gradients/math_grad_test.cc"],
deps = [
":cc_ops",
":grad_op_registry",
":grad_testutil",
":gradient_checker",
":math_grad",
":testutil",
"//tensorflow/core:lib_internal",
"//tensorflow/core:test",
"//tensorflow/core:test_main",
"//tensorflow/core:testlib",
],
)
cc_library(
name = "nn_grad",
srcs = ["gradients/nn_grad.cc"],
deps = [
":cc_ops",
":cc_ops_internal",
":grad_op_registry",
":gradients",
],
alwayslink = 1,
)
tf_cc_test(
name = "gradients_nn_grad_test",
srcs = ["gradients/nn_grad_test.cc"],
deps = [
":cc_ops",
":grad_op_registry",
":grad_testutil",
":gradient_checker",
":nn_grad",
":testutil",
"//tensorflow/core:lib_internal",
"//tensorflow/core:test",
"//tensorflow/core:test_main",
"//tensorflow/core:testlib",
],
)
cc_library(
name = "data_flow_grad",
srcs = ["gradients/data_flow_grad.cc"],
deps = [
":cc_ops",
":cc_ops_internal",
":grad_op_registry",
":gradients",
],
alwayslink = 1,
)
tf_cc_test(
name = "gradients_data_flow_grad_test",
size = "small",
srcs = ["gradients/data_flow_grad_test.cc"],
deps = [
":cc_ops",
":data_flow_grad",
":grad_op_registry",
":grad_testutil",
":gradient_checker",
":testutil",
"//tensorflow/core:lib_internal",
"//tensorflow/core:test",
"//tensorflow/core:test_main",
"//tensorflow/core:testlib",
],
)
tf_gen_op_wrappers_cc(
name = "cc_ops",
op_lib_names = [
"array_ops",
"audio_ops",
"candidate_sampling_ops",
"control_flow_ops",
"data_flow_ops",
"image_ops",
"io_ops",
"linalg_ops",
"logging_ops",
"lookup_ops",
"math_ops",
"nn_ops",
"no_op",
"parsing_ops",
"random_ops",
"sparse_ops",
"state_ops",
"string_ops",
"training_ops",
"user_ops",
],
other_hdrs = [
"ops/const_op.h",
"ops/standard_ops.h",
],
override_file = "ops/op_gen_overrides.pbtxt",
pkg = "//tensorflow/core",
)
tf_cc_test(
name = "framework_cc_ops_test",
srcs = ["framework/cc_ops_test.cc"],
deps = [
":cc_ops",
":client_session",
":test_op",
":test_op_op_lib",
":testutil",
"//tensorflow/core:framework",
"//tensorflow/core:test",
"//tensorflow/core:test_main",
"//tensorflow/core:testlib",
],
)
tf_gen_op_wrappers_cc(
name = "sendrecv_ops",
include_internal_ops = 1,
op_lib_names = [
"sendrecv_ops",
],
pkg = "//tensorflow/core",
)
tf_gen_op_wrappers_cc(
name = "function_ops",
include_internal_ops = 1,
op_lib_names = [
"function_ops",
],
pkg = "//tensorflow/core",
visibility = ["//tensorflow:internal"],
)
tf_gen_op_wrappers_cc(
name = "functional_ops",
include_internal_ops = 1,
op_lib_names = [
"functional_ops",
],
pkg = "//tensorflow/core",
visibility = ["//tensorflow:internal"],
)
tf_gen_op_wrappers_cc(
name = "resource_variable_ops",
include_internal_ops = 1,
op_lib_names = [
"resource_variable_ops",
],
pkg = "//tensorflow/core",
visibility = ["//tensorflow:internal"],
)
tf_gen_op_wrappers_cc(
name = "remote_fused_graph_ops",
op_lib_names = [
"remote_fused_graph_ops",
],
pkg = "//tensorflow/core",
)
cc_library_with_android_deps(
name = "cc_op_gen_main",
srcs = [
"framework/cc_op_gen.cc",
"framework/cc_op_gen.h",
"framework/cc_op_gen_main.cc",
],
android_deps = [
"//tensorflow/core:android_tensorflow_lib",
],
copts = tf_copts(),
deps = [
"//tensorflow/core:framework",
"//tensorflow/core:lib",
"//tensorflow/core:lib_internal",
"//tensorflow/core:op_gen_lib",
"//tensorflow/core:op_gen_overrides_proto_cc",
"//tensorflow/core:proto_text",
"//tensorflow/core:protos_all_cc",
],
)
cc_library(
name = "test_op_op_lib",
srcs = ["framework/test_op.cc"],
linkstatic = 1,
deps = ["//tensorflow/core:framework"],
alwayslink = 1,
)
cc_library(
name = "testutil",
testonly = 1,
srcs = ["framework/testutil.cc"],
hdrs = ["framework/testutil.h"],
deps = [
":client_session",
":ops",
":scope",
"//tensorflow/core:core_cpu",
"//tensorflow/core:lib_internal",
"//tensorflow/core:tensorflow",
"//tensorflow/core:testlib",
],
)
tf_gen_op_wrappers_cc(
name = "test_op",
op_lib_names = [
"test_op",
],
)
tf_cc_binary(
name = "tutorials_example_trainer",
srcs = ["tutorials/example_trainer.cc"],
copts = tf_copts(),
linkopts = select({
"//tensorflow:windows": [],
"//tensorflow:windows_msvc": [],
"//tensorflow:darwin": [
"-lm",
"-lpthread",
],
"//tensorflow:ios": [
"-lm",
"-lpthread",
],
"//conditions:default": [
"-lm",
"-lpthread",
"-lrt",
],
}),
deps = [
":cc_ops",
"//tensorflow/core:core_cpu",
"//tensorflow/core:framework",
"//tensorflow/core:lib",
"//tensorflow/core:protos_all_cc",
"//tensorflow/core:tensorflow",
],
)
filegroup(
name = "all_files",
srcs = glob(
["**/*"],
exclude = [
"**/METADATA",
"**/OWNERS",
],
),
visibility = ["//tensorflow:__subpackages__"],
)
cc_library(
name = "queue_runner",
srcs = ["training/queue_runner.cc"],
hdrs = ["training/queue_runner.h"],
deps = [
":coordinator",
"//tensorflow/core:core_cpu",
"//tensorflow/core:lib",
"//tensorflow/core:lib_internal",
"//tensorflow/core:protos_all_cc",
"//tensorflow/core/kernels:ops_util",
],
)
tf_cc_test(
name = "queue_runner_test",
srcs = ["training/queue_runner_test.cc"],
deps = [
"coordinator",
":cc_ops",
":queue_runner",
":scope",
"//tensorflow/core:core_cpu",
"//tensorflow/core:framework",
"//tensorflow/core:lib",
"//tensorflow/core:lib_internal",
"//tensorflow/core:protos_all_cc",
"//tensorflow/core:tensorflow",
"//tensorflow/core:test",
"//tensorflow/core:test_main",
"//tensorflow/core:testlib",
],
)
cc_library(
name = "coordinator",
srcs = ["training/coordinator.cc"],
hdrs = ["training/coordinator.h"],
deps = [
"//tensorflow/core:lib",
"//tensorflow/core:lib_internal",
"//tensorflow/core:protos_all_cc",
"//tensorflow/core:tensorflow",
],
)
tf_cc_test(
name = "coordinator_test",
srcs = ["training/coordinator_test.cc"],
deps = [
":cc_ops",
":coordinator",
":queue_runner",
":scope",
"//tensorflow/core:core_cpu",
"//tensorflow/core:framework",
"//tensorflow/core:lib",
"//tensorflow/core:lib_internal",
"//tensorflow/core:protos_all_cc",
"//tensorflow/core:tensorflow",
"//tensorflow/core:test",
"//tensorflow/core:test_main",
"//tensorflow/core:testlib",
],
)
tf_cc_binary：目标文件编译规则，为一个二进制可执行文件。name必须唯一，srcs指定了源文件，linkopts指定了链接规则，deps指定了依赖文件
cc_library：库文件编译规则，name指定了编译为库文件后的文件名，srcs和hdrs指定源文件和头文件，deps指定需要依赖的其他文件
tf_cc_test：测试文件规则
package：通用方法，定义的值会作用到下面的每个子rule中。default_visibility指定了这个包的默认可见规则。可见的情况下才能被其他package调用。
licenses：通用方法，默认的license
load：通用方法，加载.bzl文件
filegroup：通用方法，为多个编译目标target指定一个名字，glob是一个帮助函数，指定了目录中哪些文件会include，哪些会exclude。visibility
指定了target的可见性，也就是可以被哪些package调用
其他一些常用方法可以参看bazel文档 https://docs.bazel.build/versions/master/be/c-cpp.html#cc_binary.linkopts
2.2 目标
包package是一个容器，组成它的元素称为目标，分为文件和规则。文件分为两种，一种为程序员写的源代码，一种为构建工具生成的文件。
规则定义了如何利用输入来构建得到输出，如上面的BUILD。输入一般是源文件，库文件等，输出则一般是生成的构建目标文件。
3 使用Bazel编译项目
Bazel提供了一些编译的例子，在https://github.com/bazelbuild/examples/，可以clone到本地试一下。
其中examples/cpp-tutorial目录下包含了这么些文件：
examples
└── cpp-tutorial
├──stage1
│ └── main
│ ├── BUILD
│ ├── hello-world.cc
│ └── WORKSPACE
├──stage2
│ ├── main
│ │ ├── BUILD
│ │ ├── hello-world.cc
│ │ ├── hello-greet.cc
│ │ ├── hello-greet.h
│ └── WORKSPACE
└──stage3
├── main
│ ├── BUILD
│ ├── hello-world.cc
│ ├── hello-greet.cc
│ └── hello-greet.h
├── lib
│ ├── BUILD
│ ├── hello-time.cc
│ └── hello-time.h
└── WORKSPACE
首先进入到cpp-tutorial/stage1目录下，然后运行以下指令：
bazel build //main:hello-world
注意target中的//main:是BUILD文件相对于WORKSPACE文件的位置，hello-world则是我们在BUILD文件中命名好的target的名字。
这样在bazel-bin文件夹下会产生二进制文件。
4 查看依赖图
一个成功的build将所有的依赖都显式定义在了BUILD文件中。Bazel使用这些定义来创建项目的依赖图，这能够加速编译的过程。
让我们来可视化一下我们项目的依赖吧。首先，生成依赖图的一段文字描述（即在工作区根目录下运行下述指令）：
bazel query --nohost_deps --noimplicit_deps 'deps(//main:hello-world)' \ --output graph