TensorflowOnSpark-单机版
东门涵育
GetStarted_Standalone(单机版)
使用到的软件说明：
虚拟机 ubuntu17.04
python 2.7.13(ubuntu自带)
Java7+ jdk-8u131-linux-x64.tar.gz
spark spark-1.6.0-bin-hadoop2.6.tgz
TensorFlowOnSpark git 官网
TensorFlow 0.12.1
1、直接在虚拟机操作
成功
2、SSH连接(未设置无密码模式)
失败
3、SSH连接(设置无密码模式)
成功
注：SSH 无密码模式设置，
参考：Linux
/
在线虚拟机
/
ssh访问Linux系统
Running TensorFlowOnSpark Locally
We illustrate how to apply TensorFlowOnSpark in a standalone Spark cluster, which is installed on your local machine.
1. Clone TensorFlowOnOnSpark code.
git clone --recurse-submodules https://github.com/yahoo/TensorFlowOnSpark.git
cd TensorFlowOnSpark
git submodule init
git submodule update --force
git submodule foreach --recursive git clean -dfx
cd TensorFlowOnSpark
export TFoS_HOME=$(pwd)
pushd src
zip -r ../tfspark.zip *
popd
2. Install Spark
先配置Java环境（参考：http://blog.csdn.net/wc781708249/article/details/78223371）
${TFoS_HOME}/scripts/local-setup-spark.sh #自动下载及安装spark-1.6.0-bin-hadoop2.6.tar
# --------上面代码详解---------------------------
# vim ${TFoS_HOME}/scripts/local-setup-spark.sh
wget http://archive.apache.org/dist/spark/spark-1.6.0/spark-1.6.0-bin-hadoop2.6.tgz
gunzip spark-1.6.0-bin-hadoop2.6.tgz
tar -xvf spark-1.6.0-bin-hadoop2.6.tar
# ------------------------------------
或
wget https://d3kbcqa49mib13.cloudfront.net/spark-1.6.0-bin-hadoop2.6.tgz
tar -zxvf spark-1.6.0-bin-hadoop2.6.tgz
rm spark-1.6.0-bin-hadoop2.6.tarexport SPARK_HOME=$(pwd)/spark-1.6.0-bin-hadoop2.6export PATH=${SPARK_HOME}/bin:${PATH}
3. Installing TensorFlow from Binary
On
Mac
OS, for example, you could install TensorFlow as below:
export TF_BINARY_URL=https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-0.12.1-py2-none-any.whl
sudo pip install --upgrade $TF_BINARY_URL
Linux 安装 0.12.1版本
sudo pip install --upgrade
https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.12.1-cp27-none-linux_x86_64.whl
Test TensorFlow:
# download MNIST files, if not already done
mkdir ${TFoS_HOME}/mnist
pushd ${TFoS_HOME}/mnist
curl -O "http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz"
curl -O "http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz"
curl -O "http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz"
curl -O "http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz"
popd
python ${TFoS_HOME}/tensorflow/tensorflow/examples/tutorials/mnist/mnist_with_summaries.py --data_dir ${TFoS_HOME}/mnist
4. Launch
standalone Spark cluster
Start master:
${SPARK_HOME}/sbin/start-master.sh
Start one or more workers and connect them to the master via master-spark-URL. Go to
MasterWebUI
, make sure that you have the exact number of workers launched.
export MASTER=spark://$(hostname):7077
export SPARK_WORKER_INSTANCES=2
export CORES_PER_WORKER=1
export TOTAL_CORES=$((${CORES_PER_WORKER}*${SPARK_WORKER_INSTANCES}))
${SPARK_HOME}/sbin/start-slave.sh -c $CORES_PER_WORKER -m 3G ${MASTER}
5. Convert the MNIST zip files
cd ${TFoS_HOME}
rm -rf examples/mnist/csv
${SPARK_HOME}/bin/spark-submit \
--master ${MASTER} \
${TFoS_HOME}/examples/mnist/mnist_data_setup.py \
--output examples/mnist/csv \
--format csv
ls -lR examples/mnist/csv
6. Run distributed MNIST training (using feed_dict)
# rm -rf mnist_model
${SPARK_HOME}/bin/spark-submit \
--master ${MASTER} \
--py-files ${TFoS_HOME}/tfspark.zip,${TFoS_HOME}/examples/mnist/spark/mnist_dist.py \
--conf spark.cores.max=${TOTAL_CORES} \
--conf spark.task.cpus=${CORES_PER_WORKER} \
--conf spark.executorEnv.JAVA_HOME="$JAVA_HOME" \
${TFoS_HOME}/examples/mnist/spark/mnist_spark.py \
--cluster_size ${SPARK_WORKER_INSTANCES} \
--images examples/mnist/csv/train/images \
--labels examples/mnist/csv/train/labels \
--format csv \
--mode train \
--model mnist_model
ls -l mnist_model
7. Run distributed MNIST inference (using feed_dict)
# rm -rf predictions
${SPARK_HOME}/bin/spark-submit \
--master ${MASTER} \
--py-files ${TFoS_HOME}/tfspark.zip,${TFoS_HOME}/examples/mnist/spark/mnist_dist.py \
--conf spark.cores.max=${TOTAL_CORES} \
--conf spark.task.cpus=${CORES_PER_WORKER} \
--conf spark.executorEnv.JAVA_HOME="$JAVA_HOME" \
${TFoS_HOME}/examples/mnist/spark/mnist_spark.py \
--cluster_size ${SPARK_WORKER_INSTANCES} \
--images examples/mnist/csv/test/images \
--labels examples/mnist/csv/test/labels \
--mode inference \
--format csv \
--model mnist_model \
--output predictions
less predictions/part-00000
7. Interactive Learning with Jupyter Notebook
Install additional software required by Jupyter Notebooks.
sudo pip install jupyter jupyter[notebook]Launch IPython notebook on Master node.
pushd ${TFoS_HOME}/examples/mnist
PYSPARK_DRIVER_PYTHON="jupyter" \
PYSPARK_DRIVER_PYTHON_OPTS="notebook --no-browser --ip=`hostname`" \
pyspark --master ${MASTER} \
--conf spark.cores.max=${TOTAL_CORES} \
--conf spark.task.cpus=${CORES_PER_WORKER} \
--py-files ${TFoS_HOME}/tfspark.zip,${TFoS_HOME}/examples/mnist/spark/mnist_dist.py \
--conf spark.executorEnv.JAVA_HOME="$JAVA_HOME"
Launch a browser to access
notebook
8. Shutdown Spark cluster
${SPARK_HOME}/sbin/stop-slave.sh
${SPARK_HOME}/sbin/stop-master.sh
类似资料：