如何在独立模式下在mesos上运行Spark

Question

我已经在我的本地安装了mesos，并按照mesos设置中提到的进行了配置。现在我想在本地机器上安装的mesos上运行spark。我已经根据官方文档配置了spark，并在我的本地机器上运行了单节点hadoop集群。Spark二进制包被复制到hdfs根目录，我已经在spark-env.sh中设置了以下属性：

export MESOS_NATIVE_JAVA_LIBRARY=/usr/local/lib/libmesos.so
export SPARK_EXECUTOR_URI=hdfs://spark-2.2.0-bin-hadoop2.7.tgz

是Spark-Defaults.conf：

spark.executor.uri         hdfs://spark-2.2.0-bin-hadoop2.7.tgz

/bin/spark-shell --master mesos://host:5050

ERROR SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: Could not parse Master URL: 'mesos://host:5050'

Answer 1

我已经成功地在Mesos上安装了Apache Spark。请在您的Ubuntu机器上按照以下步骤操作。

# Update the packages.
$ sudo apt-get update

# Install a few utility tools.
$ sudo apt-get install -y tar wget git

# Install the latest OpenJDK.
$ sudo apt-get install -y openjdk-8-jdk

# Install autotools (Only necessary if building from git repository).
$ sudo apt-get install -y autoconf libtool

# Install other Mesos dependencies.
$ sudo apt-get -y install build-essential python-dev python-six python-virtualenv libcurl4-nss-dev libsasl2-dev libsasl2-modules maven libapr1-dev libsvn-dev

# Change working directory.
$ cd mesos

#If getting error "libz is required to build mesos"
$sudo apt install zlib1g-dev

# Configure and build.
$ mkdir build
$ cd build
build$../configure
$ make
$ make check
$ make install

$ cd build
#start master
./bin/mesos-master.sh --ip=127.0.0.1 --work_dir=/tmp/mesos
#start slave
./bin/mesos-slave.sh --master=127.0.0.1:5050 --work_dir=/tmp/mesos 

#If facing permission issue
./bin/mesos-slave.sh --master=127.0.0.1:5050 --work_dir=/tmp/mesos --no-systemd_enable_support

#Configuring Spark with Mesos

build spark with mesos support
./build/mvn -Pmesos -DskipTests clean package

In spark-env.sh
export MESOS_NATIVE_JAVA_LIBRARY= /usr/local/lib/libmesos.so
export SPARK_EXECUTOR_URI=/localpath/to/spark-2.2.0-bin-hadoop2.7.tgz

In spark-defaults.conf
spark.executor.uri         /localpath/to/spark-2.2.0-bin-hadoop2.7.tgz

run spark shell
./bin/spark-shell --master mesos://127.0.0.1:5050

#Mesos UI
http://127.0.0.1:5050