当前位置: 首页 > 工具软件 > Apache Giraph > 使用案例 >

【Giraph】Giraph安装配置及使用教程

景理
2023-12-01

Giraph配置教程

一、Hadoop 2.7分布式集群环境搭建
1.见:Hadoop 2.7分布式集群环境搭建

2.补充配置mapred-site.xml :

<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<value>4</value>
</property>
<property>
<name>mapred.map.tasks</name>
<value>4</value>
</property>

注:默认情况下,Hadoop允许同时运行两个映射程序。然而,Giraph的代码假设我们可以同时运行4个映射程序。因此,对于这个单节点的伪分布式部署,我们需要在mapred-site.xml中添加最后两个属性来反映这一需求。否则,一些单元测试将失败。

补充配置hdfs-site.xml:

<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/usr/local/hadoop/tmp/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/usr/local/hadoop/tmp/dfs/data</value>
</property>
</configuration>

3.运行hadoop

start-all.sh

二、maven安装并添加镜像
见博客:Linux安装Maven并配置阿里云镜像

三、Giraph配置
1.下载giraph

cd /usr/local
git clone https://github.com/apache/giraph.git

2.编译

cd giraph
mvn -Phadoop_2 -Dhadoop.version=2.7.6 -DskipTests clean package

四、运行giraph
1.创建tiny_graph.txt,输入:

[0,0,[[1,1],[3,3]]]
[1,0,[[0,1],[2,2],[3,1]]]
[2,0,[[1,2],[4,4]]]
[3,0,[[0,3],[1,1],[4,4]]]
[4,0,[[3,4],[2,4]]]

2.将文件拷贝到hdfs:

cd /usr/local
hdfs dfs -put  tiny_graph.txt  /
hdfs dfs -ls / 检查是否拷贝成功

3.提交任务
单点运行:

hadoop jar /usr/local/giraph/giraph-examples/target/giraph-examples-1.3.0-SNAPSHOT-for-hadoop-2.7.6-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPathsComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /tiny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /shorttestpaths -w 1 -ca giraph.SplitMasterWorker=false

集群运行:

hadoop jar /usr/local/giraph/giraph-examples/target/giraph-examples-1.3.0-SNAPSHOT-for-hadoop-2.7.6-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPathsComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /tiny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /shorttestpaths -w 6 -ca mapred.job.tracker=192.168.1.1

4.查看结果

hadoop dfs -cat /user/hduser/output/shortestpaths/p* | less


日志端口:master:8082

 类似资料: