Giraph 源码编译常见错误

• Giraph 源码编译命令：

        

1. giraph_basedir $ mvn -Phadoop_yarn -Dhadoop.version={指定hadoop版本} -DskipTests clean package

or

1. giraph_basedir $ mvn -Phadoop_2 -Dhadoop.version={指定hadoop版本} -DskipTests clean package

如果一切正常，在giraph_basedir/giraph-dist目录下已经生成目标 jar

• Giraph 编译中常见错误
  

1. hadoop版本问题

[ERROR] /home/cyf/giraph-release/giraph-core/target/munged/main/org/apache/giraph/job/GiraphJob.java:[213,10] 错误: setPingInterval(Configuration,int)在Client中不是公共的; 无法从外部程序包中对其进行访问

[INFO] 1 error

[INFO] -------------------------------------------------------------

[INFO] ------------------------------------------------------------------------

[INFO] Reactor Summary:

[INFO] 

[INFO] Apache Giraph Parent ............................... SUCCESS [  3.658 s]

[INFO] Apache Giraph Core ................................. FAILURE [  5.584 s]

[INFO] Apache Giraph Blocks Framework ..................... SKIPPED

[INFO] Apache Giraph Examples ............................. SKIPPED

[INFO] Apache Giraph Distribution ......................... SKIPPED

[INFO] ------------------------------------------------------------------------

[INFO] BUILD FAILURE

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 9.558 s

[INFO] Finished at: 2016-08-28T20:10:25+08:00

[INFO] Final Memory: 41M/709M

[INFO] ------------------------------------------------------------------------

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.0:compile (default-compile) on project giraph-core: Compilation failure

[ERROR] /home/cyf/giraph-release/giraph-core/target/munged/main/org/apache/giraph/job/GiraphJob.java:[213,10] 错误: setPingInterval(Configuration,int)在Client中不是公共的; 无法从外部程序包中对其进行访问

[ERROR] -> [Help 1]

[ERROR] 

[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

[ERROR] 

[ERROR] For more information about the errors and possible solutions, please read the following articles:

[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

[ERROR] 

[ERROR] After correcting the problems, you can resume the build with the command

[ERROR]   mvn <goals> -rf :giraph-core

解决方法：高版本的 Hadoop 的 hadoop/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java 文件中的 setPingInterval(Configuration,int) 方法的访问权限已经从public 变为 default， 包外已经不能直接访问。只能降低Hadoop版本，2.5.2，2.6.3，2.7.2 还都是public 类型，以上版本都变成了default。

2. SASL_PROPS 变量问题

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.0:compile (default-compile) on project giraph-core: Compilation failure: Compilation failure:

[ERROR] /home/cyf/giraph-release/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyClient.java:[84,67] 错误: 找不到符号

[ERROR] 符号:   变量 SASL_PROPS

[ERROR] 位置: 类 SaslRpcServer

[ERROR] /home/cyf/giraph-release/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyServer.java:[105,61] 错误: 找不到符号

[ERROR] -> [Help 1]

[ERROR] 

[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

[ERROR] 

[ERROR] For more information about the errors and possible solutions, please read the following articles:

[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

[ERROR] 

[ERROR] After correcting the problems, you can resume the build with the command

[ERROR]   mvn <goals> -rf :giraph-core

解决方法：

打开pom.xml文件，定位到profile为hadoop_yarn的位置

1. <profile>
2.      <id>hadoop_yarn</id>
3.      <modules>
4.        <module>giraph-dist</module>
5.      </modules>
6.      <properties>
7.        <hadoop.version>SET_HADOOP_VERSION_USING_MVN_DASH_D_OPTION</hadoop.version>
8.        <munge.symbols>PURE_YARN,STATIC_SASL_SYMBOL</munge.symbols>
9.        <!-- TODO: add these checks eventually -->
10.        <project.enforcer.skip>true</project.enforcer.skip>
11.        <giraph.maven.dependency.plugin.skip>true</giraph.maven.dependency.plugin.skip>
12.        <giraph.maven.duplicate.finder.skip>true</giraph.maven.duplicate.finder.skip>
13.      </properties>
14.      <dependencies>
15.        <!-- sorted lexicographically -->
16.        <dependency>
17.          <groupId>org.apache.hadoop</groupId>
18.          <artifactId>hadoop-common</artifactId>
19.        </dependency>
20.        <dependency>
21.          <groupId>org.apache.hadoop</groupId>
22.          <artifactId>hadoop-mapreduce-client-common</artifactId>
23.        </dependency>
24.        <dependency>
25.          <groupId>org.apache.hadoop</groupId>
26.          <artifactId>hadoop-mapreduce-client-core</artifactId>
27.        </dependency>
28.        <dependency>
29.          <groupId>org.apache.hadoop</groupId>
30.          <artifactId>hadoop-yarn-common</artifactId>
31.          <version>${hadoop.version}</version>
32.        </dependency>
33.        <dependency>
34.          <groupId>org.apache.hadoop</groupId>
35.          <artifactId>hadoop-yarn-server-common</artifactId>
36.          <version>${hadoop.version}</version>
37.        </dependency>
38.        <dependency>
39.          <groupId>org.apache.hadoop</groupId>
40.          <artifactId>hadoop-yarn-server-resourcemanager</artifactId>
41.          <version>${hadoop.version}</version>
42.        </dependency>
43.        <dependency>
44.          <groupId>org.apache.hadoop</groupId>
45.          <artifactId>hadoop-yarn-server-nodemanager</artifactId>
46.          <version>${hadoop.version}</version>
47.        </dependency>
48.        <dependency>
49.          <groupId>org.apache.hadoop</groupId>
50.          <artifactId>hadoop-yarn-server-tests</artifactId>
51.          <version>${hadoop.version}</version>
52.          <type>test-jar</type>
53.        </dependency>
54.      </dependencies>
55.    </profile>

将文件中红色部分删除即可。

3. 强制遵循编码规范

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-checkstyle-plugin:2.15:check (default) on project giraph-core: You have 1 Checkstyle violation. -> [Help 1]

[ERROR] 

[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

[ERROR] 

[ERROR] For more information about the errors and possible solutions, please read the following articles:

[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

[ERROR] 

[ERROR] After correcting the problems, you can resume the build with the command

[ERROR]   mvn <goals> -rf :giraph-core

解决方法：使用了mvn install 命令进行编译，改成mvn package就行了。

1. giraph_basedir $ mvn -Phadoop_yarn -Dhadoop.version={指定hadoop版本} -DskipTests clean package

4. 没有指定hbase等的版本？

[INFO] Reactor Summary:

[INFO] 

[INFO] Apache Giraph Parent ............................... SUCCESS [  3.750 s]

[INFO] Apache Giraph Core ................................. SUCCESS [ 49.851 s]

[INFO] Apache Giraph Blocks Framework ..................... SUCCESS [ 21.132 s]

[INFO] Apache Giraph Examples ............................. SUCCESS [ 19.574 s]

[INFO] Apache Giraph Distribution ......................... FAILURE [  0.024 s]

[INFO] ------------------------------------------------------------------------

[INFO] BUILD FAILURE

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 01:34 min

[INFO] Finished at: 2016-08-28T19:54:07+08:00

[INFO] Final Memory: 48M/766M

[INFO] ------------------------------------------------------------------------

[ERROR] Failed to execute goal on project giraph-dist: Could not resolve dependencies for project org.apache.giraph:giraph-dist:pom:1.2.0-RC0: The following artifacts could not be resolved: org.apache.giraph:giraph-hbase:jar:1.2.0-RC0, org.apache.giraph:giraph-accumulo:jar:1.2.0-RC0, org.apache.giraph:giraph-hcatalog:jar:1.2.0-RC0, org.apache.giraph:giraph-gora:jar:1.2.0-RC0, org.apache.giraph:giraph-kibble:jar:1.2.0-RC0, org.apache.giraph:giraph-rexster-io:jar:1.2.0-RC0: Failure to find org.apache.giraph:giraph-hbase:jar:1.2.0-RC0 in http://repo1.maven.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced -> [Help 1]

org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal on project giraph-dist: Could not resolve dependencies for project org.apache.giraph:giraph-dist:pom:1.2.0-RC0: The following artifacts could not be resolved: org.apache.giraph:giraph-hbase:jar:1.2.0-RC0, org.apache.giraph:giraph-accumulo:jar:1.2.0-RC0, org.apache.giraph:giraph-hcatalog:jar:1.2.0-RC0, org.apache.giraph:giraph-gora:jar:1.2.0-RC0, org.apache.giraph:giraph-kibble:jar:1.2.0-RC0, org.apache.giraph:giraph-rexster-io:jar:1.2.0-RC0: Failure to find org.apache.giraph:giraph-hbase:jar:1.2.0-RC0 in http://repo1.maven.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced

at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies(LifecycleDependencyResolver.java:221)

at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies(LifecycleDependencyResolver.java:127)

at org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved(MojoExecutor.java:245)

at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:199)

at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)

at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)

at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)

at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)

at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)

at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)

at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)

at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)

at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)

at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)

at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)

at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)

at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)

at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)

at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)

Caused by: org.apache.maven.project.DependencyResolutionException: Could not resolve dependencies for project org.apache.giraph:giraph-dist:pom:1.2.0-RC0: The following artifacts could not be resolved: org.apache.giraph:giraph-hbase:jar:1.2.0-RC0, org.apache.giraph:giraph-accumulo:jar:1.2.0-RC0, org.apache.giraph:giraph-hcatalog:jar:1.2.0-RC0, org.apache.giraph:giraph-gora:jar:1.2.0-RC0, org.apache.giraph:giraph-kibble:jar:1.2.0-RC0, org.apache.giraph:giraph-rexster-io:jar:1.2.0-RC0: Failure to find org.apache.giraph:giraph-hbase:jar:1.2.0-RC0 in http://repo1.maven.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced

at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve(DefaultProjectDependenciesResolver.java:211)

at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies(LifecycleDependencyResolver.java:195)

... 23 more

Caused by: org.eclipse.aether.resolution.DependencyResolutionException: The following artifacts could not be resolved: org.apache.giraph:giraph-hbase:jar:1.2.0-RC0, org.apache.giraph:giraph-accumulo:jar:1.2.0-RC0, org.apache.giraph:giraph-hcatalog:jar:1.2.0-RC0, org.apache.giraph:giraph-gora:jar:1.2.0-RC0, org.apache.giraph:giraph-kibble:jar:1.2.0-RC0, org.apache.giraph:giraph-rexster-io:jar:1.2.0-RC0: Failure to find org.apache.giraph:giraph-hbase:jar:1.2.0-RC0 in http://repo1.maven.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced

at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:384)

at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve(DefaultProjectDependenciesResolver.java:205)

... 24 more

Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: The following artifacts could not be resolved: org.apache.giraph:giraph-hbase:jar:1.2.0-RC0, org.apache.giraph:giraph-accumulo:jar:1.2.0-RC0, org.apache.giraph:giraph-hcatalog:jar:1.2.0-RC0, org.apache.giraph:giraph-gora:jar:1.2.0-RC0, org.apache.giraph:giraph-kibble:jar:1.2.0-RC0, org.apache.giraph:giraph-rexster-io:jar:1.2.0-RC0: Failure to find org.apache.giraph:giraph-hbase:jar:1.2.0-RC0 in http://repo1.maven.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced

at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:444)

at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:246)

at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:367)

... 25 more

Caused by: org.eclipse.aether.transfer.ArtifactNotFoundException: Failure to find org.apache.giraph:giraph-hbase:jar:1.2.0-RC0 in http://repo1.maven.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced

at org.eclipse.aether.internal.impl.DefaultUpdateCheckManager.newException(DefaultUpdateCheckManager.java:231)

at org.eclipse.aether.internal.impl.DefaultUpdateCheckManager.checkArtifact(DefaultUpdateCheckManager.java:206)

at org.eclipse.aether.internal.impl.DefaultArtifactResolver.gatherDownloads(DefaultArtifactResolver.java:585)

at org.eclipse.aether.internal.impl.DefaultArtifactResolver.performDownloads(DefaultArtifactResolver.java:503)

at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:421)

... 27 more

[ERROR] 

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

[ERROR] 

[ERROR] For more information about the errors and possible solutions, please read the following articles:

[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException

[ERROR] 

[ERROR] After correcting the problems, you can resume the build with the command

[ERROR]   mvn <goals> -rf :giraph-dist

解决方法：

好像没有影响，打开giraph-dist目录，目标 jar 已经生成。不过按上面的命令好像不会出现这个问题，不知道当时当时怎么搞的遇到了这个错误。