sqoop错误[主]工具。进口工具

Question

嗨，当我尝试将MySql表导入hdfs时，我遇到了一个错误：

sqoop导入--连接jdbc：mysql://dba-virtual-machine/test--用户名Social--表格文本行--列行--目标目录/文本行

 2014-06-16 07:43:24,308 INFO  [main] manager.MySQLManager: Preparing to use a MySQL streaming resultset.
2014-06-16 07:43:24,319 INFO  [main] tool.CodeGenTool: Beginning code generation
2014-06-16 07:43:25,004 INFO  [main] manager.SqlManager: Executing SQL statement: SELECT t.* FROM textlines AS t LIMIT 1
2014-06-16 07:43:25,026 INFO  [main] manager.SqlManager: Executing SQL statement: SELECT t.* FROM textlines AS t LIMIT 1
2014-06-16 07:43:25,060 INFO  [main] orm.CompilationManager: HADOOP_MAPRED_HOME is /home/socio/hadoop
Note: /tmp/sqoop-socio/compile/4d35eb51b72ffcde0e815ad25857257c/textlines.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
2014-06-16 07:43:26,755 INFO  [main] orm.CompilationManager: Writing jar file: /tmp/sqoop-socio/compile/4d35eb51b72ffcde0e815ad25857257c/textlines.jar
2014-06-16 07:43:26,869 WARN  [main] manager.MySQLManager: It looks like you are importing from mysql.
2014-06-16 07:43:26,870 WARN  [main] manager.MySQLManager: This transfer can be faster! Use the --direct
2014-06-16 07:43:26,870 WARN  [main] manager.MySQLManager: option to exercise a MySQL-specific fast path.
2014-06-16 07:43:26,870 INFO  [main] manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
2014-06-16 07:43:26,877 INFO  [main] mapreduce.ImportJobBase: Beginning import of textlines
2014-06-16 07:43:26,878 INFO  [main] Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/socio/hbase/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/socio/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
2014-06-16 07:43:27,894 WARN  [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2014-06-16 07:43:27,961 INFO  [main] Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
2014-06-16 07:43:28,945 INFO  [main] Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
2014-06-16 07:43:29,030 INFO  [main] Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
2014-06-16 07:43:29,031 INFO  [main] jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
2014-06-16 07:43:29,944 INFO  [main] mapreduce.JobSubmitter: Cleaning up the staging area file:/tmp/hadoop-socio/mapred/staging/socio1192068052/.staging/job_local1192068052_0001
2014-06-16 07:43:29,945 ERROR [main] security.UserGroupInformation: PriviledgedActionException as:socio (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not exist: hdfs://mac:9000/home/socio/sqoop/lib/commons-io-1.4.jar
2014-06-16 07:43:29,945 ERROR [main] tool.ImportTool: Encountered IOException running import job: java.io.FileNotFoundException: File does not exist: hdfs://mac:9000/home/socio/sqoop/lib/commons-io-1.4.jar
    at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1110)
    at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)
    at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288)
    at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:224)
    at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:93)
    at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:57)
    at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:264)
    at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:300)
    at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:387)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1286)
    at org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:186)
    at org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:159)
    at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:239)
    at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:600)
    at org.apache.sqoop.manager.MySQLManager.importTable(MySQLManager.java:118)
    at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:413)
    at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:502)
    at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
    at org.apache.sqoop.Sqoop.main(Sqoop.java:238)

此文件<代码>hdfs://mac:9000/home/socio/sqoop/lib/commons-io-1.4。jar在这里：主页/社会/sqoop/lib/commons-io-1.4。jar。我不知道我要修改什么

正在等待响应

Answer 1

在我的例子中，重新启动控制台（注销，然后登录）工作。

我这样做是因为我没有编辑mapred的权限。地点xml。文件

Answer 2

我遇到了同样的问题，我使用的是hadoop 2.5.2和sqoop 1.4.5。这个问题与不适当的hadoop配置有关。在我的情况下，地图是红色的。地点中缺少hadoop的xml/etc/hadoop目录。

我将此添加到/etc/hadoop/mapred。地点xml

    <property>
      <name>mapreduce.framework.name</name>
      <value>yarn</value>
    </property>

现在它工作得很好。

Answer 3

我解决了添加mappred-site.xml问题：

<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>

此外，hadoop和sqoop之间存在兼容性问题