Scalding初探之三：Hadoop实战

Java版本

如果在Scala工程中混入Java类导致java版本不match，可能会抛错

java.lang.UnsupportedClassVersionError: XXX Unsupported major.minor version 51.0

在build.sbt中加入

javacOptions ++= Seq("-source", "1.6", "-target", "1.6")

轻松搞定

 

特别的技巧

读一读官网的Frequently asked questions，会获得一些特别的技巧哦

1 Missing data

Pass the option --tool.partialok to your job

2 Read a single reduced value from a pipe

Job.next & Source.toIterator

3 Cases classes

Define it outside of your Job

4 Hadoop jobConf

pass parameters to my hadoop job

hadoop jar myjar \ 
com.twitter.scalding.Tool \ 
-D mapred.output.compress=false \ 
-D mapred.child.java.opts=-Xmx2048m \ 
-D mapred.reduce.tasks=20 \ 
com.class.myclass \ 
--hdfs \ 
--input $input \ 
--output $output 

append parameters to jobConf

class WordCountJob(args : Args) extends Job(args) {
// Prior to 0.9.0 we need the mode, after 0.9.0 mode is a def on Job.
override def config(implicit m: Mode): Map[AnyRef,AnyRef] = {
   super.config ++ Map ("my.job.name" -> "my new job name")
  }