带StanfordCoreNLP的Piglatin jodatime错误

Question

我试图创建一个Pig UDF，它使用通过sista Scala API接口的Stanford CoreNLP包提取tweet中提到的位置。它在本地使用“SBT run”运行时工作良好，但在从PIG调用时抛出“java.lang.NosuchMethodError”异常：

从标签器edu/stanford/nlp/models/pos-tagger/english-left3words/english-left3words/english-left3words/english-left3words-distsim.tagger从edu/stanford/nlp/models/pos-tagger/english-left3words/english-left3words/english-left3words/english-left3words-distsim.tagger从从edu/stanford/nlp/models/ner/english.muc.7class.distsim.crf.ser.gz加载分类器...2013-06-14 10:48:02,108[低内存检测器]INFO org.apache.pig.impl.util.spillablememorymanager-第一个内存处理程序调用-收集阈值init=18546688(18112k)使用=358671232(350264k)提交=366542848(357952k)max=699072512(682688k)完成[5.0秒]。从edu/stanford/nlp/models/ner/english.conll.4class.distsim.crf.ser.gz加载分类器...2013-06-14 10:48:10,522[低内存检测器]INFO org.apache.pig.impl.util.spillablememorymanager-第一个内存处理程序调用-使用阈值init=18546688(18112k)使用=590012928(576184k)提交=597786624(583776k)max=699072512(682688k)完成[5.6秒]。2013-06-14 10:48:11,469[Thread-11]警告org.apache.hadoop.mapred.localjobrunner-job_local_0001 java.lang.nosuchmethoderror：org.joda.time.duration.compareto（lorg/joda/time/readableduration；）I在edu.stanford.nlp.time.sutime$duration.compareto（sutime.java:3406)，在edu.stanford.nlp.time.sutime$duration.max（stanford.nlp.time.sutime$range.(sutime.java:3793)位于edu.stanford.nlp.time.sutime.(sutime.java:570)

以下是相关代码：

object CountryTokenizer {
  def tokenize(text: String): String = {
    val locations = TweetEntityExtractor.NERLocationFilter(text)
    println(locations)
    locations.map(x => Cities.country(x)).flatten.mkString(" ")
  }
}

class PigCountryTokenizer extends EvalFunc[String] {
  override def exec(tuple: Tuple): java.lang.String = {
    val text: java.lang.String = Util.cast[java.lang.String](tuple.get(0))
    CountryTokenizer.tokenize(text)
  }
}

object TweetEntityExtractor {
    val processor:Processor = new CoreNLPProcessor()


    def NERLocationFilter(text: String): List[String] =  {
        val doc = processor.mkDocument(text)

        processor.tagPartsOfSpeech(doc)
        processor.lemmatize(doc)
        processor.recognizeNamedEntities(doc)

        val locations = doc.sentences.map(sentence => {
            val entities = sentence.entities.map(List.fromArray(_)) match {
                case Some(l) => l
                case _ => List()
            }
            val words = List.fromArray(sentence.words)

            (words zip entities).filter(x => {
                x._1 != "" && x._2 == "LOCATION" 
            }).map(_._1)
        })
        List.fromArray(locations).flatten
    }
}

我正在使用sbt-assembly来构造一个fat-jar，因此joda-time jar文件应该是可以访问的。怎么回事？

Answer 1

Pig附带了自己的joda-time(1.6)，该版本与2.x不兼容。