Scala：异常线程"main"java.lang.NoClassDefFoundError： org/apache/log4j/LogManager

Question

我是Scala的初学者，我试图在Scala中运行一个模型，但面临一些问题：

以下是文件：

package com.salesforce.hw.titanic

import com.salesforce.op._
import com.salesforce.op.features.FeatureBuilder
import com.salesforce.op.features.types._
import com.salesforce.op.readers.DataReaders
import com.salesforce.op.stages.impl.classification._
import org.apache.spark.SparkConf
import org.apache.spark.sql.SparkSession
import org.apache.log4j.{Level, LogManager}



/**
 * A minimal Titanic Survival example with TransmogrifAI
 */



object OpTitanicMini {

  case class Passenger
  (
    id: Long,
    survived: Double,
    pClass: Option[Long],
    name: Option[String],
    sex: Option[String],
    age: Option[Double],
    sibSp: Option[Long],
    parCh: Option[Long],
    ticket: Option[String],
    fare: Option[Double],
    cabin: Option[String],
    embarked: Option[String]
  )



  def main(args: Array[String]): Unit = {
    LogManager.getLogger("com.salesforce.op").setLevel(Level.ERROR)
    implicit val spark = SparkSession.builder.config(new SparkConf()).getOrCreate()
    import spark.implicits._

    // Read Titanic data as a DataFrame
    val pathToData = Option(args(0))
    val passengersData = DataReaders.Simple.csvCase[Passenger](pathToData, key = _.id.toString).readDataset().toDF()

    // Automated feature engineering
    val (survived, features) = FeatureBuilder.fromDataFrame[RealNN](passengersData, response = "survived")
    val featureVector = features.toSeq.autoTransform()

    // Automated feature selection
    val checkedFeatures = survived.sanityCheck(featureVector, checkSample = 1.0, removeBadFeatures = true)

    // Automated model selection
    val (pred, raw, prob) = BinaryClassificationModelSelector().setInput(survived, checkedFeatures).getOutput()
    val model = new OpWorkflow().setInputDataset(passengersData).setResultFeatures(pred).train()

    println("Model summary:\n" + model.summaryPretty())








  }

}

当我试图运行它时，我得到了以下错误：

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/log4j/LogManager
  at com.salesforce.hw.titanic.OpTitanicMini$.main(OpTitanicMini.scala:72)
  at com.salesforce.hw.titanic.OpTitanicMini.main(OpTitanicMini.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.log4j.LogManager
  at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
  ... 2 more

我试图研究这个问题，发现了这篇博文，我尝试了那篇博文所说的：

我的log4j。属性文件如下所示：

log4j.rootCategory=INFO, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n

# Settings to quiet third party logs that are too verbose
log4j.logger.Remoting=ERROR
log4j.logger.org.eclipse.jetty=ERROR
log4j.logger.org.spark_project.jetty=WARN
log4j.logger.org.spark_project.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
log4j.logger.org.apache.parquet=ERROR
log4j.logger.parquet=ERROR

# Change this to set Hadoop log level
log4j.logger.org.apache.hadoop=ERROR

# SPARK-9183: Settings to avoid annoying messages when looking up nonexistent UDFs in SparkSQL with Hive support
log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL
log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR

# Set the default spark-shell log level to WARN. When running the spark-shell, the
# log level for this class is used to overwrite the root logger's log level, so that
# the user can have different defaults for the shell and regular Spark apps.
log4j.logger.org.apache.spark.repl.Main=WARN

# Change this to set Spark log level
log4j.logger.org.apache.spark=ERROR

# Breeze
log4j.logger.breeze.optimize=FATAL

# BLAS & LAPACK
log4j.logger.com.github.fommil.netlib=ERROR

# TransmogrifAI logging
log4j.logger.com.salesforce.op=INFO
log4j.logger.com.salesforce.op.utils.spark.OpSparkListener=OFF

# Helloworld logging
log4j.logger.com.salesforce.hw=INFO

我尝试了博客文章中提到的步骤，但仍然面临同样的问题，我如何解决这个问题？

Answer 1

LogManager类附带一个Spark依赖项。确保有org。阿帕奇。spark:spark core，org。阿帕奇。spark:spark mlib，org。阿帕奇。spark:spark sql及其在运行时对类路径的所有可传递依赖关系。

我们这里有一个sbt项目的例子，你可以看看。