TypeError：PySpark中的Xgboost无法调用“JavaPackage”对象

Question

我正在尝试为我的PySpark笔记本提供Scala Xgboost API。以下是本博客：https://towardsdatascience.com/pyspark-and-xgboost-integration-tested-on-the-kaggle-titanic-dataset-4e75a568bdb但是，继续遇到以下错误：

spark._jvm.ml.dmlc.xgboost4j.scala.spark.XGBoostEstimator
<py4j.java_gateway.JavaPackage at 0x7fa650fe7a58>

from sparkxgb import XGBoostEstimator

xgboost = XGBoostEstimator(
    featuresCol="features", 
    labelCol="Survival", 
    predictionCol="prediction"
)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-18-1765fb9e3344> in <module>
      4     featuresCol="features",
      5     labelCol="Survival",
----> 6     predictionCol="prediction"
      7 )

~/spark-assembly-2.4.0-twttr-kryo3-scala2128-hadoop2.9.2.t05/python/pyspark/__init__.py in wrapper(self, *args, **kwargs)
    108             raise TypeError("Method %s forces keyword arguments." % func.__name__)
    109         self._input_kwargs = kwargs
--> 110         return func(self, **kwargs)
    111     return wrapper
    112 

~/local/spark-3536cd7a-6188-4ca8-b3d0-57d42cd01531/userFiles-0a0d90bc-96b4-43f2-bf21-00ae0e6f7309/sparkxgb.zip/sparkxgb/xgboost.py in __init__(self, checkpoint_path, checkpointInterval, missing, nthread, nworkers, silent, use_external_memory, baseMarginCol, featuresCol, labelCol, predictionCol, weightCol, base_score, booster, eval_metric, num_class, num_round, objective, seed, alpha, colsample_bytree, colsample_bylevel, eta, gamma, grow_policy, max_bin, max_delta_step, max_depth, min_child_weight, reg_lambda, scale_pos_weight, sketch_eps, subsample, tree_method, normalize_type, rate_drop, sample_type, skip_drop, lambda_bias)
    113 
    114         super(XGBoostEstimator, self).__init__()
--> 115         self._java_obj = self._new_java_obj("ml.dmlc.xgboost4j.scala.spark.XGBoostEstimator", self.uid)
    116         self._create_params_from_java()
    117         self._setDefault(

~/spark-assembly-2.4.0-twttr-kryo3-scala2128-hadoop2.9.2.t05/python/pyspark/ml/wrapper.py in _new_java_obj(java_class, *args)
     65             java_obj = getattr(java_obj, name)
     66         java_args = [_py2java(sc, arg) for arg in args]
---> 67         return java_obj(*java_args)
     68 
     69     @staticmethod

TypeError: 'JavaPackage' object is not callable

我已经在谷歌上搜索了这个错误，并尝试了以下内容。我从这个博客上得到了所有的想法https://github.com/JohnSnowLabs/spark-nlp/issues/232:

1. 确保Xgboost4j位于SPARK\u DIST\u类路径中。已检查

    $echo $SPARK_DIST_CLASSPATH |  tr " " "\n" | grep 'xgboost4j' | rev | cut -d'/' -f1 | rev
    xgboost4j-0.72.jar
    xgboost4j-spark.72.jar

'export PYSPARK_SUBMIT_ARGS="--conf spark.jars=$SPARK_HOME/jars/* --conf spark.driver.extraClassPath=$SPARK_HOME/jars/* --conf spark.executor.extraClassPath=$SPARK_HOME/jars/* pyspark-shell"',

硬件信息：

• 计算机：Linux
• 使用Jupyter笔记本
• Spark版本2.4.0
• 蟒蛇3。6

Answer 1

我发现了问题，问题是sparkxbg.zip（我在网上下载的）是为xgboost4j-0.72编写的。但是，我的jars来自xgoost4j-0.9。API已完全更改。因此，0.9版本没有任何名为ml.dmlc.xgboost4j.scala.spark.XGBoost估算器的类。因此出现了错误。您可以在下面看到API的区别：

https://github.com/dmlc/xgboost/tree/release_0.72/jvm-packages/xgboost4j-spark/src/main/scala/ml/dmlc/xgboost4j/scala/spark

vs

https://github.com/dmlc/xgboost/tree/v0.90/jvm-packages/xgboost4j-spark/src/main/scala/ml/dmlc/xgboost4j/scala/spark