Java Spark collect（）JavaDD失败，内存错误（EMR集群）

Question

下面是我的代码：

JavaRDD<WebLabPurchasesDataObject> allRecords = context.textFile(inputFile).map (
        data -> {
            String[] fields = data.split(",", -1);

            String hitDay = fields[0];
            String treatmentName = fields[1];
            String sessionId = fields[2];

            return new WebLabPurchasesDataObject(hitDay,treatmentName,sessionId);   
        });

allRecords.cache();

List<WebLabPurchasesDataObject> webLabRddAllRecordsList = allRecords.collect();

每次尝试运行此代码时，我都会得到Java.lang.OutofMemoryError：Java堆空间。据我所知，Spark正在我的主节点上执行collect（）操作。那么有没有什么方法可以增加内存，使它能够运行程序呢？

18/03/15 16:35:48 WARN scheduler.TaskSetManager: Lost task 1.0 in stage 2.0 (TID 5, ip-1443-405-18-1544.us-west-2.compute.internal): java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3236)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1877)
at java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1786)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1189)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:44)
at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:80)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)    18/03/15 16:35:49 ERROR cluster.YarnScheduler: Lost executor 1 on ip-43ew55-154.us-west-2.compute.internal: remote Akka client disassociated

Answer 1

正如您所知道的，由于结果是在驱动程序上收集的，您需要增加驱动程序内存。默认值是1GB，这对您的情况来说是不够的。

在创建sparksession/sparkcontext:spark.driver.memory时添加此配置，其值更大:2g或3g。如果您正在使用spark-shell，那么在启动spark-shell时将此作为附加选项传递:--driver-memory 3g，用于3GB内存。

我还建议您阅读更多关于这里描述的配置的信息：https://spark.apache.org/docs/latest/configuration.html