RDD,DataFrames,Spark SQL：360度比较？[副本]

Question

同行,

+---------------------------+------------------+-----------------+--------------------------------------+  
|          Feature          |       RDD        | Data Frame (DF) |              Spark SQL               |  
+---------------------------+------------------+-----------------+--------------------------------------+  
| First-class Spark citizen | Yes              | Yes             | Yes                                  |  
| Native? [4]               | Core abstraction | API             | Module                               |  
| Generation [5]            | 1st              | 2nd             | 3rd                                  |  
| Abstraction [4,5,         | Low-level API    | Data processing | SQL-based                            |  
| Ansi standard SQL         | None             | Some            | near-ansi [5]                        |  
| Optimization              | None             | Catalyst [9]    | Catalyst [9]                         |  
| Performance [3,4,8        | Mix views        | Mix views       | Mix Views                            |  
| Memory                    | ?                | ?               | ?                                    |  
| Programming speed         | Slow             | Fast            | Faster if dealing with SQL workloads |  
+---------------------------+------------------+-----------------+--------------------------------------+  

[3] Introducing DataFrames in Apache Spark for Large Scale Data Science by data bricks   
[4] Spark RDDs vs DataFrames vs SparkSQL by Hortonworks  
[5] A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets When to use them and why by data bricks  
[6] Introducing Apache Spark 2.0 by data bricks  
[7] Spark RDD vs Spark SQL Performance comparison using Spark Java APIs  
[8] Spark sql queries vs dataframe functions on Stackoverflow  
[9] Spark SQL: Relational Data Processing in Spark by data bricks, MIT, UC Berkeley

我认为我修改过的问题仍然没有答案。请求取消标记为重复项。

Answer 1

我个人的看法：

• 在性能方面，您应该使用DataFrames/DataSets或Spark SQL。RDD未通过Catalyst Optimizer和Wungsten Project进行优化。
• 就灵活性而言，我认为使用Dataframe API将为您提供更多的可读性，并且比使用SQL（特别是使用Scala或Python)，更具动态性，尽管如果您愿意，您可以将它们混合使用
• 只有当您要迁移配置单元工作负载或通过ODBC从BI工具连接到spark thrift server时，我才会使用SQL