1、安装jupyter
2、安装sparkmagic
3、设置超时
"livy_server_heartbeat_timeout_seconds": 0,
4、设置集群模式
"session_configs": {
"driverMemory": "2G", "executorCores": 4, "proxyUser": "bernhard", "conf": { "spark.master": "yarn-cluster", "spark.jars.packages": "com.databricks:spark-csv_2.10:1.5.0" } }
5、配置hdp
livy.server.csrf_protection.enabled ==> false
6、启动notebook，配饰spark session
%load_ext sparkmagic.magics
%manage_spark
参考文档
https://github.com/jupyter-incubator/sparkmagic
https://github.com/bernhard-42/Sparkmagic-on-HDP