当前位置: 首页 > 工具软件 > Apache Slider > 使用案例 >

presto + slider 提交计算至yarn

淳于健
2023-12-01

10.112.28.240 prestocli

10.183.225.158 perstoser

 

hive-site.xml

useUnicode=true&characterEncoding=UTF-8&autoReconnect=true 

 

kinit -k -t /etc/presto/presto.keytab presto

hadoop fs -mkdir -p /user/presto/.slider/package/PRESTO/
hadoop fs -put ./presto-yarn-package-1.3-SNAPSHOT-0.176.zip hdfs://ns/user/presto/.slider/package/PRESTO/

hadoop fs -mkdir -p /etc/presto/
hadoop fs -put /etc/presto/presto.keytab /etc/presto/

  

 

启动协调者:

/usr/bin/nohup /xxx/soft/lesports-presto-server-0.176/bin/launcher run --node-config /etc/presto/01/node.properties --jvm-config /etc/presto/01/jvm.config --config /etc/presto/01/config.properties --log-levels-file /etc/presto/01/log.properties >/xxx/soft/lesports-presto-server-0.176/stdout.log 2>&1 &

  

netstat -nlp | grep 20000

启动slider:
/usr/bin/nohup /letv/soft/xxx-slider-0.80.0-incubating/bin/startPresto.sh cloud_presto01 /etc/presto/01/slider/appConfig.json /etc/presto/01/slider/resources.json > /xxx/soft/lecloud-slider-0.80.0-incubating/stdout.log 2>&1 &

  



container-executor.cfg有min.user.id限制1000,需要修改user id:
usermod -u 1001 presto
cat /etc/passwd


在hdfs上建文件夹:
1. kinit 获取相应票据。
kinit -k -t /var/kerberos/krb5kdc/keytab/hadoop3_root.keytab presto/hadoop3@JENKIN.COM

hdfs dfs -mkdir -p /user/presto/.slider/cluster/

 查看hdfs文件:

[root@hadoop3 ~]# hadoop fs -ls /user/presto/.slider/cluster/
Found 1 items
drwxr-xr-x   - presto supergroup          0 2017-09-15 15:25 /user/presto/.slider/cluster/cloud_presto01

  

presto-cli查询:

1. 下载presto-cli

运行: ./presto-cli --server hadoop1:20000 --catalog hive --schema jenkindb

 

问题解决:

问题:failed: No worker nodes available 

解决:presto coordinate config.properties增加 node-scheduler.include-coordinator=true

问题:org.apache.hadoop.ipc.RemoteException: User: presto@JENKIN.COM is not allowed to impersonate root

解决:hive.properties增加 hive.hdfs.impersonation.enabled=false

 

转载于:https://www.cnblogs.com/kisf/p/7524947.html

 类似资料: