org.apache.kudu.client.NonRecoverableException: Caused by: java.lang.NullPointerException
顾穆冉
Flink运行时完整的报错如下：
org.apache.kudu.client.NonRecoverableException:
at org.apache.kudu.client.KuduException.transformException(KuduException.java:132) ~[kudu-client-1.9.0.jar:1.9.0]
at org.apache.kudu.client.KuduClient.joinAndHandleException(KuduClient.java:402) ~[kudu-client-1.9.0.jar:1.9.0]
at org.apache.kudu.client.KuduClient.openTable(KuduClient.java:245) ~[kudu-client-1.9.0.jar:1.9.0]
at com.ctg.bigdata.etl.sink.KuduClientWindowOrderedSink.invoke(KuduClientWindowOrderedSink.java:82) ~[classes/:?]
at com.ctg.bigdata.etl.sink.KuduClientWindowOrderedSink.invoke(KuduClientWindowOrderedSink.java:19) ~[classes/:?]
at org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:54) ~[flink-streaming-java_2.12-1.12.1.jar:1.12.1]
at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:71) ~[flink-streaming-java_2.12-1.12.1.jar:1.12.1]
at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:46) ~[flink-streaming-java_2.12-1.12.1.jar:1.12.1]
at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:26) ~[flink-streaming-java_2.12-1.12.1.jar:1.12.1]
at org.apache.flink.streaming.runtime.tasks.BroadcastingOutputCollector.collect(BroadcastingOutputCollector.java:75) ~[flink-streaming-java_2.12-1.12.1.jar:1.12.1]
at org.apache.flink.streaming.runtime.tasks.BroadcastingOutputCollector.collect(BroadcastingOutputCollector.java:32) ~[flink-streaming-java_2.12-1.12.1.jar:1.12.1]
at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:50) ~[flink-streaming-java_2.12-1.12.1.jar:1.12.1]
at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:28) ~[flink-streaming-java_2.12-1.12.1.jar:1.12.1]
at org.apache.flink.streaming.api.operators.TimestampedCollector.collect(TimestampedCollector.java:50) ~[flink-streaming-java_2.12-1.12.1.jar:1.12.1]
at com.ctg.bigdata.etl.run.FlinkRunner$9.process(FlinkRunner.java:225) ~[classes/:?]
at com.ctg.bigdata.etl.run.FlinkRunner$9.process(FlinkRunner.java:216) ~[classes/:?]
at org.apache.flink.streaming.runtime.operators.windowing.functions.InternalSingleValueProcessWindowFunction.process(InternalSingleValueProcessWindowFunction.java:60) ~[flink-streaming-java_2.12-1.12.1.jar:1.12.1]
at org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.emitWindowContents(WindowOperator.java:577) ~[flink-streaming-java_2.12-1.12.1.jar:1.12.1]
at org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.onEventTime(WindowOperator.java:486) ~[flink-streaming-java_2.12-1.12.1.jar:1.12.1]
at org.apache.flink.streaming.api.operators.InternalTimerServiceImpl.advanceWatermark(InternalTimerServiceImpl.java:302) ~[flink-streaming-java_2.12-1.12.1.jar:1.12.1]
at org.apache.flink.streaming.api.operators.InternalTimeServiceManagerImpl.advanceWatermark(InternalTimeServiceManagerImpl.java:194) ~[flink-streaming-java_2.12-1.12.1.jar:1.12.1]
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.processWatermark(AbstractStreamOperator.java:626) ~[flink-streaming-java_2.12-1.12.1.jar:1.12.1]
at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitWatermark(OneInputStreamTask.java:193) ~[flink-streaming-java_2.12-1.12.1.jar:1.12.1]
at org.apache.flink.streaming.runtime.streamstatus.StatusWatermarkValve.findAndOutputNewMinWatermarkAcrossAlignedChannels(StatusWatermarkValve.java:196) ~[flink-streaming-java_2.12-1.12.1.jar:1.12.1]
at org.apache.flink.streaming.runtime.streamstatus.StatusWatermarkValve.inputWatermark(StatusWatermarkValve.java:105) ~[flink-streaming-java_2.12-1.12.1.jar:1.12.1]
at org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.processElement(StreamTaskNetworkInput.java:206) ~[flink-streaming-java_2.12-1.12.1.jar:1.12.1]
at org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:174) ~[flink-streaming-java_2.12-1.12.1.jar:1.12.1]
at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65) ~[flink-streaming-java_2.12-1.12.1.jar:1.12.1]
at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:395) ~[flink-streaming-java_2.12-1.12.1.jar:1.12.1]
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:191) ~[flink-streaming-java_2.12-1.12.1.jar:1.12.1]
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:609) ~[flink-streaming-java_2.12-1.12.1.jar:1.12.1]
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:573) ~[flink-streaming-java_2.12-1.12.1.jar:1.12.1]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:755) ~[flink-runtime_2.12-1.12.1.jar:1.12.1]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:570) ~[flink-runtime_2.12-1.12.1.jar:1.12.1]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_271]
Caused by: java.lang.NullPointerException查找kudu日志发现
0627 13:18:38.000976 (+2958970us) negotiation.cc:304] Negotiation complete: Timed out: Server connection negotiation failed: server connection from 192.168.18.50:57990
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":248,"thread_start_us":109,"threads_started":1}
I0627 13:18:41.942245 31806 maintenance_manager.cc:339] P b38d7989e274424cb050444d35f080ca: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.083338
I0627 13:18:41.947583 31805 maintenance_manager.cc:544] P b38d7989e274424cb050444d35f080ca: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.005s user 0.001s sys 0.001s Metrics: {"bytes_written":179,"delete_count":0,"fdatasync":3,"fdatasync_us":2608,"lbm_write_time_us":121,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I0627 13:18:41.947857 31806 maintenance_manager.cc:339] P b38d7989e274424cb050444d35f080ca: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I0627 13:18:41.952919 31805 maintenance_manager.cc:544] P b38d7989e274424cb050444d35f080ca: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.005s user 0.001s sys 0.001s Metrics: {"cfile_cache_hit":2,"cfile_cache_hit_bytes":2873,"cfile_cache_miss":2,"cfile_cache_miss_bytes":79,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"fdatasync":3,"fdatasync_us":2294,"lbm_read_time_us":39,"lbm_reads_lt_1ms":6,"lbm_write_time_us":86,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":162,"threads_started":1,"update_count":77}
I0627 13:18:45.168354 31865 hms_notification_log_listener.cc:227] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: b38d7989e274424cb050444d35f080ca, Raft Consensus state: current_term: 85 leader_uuid: "d11ee060e05840af9cc412908a90ba72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38d7989e274424cb050444d35f080ca" member_type: VOTER last_known_addr { host: "bigdsrv001" port: 7051 } } peers { permanent_uuid: "d11ee060e05840af9cc412908a90ba72" member_type: VOTER last_known_addr { host: "bigdsrv002" port: 7051 } } peers { permanent_uuid: "64635ac2b4ab4f2eb69960936182bc38" member_type: VOTER last_known_addr { host: "bigdsrv003" port: 7051 } } }
I0627 13:19:00.168740 31865 hms_notification_log_listener.cc:227] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: b38d7989e274424cb050444d35f080ca, Raft Consensus state: current_term: 85 leader_uuid: "d11ee060e05840af9cc412908a90ba72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38d7989e274424cb050444d35f080ca" member_type: VOTER last_known_addr { host: "bigdsrv001" port: 7051 } } peers { permanent_uuid: "d11ee060e05840af9cc412908a90ba72" member_type: VOTER last_known_addr { host: "bigdsrv002" port: 7051 } } peers { permanent_uuid: "64635ac2b4ab4f2eb69960936182bc38" member_type: VOTER last_known_addr { host: "bigdsrv003" port: 7051 } } }
I0627 13:19:15.169083 31865 hms_notification_log_listener.cc:227] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: b38d7989e274424cb050444d35f080ca, Raft Consensus state: current_term: 85 leader_uuid: "d11ee060e05840af9cc412908a90ba72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38d7989e274424cb050444d35f080ca" member_type: VOTER last_known_addr { host: "bigdsrv001" port: 7051 } } peers { permanent_uuid: "d11ee060e05840af9cc412908a90ba72" member_type: VOTER last_known_addr { host: "bigdsrv002" port: 7051 } } peers { permanent_uuid: "64635ac2b4ab4f2eb69960936182bc38" member_type: VOTER last_known_addr { host: "bigdsrv003" port: 7051 } } }
I0627 13:19:30.169376 31865 hms_notification_log_listener.cc:227] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: b38d7989e274424cb050444d35f080ca, Raft Consensus state: current_term: 85 leader_uuid: "d11ee060e05840af9cc412908a90ba72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38d7989e274424cb050444d35f080ca" member_type: VOTER last_known_addr { host: "bigdsrv001" port: 7051 } } peers { permanent_uuid: "d11ee060e05840af9cc412908a90ba72" member_type: VOTER last_known_addr { host: "bigdsrv002" port: 7051 } } peers { permanent_uuid: "64635ac2b4ab4f2eb69960936182bc38" member_type: VOTER last_known_addr { host: "bigdsrv003" port: 7051 } } }
I0627 13:19:45.169740 31865 hms_notification_log_listener.cc:227] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: b38d7989e274424cb050444d35f080ca, Raft Consensus state: current_term: 85 leader_uuid: "d11ee060e05840af9cc412908a90ba72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38d7989e274424cb050444d35f080ca" member_type: VOTER last_known_addr { host: "bigdsrv001" port: 7051 } } peers { permanent_uuid: "d11ee060e05840af9cc412908a90ba72" member_type: VOTER last_known_addr { host: "bigdsrv002" port: 7051 } } peers { permanent_uuid: "64635ac2b4ab4f2eb69960936182bc38" member_type: VOTER last_known_addr { host: "bigdsrv003" port: 7051 } } }
I0627 13:20:00.170046 31865 hms_notification_log_listener.cc:227] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: b38d7989e274424cb050444d35f080ca, Raft Consensus state: current_term: 85 leader_uuid: "d11ee060e05840af9cc412908a90ba72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38d7989e274424cb050444d35f080ca" member_type: VOTER last_known_addr { host: "bigdsrv001" port: 7051 } } peers { permanent_uuid: "d11ee060e05840af9cc412908a90ba72" member_type: VOTER last_known_addr { host: "bigdsrv002" port: 7051 } } peers { permanent_uuid: "64635ac2b4ab4f2eb69960936182bc38" member_type: VOTER last_known_addr { host: "bigdsrv003" port: 7051 } } }
I0627 13:20:15.170383 31865 hms_notification_log_listener.cc:227] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: b38d7989e274424cb050444d35f080ca, Raft Consensus state: current_term: 85 leader_uuid: "d11ee060e05840af9cc412908a90ba72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38d7989e274424cb050444d35f080ca" member_type: VOTER last_known_addr { host: "bigdsrv001" port: 7051 } } peers { permanent_uuid: "d11ee060e05840af9cc412908a90ba72" member_type: VOTER last_known_addr { host: "bigdsrv002" port: 7051 } } peers { permanent_uuid: "64635ac2b4ab4f2eb69960936182bc38" member_type: VOTER last_known_addr { host: "bigdsrv003" port: 7051 } } }
I0627 13:20:30.170612 31865 hms_notification_log_listener.cc:227] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: b38d7989e274424cb050444d35f080ca, Raft Consensus state: current_term: 85 leader_uuid: "d11ee060e05840af9cc412908a90ba72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38d7989e274424cb050444d35f080ca" member_type: VOTER last_known_addr { host: "bigdsrv001" port: 7051 } } peers { permanent_uuid: "d11ee060e05840af9cc412908a90ba72" member_type: VOTER last_known_addr { host: "bigdsrv002" port: 7051 } } peers { permanent_uuid: "64635ac2b4ab4f2eb69960936182bc38" member_type: VOTER last_known_addr { host: "bigdsrv003" port: 7051 } } }
W0627 13:20:34.300704 31793 service_pool.cc:129] ConnectToMaster request on kudu.master.MasterService from 192.168.18.50:58105 dropped due to backpressure. The service queue is full; it has 50 items.
W0627 13:20:37.251070 2350 negotiation.cc:313] Failed RPC negotiation. Trace:错误关键字：
0627 13:18:38.000976 (+2958970us) negotiation.cc:304] Negotiation complete: Timed out: Server connection negotiation failed: server connection from 192.168.18.50:57990
W0627 13:20:34.300704 31793 service_pool.cc:129] ConnectToMaster request on kudu.master.MasterService from 192.168.18.50:58105 dropped due to backpressure. The service queue is full; it has 50 items.
原因：高峰期单个Tablet的rpc请求队列达到上限，导致TabletServer无法提供服务，临时解决方案是重启该TabletServer。 可在gflagfile增加参数：–rpc_service_queue_length=120 (适当调大，默认值100)
类似资料：